Apache Kafka is an open-source stream processing platform developed by the Apache Software Foundation written in Scala and Java. The Kafka event streaming platform is used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.
N/A
PostgreSQL
Score 8.5 out of 10
N/A
PostgreSQL (alternately Postgres) is a free and open source object-relational database system boasting over 30 years of active development, reliability, feature robustness, and performance. It supports SQL and is designed to support various workloads flexibly.
Apache Kafka is well-suited for most data-streaming use cases. Amazon Kinesis and Azure EventHubs, unless you have a specific use case where using those cloud PaAS for your data lakes, once set up well, Apache Kafka will take care of everything else in the background. Azure EventHubs, is good for cross-cloud use cases, and Amazon Kinesis - I have no real-world experience. But I believe it is the same.
PostgreSQL, unlike other databases, is user-friendly and uses an open-source database. Ideal for relational databases, they can be accessed when speed and efficiency are required. It enables high-availability and disaster recovery replication from instance to instance. PostgreSQL can store data in a JSON format, including hashes, keys, and values. Multi-platform compatibility is also a big selling point. We could, however, use all the DBMS’s cores. While it works well in fast environments, it can be problematic in slower ones or cause multiple master replication.
Really easy to configure. I've used other message brokers such as RabbitMQ and compared to them, Kafka's configurations are very easy to understand and tweak.
Very scalable: easily configured to run on multiple nodes allowing for ease of parallelism (assuming your queues/topics don't have to be consumed in the exact same order the messages were delivered)
Not exactly a feature, but I trust Kafka will be around for at least another decade because active development has continued to be strong and there's a lot of financial backing from Confluent and LinkedIn, and probably many other companies who are using it (which, anecdotally, is many).
The stability it offers, its speed of response and its resource management is excellent even in complex database environments and with low-resource machines.
The large amount of resources it has in addition to the many own and third-party tools that are compatible that make productivity greatly increase.
The adaptability in various environments, whether distributed or not, [is a] complete set of configuration options which allows to greatly customize the work configuration according to the needs that are required.
The excellent handling of referential and transactional integrity, its internal security scheme, the ease with which we can create backups are some of the strengths that can be mentioned.
Sometimes it becomes difficult to monitor our Kafka deployments. We've been able to overcome it largely using AWS MSK, a managed service for Apache Kafka, but a separate monitoring dashboard would have been great.
Simplify the process for local deployment of Kafka and provide a user interface to get visibility into the different topics and the messages being processed.
Learning curve around creation of broker and topics could be simplified
The query syntax for JSON fields is unwieldy when you start getting into complex queries with many joins.
I wish there was a distinction (a flag) you could set for automated scripts vs working in the psql CLI, which would provide an 'Are you sure you want to do X?' type prompt if your query is likely to affect more than a certain number of rows. Especially on updates/deletes. Setting the flag in the headless(scripted) flow would disable the prompt.
Better documentation around JSON and Array aggregation, with more examples of how the data is transformed.
Apache Kafka is highly recommended to develop loosely coupled, real-time processing applications. Also, Apache Kafka provides property based configuration. Producer, Consumer and broker contain their own separate property file
Postgresql is the best tool out there for relational data so I have to give it a high rating when it comes to analytics, data availability and consistency, so on and so forth. SQL is also a relatively consistent language so when it comes to building new tables and loading data in from the OLTP database, there are enough tools where we can perform ETL on a scalable basis.
The data queries are relatively quick for a small to medium sized table. With complex joins, and a wide and deep table however, the performance of the query has room for improvement.
Support for Apache Kafka (if willing to pay) is available from Confluent that includes the same time that created Kafka at Linkedin so they know this software in and out. Moreover, Apache Kafka is well known and best practices documents and deployment scenarios are easily available for download. For example, from eBay, Linkedin, Uber, and NYTimes.
There are several companies that you can contract for technical support, like EnterpriseDB or Percona, both first level in expertise and commitment to the software.
But we do not have contracts with them, we have done all the way from googling to forums, and never have a problem that we cannot resolve or pass around. And for dozens of projects and more than 15 years now.
The online training is request based. Had there been recorded videos available online for potential users to benefit from, I could have rated it higher. The online documentation however is very helpful. The online documentation PDF is downloadable and allows users to pace their own learning. With examples and code snippets, the documentation is great starting point.
I used other messaging/queue solutions that are a lot more basic than Confluent Kafka, as well as another solution that is no longer in the market called Xively, which was bought and "buried" by Google. In comparison, these solutions offer way fewer functionalities and respond to other needs.
Postgres stacks up just [fine] along the other big players in the RDBMS world. It's very popular for a reason. It's very close to MySQL in terms of cost and features - I'd pick either solution and be just as happy. Compared to Oracle it is a MUCH cheaper solution that is just as usable.
Positive: Get a quick and reliable pub/sub model implemented - data across components flows easily.
Positive: it's scalable so we can develop small and scale for real-world scenarios
Negative: it's easy to get into a confusing situation if you are not experienced yet or something strange has happened (rare, but it does). Troubleshooting such situations can take time and effort.
The user-role system has saved us tons of time and thus money. As I mentioned in the "Use Case" section, Postgres is not only used by engineering but also finance to measure how much to charge customers and customer support to debug customer issues. Sure, it's not easy for non-technical employees to psql in and view raw tables, but it has saved engineering hundreds of man-hours that would have had to be spent on building equivalent tools to serve finance or customer support.
It provides incredibly trustworthy storage for wherever customer data dumped in. In our 6 years of Postgres existence, we have not lost a byte of customer data due to Postgres messing up a transaction or during the multiple times the hard-drives failed (thanks to ACID compliance!).
This is less significant, but Postgres is also quite easy to manage (unless you are going above and beyond to squeeze out every last bit of performance). There's not much to configure, and the out of the box settings are quite sane. That has saved us engineers lots of time that would have gone into Postgres administration.
