Apache Kafka is an open-source stream processing platform developed by the Apache Software Foundation written in Scala and Java. The Kafka event streaming platform is used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.
N/A
IBM API Connect
Score 9.1 out of 10
N/A
IBM API Connect is a scalable API solution that helps organizations implement a robust API strategy by creating, exposing, managing and monetizing an entire API ecosystem across multiple clouds. As businesses embrace their digital transformation journey, APIs become critical to unlock the value of business data and assets. With increasing adoption of APIs, consistency and governance are needed across the enterprise. API Connect aims to help businesses…
Apache Kafka is well-suited for most data-streaming use cases. Amazon Kinesis and Azure EventHubs, unless you have a specific use case where using those cloud PaAS for your data lakes, once set up well, Apache Kafka will take care of everything else in the background. Azure EventHubs, is good for cross-cloud use cases, and Amazon Kinesis - I have no real-world experience. But I believe it is the same.
I love the IBM API Connect features, performance, and security level for all our business data. The workload balancing and integration with other third-party products are very simple. The data migration speed is beneficial, especially for time management, and creating process reports through IBM API Connect is incredible.
Really easy to configure. I've used other message brokers such as RabbitMQ and compared to them, Kafka's configurations are very easy to understand and tweak.
Very scalable: easily configured to run on multiple nodes allowing for ease of parallelism (assuming your queues/topics don't have to be consumed in the exact same order the messages were delivered)
Not exactly a feature, but I trust Kafka will be around for at least another decade because active development has continued to be strong and there's a lot of financial backing from Confluent and LinkedIn, and probably many other companies who are using it (which, anecdotally, is many).
Sometimes it becomes difficult to monitor our Kafka deployments. We've been able to overcome it largely using AWS MSK, a managed service for Apache Kafka, but a separate monitoring dashboard would have been great.
Simplify the process for local deployment of Kafka and provide a user interface to get visibility into the different topics and the messages being processed.
Learning curve around creation of broker and topics could be simplified
The first thing challenge we faced with the product was that if you deploying it to a third party cloud, that was very challenging and we need IBM team help at every step of the way and as well all know IBM support doesn't come cheap. A reason for that is that there isn't enough documentation done on the subject from IBM side.
The upgrade process is not that seamless and involves a lot of hassles.
You really need to have our requirements sorted out clearly because it is not very easy to customize the UI according to your needs, So you need to involve IBM from the start and give them clear requirements and then work with them to achieve it.
Apache Kafka is highly recommended to develop loosely coupled, real-time processing applications. Also, Apache Kafka provides property based configuration. Producer, Consumer and broker contain their own separate property file
IBM API Connect may be less appropriate for small-scale projects with minimal API management requirements, where simpler and more cost-effective solutions suffice. Organizations lacking the necessary technical expertise or resources to harness its full potential may face implementation challenges. In static environments with infrequent API changes or limited developer engagement, the platform's comprehensive features may be excessive for the task at hand.
Support for Apache Kafka (if willing to pay) is available from Confluent that includes the same time that created Kafka at Linkedin so they know this software in and out. Moreover, Apache Kafka is well known and best practices documents and deployment scenarios are easily available for download. For example, from eBay, Linkedin, Uber, and NYTimes.
I used other messaging/queue solutions that are a lot more basic than Confluent Kafka, as well as another solution that is no longer in the market called Xively, which was bought and "buried" by Google. In comparison, these solutions offer way fewer functionalities and respond to other needs.
There are two main reasons for choosing IBM over others. 1) Pricing 2) The conversation during the sales stage. The team at IBM understood our requirements and acted as consultants instead of sales people. They genuinely focused on providing a solution to our pain points which reflected during the implementation and continued after go-live in the form of technical support
Positive: Get a quick and reliable pub/sub model implemented - data across components flows easily.
Positive: it's scalable so we can develop small and scale for real-world scenarios
Negative: it's easy to get into a confusing situation if you are not experienced yet or something strange has happened (rare, but it does). Troubleshooting such situations can take time and effort.
I consider IBM API Connect as a business capability enabler - the ROI level is practically secondary.
With this platform at the core, associated architectural framework and guardrails ensure that we can progress with distributed development and automation in autonomous teams - a key factor to deliver required time to market performance.
At this time, security and trust is key. A flexible yet secure API manager layer is necessary to ensure our relationships with partners and customers.
