Apache Airflow is an open source tool that can be used to programmatically author, schedule and monitor data pipelines using Python and SQL.
N/A
Apache Kafka
Score 8.7 out of 10
N/A
Apache Kafka is an open-source stream processing platform developed by the Apache Software Foundation written in Scala and Java. The Kafka event streaming platform is used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.
N/A
IBM MQ
Score 9.0 out of 10
N/A
IBM MQ (formerly WebSphere MQ and MQSeries) is messaging middleware.
digdag (https://www.digdag.io/)- Digdag is a very simple build, run, schedule, and monitor complex pipelines of tasks with a simple implementation and no configuration. Easy to write YAMLs
Airflow has a better community and widely adopted. Has a better UI and better documentation
Using Jenkins and Kafka, it is not for the same purpose, although it might be similar. I would say AirFlow is really what it says on the can - workflow management. For our organisation, the purpose is clear. So long your aim is to have a rich workflow scheduler and job …
Confluent Cloud is still based on Apache Kafka but it has a subscription fee so, from a long term perspective, it is wiser to deploy your own Kafka instance that spans public and private cloud. Amazon Kinesis, Google Cloud Pub/Sub do not do well for a very number of messages …
Kafka is faster and more scalable, also "free" as opensource (albeit we deploy using a commercial distribution). Infrastructure tends to be cheaper. On the other hand, projects must adapt to Kafka APIs that sometimes change and BAU increases until a major 1.x version comes out …
Apache Kafka may be a better option in comparison with IBM MQ its real-time data streaming and large data payload service. It depends upon the specific requirement and meets those needs. MuleSoft any point platform is very easy to connect to various other types of platforms in …
I've also used Apache Kafka and RabbitMQ. Compared to these, IBM MQ offers superior reliability and transactional integrity, making it a better choice for complex, mission-critical enterprise environments where message delivery and security are paramount. We chose IBM MQ for …
Kafka is renowned for its impressive throughput, fault tolerance, and real-time data streaming capabilities. Nonetheless, IBM MQ remains the preferred choice due to its unwavering commitment to guaranteed delivery and exceptional reliability. Fault-Tolerant Architectures of IBM …
We found IBM MQ very easy to get started and quick to learn by the new users with a short learning curve and seamlessly integrates with IBM products, and quick to perform self-service analytics and make informed business decisions. IBM MQ is also very straightforward in …
IBM MQ is very stable and a proven product compared to other Messaging platforms available. Performance was better than WSO2 product and also the RabbitMQ. Though Kafka and IBM MQ is not directly comparable, Kafka is more suited for event based systems and also where there is …
Airflow is well-suited for data engineering pipelines, creating scheduled workflows, and working with various data sources. You can implement almost any kind of DAG for any use case using the different operators or enforce your operator using the Python operator with ease. The MLOps feature of Airflow can be enhanced to match MLFlow-like features, making Airflow the go-to solution for all workloads, from data science to data engineering.
Apache Kafka is well-suited for most data-streaming use cases. Amazon Kinesis and Azure EventHubs, unless you have a specific use case where using those cloud PaAS for your data lakes, once set up well, Apache Kafka will take care of everything else in the background. Azure EventHubs, is good for cross-cloud use cases, and Amazon Kinesis - I have no real-world experience. But I believe it is the same.
In the context of Internet of Things (IoT) applications, IBM MQ plays a pivotal role in managing the substantial data streams emanating from interconnected devices. Its primary function is to guarantee the dependable transmission and processing of data, catering to a diverse range of IoT use cases, including but not limited to smart city initiatives, healthcare monitoring systems, and industrial automation solutions. In the telecommunications sector, IBM MQ is employed for message routing, call detail record (CDR) processing, and network management to ensure real-time data exchange and fault tolerance. When managing the supply chain and logistics, IBM MQ is used to ensure timely and accurate communication between different entities, including suppliers, warehouses, and transportation providers. IBM MQ can be cost-prohibitive for smaller organizations due to licensing and maintenance costs. In such cases, open-source or lightweight messaging solutions may be more appropriate. For scenarios requiring extremely low-latency, real-time data exchange, and high throughput, other messaging technologies, like Apache Kafka, may be more suitable due to their specialized design for such use cases.
Apache Airflow is one of the best Orchestration platforms and a go-to scheduler for teams building a data platform or pipelines.
Apache Airflow supports multiple operators, such as the Databricks, Spark, and Python operators. All of these provide us with functionality to implement any business logic.
Apache Airflow is highly scalable, and we can run a large number of DAGs with ease. It provided HA and replication for workers. Maintaining airflow deployments is very easy, even for smaller teams, and we also get lots of metrics for observability.
Really easy to configure. I've used other message brokers such as RabbitMQ and compared to them, Kafka's configurations are very easy to understand and tweak.
Very scalable: easily configured to run on multiple nodes allowing for ease of parallelism (assuming your queues/topics don't have to be consumed in the exact same order the messages were delivered)
Not exactly a feature, but I trust Kafka will be around for at least another decade because active development has continued to be strong and there's a lot of financial backing from Confluent and LinkedIn, and probably many other companies who are using it (which, anecdotally, is many).
The documentation is very clear,It is understandable and the support helps to configure it in the best way.
Server guidelines make it possible to get the most out of work management. It's broad, we can work with different operating systems, I really recommend using linux.
It is highly compatible with systems, brockers, applications, and data accumulation programs, it is possible to configure everything so that after the installation of programs, they can communicate with each other and then throw data to an external program that accumulates it and represents in clear details of steps to follow and make business decisions.
UI/Dashboard can be updated to be customisable, and jobs summary in groups of errors/failures/success, instead of each job, so that a summary of errors can be used as a starting point for reviewing them.
Navigation - It's a bit dated. Could do with more modern web navigation UX. i.e. sidebars navigation instead of browser back/forward.
Again core functional reorg in terms of UX. Navigation can be improved for core functions as well, instead of discovery.
Sometimes it becomes difficult to monitor our Kafka deployments. We've been able to overcome it largely using AWS MSK, a managed service for Apache Kafka, but a separate monitoring dashboard would have been great.
Simplify the process for local deployment of Kafka and provide a user interface to get visibility into the different topics and the messages being processed.
Learning curve around creation of broker and topics could be simplified
There is limitation on number of svrconn connections you can have to MQ on the mainframe which has been an major issue for us. This has been an issue for us for over 4 years and still no fix although I am aware IBM have been working on a solution over the last year.
When upgrading to MQ V9.3 on our MQ appliances there is no fall-back option. This was the same for MQ V9.2 upgrade from MQ V9.0. For production upgrades this I believe is not acceptable.
AMS is not supplied as part of the standard mainframe MQ licence. You need an extra licence. IBM tell customers how important security and protecting data is yet they still want to charge for this software. The cost of MQ on the mainframe is not cheap so I would expect AMS to be part of the base product.
For its capability to connect with multicloud environments. Access Control management is something that we don't get in all the schedulers and orchestrators. But although it provides so many flexibility and options to due to python , some level of knowledge of python is needed to be able to build workflows.
Apache Kafka is highly recommended to develop loosely coupled, real-time processing applications. Also, Apache Kafka provides property based configuration. Producer, Consumer and broker contain their own separate property file
I give it a nine because it has significantly improved my team's data reliability and operational efficiency. Its great security features give us peace of mind, knowing our sensitive data is well protected. While the setup might initially be complex, I believe the long-term benefits far outweigh this hurdle.
The messages are delivered instantly with this software and it integrates with our technology stack, in terms of availability we only had one failure when we were doing some testing and integration with third parties, the features of this software make it always available and its deployment is easy for the company, it does not generate expenses due to failures
Support for Apache Kafka (if willing to pay) is available from Confluent that includes the same time that created Kafka at Linkedin so they know this software in and out. Moreover, Apache Kafka is well known and best practices documents and deployment scenarios are easily available for download. For example, from eBay, Linkedin, Uber, and NYTimes.
There are very specific things that must be elevated to more specialized areas of support, but the common support is very agile when receiving questions or when we leave concerns in real time. I recommend the support of the program in this regard.
Multiple DAGs can be orchestrated simultaneously at varying times, and runs can be reproduced or replicated with relative ease. Overall, utilizing Apache Airflow is easier to use than other solutions now on the market. It is simple to integrate in Apache Airflow, and the workflow can be monitored and scheduling can be done quickly using Apache Airflow. We advocate using this tool for automating the data pipeline or process.
I used other messaging/queue solutions that are a lot more basic than Confluent Kafka, as well as another solution that is no longer in the market called Xively, which was bought and "buried" by Google. In comparison, these solutions offer way fewer functionalities and respond to other needs.
We found IBM MQ very easy to get started and quick to learn by the new users with a short learning curve and seamlessly integrates with IBM products, and quick to perform self-service analytics and make informed business decisions. IBM MQ is also very straightforward in creating simple and best reports, which are very profitable and productive.
Impact Depends on number of workflows. If there are lot of workflows then it has a better usecase as the implementation is justified as it needs resources , dedicated VMs, Database that has a cost
Positive: Get a quick and reliable pub/sub model implemented - data across components flows easily.
Positive: it's scalable so we can develop small and scale for real-world scenarios
Negative: it's easy to get into a confusing situation if you are not experienced yet or something strange has happened (rare, but it does). Troubleshooting such situations can take time and effort.
