Apache Kafka vs. Azure Data Factory

Apache Kafka is well-suited for most data-streaming use cases. Amazon Kinesis and Azure EventHubs, unless you have a specific use case where using those cloud PaAS for your data lakes, once set up well, Apache Kafka will take care of everything else in the background. Azure EventHubs, is good for cross-cloud use cases, and Amazon Kinesis - I have no real-world experience. But I believe it is the same.

Well-suited Scenarios for Azure Data Factory (ADF): When an organization has data sources spread across on-premises databases and cloud storage solutions, I think Azure Data Factory is excellent for integrating these sources. Azure Data Factory's integration with Azure Databricks allows it to handle large-scale data transformations effectively, leveraging the power of distributed processing. For regular ETL or ELT processes that need to run at specific intervals (daily, weekly, etc.), I think Azure Data Factory's scheduling capabilities are very handy. Less Appropriate Scenarios for Azure Data Factory: Real-time Data Streaming - Azure Data Factory is primarily batch-oriented. Simple Data Copy Tasks - For straightforward data copy tasks without the need for transformation or complex workflows, in my opinion, using Azure Data Factory might be overkill; simpler tools or scripts could suffice. Advanced Data Science Workflows: While Azure Data Factory can handle data prep and transformation, in my experience, it's not designed for in-depth data science tasks. I think for advanced analytics, machine learning, or statistical modeling, integration with specialized tools would be necessary.

Incentivized

• Really easy to configure. I've used other message brokers such as RabbitMQ and compared to them, Kafka's configurations are very easy to understand and tweak.
• Very scalable: easily configured to run on multiple nodes allowing for ease of parallelism (assuming your queues/topics don't have to be consumed in the exact same order the messages were delivered)
• Not exactly a feature, but I trust Kafka will be around for at least another decade because active development has continued to be strong and there's a lot of financial backing from Confluent and LinkedIn, and probably many other companies who are using it (which, anecdotally, is many).

Incentivized

• It allows copying data from various types of data sources like on-premise files, Azure Database, Excel, JSON, Azure Synapse, API, etc. to the desired destination.
• We can use linked service in multiple pipeline/data load.
• It also allows the running of SSIS & SSMS packages which makes it an easy-to-use ETL & ELT tool.

Incentivized

• Sometimes it becomes difficult to monitor our Kafka deployments. We've been able to overcome it largely using AWS MSK, a managed service for Apache Kafka, but a separate monitoring dashboard would have been great.
• Simplify the process for local deployment of Kafka and provide a user interface to get visibility into the different topics and the messages being processed.
• Learning curve around creation of broker and topics could be simplified

• Limited source/sink (target) connectors depending on which area of Azure Data Factory you are using.
• Does not yet have parity with SSIS as far as the transforms available.

Incentivized

Kafka is quickly becoming core product of the organization, indeed it is replacing older messaging systems. No better alternatives found yet

Incentivized

Apache Kafka is highly recommended to develop loosely coupled, real-time processing applications. Also, Apache Kafka provides property based configuration. Producer, Consumer and broker contain their own separate property file

Incentivized

So far product has performed as expected. We were noticing some performance issues, but they were largely Synapse related. This has led to a shift from Synapse to Databricks. Overall this has delayed our analytic platform. Once databricks becomes fully operational, Azure Data Factory will be critical to our environment and future success.

Incentivized

Support for Apache Kafka (if willing to pay) is available from Confluent that includes the same time that created Kafka at Linkedin so they know this software in and out. Moreover, Apache Kafka is well known and best practices documents and deployment scenarios are easily available for download. For example, from eBay, Linkedin, Uber, and NYTimes.

Incentivized

We have not had need to engage with Microsoft much on Azure Data Factory, but they have been responsive and helpful when needed. This being said, we have not had a major emergency or outage requiring their intervention. The score of seven is a representation that they have done well for now, but have not proved out their support for a significant issue

Incentivized

I used other messaging/queue solutions that are a lot more basic than Confluent Kafka, as well as another solution that is no longer in the market called Xively, which was bought and "buried" by Google. In comparison, these solutions offer way fewer functionalities and respond to other needs.

Incentivized

The easy integration with other Microsoft software as well as high processing speed, very flexible cost, and high level of security of Microsoft Azure products and services stack up against other similar products.

Incentivized

• Positive: Get a quick and reliable pub/sub model implemented - data across components flows easily.
• Positive: it's scalable so we can develop small and scale for real-world scenarios
• Negative: it's easy to get into a confusing situation if you are not experienced yet or something strange has happened (rare, but it does). Troubleshooting such situations can take time and effort.

Incentivized

• It is very useful and make things easier
• Debugging can improve
• Its better suited than other products with the same objective

Incentivized