Azure Data Factory vs. Azure Service Bus

Best scenario is for ETL process. The flexibility and connectivity is outstanding. For our environment, SAP data connectivity with Azure Data Factory offers very limited features compared to SAP Data Sphere. Due to the limited modelling capacity of the tool, we use Databricks for data modelling and cleaning. Usage of multiple tools could have been avoided if adf has modelling capabilities.

Incentivized

If you need a cloud-based service bus or a simple to use queue/topic/routing/pub-sub service, then Azure Service Bus is a very good choice at a reasonable price and performance. Typically on-premise we'd use RabbitMQ because it "just works", but if you're building a "cloud-first" application, then this is the one to go with. It's especially easy to integrate with if you're already embedded in the Microsoft ecosystem.

Incentivized

• Data Ingestion - it works very well with numerous data sources.
• Data pipeline orchestration: It is a generic, popular tool for orchestrating data pipelines.
• Works well in Azure ecosystem, Azure services and data platforms like Databricks.
• It is a serverless and scalable solution for cloud data integration.

Incentivized

• Acting as a basic queuing service it works very well.
• One of the best parts is that Azure Service Bus can work over HTTPS which helps in strict firewall situations. There is a performance hit if you choose to use HTTPS.
• The routing capabilities are quite good when using topics and subscriptions. You can apply filters using a pseudo-SQL-like language though the correlation filters are quick and easy options.
• Costs are very reasonable at low-ish volumes. If you're processing 10's of millions of messages a month... it may be a different story.

Incentivized

• Granularity of Errors: Sometimes, Azure Data Factory provides error messages that are too generic or vague for us, making it challenging to pinpoint the exact cause of a pipeline failure. Enhanced error messages with more actionable details would greatly assist us as users in debugging their pipelines.
• Pipeline Design UI: In my experience, the visual interface for designing pipelines, especially when dealing with complex workflows or numerous activities, can become cluttered. I think a more intuitive and scalable design interface would improve usability. In my opinion, features like zoom, better alignment tools, or grouping capabilities could make managing intricate designs more manageable.
• Native Support: While Azure Data Factory does support incremental data loads, in my experience, the setup can be somewhat manual and complex. I think native and more straightforward support for Change Data Capture, especially from popular databases, would simplify the process of capturing and processing only the changed data, making regular data updates more efficient

Incentivized

• The SqlFilter could be a little easier to use, but it's not terrible.
• The performance while using HTTPS for the connection is a little slow compared to direct connections using AMQP ports.
• There is a size limit to the message - unlike RMQ for instance, Azure Service Bus caps messages to 256kb on the standard tier.

Incentivized

So far product has performed as expected. We were noticing some performance issues, but they were largely Synapse related. This has led to a shift from Synapse to Databricks. Overall this has delayed our analytic platform. Once databricks becomes fully operational, Azure Data Factory will be critical to our environment and future success.

Incentivized

We have not had need to engage with Microsoft much on Azure Data Factory, but they have been responsive and helpful when needed. This being said, we have not had a major emergency or outage requiring their intervention. The score of seven is a representation that they have done well for now, but have not proved out their support for a significant issue

Incentivized

Azure Data Factory helps us automate to schedule jobs as per customer demands to make ETL triggers when the need arises. Anyone can define the workflow with the Azure Data Factory UI designer tool and easily test the systems. It helped us automate the same workflow with programming languages like Python or automation tools like ansible. Numerous options for connectivity be it a database or storage account helps us move data transfer to the cloud or on-premise systems.

Incentivized

RabbitMQ is simple and awesome... but so is Azure Service Bus. Both accomplish the same thing but in different environments. If you're building a cloud-native application - especially one that is serverless by design - Azure Service Bus is the only real choice in Azure. It works well, it's performance, and it's reasonably priced in the Standard tier. From our testing, RMQ is more performant, but it's hard to compare service-based implementations vs RMQ installed on VMs.

Incentivized

• Facilitate better decision-making and improve business processes.
• Optimize business process outcomes by increasing internal efficiency and operational effectiveness.
• Boosts revenue growth while improving business process agility.

Incentivized

• Compared to open-source free software like RMQ, Azure Service Bus does have some costs to it. But the cost is reasonable.
• Also unlike RMQ, Azure Service Bus doesn't require you to stand up any hardware - so it's very easy to use and saves time/money from that perspective.

Incentivized