Apache Airflow vs. StreamSets DataOps Platform

Overview
ProductRatingMost Used ByProduct SummaryStarting Price
Apache Airflow
Score 8.4 out of 10
N/A
Apache Airflow is an open source tool that can be used to programmatically author, schedule and monitor data pipelines using Python and SQL. Created at Airbnb as an open-source project in 2014, Airflow was brought into the Apache Software Foundation’s Incubator Program 2016 and announced as Top-Level Apache Project in 2019. It is used as a data orchestration solution, with over 140 integrations and community support.N/A
StreamSets
Score 8.1 out of 10
N/A
StreamSets in San Francisco offers their DataOps Platform, a subscription based streaming analytics platform including StreamSets Data Collector data source management, Control Hub for data movement architecture management, StreamSets Data Collector Edge IoT manager, DataFlow Performance Manager (DPM), and StreamSets Data Protector compliance (e.g. GDPR) compliance module.N/A
Pricing
Apache AirflowStreamSets DataOps Platform
Editions & Modules
No answers on this topic
No answers on this topic
Offerings
Pricing Offerings
Apache AirflowStreamSets
Free Trial
NoNo
Free/Freemium Version
YesNo
Premium Consulting/Integration Services
NoNo
Entry-level Setup FeeNo setup feeNo setup fee
Additional Details
More Pricing Information
Community Pulse
Apache AirflowStreamSets DataOps Platform
Top Pros
Top Cons
Features
Apache AirflowStreamSets DataOps Platform
Workload Automation
Comparison of Workload Automation features of Product A and Product B
Apache Airflow
8.2
9 Ratings
0% above category average
StreamSets DataOps Platform
-
Ratings
Multi-platform scheduling8.89 Ratings00 Ratings
Central monitoring8.49 Ratings00 Ratings
Logging8.19 Ratings00 Ratings
Alerts and notifications7.99 Ratings00 Ratings
Analysis and visualization7.99 Ratings00 Ratings
Application integration8.49 Ratings00 Ratings
Streaming Analytics
Comparison of Streaming Analytics features of Product A and Product B
Apache Airflow
-
Ratings
StreamSets DataOps Platform
9.0
1 Ratings
11% above category average
Visualization Dashboards00 Ratings7.01 Ratings
Low Latency00 Ratings8.01 Ratings
Integrated Development Tools00 Ratings10.01 Ratings
Data wrangling and preparation00 Ratings10.01 Ratings
Data Enrichment00 Ratings10.01 Ratings
Best Alternatives
Apache AirflowStreamSets DataOps Platform
Small Businesses

No answers on this topic

IBM Streams
IBM Streams
Score 9.0 out of 10
Medium-sized Companies
ActiveBatch Workload Automation
ActiveBatch Workload Automation
Score 8.6 out of 10
Confluent
Confluent
Score 7.4 out of 10
Enterprises
Redwood RunMyJobs
Redwood RunMyJobs
Score 9.4 out of 10
Spotfire Streaming
Spotfire Streaming
Score 7.9 out of 10
All AlternativesView all alternativesView all alternatives
User Ratings
Apache AirflowStreamSets DataOps Platform
Likelihood to Recommend
7.9
(9 ratings)
9.0
(1 ratings)
User Testimonials
Apache AirflowStreamSets DataOps Platform
Likelihood to Recommend
Apache
For a quick job scanning of status and deep-diving into job issues, details, and flows, AirFlow does a good job. No fuss, no muss. The low learning curve as the UI is very straightforward, and navigating it will be familiar after spending some time using it. Our requirements are pretty simple. Job scheduler, workflows, and monitoring. The jobs we run are >100, but still is a lot to review and troubleshoot when jobs don't run. So when managing large jobs, AirFlow dated UI can be a bit of a drawback.
Read full review
StreamSets
Majorly for all Batch and Streaming Scenarios we are designing StreamSets pipelines, few best suited and tried out use cases below : 1. JDBC to ADLS data transfer based on source refresh frequency. 2. Kafka to GCS. 3. Kafka to Azure Event. 4. Hub HDFS to ADLS data transfer. 5. Schema generation to generate Avro. The easy to design Canvas, Scheduling Jobs, Fragment creation and utilization, an inbuilt wide range of Stage availability makes it an even more favorable tool for me to design data engineering pipelines.
Read full review
Pros
Apache
  • In charge of the ETL processes.
  • As there is no incoming or outgoing data, we may handle the scheduling of tasks as code and avoid the requirement for monitoring.
Read full review
StreamSets
  • A easy to use canvas to create Data Engineering Pipeline.
  • A wide range of available Stages ie. Sources, Processors, Executors, and Destinations.
  • Supports both Batch and Streaming Pipelines.
  • Scheduling is way easier than cron.
  • Integration with Key-Vaults for Secrets Fetching.
Read full review
Cons
Apache
  • they should bring in some time based scheduling too not only event based
  • they do not store the metadata due to which we are not able to analyze the workflows
  • they only support python as of now for scripted pipeline writing
Read full review
StreamSets
  • Monitoring/Visualization can be improvised and enhanced a lot (e.g. to monitor a Job to see what happened 7 days back with data transfer).
  • The logging mechanism can be simplified (Logs can be filtered with "ERROR", "DEBUG", "ALL" etc but still takes some time to get familiar for understanding).
  • Auto Scalability for heavy load transfer (Taking much time for >5 million record transfer from JDBC to ADLS destination in Avro file transfer).
  • There should be a concept of creating Global variables which is missing.
Read full review
Alternatives Considered
Apache
There are a number of reasons to choose Apache Airflow over other similar platforms- Integrations—ready-to-use operators allow you to integrate Airflow with cloud platforms (Google, AWS, Azure, etc) Apache Airflow helps with backups and other DevOps tasks, such as submitting a Spark job and storing the resulting data on a Hadoop cluster It has machine learning model training, such as triggering a Sage maker job.
Read full review
StreamSets
StreamSets is a one-stop solution to design Data engineering Pipelines and doesn't require deep Programming knowledge, It's so user-friendly that anyone in Team can contribute to the Idea of pipeline design. In Hadoop One has to be programming proficient to use its various components like Hive, HDFS, Kafka, etc but in StreamSets all these stages are built-in and ready to use with minor configuration.
Read full review
Return on Investment
Apache
  • A lot of helpful features out-of-the-box, such as the DAG visualizations and task trees
  • Allowed us to implement complex data pipelines easily and at a relatively low cost
Read full review
StreamSets
  • Simplified Improvised Overall data ingestion and Integration Process.
  • Support to various Hetrogenous Source systems like RDBMS< Kafka, Salesforce, Key Vault.
  • Secure, easy to launch Integration tool.
Read full review
ScreenShots