AWS Data Pipeline - Data engineer's time saver
Rating: 10 out of 10
IncentivizedUse Cases and Deployment Scope
We are using AWS data pipeline to create data flows to extract, transform and load data to redshift, Basically creating ETL job flows using AWS data pipeline. It is helping data engineers to effectively and quickly create and manage complex data processing flows.
Pros
- Helps you easily create complex data processing workloads
- Fault tolerant
- Highly available
Cons
- Pipeline Stuck in Pending Status
- Pipeline Component Stuck in Waiting for Runner Status
- EMR Cluster Fails With Error
Likelihood to Recommend
AWS Data Pipeline is a web service that helps you reliably process and move data between different AWS compute and storage services, as well as on-premises data sources, at specified intervals. With AWS Data Pipeline, you can regularly access your data where it’s stored, transform and process it at scale, and efficiently transfer the results to AWS services such as Amazon S3, Amazon RDS, Amazon DynamoDB, and Amazon EMR.
VW
Vaibhav Wadikar
Data Engineer in Engineering at Innova Solutions (501-1000 employees)
Vetted Review
1 year of experience