AWS Glue vs. PyCharm

When the data which requires ETL has different formats, schema, and volume, this service suits them best. So, when the volume is not consistent (typical use-case of healthcare and online shopping), AWS Glue can be the prime choice. When the data is available in both batch and streaming mode, the developer needs to generate a separate codebase. This increases the source code management efforts. So, prefer to go with Glue when the nature of the data is the same (either batched or streamed).

Incentivized

It's easy to create virtual environments and install packages for different projects as we may need project-specific packages for doing our experiments, also it's easy to see what changes we have made and create pull requests faster. But sometimes we want some light python editor like Jupiter notebook as PyCharm is relatively heavier, also Jupiter notebooks are a good option when we need to run remote code on local machines.

Incentivized

• After data cleansing, the team also implemented the best practices for using AWS platform services as a Data Lake, such as job bookmarking for AWS Glue jobs, proper delimiter for the AWS Glue crawlers, partitioning in AWS S3, and transformation to parquet file for compression and faster querying time in Amazon Athena.
• Data modernization through combining data from multiple sources into a functioning datasets, rebuilding DW, and resctructuring data sources.
• Aims to lessen customer complaints, eliminate manual data extraction requests via SR from different data sources, and Increase accuracy, consistency and speed up reconciliation process.

Incentivized

• Git integration is really essential as it allows anyone to visually see the local and remote changes, compare revisions without the need for complex commands.
• Complex debugging tools are basked into the IDE. Controls like break on exception are sometimes very helpful to identify errors quickly.
• Multiple runtimes - Python, Flask, Django, Docker are native the to IDE. This makes development and debugging and even more seamless.
• Integrates with Jupyter and Markdown files as well. Side by side rendering and editing makes it simple to develop such files.

Incentivized

• It’s integration with other cloud vendors is bit difficult
• If it can support non SQL based databases as well, it would be powerful.
• Real time data synchronisation in data source is missing

Incentivized

• PyCharm text editor automatically inserts whitespace at the end of each line which can cause issues when doing text comparisons.
• The package requirement checker and installer does not work well all the time and can be improved
• Integration with GitLab pipelines can be made better.

Incentivized

It's perfect for our needs, cuts development time, is really helpful for newbies to understand projects structure

Incentivized

I personally found it very usable for a data engineer's day job, particularly for performing ETL and managing the data pipelines.

Incentivized

It's pretty easy to use, but if it's your first time using it, you need time to adapt. Nevertheless, it has a lot of options, and everything is pretty easy to find. The console has a lot of advantages and lets you accelerate your development from the first day.

Incentivized

Amazon responds in good time once the ticket has been generated but needs to generate tickets frequent because very few sample codes are available, and it's not cover all the scenarios.

Incentivized

I rate 10/10 because I have never needed a direct customer support from the JetBrains so far. Whenever and for whatever kind of problems I came across, I have been able to resolve it within the internet community, simply by Googling because turns out most of the time, it was me who lacked the proper information to use the IDE or simply make the proper configuration. I have never came across a bug in PyCharm either so it deserves 10/10 for overall support

Incentivized

The cataloging of data objects is the best in the case of AWS Glue. We use AWS Glue in all of our data pipelines to sync external and internal data sources and to automatically produce SQL-based ETL based on AWS Glue catalog objects. Integration with Amazon products is the other advantage.

Incentivized

It is more complete and can handle more projects at the same time. On the other hand, Visual Studio Code has better integration with LMS to help you code. PyCharm allows you to integrate with many external tools and external servers that Visual Studio Code has difficulties with.

Incentivized

• Positive Impact :- after ETL we can able to do some kind of automation
• Negative :- At some point of time it can hamper the cost but not really

Incentivized

• Improved efficiency with coding assistance (templates, code completion, documentation), which helps us avoid 'reinventing the wheel' with new projects.
• Extensive support for other packages/integrations: Docker support to test code, Git repo creation (for version control), and integration with different database systems (Postgres, MySQL).

Incentivized