Apache Sqoop is a tool for use with Hadoop, used to transfer data between Apache Hadoop and other, structured data stores.
N/A
Informatica PowerCenter
Score 8.6 out of 10
N/A
Informatica PowerCenter is a metadata driven data integration technology designed to form the foundation for data integration initiatives, including analytics and data warehousing, application migration, or consolidation and data governance.
Sqoop is great for sending data between a JDBC compliant database and a Hadoop environment. Sqoop is built for those who need a few simple CLI options to import a selection of database tables into Hadoop, do large dataset analysis that could not commonly be done with that database system due to resource constraints, then export the results back into that database (or another). Sqoop falls short when there needs to be some extra, customized processing between database extract, and Hadoop loading, in which case Apache Spark's JDBC utilities might be preferred
Informatica Powercenter is the centerpiece of our overall enterprise data warehouse strategy. It's a critical enablement to ensure we can feed in multiple data stream and transform them into digestible data within our data warehouse. With its flexible capabilities and API availability, we were able to feed in industry standard data format as well as home grown data structure. Overall, we are very pleased with their capability and contribution to our data warehouse strategy.
Sqoop2 development seems to have stalled. I have set it up outside of a Cloudera CDH installation, and I actually prefer it's "Sqoop Server" model better than just the CLI client version that is Sqoop1. This works especially well in a microservices environment, where there would be only one place to maintain the JDBC drivers to use for Sqoop.
There are too many ways to perform the same or similar functions which in turn makes it challenging to trace what a workflow is doing and at which point (ex. sessions can be designed as static or re-usable and the override can occur at the session or workflow, or both which can be counter productive and confusing when troubleshooting).
The power in structured design is a double edged sword. Simple tasks for a POC can become cumbersome. Ex. if you want to move some data to test a process, you first have to create your sources by importing them which means an ODBC connection or similar will need to be configured, you in turn have to develop your targets and all of the essential building blocks before being able to begin actual development. While I am on sources and targets, I think of a table definition as just that and find it counter intuitive to have to design a table as both a source and target and manage them as different objects. It would be more intuitive to have a table definition and its source/target properties defined by where you drag and drop it in the mapping.
There are no checkpoints or data viewer type functions without designing an entire mapping and workflow. If you would like to simply run a job up to a point and check the throughput, an entire mapping needs to be completed and you would workaround this by creating a flat file target.
Positives; - Multi User Development Environment - Speed of transformation - Seamless integration between other Informatica products. Negatives; - There should be less windows to maintain developers' focus while using. You probably need 2 big monitors when you start development with Informatica Power Center. - Oracle Analytical functions should be natively used. - E-LT support as well as ETL support.
PowerCenter is robust and fast, and it does a great job meeting all the needs, not just the most commercially vocal needs. In the hands of an expert power user, you can accomplish almost anything with your data. It is not for new users or intermittent users-- for that the Cloud version is a better fit. Be prepared for costly connectors (priced differently for each source or destination you are working with), and just be planful of your projects so you are not paying for connectors you no longer need or want
Informatica power center is a leader of the pack of ETL tools and has some great abilities that make it stand out from other ETL tools. It has been a great partner to its clients over a long time so it's definitely dependable. With all the great things about Informatica, it has a bit of tech burden that should be addressed to make it more nimble, reduce the learning curve for new developers, provide better connectivity with visualization tools.
Sqoop comes preinstalled on the major Hadoop vendor distributions as the recommended product to import data from relational databases. The ability to extend it with additional JDBC drivers makes it very flexible for the environment it is installed within.
Spark also has a useful JDBC reader, and can manipulate data in more ways than Sqoop, and also upload to many other systems than just Hadoop.
Kafka Connect JDBC is more for streaming database updates using tools such as Oracle GoldenGate or Debezium.
Streamsets and Apache NiFi both provide a more "flow based programming" approach to graphically laying out connectors between various systems, including JDBC and Hadoop.
PowerCenter is the industry leader when it comes to interfacing with multiple source and target systems. The graphical interface increases employee productivity while reducing human resource expenditures and training requirements. These other tools offer some similar capabilities, but lack the range and depth when compared with the PowerCenter platform.
When combined with Cloudera's HUE, it can enable non-technical users to easily import relational data into Hadoop.
Being able to manipulate large datasets in Hadoop, and them load them into a type of "materialized view" in an external database system has yielded great insights into the Hadoop datalake without continuously running large batch jobs.
Sqoop isn't very user-friendly for those uncomfortable with a CLI.
The data pipeline automation capability of Informatica means that few resources are needed to pre-process the data that ultimately resides in a Data Warehouse. Once a workflow is implemented, manual intervention is not needed.
PowerCenter did require more resources and time for installation and configuration than was expected/planned for.
The lack of or minimal support of unstructured data means that newer sources of dynamic/changing data cannot be easily processed/transformed through PowerCenter workflows.
