Apache Sqoop vs Databricks Lakehouse Platform

Likelihood to Recommend

Sqoop is great for sending data between a JDBC compliant database and a Hadoop environment. Sqoop is built for those who need a few simple CLI options to import a selection of database tables into Hadoop, do large dataset analysis that could not commonly be done with that database system due to resource constraints, then export the results back into that database (or another). Sqoop falls short when there needs to be some extra, customized processing between database extract, and Hadoop loading, in which case Apache Spark's JDBC utilities might be preferred

Read full review

Jordan Moore

Consultant

Avalon Consulting, LLCInformation Technology and Services, 51-200 employees

View all 1 answers on this topic

Databricks has helped my teams write PySpark and Spark SQL jobs and test them out before formally integrating them in Spark jobs. Through Databricks we can create parquet and JSON output files. Datamodelers and scientists who are not very good with coding can get good insight into the data using the notebooks that can be developed by the engineers.

Read full review

Verified User

Team Lead in Engineering

Financial Services Company, 10,001+ employees

View all 5 answers on this topic

Feature Rating Comparison

Apache Sqoop

Databricks Lakehouse Platform

Connect to Multiple Data Sources

Extend Existing Data Sources

Automatic Data Format Detection

Apache Sqoop

Databricks Lakehouse Platform

Visualization

Interactive Data Analysis

Apache Sqoop

Databricks Lakehouse Platform

Interactive Data Cleaning and Enrichment

Data Transformations

Data Encryption

Built-in Processors

Apache Sqoop

Databricks Lakehouse Platform

Multiple Model Development Languages and Tools

Automated Machine Learning

Single platform for multiple model development

Self-Service Model Delivery

Apache Sqoop

Databricks Lakehouse Platform

Flexible Model Publishing Options

Security, Governance, and Cost Controls

Pros

Provides generalized JDBC extensions to migrate data between most database systems
Generates Java classes upon reading database records for use in other code utilizing Hadoop's client libraries
Allows for both import and export features

Read full review

Jordan Moore

Consultant

Avalon Consulting, LLCInformation Technology and Services, 51-200 employees

View all 1 answers on this topic

Extremely Flexible in Data Scenarios
Fantastic Performance
DB is always updating the system so we can have latest features.

Read full review

Verified User

Director in Information Technology

Financial Services Company, 201-500 employees

View all 5 answers on this topic

Cons

Sqoop2 development seems to have stalled. I have set it up outside of a Cloudera CDH installation, and I actually prefer it's "Sqoop Server" model better than just the CLI client version that is Sqoop1. This works especially well in a microservices environment, where there would be only one place to maintain the JDBC drivers to use for Sqoop.

Read full review

Jordan Moore

Consultant

Avalon Consulting, LLCInformation Technology and Services, 51-200 employees

View all 1 answers on this topic

The navigation through which one would create a workspace is a bit confusing at first. It takes a couple minutes to figure out how to create a folder and upload files since it is not the same as traditional file systems such as box.com
Also, when you create a table, if you forgot to copy the link where the table is stored, it is hard to relocate it. Most of the time I would have to delete the table and re-created.

Read full review

Ann Le

Freelance Translator

ZOO Digital Group plcEntertainment, 501-1000 employees

View all 5 answers on this topic

Usability

No score

No answers yet

No answers on this topic

Databricks Lakehouse Platform 9.0

Based on 1 answer

This has been very useful in my organization for shared notebooks, integrated data pipeline automation and data sources integrations. Integration with AWS is seamless. Non tech users can easily learn how to use Databricks. You can have your company LDAP connect to it for login based access controls to some extent

Read full review

Verified User

Team Lead in Engineering

Financial Services Company, 10,001+ employees

View all 1 answers on this topic

Alternatives Considered

Sqoop comes preinstalled on the major Hadoop vendor distributions as the recommended product to import data from relational databases. The ability to extend it with additional JDBC drivers makes it very flexible for the environment it is installed within.
Spark also has a useful JDBC reader, and can manipulate data in more ways than Sqoop, and also upload to many other systems than just Hadoop.
Kafka Connect JDBC is more for streaming database updates using tools such as Oracle GoldenGate or Debezium.
Streamsets and Apache NiFi both provide a more "flow based programming" approach to graphically laying out connectors between various systems, including JDBC and Hadoop.

Read full review

Jordan Moore

Consultant

Avalon Consulting, LLCInformation Technology and Services, 51-200 employees

View all 1 answers on this topic

Easier to set up and get started. Less of a learning curve.

Read full review

Verified User

Director in Engineering

Financial Services Company, 10,001+ employees

View all 5 answers on this topic

Return on Investment

When combined with Cloudera's HUE, it can enable non-technical users to easily import relational data into Hadoop.
Being able to manipulate large datasets in Hadoop, and them load them into a type of "materialized view" in an external database system has yielded great insights into the Hadoop datalake without continuously running large batch jobs.
Sqoop isn't very user-friendly for those uncomfortable with a CLI.

Read full review

Jordan Moore

Consultant

Avalon Consulting, LLCInformation Technology and Services, 51-200 employees

View all 1 answers on this topic

Rapid growth of analytics within our company.
Cost model aligns with usage allowing us to make a reasonable initial investment and scale the cost as we realize the value.
Platform is easy to learn and Databricks provides excellent support and training.
Platform does not require a large DevOPs investment

Read full review

Verified User

Strategist in Engineering

Computer Hardware Company, 10,001+ employees

View all 5 answers on this topic

Pricing Details

General

Free Trial

—

Free/Freemium Version

—

Premium Consulting/Integration Services

—

Entry-level set up fee?

No

Apache Sqoop Editions & Modules

—

Additional Pricing Details

—

General

Free Trial

—

Free/Freemium Version

—

Premium Consulting/Integration Services

—

Entry-level set up fee?

No

Databricks Lakehouse Platform Editions & Modules

Edition

Standard	$0.07¹
Premium	$0.10¹
Enterprise	$0.13¹

Per DBU

Additional Pricing Details

—

Rating Summary

Apache Sqoop

Databricks Lakehouse Platform

Apache Sqoop

Databricks Lakehouse Platform

Likelihood to Recommend

Apache Sqoop

Databricks Lakehouse Platform

Feature Rating Comparison

Connect to Multiple Data Sources

Extend Existing Data Sources

Automatic Data Format Detection

Visualization

Interactive Data Analysis

Interactive Data Cleaning and Enrichment

Data Transformations

Data Encryption

Built-in Processors

Multiple Model Development Languages and Tools

Automated Machine Learning

Single platform for multiple model development

Self-Service Model Delivery

Flexible Model Publishing Options

Security, Governance, and Cost Controls

Pros

Apache Sqoop

Databricks Lakehouse Platform

Cons

Apache Sqoop

Databricks Lakehouse Platform

Usability

Apache Sqoop

Databricks Lakehouse Platform

Alternatives Considered

Apache Sqoop

Databricks Lakehouse Platform

Return on Investment

Apache Sqoop

Databricks Lakehouse Platform

Pricing Details

General

Free Trial

Free/Freemium Version

Premium Consulting/Integration Services

Entry-level set up fee?

Apache Sqoop Editions & Modules

Additional Pricing Details

General

Free Trial

Free/Freemium Version

Premium Consulting/Integration Services

Entry-level set up fee?

Databricks Lakehouse Platform Editions & Modules

Edition

Additional Pricing Details

Rating Summary

Add comparison