Cloudera Data Hub

Cloudera Data Hub is a cloud-native compute orchestration and analytics provisioning service within the Cloudera Data Platform (CDP) that deploys distributed Big Data Processing and Distribution Systems across multi-cloud environments. The service automates the provisioning, lifecycle management, and scaling of tailored multi-node workload clusters powered by Cloudera Runtime, executing distributed batch processing, interactive Structured Query Language (SQL) analytics, real-time data streaming, and operational NoSQL workloads on public cloud infrastructure.

Key Capabilities

• Workload-Tailored Cluster Templates: Provisions dedicated compute clusters utilizing declarative JavaScript Object Notation (JSON) templates that define host groups, node layout, service configurations, and hardware sizing. Workload templates include:
  • Real-Time Data Mart: Combines the Apache Kudu columnar storage engine and the Apache Impala distributed massively parallel processing (MPP) SQL query engine for low-latency analytical queries and change data capture (CDC) reconciliation.
  • Data Engineering: Deploys Apache Spark and Apache Hive execution runtimes to process large-scale batch transformations and distributed extract, transform, load (ETL/ELT) data pipelines.
  • Streaming & Ingestion (DataFlow for Data Hub): Leverages Apache NiFi and Apache Kafka to ingest, route, parse, and transform high-throughput edge and enterprise event streams.
  • Operational Database: Provisions Apache HBase alongside Apache Phoenix to support low-latency key-value storage and ANSI SQL transactions across petabyte-scale operational datasets.
• Decoupled Cloud Compute & Storage: Separates ephemeral compute resources from object storage layers. Clusters scale worker nodes dynamically while persisting datasets to cloud-native object storage, including Amazon Simple Storage Service (Amazon S3), Azure Data Lake Storage Gen2 (ADLS Gen2), and Google Cloud Storage (GCS), alongside Apache Ozone object stores.
• Shared Data Governance (Cloudera SDX): Integrates automatically with the centralized Shared Data Experience (SDX) data lake layer running within each customer environment. Workload clusters inherit centralized security, role-based access control policies via Apache Ranger, end-to-end data lineage tracking and cataloging via Apache Atlas, and unified Kerberos authentication.
• Automated Lifecycle & Operational Management: Interfaces with Cloudera Manager and the CDP Control Plane to manage cluster scaling, health telemetry, automated software patching, and pre-warmed image deployments across public cloud zones.

Audience & Use Cases

• Audience: Big data architects, enterprise data engineers, database administrators, and cloud infrastructure engineers managing large-scale distributed analytics environments.
• Use Case: Migrating legacy on-premises Hadoop deployments (CDH and HDP) to public cloud providers, executing high-throughput batch ETL transformations, performing sub-second interactive SQL queries on fast-arriving event streams, and managing distributed messaging pipelines.

Technical Specifications

• Supported Cloud Infrastructure: Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP).
• Underlying Engine: Cloudera Runtime (integrating Apache Spark, Apache Impala, Apache Hive, Apache Kudu, Apache Kafka, Apache NiFi, Apache HBase, and Apache Ozone).
• Control Interface: Accessible via the Cloudera Management Console web interface, the CDP Command-Line Interface (CLI), and Cloudera Manager RESTful Application Programming Interfaces (APIs).
• Security & Directory Services: Apache Ranger for fine-grained authorization and dynamic data masking, Apache Atlas for metadata governance, Kerberos for cluster authentication, and integration with enterprise LDAP / Active Directory identity providers.