Apache Kafka is an open-source stream processing platform developed by the Apache Software Foundation written in Scala and Java. The Kafka event streaming platform is used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.
N/A
Oracle Data Integrator (ODI)
Score 8.4 out of 10
N/A
Oracle Data Integrator is an ELT data integrator designed with interoperability other Oracle programs. The program focuses on a high-performance capacity to support Big Data use within Oracle.
ODI is the naturel successor of OWB, adopting the same EL-T approach but supporting a lot more technologies as source and target. The overall product is much more stable and not tied to the Oracle Database. Unlike Informatica, ODI generates all the code in the native underlying …
Apache Kafka is well-suited for most data-streaming use cases. Amazon Kinesis and Azure EventHubs, unless you have a specific use case where using those cloud PaAS for your data lakes, once set up well, Apache Kafka will take care of everything else in the background. Azure EventHubs, is good for cross-cloud use cases, and Amazon Kinesis - I have no real-world experience. But I believe it is the same.
Oracle Data Integrator is well suited in all the situations where you need to integrate data from and to different systems/technologies/environments or to schedule some tasks. I've used it on Oracle Database (Data Warehouses or Data Marts), with great loading and transforming performances to accomplish any kind of relational task. This is true for all Oracle applications (like Hyperion Planning, Hyperion Essbase, Hyperion Financial Management, and so on). I've also used it to manage files on different operating systems, to execute procedures in various languages and to read and write data from and to non-Oracle technologies, and I can confirm that its performances have always been very good. It can become less appropriate depending on the expenses that can be afforded by the customer since its license costs are quite high.
Really easy to configure. I've used other message brokers such as RabbitMQ and compared to them, Kafka's configurations are very easy to understand and tweak.
Very scalable: easily configured to run on multiple nodes allowing for ease of parallelism (assuming your queues/topics don't have to be consumed in the exact same order the messages were delivered)
Not exactly a feature, but I trust Kafka will be around for at least another decade because active development has continued to be strong and there's a lot of financial backing from Confluent and LinkedIn, and probably many other companies who are using it (which, anecdotally, is many).
Oracle Data Integrator nearly addresses every data issue that one can expect. Oracle Data Integrator is tightly integrated to the Oracle Suite of products. This is one of the major strengths of Oracle Data Integrator. Oracle Data Integrator is part of the Oracle Business Intelligence Applications Suite - which is highly used by various industries. This tool replaced Informatica ETL in Oracle Business Intelligence Applications Suite.
Oracle Data Integrator comes with many pre-written data packages. If one has to load data from Excel to Oracle Database, there is a package that is ready available for them - cutting down lot of effort on writing the code. Similarly, there are packages for Oracle to SQL, SQL to Oracle and all other possible combinations. Developers love this feature.
Oracle Data Integrator relies highly on the database for processing. This is actually an ELT tool rather than an ETL tool. It first loads all the data into target instance and then transforms it at the expense of database resources. This light footprint makes this tool very special.
The other major advantage of Oracle Data Integrator, like any other Oracle products, is a readily available developer pool. As all Oracle products are free to download for demo environments, many organizations prefer to play around with a product before purchasing it. Also, Oracle support and community is a big advantage compared to other vendors.
Sometimes it becomes difficult to monitor our Kafka deployments. We've been able to overcome it largely using AWS MSK, a managed service for Apache Kafka, but a separate monitoring dashboard would have been great.
Simplify the process for local deployment of Kafka and provide a user interface to get visibility into the different topics and the messages being processed.
Learning curve around creation of broker and topics could be simplified
ODI does not have an intuitive user interface. It is powerful, but difficult to figure out at first. There is a significant learning curve between usability, proficiency, and mastery of the tool.
ODI contains some frustrating bugs. It is Java based and has some caching issues, often requiring you to restart the program before you see your code changes stick.
ODI does not have a strong versioning process. It is not intuitive to keep an up to date repository of versioned code packages. This can create versioning issues between environments if you do not have a strong external code versioning process.
It is maturing and over time will have a good pool of resources. Each new version has addressed the issues of the previous ones. Its getting better and bigger.
Apache Kafka is highly recommended to develop loosely coupled, real-time processing applications. Also, Apache Kafka provides property based configuration. Producer, Consumer and broker contain their own separate property file
Support for Apache Kafka (if willing to pay) is available from Confluent that includes the same time that created Kafka at Linkedin so they know this software in and out. Moreover, Apache Kafka is well known and best practices documents and deployment scenarios are easily available for download. For example, from eBay, Linkedin, Uber, and NYTimes.
I used other messaging/queue solutions that are a lot more basic than Confluent Kafka, as well as another solution that is no longer in the market called Xively, which was bought and "buried" by Google. In comparison, these solutions offer way fewer functionalities and respond to other needs.
I have used Trifacta Google Data Prep quite a bit. We use Google Cloud Platform across our organization. The tools are very comparable in what they offer. I would say Data Prep has a slight edge in usability and a cleaner UI, but both of the tools have comparable toolsets.
Positive: Get a quick and reliable pub/sub model implemented - data across components flows easily.
Positive: it's scalable so we can develop small and scale for real-world scenarios
Negative: it's easy to get into a confusing situation if you are not experienced yet or something strange has happened (rare, but it does). Troubleshooting such situations can take time and effort.
