Apache Kafka is an open-source stream processing platform developed by the Apache Software Foundation written in Scala and Java. The Kafka event streaming platform is used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.
N/A
TeamCity
Score 7.1 out of 10
N/A
TeamCity is a continuous integration server from Czeck company JetBrains.
Apache Kafka is well-suited for most data-streaming use cases. Amazon Kinesis and Azure EventHubs, unless you have a specific use case where using those cloud PaAS for your data lakes, once set up well, Apache Kafka will take care of everything else in the background. Azure EventHubs, is good for cross-cloud use cases, and Amazon Kinesis - I have no real-world experience. But I believe it is the same.
TeamCity is very quick and straightforward to get up and running. A new server and a handful of agents could be brought online in easily under an hour. The professional tier is completely free, full-featured, and offers a huge amount of growth potential. TeamCity does exceptionally well in a small-scale business or enterprise setting.
Really easy to configure. I've used other message brokers such as RabbitMQ and compared to them, Kafka's configurations are very easy to understand and tweak.
Very scalable: easily configured to run on multiple nodes allowing for ease of parallelism (assuming your queues/topics don't have to be consumed in the exact same order the messages were delivered)
Not exactly a feature, but I trust Kafka will be around for at least another decade because active development has continued to be strong and there's a lot of financial backing from Confluent and LinkedIn, and probably many other companies who are using it (which, anecdotally, is many).
Sometimes it becomes difficult to monitor our Kafka deployments. We've been able to overcome it largely using AWS MSK, a managed service for Apache Kafka, but a separate monitoring dashboard would have been great.
Simplify the process for local deployment of Kafka and provide a user interface to get visibility into the different topics and the messages being processed.
Learning curve around creation of broker and topics could be simplified
The customization is still fairly complex and is best managed by a dev support team. There is great flexibility, but with flexibility comes responsibility. It isn't always obvious to a developer how to make simple customizations.
Sometimes the process for dealing with errors in the process isn't obvious. Some paths to rerunning steps redo dependencies unnecessarily while other paths that don't are less obvious.
Apache Kafka is highly recommended to develop loosely coupled, real-time processing applications. Also, Apache Kafka provides property based configuration. Producer, Consumer and broker contain their own separate property file
TeamCity runs really well, even when sharing a small instance with other applications. The user interface adequately conveys important information without being overly bloated, and it is snappy. There isn't any significant overhead to build agents or unit test runners that we have measured.
Support for Apache Kafka (if willing to pay) is available from Confluent that includes the same time that created Kafka at Linkedin so they know this software in and out. Moreover, Apache Kafka is well known and best practices documents and deployment scenarios are easily available for download. For example, from eBay, Linkedin, Uber, and NYTimes.
I used other messaging/queue solutions that are a lot more basic than Confluent Kafka, as well as another solution that is no longer in the market called Xively, which was bought and "buried" by Google. In comparison, these solutions offer way fewer functionalities and respond to other needs.
TeamCity is a great on-premise Continuous Integration tool. Visual Studio Team Services (VSTS) is a hosted SAAS application in Microsoft's Cloud. VSTS is a Source Code Repository, Build and Release System, and Agile Project Management Platform - whereas TeamCity is a Build and Release System only. TeamCity's interface is easier to use than VSTS, and neither have a great deployment pipeline solution. But VSTS's natural integration with Microsoft products, Microsoft's Cloud, Integration with Azure Active Directory, and free, private, Source Code repository - offer additional features and capabilities not available with Team City alone.
Positive: Get a quick and reliable pub/sub model implemented - data across components flows easily.
Positive: it's scalable so we can develop small and scale for real-world scenarios
Negative: it's easy to get into a confusing situation if you are not experienced yet or something strange has happened (rare, but it does). Troubleshooting such situations can take time and effort.
TeamCity has greatly improved team efficiency by streamlining our production and pre-production pipelines. We moved to TeamCity after seeing other teams have more success with it than we had with other tools.
TeamCity has helped the reliability of our product by easily allowing us to integrate unit testing, as well as full integration testing. This was not possible with other tools given our corporate firewall.
TeamCity's ability to include Docker containers in the pipeline steps has been crucial in improving our efficiency and reliability.
