Apache Kafka is an open-source stream processing platform developed by the Apache Software Foundation written in Scala and Java. The Kafka event streaming platform is used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.
N/A
CircleCI
Score9.3 out of 10
N/A
CircleCI is a software delivery engine from the company of the same name in San Francisco, that helps teams ship software faster, offering their platform for Continuous Integration and Continuous Delivery (CI/CD). Ultimately, the solution helps to map every source of change for software teams, so they can accelerate innovation and growth.
$0
for up to 6,000 build minutes and up to 5 active users per month
Apache Kafka is well-suited for most data-streaming use cases. Amazon Kinesis and Azure EventHubs, unless you have a specific use case where using those cloud PaAS for your data lakes, once set up well, Apache Kafka will take care of everything else in the background. Azure EventHubs, is good for cross-cloud use cases, and Amazon Kinesis - I have no real-world experience. But I believe it is the same.
Based on our experience, CircleCI is well-suited for automating mobile app release cycles. For example, to release an iOS app, you would need to build, sign, and upload it to TestFlight, which requires a dedicated Mac in the office. But with CircleCI, you can have macOS executors, so you don't have to manage a physical build machine. Another benefit is that CircleCI's certified AWS Orbs abstract away complex authentication and deployment logic, allowing us to build, push, and deploy Docker containers to Amazon ECS with minimal configuration and high reliability. CircleCI is less suited for smaller projects where the development and deployment are not that extensive, for example, a static site. Once you have built a static site, you probably won't make any further changes, so there's no point in paying for it.
A de minimis incentive was given to thank the reviewer for their time. The incentive was not used to bias or drive a particular response, nor was the incentive contingent on a positive endorsement. More Info
Really easy to configure. I've used other message brokers such as RabbitMQ and compared to them, Kafka's configurations are very easy to understand and tweak.
Very scalable: easily configured to run on multiple nodes allowing for ease of parallelism (assuming your queues/topics don't have to be consumed in the exact same order the messages were delivered)
Not exactly a feature, but I trust Kafka will be around for at least another decade because active development has continued to be strong and there's a lot of financial backing from Confluent and LinkedIn, and probably many other companies who are using it (which, anecdotally, is many).
A de minimis incentive was given to thank the reviewer for their time. The incentive was not used to bias or drive a particular response, nor was the incentive contingent on a positive endorsement. More Info
Automated builds! This is really why you get CircleCI, to automate the build process. This makes building your application far more reliable and repeatable. It can also run tests and verify your application is working as expected.
Simple. Unlike Jenkins, Teamcity, or other platforms, CircleCI doesn't need a lot of setup. It's completely hosted, so there's no infrastructure to set up. The config file does take a bit to understand, but if you follow their example and start with something small and add to it, you can get it up and going quicker than it first looks.
Scales easily. Again, since it's all cloud-based, you don't have to manage or scale infrastructure. Simply subscribe to the number of containers you want, and scaling up just means buying more containers.
A de minimis incentive was given to thank the reviewer for their time. The incentive was not used to bias or drive a particular response, nor was the incentive contingent on a positive endorsement. More Info
Sometimes it becomes difficult to monitor our Kafka deployments. We've been able to overcome it largely using AWS MSK, a managed service for Apache Kafka, but a separate monitoring dashboard would have been great.
Simplify the process for local deployment of Kafka and provide a user interface to get visibility into the different topics and the messages being processed.
Learning curve around creation of broker and topics could be simplified
A de minimis incentive was given to thank the reviewer for their time. The incentive was not used to bias or drive a particular response, nor was the incentive contingent on a positive endorsement. More Info
A de minimis incentive was given to thank the reviewer for their time. The incentive was not used to bias or drive a particular response, nor was the incentive contingent on a positive endorsement. More Info
Apache Kafka is highly recommended to develop loosely coupled, real-time processing applications. Also, Apache Kafka provides property based configuration. Producer, Consumer and broker contain their own separate property file
A de minimis incentive was given to thank the reviewer for their time. The incentive was not used to bias or drive a particular response, nor was the incentive contingent on a positive endorsement. More Info
The reliability & speed, it just works. The ability to spin up macOS runners and Docker containers on demand without managing hardware is a huge win. The Orbs system makes integrating with AWS and Slack incredibly easy, saving us weeks of custom scripting and providing real-time updates in our Slack channel. This makes it easy for us to track and ensures that everyone involved knows the status. Of course, it has drawbacks related to configuration complexity and, in some cases, cost transparency, but overall, it is an industry-standard, robust tool that solves our core infrastructure problems well.
A de minimis incentive was given to thank the reviewer for their time. The incentive was not used to bias or drive a particular response, nor was the incentive contingent on a positive endorsement. More Info
It's pretty snappy, even with using workflows with multiple steps and different docker images. I've seen builds take a long time if it's really involved, but from what I can tell, it's still at least on par if not faster than other build tools.
A de minimis incentive was given to thank the reviewer for their time. The incentive was not used to bias or drive a particular response, nor was the incentive contingent on a positive endorsement. More Info
Support for Apache Kafka (if willing to pay) is available from Confluent that includes the same time that created Kafka at Linkedin so they know this software in and out. Moreover, Apache Kafka is well known and best practices documents and deployment scenarios are easily available for download. For example, from eBay, Linkedin, Uber, and NYTimes.
A de minimis incentive was given to thank the reviewer for their time. The incentive was not used to bias or drive a particular response, nor was the incentive contingent on a positive endorsement. More Info
Unless you have a reasonably large account, you're going to be mainly stuck reading their documentation. Which has improved somewhat over the years but is still extremely limited compared to a platform like Digital Ocean who invested in the documentation and a community to ensure it's kept up to date. If you can't find your answer there, you can be stuck.
A de minimis incentive was given to thank the reviewer for their time. The incentive was not used to bias or drive a particular response, nor was the incentive contingent on a positive endorsement. More Info
I used other messaging/queue solutions that are a lot more basic than Confluent Kafka, as well as another solution that is no longer in the market called Xively, which was bought and "buried" by Google. In comparison, these solutions offer way fewer functionalities and respond to other needs.
A de minimis incentive was given to thank the reviewer for their time. The incentive was not used to bias or drive a particular response, nor was the incentive contingent on a positive endorsement. More Info
Jenkins is usually self-hosted, Travis CI's infrastructure is largely unreliable (lots of tests time out for no discernable reason), and Semaphore encourages you to configure your CI/CD from a web UI. We like CircleCI because its hosted, our tests run largely as expected on their infrastructure, and we can configure it from a config file that we track in GitHub.
A de minimis incentive was given to thank the reviewer for their time. The incentive was not used to bias or drive a particular response, nor was the incentive contingent on a positive endorsement. More Info
Positive: Get a quick and reliable pub/sub model implemented - data across components flows easily.
Positive: it's scalable so we can develop small and scale for real-world scenarios
Negative: it's easy to get into a confusing situation if you are not experienced yet or something strange has happened (rare, but it does). Troubleshooting such situations can take time and effort.
A de minimis incentive was given to thank the reviewer for their time. The incentive was not used to bias or drive a particular response, nor was the incentive contingent on a positive endorsement. More Info
We pay over $5K/ month and we have high expectations for service. Sometimes I feel that we don't get the value, but only sometimes.
We have had to build our own application to keep state and broker releases and deployments. We call our app deployer. I feel that CircleCI could do more to understand our needs and possibly build additional features that would enable us to invest less in build and deployment infrastructure and justify paying more for Circle.
A de minimis incentive was given to thank the reviewer for their time. The incentive was not used to bias or drive a particular response, nor was the incentive contingent on a positive endorsement. More Info
