Don't confuse Kinesis with a Queue system.
Anonymous | TrustRadius Reviewer
August 22, 2018

Don't confuse Kinesis with a Queue system.

Score 1 out of 10
Vetted Review
Verified User
Review Source

Overall Satisfaction with Amazon Kinesis

We currently use Kinesis as a trigger for Amazon Lambda functions, however we're working at switching back to SQS since Lambda now works with SQS directly. We simply used Kinesis as a temporary method of plumbing requests from one Lambda function to another.

We also use it as a byproduct of using DynamoDB Streams, but only in so far as that's used under-the-hood to link changes from DynamoDB to a Lambda trigger.
  • Link DynamoDB change events directly to Lambda
  • Fast streaming of events
  • Easy to "tail" changes/latest events
  • Not a queue system, so little visibility into "backlog" if there is any
  • Confusing terminology to make sure events aren't missed
  • Sometimes didn't seem to trigger Lambda functions, or dropped events when a lot came in
  • Caused us to need to re-engineer some basic re-try logic
  • Caused us to drop some content without knowing it
  • Made monitoring much more difficult
  • We eventually switched back to SQS because Kinesis is not the same as a Queue system
  • Amazon Simple Queue Service (SQS)
Actually we didn't select Kinesis, we were forced into using it because SQS wasn't yet supported by Lambda. Unlike Kinesis, SQS supports both FIFO and standard queues which let us control order of events processed, as well as handle retry logic, failover logic, and set up CloudWatch Alarms when we have too many events in our backlog. It also lets us know when something didn't process properly and we can alert DevOps to that issue and/or retry the processes automatically.

With SQS, we have much more visibility then we ever did with Kinesis. This is very important to our use cases, as we aren't processing millions of events per second, but for us every event is very important.
It's great for big data applications, where it's not as important to make sure each event is processed, but you're more looking for overall analytics and speed is more important than absolute detail. It's added quite a few features like analytics and other tools specifically designed around big data, but it's not the same thing as a queue system.

Originally we were forced into using Kinesis since Lambda didn't support SQS directly, but now that there is native SQS support for Lambda, we'll be switching almost all of our implementation over to use that instead. Kinesis has its strengths, but monitoring and error retry logic is not one of those.