Don't confuse Kinesis with a Queue system.
August 22, 2018
Don't confuse Kinesis with a Queue system.
Score 1 out of 10
Vetted Review
Verified User
Overall Satisfaction with Amazon Kinesis
We currently use Kinesis as a trigger for Amazon Lambda functions, however we're working at switching back to SQS since Lambda now works with SQS directly. We simply used Kinesis as a temporary method of plumbing requests from one Lambda function to another.
We also use it as a byproduct of using DynamoDB Streams, but only in so far as that's used under-the-hood to link changes from DynamoDB to a Lambda trigger.
We also use it as a byproduct of using DynamoDB Streams, but only in so far as that's used under-the-hood to link changes from DynamoDB to a Lambda trigger.
- Link DynamoDB change events directly to Lambda
- Fast streaming of events
- Easy to "tail" changes/latest events
- Not a queue system, so little visibility into "backlog" if there is any
- Confusing terminology to make sure events aren't missed
- Sometimes didn't seem to trigger Lambda functions, or dropped events when a lot came in
- Caused us to need to re-engineer some basic re-try logic
- Caused us to drop some content without knowing it
- Made monitoring much more difficult
- We eventually switched back to SQS because Kinesis is not the same as a Queue system
- Amazon Simple Queue Service (SQS)
Actually we didn't select Kinesis, we were forced into using it because SQS wasn't yet supported by Lambda. Unlike Kinesis, SQS supports both FIFO and standard queues which let us control order of events processed, as well as handle retry logic, failover logic, and set up CloudWatch Alarms when we have too many events in our backlog. It also lets us know when something didn't process properly and we can alert DevOps to that issue and/or retry the processes automatically.
With SQS, we have much more visibility then we ever did with Kinesis. This is very important to our use cases, as we aren't processing millions of events per second, but for us every event is very important.
With SQS, we have much more visibility then we ever did with Kinesis. This is very important to our use cases, as we aren't processing millions of events per second, but for us every event is very important.