CloudWatch - Flexible Log Management At A Great Price Point
September 17, 2018
CloudWatch - Flexible Log Management At A Great Price Point
Score 7 out of 10
Vetted Review
Verified User
Overall Satisfaction with Amazon CloudWatch
We use Amazon CloudWatch to aggregate and retain logs across all of the different services that make up our infrastructure. It is primarily used across our engineering and dev ops departments. Using CloudWatch logs allows us to address compliance issues associated with log retention because it is very easy to configure an expiration (if any) for log files. We also use CloudWatch metrics to monitor important KPIs and performance metrics for our business.
- Managing log retention periods is very simple with CloudWatch, and can be configured on a per-group basis.
- Monitoring host performance is very easy when coupled with the CloudWatch Agent on an EC2 instance. A simple installation and configuration replaces an entire 3rd-party host monitoring stack.
- CloudWatch is flexible enough for not just host monitoring, but application monitoring as well. It's easy to pipe local logs up to CloudWatch and extract structured data in order to monitor and set alerts on custom app metrics.
- Unfortunately, the CloudWatch dashboard does not provide the ability to create histograms of discrete counts. This makes it difficult to, for instance, use CloudWatch to quickly identify specific IP addresses that have a high request volume in a certain period.
- The UX for creating a custom metric from a CloudWatch log group is somewhat confusing. Every time I need to create a new metric I find myself fumbling around the interface for a few minutes while I try to remember how to do it.
- The alerting options for CloudWatch are not as extensive as are available with some 3rd-party services.
- We were able to set up log streaming, retention, and simple downtime alerts within a few hours, having no prior experience with CloudWatch, freeing up our engineers to focus on more important business goals.
- CloudWatch log groups have made it relatively easy to detect and diagnose issues in production by allowing us to aggregate logs across servers, correlate failures, isolate misbehaving servers, etc. Thanks to CloudWatch, we are generally able to identify, understand and mitigate most production fires within 10-15 minutes.
- Choosing CloudWatch to manage log aggregation has saved us quite a bit of time and money over the past year. Generally, 3rd-party log aggregation solutions tend to get quite expensive unless you self-host, in which case you typically need to spend a fair amount of time setting up, maintaining, and monitoring these services.
We found that CloudWatch provides great value in terms of cost and maintenance time. It is cheap and requires virtually 0 upkeep. Of the other solutions we evaluated, Loggly and New Relic get quite expensive when you reach the volume of log data that we are processing even though in terms of set up and maintenance effort they are on par with CloudWatch. As dedicated log aggregation (Loggly) and application monitoring (New Relic) solutions, they do offer great features that are not exactly easy to replicate with CloudWatch, but we found that CloudWatch does more than enough to meet our needs. PaperTrail is also a relatively cheap option but has a very limited feature set. Finally, we have successfully used Graylog as part of a log aggregation stack (in tandem with fluentd and elasticsearch), and the dashboards you can create with this combination are simply spectacular. Unfortunately, even though the cost if self-hosting is relatively cheap, we found that maintaining that stack was difficult without having a dedicated engineer for it.