Our use of PagerDuty

PagerDuty is used across the whole organization to help us solve incident response and triage. We integrated with Slack, which is our main information gathering and triage platform. We are also using analytics and Response Play to help us analyze our incident, alert frequency, and trigger actions. We tried to use the postmortem feature but decided not to opt-in due to usability challenges particularly on the collaboration area.

We appreciate the high uptime of PD as it plays a critical role in our overall incident response process. It is the main alert trigger and triage mechanism for our organization.

PagerDuty allows us to trigger an alert by means of API token and email, which we used to integrate with Slack, Grafana, and Icinga. We have quite a few tools we use internally and we are able to integrate them all with PagerDuty.

In our stage environment, we often have problems with memory in Sandbox. In the past, we have to manually intervene whenever our Sandbox starts swapping due to bug or overutilization. With Response Play, we are able to create and kickoff an auto-remediation process to fix the problem without human intervention.

It gives us insight into how often we receive alerts and the nature of it. It also helps us to look at the workload of my team members and identify when we receive most of the alerts which might be related to capacity and traffic. We are also able to us analyze and identify services that are more vulnerable to high load.