A Dream for Service Providers with Complicated On-Call Scheduling and Event Management

The PagerDuty platform is used across the organization for on-call management, event/alert management and automated escalations. The platform solved a significant inefficiency within the business related to on-call and escalations. Prior to PagerDuty, escalations were manual and not well documented. This made it difficult to hold people accountable and properly document problems. Through PagerDuty's escalation policies, on-call schedules and general workflow automation capabilities, the problem was solved fairly quick. Because our on-call team members are paid each time they answer an on-call request, this made tracking the notifications much easier.

Generally speaking the PagerDuty platform is reliable. Outages do happen on occasion. These outages are significantly felt by our organization even if it is for a few minutes. Because we leverage PagerDuty as our "eyes on glass" platform, we often see problems well before PagerDuty support identifies the problem. I can only recall one situation where the platform was completely down. Typically it is one or two minor components in a degraded state.

We do not use any of the native/pre-built integration capabilities. We have built our own integrations using the events API and webhooks. This gives us additional flexibility how information is sent/received from the PagerDuty platform. We have tested some of the native integrations but found they lacked flexibility. The custom code we developed is deployed on a web services engine backed by service bus technology. For most organizations this is likely overkill but necessary if you are integrated multiple tools together.

PagerDuty is our first response platform. Therefore, all alerts and events flow through the platform first before being escalated to a person. All events stream in to the front-end UI and are picked up manually by our tier 1 staff. If the tier 1 staff decide the problem requires immediate escalation to a higher tier team member, they will invoke an escalation policy for the appropriate resource. The higher tier staff decide how they want to be notified when a low or high priority incident is escalated to them i.e. notifications via phone, email, text and push notification or some combination of them all. PagerDuty is our system of record for all events and alerts. A separate tool is used for our incident system of record. We have build a webhook that will generate an incident in our ITSM platform.