Integrate Your Monitoring System With PagerDuty Using Events API V2

Introduction

PagerDuty's Events API V2 lets you push events from your monitoring systems to PagerDuty. You can send such events when an incident is triggered, updated, or resolved. This article is a short guide on the different options to integrate PagerDuty with your monitoring and alerting systems.

• Introduction
• Lifecycle of an Incident
• The PagerDuty Events API v2
  • Prerequisites for Integrating with the Events API V2
  • Getting Your PagerDuty Integration Key
  • A Simple Example in NodeJS
  • The Dedup Key
• Conclusion

Lifecycle of an Incident

If you are using PagerDuty for on-call management, an incident will typically go through these states:

State	Triggered By	Source
Triggered	Automatic	Monitoring system
Acknowledged	On-call Engineer	PagerDuty app/Phone call
Updated	Automatic	Monitoring system
Resolved	On-call Engineer	PagerDuty app/Phone call

Once an incident is created in PagerDuty's system, its state will keep changing either in response to events sent by your alerting systems, or by an on-call engineer changing it manually from the PagerDuty app or dashboard.

The PagerDuty Events API v2

The Events API V2 allows you manage incidents directly using an API call over HTTPS. You might want to do this if you have one or more of the following:

• A off-the-shelf monitoring systems that generates, updates, and resolves alerts, and you want these to be sent to PagerDuty.
• A custom monitoring system that creates, updates, and resolves alerts, and you want these to be sent to PagerDuty.

The Events API V2 is an asynchronous API - i.e. it will return a response immediately and then process the call in a separate thread.

Prerequisites for Integrating with the Events API V2

You need a routing key, also called an integration key, to invoke the Events API V2.

Getting Your PagerDuty Integration Key

Follow these steps to get your integration key:

• Login to your PagerDuty account
• Go to Services -> Service Directory
• Click on "New Service"
• Add Name and Description
• Choose the Escalation Policy to apply to the service
• Under "Integrations", choose "Events API V2" and click on Create Service
• Once you create the service, copy the "Integration Key"

A Simple Example in NodeJS

To run this example, you have to install the official Javascript SDK from https://www.npmjs.com/package/@pagerduty/pdjs.

A simple event push will look look like this:

import { event } from "@pagerduty/pdjs";

.....
    event({
        "data": {
            "routing_key": "Your-Routing-Key-Here",
            "event_action": "trigger",
            "dedup_key": DEDUP_KEY,
            "payload": {
                "summary": "Event processor in us-east-1",
                "source": "rnmd-2398.xyzcloud.io",
                "severity": "critical",
                "timestamp": "2024-07-17T08:42:58.315+0000",
            },
            "links": [
                {
                    "href": "https://incidenthub.cloud/dashboard",
                    "text": "Go to dashboard",
                },
            ],
        },
.....

The Dedup Key

When your monitoring system sends an event to trigger an incident, it's important to have a unique DEDUP_KEY. This field determines whether subsequent events for this incident will be grouped together in PagerDuty. When your system sends an update, or a resolved event, the DEDUP_KEY must match the one sent during the trigger call. In other words, the DEDUP_KEY must be unique per incident. This allows PagerDuty to correctly distinguish between events sent for different incidents.

IncidentHub integrates with PagerDuty and uses the incident's public URL as the DEDUP_KEY as that is unique globally, and also remains the same for an incident. Each incident update event has the same DEDUP_KEY.

If you are integrating PagerDuty with some other monitoring system, the same rules apply. Make sure that you use an identifier that is unique in your system for the DEDUP_KEY value.

Let us look at a Google Cloud example. An incident affecting Anthos Service Mesh in Nov 2023 went through 4 updates including trigger and resolve. The URL remained the same for the incident as it went through the lifecycle.

Conclusion

Integrating with the PagerDuty API is straightforward. While integrating make sure you keep these things in mind:

• Map your monitoring system's alert priorities to the appropriate PagerDuty levels.
• Use the correct DEDUP_KEY for each case.
• Provide additional details about the incident using the custom fields. E.g. in the example above, you can provide more information about the incident using a custom link pointing to the dashboard.

For more details about the API, its rate limits, response codes, and request structure, please refer to the official PagerDuty documentation. You can use any of the PagerDuty client SDKs to send events or roll out your own.

The PagerDuty brand name and logo are registered trademarks of PagerDuty, Inc.
IncidentHub is not affiliated with PagerDuty in any way.