Incident alerts

Connect your alerting tools to see alerts in a chat channel, where the team is already talking. Each alert is a card that changes as the alert does (Triggered → Acknowledged → Resolved), so the channel shows what is still on fire. From a card you can create an issue, and with PagerDuty and Opsgenie you can also acknowledge and resolve the alert without leaving Deckbell.

Deckbell works with:

• PagerDuty incidents
• Opsgenie alerts
• Datadog monitor alerts
• Grafana Alerting (as a webhook contact point)
• Better Stack Uptime incidents

Open Integrations, pick the tool and choose Configure. You need permission to manage integrations.

Adding a connection

Choose Add connection and give it a name (for example "PagerDuty production"). Deckbell makes a webhook URL and, for every tool except PagerDuty, a token. Paste them into the tool as described below for each one. Until a delivery comes with the right token or signature, Deckbell refuses it.

You can add several connections for the same tool, one per account, team or environment. Each has its own URL, token, channel and rules.

Each connection card shows when the last alert arrived and how many are still open.

Alert cards in chat

Pick a channel under Post alerts to. Each new alert shows up there as a card with its status, severity, service, the time it triggered (and who acknowledged or resolved it), and a link to the alert in the tool.

The same alert is never posted twice. When the tool sends an update for it (acknowledged, resolved, priority changed, re-triggered), the card changes in place instead of a new message being posted.

Use Which alerts to only post alerts of a certain severity or higher. Deckbell sorts alerts into four severities: Info, Warning, High and Critical. PagerDuty and Opsgenie priorities P1 to P5 map to Critical, High, Warning and Info (P4 and P5 are Info). Without a priority, PagerDuty's high urgency is High and low urgency is Warning. Grafana uses the alert's severity label, and Better Stack incidents are High.

Creating issues from alerts

• By hand: choose Create issue on the card. The issue gets the alert's title, severity, service and a link to the alert, and the card then shows the issue.
• Automatically: under Create issues automatically, pick a board and a severity. Every new alert at that severity or higher becomes an issue in the board's first to-do column, with a priority to match (Critical is Urgent).

With a PagerDuty or Opsgenie API key (below), Deckbell also adds a note to the incident with a link to the issue, so the link goes both ways.

Under When the alert resolves, choose what happens to the alert's issue:

• Comment on its issue (the default): a comment says the alert resolved and who resolved it.
• Comment and move its issue to done.
• Leave its issue alone.

Acknowledge and resolve from Deckbell

PagerDuty and Opsgenie connections can also get Acknowledge and Resolve buttons on their cards. Add an API key under Acknowledge and resolve from Deckbell. Deckbell checks the key before saving it, stores it encrypted, and never shows it again. Remove key turns the buttons off.

Anyone who can manage issues can use the buttons. The card updates right away, and the tool confirms the change through its webhook.

PagerDuty

1. In PagerDuty, open Integrations → Generic Webhooks (v3) and choose New Webhook.
2. Paste the Deckbell webhook URL, pick a service (or the whole account), and select the incident events: triggered, acknowledged, unacknowledged, resolved, reopened, escalated and priority updated.
3. PagerDuty shows a signing secret once. Paste it into Deckbell under Signing secret and save. Deckbell checks the signature on every delivery.

To get the buttons on cards, create a REST API key with read and write access in PagerDuty (Integrations → API Access Keys). Paste it into Deckbell along with the email address of the PagerDuty user the changes should be made as.

If you regenerate the signing secret in PagerDuty, paste the new one into Deckbell.

Opsgenie

1. In Opsgenie, open Settings → Integrations, add a Webhook integration, and paste the Deckbell webhook URL.
2. Add a custom header named X-Deckbell-Token with the Deckbell token as its value.
3. Turn on Add Alert Description to Payload, then save and turn on the integration.

For the buttons on cards, create an API integration in Opsgenie with Read, Create and Update access. Paste its API key into Deckbell and pick your region (US or EU).

Datadog

1. In Datadog, open Integrations → Webhooks and add a webhook. Name it, for example, deckbell, and paste the Deckbell webhook URL.

2. Under Custom Headers, enter {"X-Deckbell-Token": "<your token>"}.

3. Under Payload, use this payload so Deckbell can tell triggers from recoveries and read the priority:

   {
     "id": "$ID",
     "alert_id": "$ALERT_ID",
     "aggregate": "$AGGREG_KEY",
     "title": "$EVENT_TITLE",
     "transition": "$ALERT_TRANSITION",
     "priority": "$ALERT_PRIORITY",
     "alert_type": "$ALERT_TYPE",
     "link": "$LINK",
     "date": "$DATE",
     "last_updated": "$LAST_UPDATED",
     "hostname": "$HOSTNAME",
     "body": "$EVENT_MSG",
     "alert_scope": "$ALERT_SCOPE"
   }
   

4. Mention @webhook-deckbell in the message of each monitor whose alerts should go to Deckbell.

A monitor's alert and its recovery update the same card. Datadog monitors can't be acknowledged from Deckbell.

Grafana

1. In Grafana, open Alerting → Contact points and add a contact point with the Webhook integration.
2. Paste the Deckbell webhook URL.
3. Under the optional settings, authenticate with the Deckbell token in one of these ways:
   • Authorization header: scheme Bearer, credentials the token.
   • Basic auth: any username, the token as the password.
   • HMAC signature: the token as the secret.
4. Send alerts to the contact point from a notification policy or an alert rule.

Grafana groups alerts into one notification. Deckbell makes a separate card for each alert in the group, and updates it when that alert resolves. Set a severity label (critical, high, warning or info) on your alert rules to control the severity. Alerts without one are High.

Better Stack

1. In Better Stack, open Uptime → Integrations and add a Webhook integration.
2. Paste the Deckbell webhook URL. It already includes the token, so keep it private.
3. Add the integration to the escalation policies whose incidents should go to Deckbell.

Incidents update their card when they're acknowledged and resolved.

Tokens and security

• Copy the token from the connection card. Rotate makes a new token, and the old one stops working immediately, so update it in the tool right after.
• PagerDuty deliveries are checked against PagerDuty's signature. The others must carry the token (Grafana can sign with it instead).
• Each webhook URL accepts up to 120 deliveries a minute. Extra deliveries are turned away and the tool retries them later.
• Remove deletes a connection. Its URL stops working, and cards already in chat stay where they are.

Adding, changing, rotating and removing connections are recorded in the workspace's audit log.