Skip to content
Polo docs

From a Grafana alert to a verified recovery

This guide follows the acceptance journey “Grafana posted an outage alert to Slack”. An alert reaches Polo without anyone mentioning it. Polo opens an incident, investigates with read-only tools, proposes one exact fix, waits for approval, runs it, and checks that the service recovered.

VideoFrom an alert to a verified recovery

There are two ways for a Grafana alert to reach Polo. The direct webhook is the more complete path.

Path How it works Limits
Direct webhook Grafana sends signed alert notifications to Polo. Repeated deliveries of one alert update the same incident, and Polo records when Grafana resolves it.
Slack alert channel Polo reads messages that an approved alert bot posts in a channel you name. One task per message. Polo does not see the alert resolve.

Before you start

Section titled “Before you start”
  • An agent with the read tools an investigation needs, such as metrics, logs, Kubernetes status or uptime reads. The Monitoring agent template is a starting point. See Build your first agent.
  • The connections those tools read from. See Connections.
  • Runbooks for the service under Apps → Knowledge, tagged with the service and environment. The investigation reads them first.

Set up the direct webhook

Section titled “Set up the direct webhook”
  1. Open Apps → Intake and choose Connect source.
  2. Choose Grafana.
  3. Enter a Connection name and choose the Agent that investigates.
  4. Enter the Grafana organization ID and the Grafana receiver. Use the exact receiver name from the contact point.
  5. Choose Add alert rule and enter the Alert name from your rule. Add exact label matches, one label=value per line, to narrow it. Only the rules you list open work.
  6. Open Task context and permissions. Set the Environment and Service, and tick the Tools available to incoming work.
  7. Under Fixes Polo may propose, tick the changes the agent may propose for these alerts. Leave it empty for investigation only.
  8. Choose Create connection.

Polo shows a Webhook URL and a Grafana signing secret. Copy the secret before closing; it is shown once.

Then, in Grafana Alerting:

  1. Add a Webhook integration to the contact point. Use the webhook URL, the POST method and a maximum of 20 alerts.
  2. In the HMAC signature settings, enter the signing secret and use the X-Grafana-Alerting-Signature header.
  3. Set the timestamp header to X-Grafana-Alerting-Signature-Timestamp.
  4. Route the approved alert rules to this contact point.

The source reads as registered, with no delivery yet, until its first signed alert arrives.

Or use a Slack alert channel

Section titled “Or use a Slack alert channel”
  1. Open Apps → Intake, choose Connect source, then Slack, and connect your Slack workspace.
  2. In the source’s routes, open Alert bot messages.
  3. Enter the Alert channels and the Approved alert bot IDs. Both are required. The bot can post without tagging Polo.
  4. Set the task context, the tools and Fixes Polo may propose as above.

What happens when an alert fires

Section titled “What happens when an alert fires”
  1. The alert arrives. Polo checks the signature and drops duplicates before any model is called. A message that matches no approved rule opens nothing.
  2. An incident opens. It appears on Incidents under Active incidents, and the delivery is listed under Alerts received.
  3. Polo investigates. The task runs with read-only tools: it reads the runbooks that apply, then the uptime, metrics, logs and recent changes its tools allow, and reports the likely cause with evidence.
  4. Polo proposes one exact fix, if you allowed any under Fixes Polo may propose. It appears on Approvals under Waiting for you.
  5. Someone approves. In a production incident the rule can require an approver other than the person who set up the source. A standing approval for that exact change lets it run without waiting.
  6. The fix runs and is verified with a separate read.
  7. Recovery is confirmed. For a Grafana webhook alert, the incident counts as recovered once Grafana resolves the alert.

A Slack alert thread hears the proposal, the verified change and the recovery, once each.

Read how it ended

Section titled “Read how it ended”

Each incident on Incidents is told as a short story: opened, investigated, fix proposed, approved and run, recovered. The ending says whether the fix is why:

  • Recovered after Polo’s verified fix: the fix was verified before the alert resolved.
  • Recovered before Polo confirmed its fix worked: the fix had started but was not confirmed, so it may or may not be why.
  • Recovered on its own: Polo made no change, or its fix was not applied.

An alert that resolves alone does not prove a root cause was fixed. After a fix, Polo publishes an incident report covering the timeline, cause, fix, verification and follow-up.

If an incident has no investigation yet, choose Investigate now on it.

Limits to know

Section titled “Limits to know”
  • Alerts are not correlated across different sources or rules.
  • The service owner is found only if the agent reads the service catalog.
  • There is no postmortem workflow beyond the incident report.
Section titled “Related”