Link Alerts to External Docs

When troubleshooting production incidents, quick access to relevant documentation and runbooks is crucial!

This guide demonstrates how to link Prometheus alerts with external links to your technical documentation, GitHub repositories, and internal wikis.

Implementation

In this example, we'll add links to a Prometheus alert. We're using KubeContainerCPURequestAlert as an example, but you can apply this to any alert.

Below there are three alternatives ways to enrich the alert with links. Apply the YAML to the customPlaybooks Helm value and update Robusta.

customPlaybooks:
- triggers:
  - on_prometheus_alert:
      alert_name: KubeContainerCPURequestAlert
  actions:
  - custom_graph_enricher:
      graph_title: CPU Usage for this container
      graph_duration_minutes: 5
      chart_values_format: Plain
      promql_query: 'sum(rate(container_cpu_usage_seconds_total{container="stress"}[5m])) by (pod)'
  - template_enricher:
      template: | # (1)
        :scroll: Playbook <https://playbook-url/|Handling High Resource Utilization>
        :github: Adjust CPU requests <https://github.com/YourRepository/|in the `Prod-sre` repository>
        :notion: Internal Docs on <https://notion.com/path-to-docs/|Customizing CPU requests>
  - button_enricher:
      button_text: "Playbook for ${team}"
      button_url: "https://playbook-url/team/${team}"

  1. We're using custom emojis here that correspond to GitHub and Notion logos. Before you configure this, follow this guide to add emojis to your workspace.

Testing

To test, deploy a resource-intensive pod to intentionally trigger the defined alert:

kubectl apply -f https://raw.githubusercontent.com/robusta-dev/kubernetes-demos/main/cpu_throttling/throttling.yaml

We can wait for the alert to fire, or we can speed things up and simulate the alert, as if it fired immediately:

robusta demo-alert --alert=KubeContainerCPURequestAlert --labels=label1=test,label2=alert

Once the alert fires, a notification will arrive with external links included.

Sample Alert

Further Reading