Enrich Prometheus AlertsΒΆ
Robusta can add extra context to your Prometheus alerts, so you can respond to alerts faster and without digging elsewhere for information.
In this tutorial, you will learn how to enrich alerts with two practical use cases.
Use Case 1: Enrich Alerts by Running a Bash ScriptΒΆ
Implementation:
Add the following YAML to the customPlaybooks
Helm value and update Robusta. This configures Robusta to execute the ps aux
command in response to the CPUThrottlingHigh
alert.
customPlaybooks:
- triggers:
- on_prometheus_alert:
alert_name: CPUThrottlingHigh
actions:
- node_bash_enricher:
bash_command: ps aux | head -n 5
Testing:
Trigger the alert we defined by deploying a Pod that consumes a lot of CPU:
kubectl apply -f https://raw.githubusercontent.com/robusta-dev/kubernetes-demos/main/cpu_throttling/throttling.yaml
Sample Alert:
Use Case 2: Enhance Alerts with Links to External DocumentationΒΆ
Implementation:
In this example, we add links to the alert KubeContainerCPURequestAlert
that we created in a previous tutorial.
Add the following YAML to the customPlaybooks
Helm value and update Robusta.
customPlaybooks:
- triggers:
- on_prometheus_alert:
alert_name: KubeContainerCPURequestAlert
actions:
- custom_graph_enricher:
graph_title: CPU Usage for this container
graph_duration_minutes: 5
chart_values_format: Plain
promql_query: 'sum(rate(container_cpu_usage_seconds_total{container="stress"}[5m])) by (pod)'
- template_enricher:
template: | # (1)
:scroll: Playbook <https://playbook-url/|Handling High Resource Utilization>
:github: Adjust CPU requests <https://github.com/YourRepository/|in the `Prod-sre` repository>
:notion: Internal Docs on <https://notion.com/path-to-docs/|Customizing CPU requests>
We're using custom emojis here that correspond to GitHub and Notion logos. Before you configure this, follow this guide to add emojis to your workspace.
Testing:
To test, deploy a resource-intensive pod to intentionally trigger the defined alert:
kubectl apply -f https://raw.githubusercontent.com/robusta-dev/kubernetes-demos/main/cpu_throttling/throttling.yaml
We can wait for the alert to fire, or we can speed things up and simulate the alert, as if it fired immediately:
robusta demo-alert --alert=KubeContainerCPURequestAlert --labels=label1=test,label2=alert
Once the alert fires, a notification will arrive with external links included.
Sample Alert:
Further ReadingΒΆ
View all Prometheus enrichment actions