Remediate Prometheus Alerts¶

Robusta can respond to Prometheus alerts and automatically remediate them.

Using Kubernetes Jobs for Alert Remediation¶

A popular remediation method involves running Kubernetes Jobs when alerts fire.

Add the following to your customPlaybooks:

customPlaybooks:
- triggers:
  - on_prometheus_alert:
      alert_name: SomeAlert
  actions:
  - alert_handling_job:
      command:
      - "echo"
      - "do something to fix it"
      image: "a_docker_image"

Perform a Helm Upgrade.

Test this playbook by simulating a Prometheus alert:

robusta playbooks trigger prometheus_alert alert_name=SomeAlert

Running Bash Commands for Alert Remediation¶

Alerts can also be remediated with bash commands:

customPlaybooks:
- triggers:
  - on_prometheus_alert:
      alert_name: SomeAlert
  actions:
  - node_bash_enricher:
     bash_command: do something

Further Reading¶

• Reference for the alert_handling_job action

• Reference for the node_bash_enricher action

• More remediation actions