Event Enrichment

The actions are used to gather extra data on errors, alerts, and other cluster events.

Use them as building blocks in your own automations.

Node Enrichers

These actions can add context to any node-related event, be it from on_prometheus_alert or on_node_update.

Node bash enricher

Playbook Action: node_bash_enricher

Execute the specified bash command on the target node. Enrich the finding with the command results.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- node_bash_enricher:
    bash_command: ls -l /etc/data/db
triggers:
- on_node_all_changes: {}

The above is an example. Try customizing the trigger and parameters.

required:
bash_command (str)

Bash command to execute on the target.

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger node_bash_enricher name=NODE_NAME  bash_command=BASH_COMMAND

Node status enricher

Playbook Action: node_status_enricher

Enrich the finding with the node's status conditions.

Can help troubleshooting Node issues.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- node_status_enricher: {}
triggers:
- on_node_all_changes: {}

The above is an example. Try customizing the trigger and parameters.

No action parameters

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger node_status_enricher name=NODE_NAME 

Node running pods enricher

Playbook Action: node_running_pods_enricher

Enrich the finding with pods running on this node, along with the 'Ready' status of each pod.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- node_running_pods_enricher: {}
triggers:
- on_node_all_changes: {}

The above is an example. Try customizing the trigger and parameters.

No action parameters

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger node_running_pods_enricher name=NODE_NAME 

Node allocatable resources enricher

Playbook Action: node_allocatable_resources_enricher

Enrich the finding with the node resources available for allocation.

Can help troubleshooting node issues.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- node_allocatable_resources_enricher: {}
triggers:
- on_node_all_changes: {}

The above is an example. Try customizing the trigger and parameters.

No action parameters

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger node_allocatable_resources_enricher name=NODE_NAME 

Node graph enricher

Playbook Action: node_graph_enricher

Get a graph of a specific resource for this node.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- node_graph_enricher:
    prometheus_url: http://prometheus-k8s.monitoring.svc.cluster.local:9090
    resource_type: Memory
triggers:
- on_node_all_changes: {}

The above is an example. Try customizing the trigger and parameters.

required:
resource_type (str)

one of: CPU, Memory, Disk (see ResourceChartResourceType)

optional:
prometheus_url (str)

Prometheus url. If omitted, we will try to find a prometheus instance in the same cluster

graph_duration_minutes (int) = 60

Graph duration is minutes. Default is 60.

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger node_graph_enricher name=NODE_NAME  resource_type=RESOURCE_TYPE

Node cpu enricher

Playbook Action: node_cpu_enricher

Enrich the finding with analysis of the node's CPU usage. Collect information about pods running on this node, their CPU request configuration, their actual cpu usage etc. Provides insightful information regarding node high CPU usage.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- node_cpu_enricher:
    prometheus_url: http://prometheus-k8s.monitoring.svc.cluster.local:9090
triggers:
- on_node_all_changes: {}

The above is an example. Try customizing the trigger and parameters.

optional:
prometheus_url (str)

Prometheus url. If omitted, we will try to find a prometheus instance in the same cluster

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger node_cpu_enricher name=NODE_NAME 

Pod Enrichers

These actions can add context to any pod-related event, be it from on_prometheus_alert or on_pod_update.

Logs enricher

Playbook Action: logs_enricher

Enrich the alert with pod logs The pod to fetch logs for is determined by the alert’s pod label from Prometheus.

By default, if the alert has no pod this enricher will silently do nothing.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- logs_enricher: {}
triggers:
- on_pod_crash_loop: {}

The above is an example. Try customizing the trigger and parameters.

optional:
warn_on_missing_label (bool)

Send a warning if the alert doesn't have a pod label

regex_replacer_patterns (complex list)

regex patterns to replace text, for example for security reasons (Note: Replacements are executed in the given order)

each entry contains:

required:
regex (str)
optional:
name (str) = Redacted
regex_replacement_style (str)

one of SAME_LENGTH_ASTERISKS or NAMED (See RegexReplacementStyle)

previous (bool)

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger logs_enricher name=POD_NAME namespace=POD_NAMESPACE 

Pod events enricher

Playbook Action: pod_events_enricher

Given a Kubernetes pod, fetch related events in the near past

Add this to your Robusta configuration (Helm values.yaml):

actions:
- pod_events_enricher: {}
triggers:
- on_pod_all_changes: {}

The above is an example. Try customizing the trigger and parameters.

optional:
max_events (int) = 8
included_types (str list) = ['Warning', 'Normal']

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger pod_events_enricher name=POD_NAME namespace=POD_NAMESPACE 

Pod bash enricher

Playbook Action: pod_bash_enricher

Execute the specified bash command on the target pod. Enrich the finding with the command results.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- pod_bash_enricher:
    bash_command: ls -l /etc/data/db
triggers:
- on_pod_all_changes:
    alert_name: ExampleLowDiskAlert

The above is an example. Try customizing the trigger and parameters.

required:
bash_command (str)

Bash command to execute on the target.

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger pod_bash_enricher name=POD_NAME namespace=POD_NAMESPACE  bash_command=BASH_COMMAND

Pod graph enricher

Playbook Action: pod_graph_enricher

Get a graph of a specific resource for this pod. Note: "Disk" Resource is not supported.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- pod_graph_enricher:
    prometheus_url: http://prometheus-k8s.monitoring.svc.cluster.local:9090
    resource_type: Memory
triggers:
- on_pod_all_changes: {}

The above is an example. Try customizing the trigger and parameters.

required:
resource_type (str)

one of: CPU, Memory, Disk (see ResourceChartResourceType)

optional:
prometheus_url (str)

Prometheus url. If omitted, we will try to find a prometheus instance in the same cluster

graph_duration_minutes (int) = 60

Graph duration is minutes. Default is 60.

display_limits (bool)

displays on the graph the pod limit for the resource if true (only CPU and Memory are supported)

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger pod_graph_enricher name=POD_NAME namespace=POD_NAMESPACE  resource_type=RESOURCE_TYPE

Pod node graph enricher

Playbook Action: pod_node_graph_enricher

Get a graph of a specific resource for the node the pod resides on.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- pod_node_graph_enricher:
    prometheus_url: http://prometheus-k8s.monitoring.svc.cluster.local:9090
    resource_type: Memory
triggers:
- on_pod_all_changes: {}

The above is an example. Try customizing the trigger and parameters.

required:
resource_type (str)

one of: CPU, Memory, Disk (see ResourceChartResourceType)

optional:
prometheus_url (str)

Prometheus url. If omitted, we will try to find a prometheus instance in the same cluster

graph_duration_minutes (int) = 60

Graph duration is minutes. Default is 60.

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger pod_node_graph_enricher name=POD_NAME namespace=POD_NAMESPACE  resource_type=RESOURCE_TYPE

Pod ps

Playbook Action: pod_ps

Fetch the list of running processes in a pod.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- pod_ps: {}
triggers:
- on_pod_all_changes: {}

The above is an example. Try customizing the trigger and parameters.

No action parameters

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger pod_ps name=POD_NAME namespace=POD_NAMESPACE 

Pod oom killer enricher

Playbook Action: pod_oom_killer_enricher

Retrieves pod and node information for an OOMKilled pod

Add this to your Robusta configuration (Helm values.yaml):

actions:
- pod_oom_killer_enricher: {}
triggers:
- on_pod_all_changes: {}

The above is an example. Try customizing the trigger and parameters.

No action parameters

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger pod_oom_killer_enricher name=POD_NAME namespace=POD_NAMESPACE 

Daemonset Enrichers

These actions can add context to any daemonset-related event, be it from on_prometheus_alert or on_daemonset_update.

Daemonset status enricher

Playbook Action: daemonset_status_enricher

Enrich the finding with daemon set stats.

Includes recommendations for the identified cause.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- daemonset_status_enricher: {}
triggers:
- on_daemonset_delete: {}

The above is an example. Try customizing the trigger and parameters.

No action parameters

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger daemonset_status_enricher name=DAEMONSET_NAME namespace=DAEMONSET_NAMESPACE 

Deployment Enrichers

These actions can add context to any deployment-related event, be it from on_prometheus_alert or on_deployment_update.

Deployment status enricher

Playbook Action: deployment_status_enricher

Enrich the finding with deployment status conditions.

Usually these conditions can provide important information regarding possible issues.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- deployment_status_enricher: {}
triggers:
- on_deployment_delete: {}

The above is an example. Try customizing the trigger and parameters.

No action parameters

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger deployment_status_enricher name=DEPLOYMENT_NAME namespace=DEPLOYMENT_NAMESPACE 

Deployment events enricher

Playbook Action: deployment_events_enricher

Given a deployment, fetch related events in the near past.

Can optionally fetch events for related pods instead.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- deployment_events_enricher: {}
triggers:
- on_deployment_delete: {}

The above is an example. Try customizing the trigger and parameters.

optional:
max_events (int) = 8
included_types (str list) = ['Warning', 'Normal']
dependent_pod_mode (bool)

when True, instead of fetching events for the deployment itself, fetch events for pods in the deployment.

max_pods (int) = 1

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger deployment_events_enricher name=DEPLOYMENT_NAME namespace=DEPLOYMENT_NAMESPACE 

Job Enrichers

These actions can add context to any job-related event, be it from on_prometheus_alert or on_job_update.

Job events enricher

Playbook Action: job_events_enricher

Given a Kubernetes job, fetch related events in the near past

Add this to your Robusta configuration (Helm values.yaml):

actions:
- job_events_enricher: {}
triggers:
- on_job_update: {}

The above is an example. Try customizing the trigger and parameters.

optional:
max_events (int) = 8
included_types (str list) = ['Warning', 'Normal']

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger job_events_enricher name=JOB_NAME namespace=JOB_NAMESPACE 

Job info enricher

Playbook Action: job_info_enricher

Given a Kubernetes job, add information about the job, from the job spec and status.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- job_info_enricher: {}
triggers:
- on_job_update: {}

The above is an example. Try customizing the trigger and parameters.

No action parameters

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger job_info_enricher name=JOB_NAME namespace=JOB_NAMESPACE 

Job pod enricher

Playbook Action: job_pod_enricher

Given a Kubernetes job, get information about the latest job pod.

Used to get the related pod's events and/or logs

Add this to your Robusta configuration (Helm values.yaml):

actions:
- job_pod_enricher: {}
triggers:
- on_job_update: {}

The above is an example. Try customizing the trigger and parameters.

optional:
max_events (int) = 8
included_types (str list) = ['Warning', 'Normal']
events (bool) = True

Add the events of the related pod

logs (bool) = True

Add the logs of the related pod

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger job_pod_enricher name=JOB_NAME namespace=JOB_NAMESPACE 

Kubernetes Resource Enrichers

These actions can add context to more than one Kubernetes resource type

Event Enrichers

Event resource events

Playbook Action: event_resource_events

Given a Kubernetes event, gather all other events on the same resource in the near past

Add this to your Robusta configuration (Helm values.yaml):

actions:
- event_resource_events: {}
triggers:
- on_kubernetes_warning_event_update: {}

The above is an example. Try customizing the trigger and parameters.

No action parameters

Deployment events enricher

Playbook Action: deployment_events_enricher

Given a deployment, fetch related events in the near past.

Can optionally fetch events for related pods instead.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- deployment_events_enricher: {}
triggers:
- on_deployment_delete: {}

The above is an example. Try customizing the trigger and parameters.

optional:
max_events (int) = 8
included_types (str list) = ['Warning', 'Normal']
dependent_pod_mode (bool)

when True, instead of fetching events for the deployment itself, fetch events for pods in the deployment.

max_pods (int) = 1

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger deployment_events_enricher name=DEPLOYMENT_NAME namespace=DEPLOYMENT_NAMESPACE 

Pod events enricher

Playbook Action: pod_events_enricher

Given a Kubernetes pod, fetch related events in the near past

Add this to your Robusta configuration (Helm values.yaml):

actions:
- pod_events_enricher: {}
triggers:
- on_pod_all_changes: {}

The above is an example. Try customizing the trigger and parameters.

optional:
max_events (int) = 8
included_types (str list) = ['Warning', 'Normal']

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger pod_events_enricher name=POD_NAME namespace=POD_NAMESPACE 

Prometheus Enrichers

These actions enrich Prometheus alerts. They only work with the on_prometheus_alert trigger:

Graph enricher

Playbook Action: graph_enricher

Enrich the alert with a graph of the Prometheus query which triggered the alert.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- graph_enricher:
    prometheus_url: http://prometheus-k8s.monitoring.svc.cluster.local:9090
triggers:
- on_prometheus_alert: {}

The above is an example. Try customizing the trigger and parameters.

optional:
prometheus_url (str)

Prometheus url. If omitted, we will try to find a prometheus instance in the same cluster

Custom graph enricher

Playbook Action: custom_graph_enricher

Enrich the alert with a graph of a custom Prometheus query

Add this to your Robusta configuration (Helm values.yaml):

actions:
- custom_graph_enricher:
    graph_title: CPU Usage for this nod
    prometheus_url: http://prometheus-k8s.monitoring.svc.cluster.local:9090
    promql_query: instance:node_cpu_utilisation:rate5m{job="node-exporter", instance=~"$node_internal_ip:[0-9]+",
      cluster=""} != 0
triggers:
- on_prometheus_alert: {}

The above is an example. Try customizing the trigger and parameters.

required:
promql_query (str)

Promql query. You can use $pod, $node and $node_internal_ip to template (see example). For more information, see https://prometheus.io/docs/prometheus/latest/querying/basics/

optional:
prometheus_url (str)

Prometheus url. If omitted, we will try to find a prometheus instance in the same cluster

graph_title (str)

A nicer name for the Prometheus query.

graph_duration_minutes (int) = 60

Graph duration is minutes.

chart_values_format (str) = Plain

Customize the y-axis labels with one of: Plain, Bytes, Percentage (see ChartValuesFormat)

Alert graph enricher

Playbook Action: alert_graph_enricher

Enrich the alert with a graph of a relevant resource (Pod or Node).

Add this to your Robusta configuration (Helm values.yaml):

actions:
- alert_graph_enricher:
    item_type: Pod
    prometheus_url: http://prometheus-k8s.monitoring.svc.cluster.local:9090
    resource_type: Memory
triggers:
- on_prometheus_alert: {}

The above is an example. Try customizing the trigger and parameters.

required:
resource_type (str)

one of: CPU, Memory, Disk (see ResourceChartResourceType)

item_type (str)

one of: Pod, Node (see ResourceChartItemType)

optional:
prometheus_url (str)

Prometheus url. If omitted, we will try to find a prometheus instance in the same cluster

graph_duration_minutes (int) = 60

Graph duration is minutes. Default is 60.

Template enricher

Playbook Action: template_enricher

Enrich an alert with a paragraph to the alert’s description containing templated markdown. You can inject any of the alert’s Prometheus labels into the markdown.

A variable like $foo will be replaced by the value of the Prometheus label foo. If a label isn’t present then the text “<missing>” will be used instead.

Common variables to use are $alertname, $deployment, $namespace, and $node

The template can include all markdown directives supported by Slack. Note that Slack markdown links use a different format than GitHub.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- template_enricher:
    template: The alertname is $alertname and the pod is $pod
triggers:
- on_prometheus_alert: {}

The above is an example. Try customizing the trigger and parameters.

optional:
template (str)

The enrichment templated markdown text

Stack overflow enricher

Playbook Action: stack_overflow_enricher

Add a button to the alert - clicking it will show the top StackOverflow search results on this alert name.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- stack_overflow_enricher: {}
triggers:
- on_prometheus_alert: {}

The above is an example. Try customizing the trigger and parameters.

No action parameters

Default enricher

Playbook Action: default_enricher

Enrich an alert with the original message and labels.

By default, this enricher is last in the processing order, so it will be added to all alerts, that aren't silenced.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- default_enricher: {}
triggers:
- on_prometheus_alert: {}

The above is an example. Try customizing the trigger and parameters.

No action parameters

Alert definition enricher

Playbook Action: alert_definition_enricher

Enrich an alert with the Prometheus query that triggered the alert.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- alert_definition_enricher: {}
triggers:
- on_prometheus_alert: {}

The above is an example. Try customizing the trigger and parameters.

No action parameters

Prometheus Silencers

These actions can selectively silence Prometheus alerts. They only work with the on_prometheus_alert trigger:

Node restart silencer

Playbook Action: node_restart_silencer

Silence alerts for pods on a node that recently restarted.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- node_restart_silencer: {}
triggers:
- on_prometheus_alert: {}

The above is an example. Try customizing the trigger and parameters.

optional:
post_restart_silence (int) = 300

Period after restart to silence alerts. Seconds.

Severity silencer

Playbook Action: severity_silencer

Silence alerts with the specified severity level.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- severity_silencer:
    severity: warning
triggers:
- on_prometheus_alert: {}

The above is an example. Try customizing the trigger and parameters.

optional:
severity (str) = none

severity level that should be silenced.

Name silencer

Playbook Action: name_silencer

Silence named alerts.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- name_silencer:
    names:
    - string
    - string
triggers:
- on_prometheus_alert: {}

The above is an example. Try customizing the trigger and parameters.

required:
names (str list)

List of alert names that should be silenced.