Event Enrichment

The actions are used to gather extra data on errors, alerts, and other cluster events.

Use them as building blocks in your own automations.

Node Enrichers

These actions can add context to any node-related event, be it from on_prometheus_alert or on_node_update.

Node bash enricher

Playbook Action

Execute the specified bash command on the target node. Enrich the finding with the command results.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- node_bash_enricher:
    bash_command: ls -l /etc/data/db
triggers:
- on_node_create: {}

The above is an example. Try customizing the trigger and parameters.

required:
bash_command (str)

Bash command to execute on the target.

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger node_bash_enricher name=NODE_NAME  bash_command=BASH_COMMAND

Node status enricher

Playbook Action

Enrich the finding with the node's status conditions.

Can help troubleshooting Node issues.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- node_status_enricher: {}
triggers:
- on_node_create: {}

The above is an example. Try customizing the trigger and parameters.

No action parameters

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger node_status_enricher name=NODE_NAME 

Node running pods enricher

Playbook Action

Enrich the finding with pods running on this node, along with the 'Ready' status of each pod.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- node_running_pods_enricher: {}
triggers:
- on_node_create: {}

The above is an example. Try customizing the trigger and parameters.

No action parameters

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger node_running_pods_enricher name=NODE_NAME 

Node allocatable resources enricher

Playbook Action

Enrich the finding with the node resources available for allocation.

Can help troubleshooting node issues.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- node_allocatable_resources_enricher: {}
triggers:
- on_node_create: {}

The above is an example. Try customizing the trigger and parameters.

No action parameters

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger node_allocatable_resources_enricher name=NODE_NAME 

Node graph enricher

Playbook Action

Get a graph of a specific resource for this node.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- node_graph_enricher:
    prometheus_url: http://prometheus-k8s.monitoring.svc.cluster.local:9090
    resource_type: Memory
triggers:
- on_node_create: {}

The above is an example. Try customizing the trigger and parameters.

required:
resource_type (str)

one of: CPU, Memory, Disk (see ResourceChartResourceType)

optional:
prometheus_url (str)

Prometheus url. If omitted, we will try to find a prometheus instance in the same cluster

graph_duration_minutes (int) = 60

Graph duration is minutes. Default is 60.

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger node_graph_enricher name=NODE_NAME  resource_type=RESOURCE_TYPE

Node cpu enricher

Playbook Action

Enrich the finding with analysis of the node's CPU usage. Collect information about pods running on this node, their CPU request configuration, their actual cpu usage etc. Provides insightful information regarding node high CPU usage.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- node_cpu_enricher:
    prometheus_url: http://prometheus-k8s.monitoring.svc.cluster.local:9090
triggers:
- on_node_create: {}

The above is an example. Try customizing the trigger and parameters.

optional:
prometheus_url (str)

Prometheus url. If omitted, we will try to find a prometheus instance in the same cluster

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger node_cpu_enricher name=NODE_NAME 

Pod Enrichers

These actions can add context to any pod-related event, be it from on_prometheus_alert or on_pod_update.

Logs enricher

Playbook Action

Enrich the alert with pod logs The pod to fetch logs for is determined by the alert’s pod label from Prometheus.

By default, if the alert has no pod this enricher will silently do nothing.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- logs_enricher: {}
triggers:
- on_pod_all_changes: {}

The above is an example. Try customizing the trigger and parameters.

optional:
warn_on_missing_label (bool)

Send a warning if the alert doesn't have a pod label

regex_replacer_patterns (complex list)

regex patterns to replace text, for example for security reasons (Note: Replacements are executed in the given order)

each entry contains:

required:
regex (str)
optional:
name (str) = Redacted
regex_replacement_style (str)

one of SAME_LENGTH_ASTERISKS or NAMED (See RegexReplacementStyle)

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger logs_enricher name=POD_NAME namespace=POD_NAMESPACE 

Pod bash enricher

Playbook Action

Execute the specified bash command on the target pod. Enrich the finding with the command results.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- pod_bash_enricher:
    bash_command: ls -l /etc/data/db
triggers:
- on_pod_all_changes: {}

The above is an example. Try customizing the trigger and parameters.

required:
bash_command (str)

Bash command to execute on the target.

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger pod_bash_enricher name=POD_NAME namespace=POD_NAMESPACE  bash_command=BASH_COMMAND

Pod events enricher

Playbook Action

Enrich the finding with the pod events.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- pod_events_enricher: {}
triggers:
- on_pod_all_changes: {}

The above is an example. Try customizing the trigger and parameters.

No action parameters

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger pod_events_enricher name=POD_NAME namespace=POD_NAMESPACE 

Pod graph enricher

Playbook Action

Get a graph of a specific resource for this pod. Note: "Disk" Resource is not supported.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- pod_graph_enricher:
    prometheus_url: http://prometheus-k8s.monitoring.svc.cluster.local:9090
    resource_type: Memory
triggers:
- on_pod_all_changes: {}

The above is an example. Try customizing the trigger and parameters.

required:
resource_type (str)

one of: CPU, Memory, Disk (see ResourceChartResourceType)

optional:
prometheus_url (str)

Prometheus url. If omitted, we will try to find a prometheus instance in the same cluster

graph_duration_minutes (int) = 60

Graph duration is minutes. Default is 60.

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger pod_graph_enricher name=POD_NAME namespace=POD_NAMESPACE  resource_type=RESOURCE_TYPE

Pod ps

Playbook Action

Fetch the list of running processes in a pod.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- pod_ps: {}
triggers:
- on_pod_all_changes: {}

The above is an example. Try customizing the trigger and parameters.

No action parameters

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger pod_ps name=POD_NAME namespace=POD_NAMESPACE 

Daemonset Enrichers

These actions can add context to any daemonset-related event, be it from on_prometheus_alert or on_daemonset_update.

Daemonset status enricher

Playbook Action

Enrich the finding with daemon set stats.

Includes recommendations for the identified cause.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- daemonset_status_enricher: {}
triggers:
- on_daemonset_all_changes: {}

The above is an example. Try customizing the trigger and parameters.

No action parameters

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger daemonset_status_enricher name=DAEMONSET_NAME namespace=DAEMONSET_NAMESPACE 

Deployment Enrichers

These actions can add context to any deployment-related event, be it from on_prometheus_alert or on_deployment_update.

Deployment status enricher

Playbook Action

Enrich the finding with deployment status conditions.

Usually these conditions can provide important information regarding possible issues.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- deployment_status_enricher: {}
triggers:
- on_deployment_all_changes: {}

The above is an example. Try customizing the trigger and parameters.

No action parameters

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger deployment_status_enricher name=DEPLOYMENT_NAME namespace=DEPLOYMENT_NAMESPACE 

Kubernetes Resource Enrichers

These actions can add context to more than one Kubernetes resource type

Event Enrichers

Event resource events

Playbook Action

Given a Kubernetes event, gather all other events on the same resource in the near past

Add this to your Robusta configuration (Helm values.yaml):

actions:
- event_resource_events: {}
triggers:
- on_kubernetes_warning_event_update: {}

The above is an example. Try customizing the trigger and parameters.

optional:
finding_key (str) = DEFAULT

Specify the finding identifier, to reference it in other actions.

Prometheus Enrichers

These actions enrich Prometheus alerts. They only work with the on_prometheus_alert trigger:

Graph enricher

Playbook Action

Enrich the alert with a graph of the Prometheus query which triggered the alert.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- graph_enricher:
    prometheus_url: http://prometheus-k8s.monitoring.svc.cluster.local:9090
triggers:
- on_prometheus_alert: {}

The above is an example. Try customizing the trigger and parameters.

optional:
prometheus_url (str)

Prometheus url. If omitted, we will try to find a prometheus instance in the same cluster

Custom graph enricher

Playbook Action

Enrich the alert with a graph of a custom Prometheus query

Add this to your Robusta configuration (Helm values.yaml):

actions:
- custom_graph_enricher:
    graph_title: CPU Usage for this nod
    prometheus_url: http://prometheus-k8s.monitoring.svc.cluster.local:9090
    promql_query: instance:node_cpu_utilisation:rate5m{job="node-exporter", instance=~"$node_internal_ip:[0-9]+",
      cluster=""} != 0
triggers:
- on_prometheus_alert: {}

The above is an example. Try customizing the trigger and parameters.

required:
promql_query (str)

Promql query. You can use $pod, $node and $node_internal_ip to template (see example). For more information, see https://prometheus.io/docs/prometheus/latest/querying/basics/

optional:
prometheus_url (str)

Prometheus url. If omitted, we will try to find a prometheus instance in the same cluster

graph_title (str)

A nicer name for the Prometheus query.

graph_duration_minutes (int) = 60

Graph duration is minutes.

chart_values_format (str) = Plain

Customize the y-axis labels with one of: Plain, Bytes, Percentage (see ChartValuesFormat)

Alert graph enricher

Playbook Action

Enrich the alert with a graph of a relevant resource (Pod or Node).

Add this to your Robusta configuration (Helm values.yaml):

actions:
- alert_graph_enricher:
    item_type: Pod
    prometheus_url: http://prometheus-k8s.monitoring.svc.cluster.local:9090
    resource_type: Memory
triggers:
- on_prometheus_alert: {}

The above is an example. Try customizing the trigger and parameters.

required:
resource_type (str)

one of: CPU, Memory, Disk (see ResourceChartResourceType)

item_type (str)

one of: Pod, Node (see ResourceChartItemType)

optional:
prometheus_url (str)

Prometheus url. If omitted, we will try to find a prometheus instance in the same cluster

graph_duration_minutes (int) = 60

Graph duration is minutes. Default is 60.

Template enricher

Playbook Action

Enrich an alert with a paragraph to the alert’s description containing templated markdown. You can inject any of the alert’s Prometheus labels into the markdown.

A variable like $foo will be replaced by the value of the Prometheus label foo. If a label isn’t present then the text “<missing>” will be used instead.

Common variables to use are $alertname, $deployment, $namespace, and $node

The template can include all markdown directives supported by Slack. Note that Slack markdown links use a different format than GitHub.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- template_enricher:
    template: The alertname is $alertname and the pod is $pod
triggers:
- on_prometheus_alert: {}

The above is an example. Try customizing the trigger and parameters.

optional:
template (str)

The enrichment templated markdown text

Stack overflow enricher

Playbook Action

Add a button to the alert - clicking it will show the top StackOverflow search results on this alert name.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- stack_overflow_enricher: {}
triggers:
- on_prometheus_alert: {}

The above is an example. Try customizing the trigger and parameters.

No action parameters

Default enricher

Playbook Action

Enrich an alert with the original message and labels.

By default, this enricher is last in the processing order, so it will be added to all alerts, that aren't silenced.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- default_enricher: {}
triggers:
- on_prometheus_alert: {}

The above is an example. Try customizing the trigger and parameters.

No action parameters

Alert definition enricher

Playbook Action

Enrich an alert with the Prometheus query that triggered the alert.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- alert_definition_enricher: {}
triggers:
- on_prometheus_alert: {}

The above is an example. Try customizing the trigger and parameters.

No action parameters

Prometheus Silencers

These actions can selectively silence Prometheus alerts. They only work with the on_prometheus_alert trigger:

Node restart silencer

Playbook Action

Silence alerts for pods on a node that recently restarted.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- node_restart_silencer: {}
triggers:
- on_prometheus_alert: {}

The above is an example. Try customizing the trigger and parameters.

optional:
post_restart_silence (int) = 300

Period after restart to silence alerts. Seconds.

Severity silencer

Playbook Action

Silence alerts with the specified severity level.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- severity_silencer:
    severity: warning
triggers:
- on_prometheus_alert: {}

The above is an example. Try customizing the trigger and parameters.

optional:
severity (str) = none

severity level that should be silenced.

Name silencer

Playbook Action

Silence named alerts.

Add this to your Robusta configuration (Helm values.yaml):

actions:
- name_silencer:
    names:
    - string
    - string
triggers:
- on_prometheus_alert: {}

The above is an example. Try customizing the trigger and parameters.

required:
names (str list)

List of alert names that should be silenced.