Event Enrichment

The actions are used to gather extra data on errors, alerts, and other cluster events.

Use them as building blocks in your own automations, or write your own enrichment actions in Python.

Node Enrichers

These actions can add context to any node-related event, be it from on_prometheus_alert or on_node_update.

Node bash enricher

Playbook Action: node_bash_enricher

Execute the specified bash command on the target node. Enrich the finding with the command results.

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - node_bash_enricher:
      bash_command: ls -l /etc/data/db
  triggers:
  - on_node_delete: {}

The above is an example. Try customizing the trigger and parameters.

required:
bash_command (str)

Bash command to execute on the target.

optional:
custom_annotations (str dict)

custom annotations to be used for the running pod/job

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger node_bash_enricher name=NODE_NAME  bash_command=BASH_COMMAND

Node status enricher

Playbook Action: node_status_enricher

Enrich the finding with the node's status conditions.

Can help troubleshooting Node issues.

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - node_status_enricher: {}
  triggers:
  - on_node_delete: {}

The above is an example. Try customizing the trigger and parameters.

No action parameters

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger node_status_enricher name=NODE_NAME

Node running pods enricher

Playbook Action: node_running_pods_enricher

Enrich the finding with pods running on this node, along with the 'Ready' status of each pod.

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - node_running_pods_enricher: {}
  triggers:
  - on_node_delete: {}

The above is an example. Try customizing the trigger and parameters.

No action parameters

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger node_running_pods_enricher name=NODE_NAME

Node allocatable resources enricher

Playbook Action: node_allocatable_resources_enricher

Enrich the finding with the node resources available for allocation.

Can help troubleshooting node issues.

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - node_allocatable_resources_enricher: {}
  triggers:
  - on_node_delete: {}

The above is an example. Try customizing the trigger and parameters.

No action parameters

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger node_allocatable_resources_enricher name=NODE_NAME

Node graph enricher

Playbook Action: node_graph_enricher

Get a graph of a specific resource for this node.

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - node_graph_enricher:
      prometheus_additional_labels: 'cluster: ''cluster-2-test''env: ''prod'''
      prometheus_auth: Basic YWRtaW46cGFzc3dvcmQ=
      prometheus_url: http://prometheus-k8s.monitoring.svc.cluster.local:9090
      prometheus_url_query_string: demo-query=example-data
      resource_type: Memory
  triggers:
  - on_node_delete: {}

The above is an example. Try customizing the trigger and parameters.

required:
resource_type (str)

one of: CPU, Memory, Disk (see ResourceChartResourceType)

optional:
prometheus_url (str)

Prometheus url. If omitted, we will try to find a prometheus instance in the same cluster

prometheus_auth (str)

Prometheus auth header to be used in Authorization header. If omitted, we will not add any auth header

prometheus_url_query_string (str)

Additional query string parameters to be appended to the Prometheus connection URL

prometheus_additional_labels (str dict)

A dictionary of additional labels needed for multi-cluster prometheus

add_additional_labels (bool) = True

adds the additional labels (if defined) to the query

prometheus_graphs_overrides (complex list)

each entry contains:

required:
resource_type (str)
item_type (str)
query (str)
optional:
values_format (str)
graph_duration_minutes (int) = 60

Graph duration is minutes. Default is 60.

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger node_graph_enricher name=NODE_NAME  resource_type=RESOURCE_TYPE

Node cpu enricher

Playbook Action: node_cpu_enricher

Enrich the finding with analysis of the node's CPU usage. Collect information about pods running on this node, their CPU request configuration, their actual cpu usage etc. Provides insightful information regarding node high CPU usage.

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - node_cpu_enricher:
      prometheus_additional_labels: 'cluster: ''cluster-2-test''env: ''prod'''
      prometheus_auth: Basic YWRtaW46cGFzc3dvcmQ=
      prometheus_url: http://prometheus-k8s.monitoring.svc.cluster.local:9090
      prometheus_url_query_string: demo-query=example-data
  triggers:
  - on_node_delete: {}

The above is an example. Try customizing the trigger and parameters.

optional:
prometheus_url (str)

Prometheus url. If omitted, we will try to find a prometheus instance in the same cluster

prometheus_auth (str)

Prometheus auth header to be used in Authorization header. If omitted, we will not add any auth header

prometheus_url_query_string (str)

Additional query string parameters to be appended to the Prometheus connection URL

prometheus_additional_labels (str dict)

A dictionary of additional labels needed for multi-cluster prometheus

add_additional_labels (bool) = True

adds the additional labels (if defined) to the query

prometheus_graphs_overrides (complex list)

each entry contains:

required:
resource_type (str)
item_type (str)
query (str)
optional:
values_format (str)

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger node_cpu_enricher name=NODE_NAME

Pod Enrichers (General)

These actions can add context to any pod-related event, be it from on_prometheus_alert or on_pod_update.

Logs enricher

Playbook Action: logs_enricher

Fetch and attach Pod logs. The pod to fetch logs for is determined by the alert’s pod label from Prometheus.

By default, if the alert has no pod this enricher will silently do nothing.

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - logs_enricher: {}
  triggers:
  - on_pod_crash_loop: {}

The above is an example. Try customizing the trigger and parameters.

optional:
container_name (str)

Specific container to get logs from

warn_on_missing_label (bool)

Send a warning if the alert doesn't have a pod label

regex_replacer_patterns (complex list)

regex patterns to replace text, for example for security reasons (Note: Replacements are executed in the given order)

each entry contains:

required:
regex (str)
optional:
name (str) = Redacted
regex_replacement_style (str)

one of SAME_LENGTH_ASTERISKS or NAMED (See RegexReplacementStyle)

previous (bool)
filter_regex (str)

only shows lines that match the regex

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger logs_enricher name=POD_NAME namespace=POD_NAMESPACE

Pod events enricher

Playbook Action: pod_events_enricher

Given a Kubernetes pod, fetch related events in the near past

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - pod_events_enricher: {}
  triggers:
  - on_pod_all_changes: {}

The above is an example. Try customizing the trigger and parameters.

optional:
max_events (int) = 8
included_types (str list) = ['Warning', 'Normal']

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger pod_events_enricher name=POD_NAME namespace=POD_NAMESPACE

Pod bash enricher

Playbook Action: pod_bash_enricher

Execute the specified bash command on the target pod. Enrich the finding with the command results.

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - pod_bash_enricher:
      bash_command: ls -l /etc/data/db
  triggers:
  - on_pod_all_changes:
      alert_name: ExampleLowDiskAlert

The above is an example. Try customizing the trigger and parameters.

required:
bash_command (str)

Bash command to execute on the target.

optional:
custom_annotations (str dict)

custom annotations to be used for the running pod/job

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger pod_bash_enricher name=POD_NAME namespace=POD_NAMESPACE  bash_command=BASH_COMMAND

Pod graph enricher

Playbook Action: pod_graph_enricher

Get a graph of a specific resource for this pod. Note: "Disk" Resource is not supported.

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - pod_graph_enricher:
      prometheus_additional_labels: 'cluster: ''cluster-2-test''env: ''prod'''
      prometheus_auth: Basic YWRtaW46cGFzc3dvcmQ=
      prometheus_url: http://prometheus-k8s.monitoring.svc.cluster.local:9090
      prometheus_url_query_string: demo-query=example-data
      resource_type: Memory
  triggers:
  - on_pod_all_changes: {}

The above is an example. Try customizing the trigger and parameters.

required:
resource_type (str)

one of: CPU, Memory, Disk (see ResourceChartResourceType)

optional:
prometheus_url (str)

Prometheus url. If omitted, we will try to find a prometheus instance in the same cluster

prometheus_auth (str)

Prometheus auth header to be used in Authorization header. If omitted, we will not add any auth header

prometheus_url_query_string (str)

Additional query string parameters to be appended to the Prometheus connection URL

prometheus_additional_labels (str dict)

A dictionary of additional labels needed for multi-cluster prometheus

add_additional_labels (bool) = True

adds the additional labels (if defined) to the query

prometheus_graphs_overrides (complex list)

each entry contains:

required:
resource_type (str)
item_type (str)
query (str)
optional:
values_format (str)
graph_duration_minutes (int) = 60

Graph duration is minutes. Default is 60.

display_limits (bool)

displays on the graph the pod limit for the resource if true (only CPU and Memory are supported)

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger pod_graph_enricher name=POD_NAME namespace=POD_NAMESPACE  resource_type=RESOURCE_TYPE

Pod node graph enricher

Playbook Action: pod_node_graph_enricher

Get a graph of a specific resource for the node the pod resides on.

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - pod_node_graph_enricher:
      prometheus_additional_labels: 'cluster: ''cluster-2-test''env: ''prod'''
      prometheus_auth: Basic YWRtaW46cGFzc3dvcmQ=
      prometheus_url: http://prometheus-k8s.monitoring.svc.cluster.local:9090
      prometheus_url_query_string: demo-query=example-data
      resource_type: Memory
  triggers:
  - on_pod_all_changes: {}

The above is an example. Try customizing the trigger and parameters.

required:
resource_type (str)

one of: CPU, Memory, Disk (see ResourceChartResourceType)

optional:
prometheus_url (str)

Prometheus url. If omitted, we will try to find a prometheus instance in the same cluster

prometheus_auth (str)

Prometheus auth header to be used in Authorization header. If omitted, we will not add any auth header

prometheus_url_query_string (str)

Additional query string parameters to be appended to the Prometheus connection URL

prometheus_additional_labels (str dict)

A dictionary of additional labels needed for multi-cluster prometheus

add_additional_labels (bool) = True

adds the additional labels (if defined) to the query

prometheus_graphs_overrides (complex list)

each entry contains:

required:
resource_type (str)
item_type (str)
query (str)
optional:
values_format (str)
graph_duration_minutes (int) = 60

Graph duration is minutes. Default is 60.

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger pod_node_graph_enricher name=POD_NAME namespace=POD_NAMESPACE  resource_type=RESOURCE_TYPE

Pod ps

Playbook Action: pod_ps

Fetch the list of running processes in a pod.

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - pod_ps: {}
  triggers:
  - on_pod_all_changes: {}

The above is an example. Try customizing the trigger and parameters.

optional:
custom_annotations (str dict)

custom annotations to be used for the running pod/job

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger pod_ps name=POD_NAME namespace=POD_NAMESPACE

Image pull backoff reporter

Playbook Action: image_pull_backoff_reporter

Notify when an ImagePullBackoff occurs and determine the reason why.

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - image_pull_backoff_reporter: {}
  triggers:
  - on_pod_all_changes: {}

The above is an example. Try customizing the trigger and parameters.

optional:
rate_limit (int) = 3600

Rate limit the execution of this action (Seconds).

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger image_pull_backoff_reporter name=POD_NAME namespace=POD_NAMESPACE

Pod Enrichers (Crashes)

These actions add context for specific Pod-related errors.

They're less general than the above actions, and usually designed for a specific error like CrashLoopBackOff.

Report crash loop

Playbook Action: report_crash_loop

No description

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - report_crash_loop: {}
  triggers:
  - on_pod_all_changes: {}

The above is an example. Try customizing the trigger and parameters.

optional:
regex_replacer_patterns (complex list)

regex patterns to replace text, for example for security reasons (Note: Replacements are executed in the given order)

each entry contains:

required:
regex (str)
optional:
name (str) = Redacted
regex_replacement_style (str) = SAME_LENGTH_ASTERISKS

one of SAME_LENGTH_ASTERISKS or NAMED (See RegexReplacementStyle)

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger report_crash_loop name=POD_NAME namespace=POD_NAMESPACE

Pod oom killer enricher

Playbook Action: pod_oom_killer_enricher

Retrieves pod and node information for an OOMKilled pod

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - pod_oom_killer_enricher:
      prometheus_additional_labels: 'cluster: ''cluster-2-test''env: ''prod'''
      prometheus_auth: Basic YWRtaW46cGFzc3dvcmQ=
      prometheus_url: http://prometheus-k8s.monitoring.svc.cluster.local:9090
      prometheus_url_query_string: demo-query=example-data
      resource_type: Memory
  triggers:
  - on_pod_oom_killed: {}

The above is an example. Try customizing the trigger and parameters.

required:
resource_type (str)

one of: CPU, Memory, Disk (see ResourceChartResourceType)

optional:
custom_annotations (str dict)

custom annotations to be used for the running pod/job

prometheus_url (str)

Prometheus url. If omitted, we will try to find a prometheus instance in the same cluster

prometheus_auth (str)

Prometheus auth header to be used in Authorization header. If omitted, we will not add any auth header

prometheus_url_query_string (str)

Additional query string parameters to be appended to the Prometheus connection URL

prometheus_additional_labels (str dict)

A dictionary of additional labels needed for multi-cluster prometheus

add_additional_labels (bool) = True

adds the additional labels (if defined) to the query

prometheus_graphs_overrides (complex list)

each entry contains:

required:
resource_type (str)
item_type (str)
query (str)
optional:
values_format (str)
graph_duration_minutes (int) = 60

Graph duration is minutes. Default is 60.

delay_graph_s (int)

the amount of seconds to delay getting the graph inorder to record the memory spike

attach_logs (bool)
container_memory_graph (bool)
node_memory_graph (bool)
dmesg_log (bool)

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger pod_oom_killer_enricher name=POD_NAME namespace=POD_NAMESPACE  resource_type=RESOURCE_TYPE

Daemonset Enrichers

These actions can add context to any daemonset-related event, be it from on_prometheus_alert or on_daemonset_update.

Daemonset status enricher

Playbook Action: daemonset_status_enricher

Enrich the finding with daemon set stats.

Includes recommendations for the identified cause.

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - daemonset_status_enricher: {}
  triggers:
  - on_daemonset_delete: {}

The above is an example. Try customizing the trigger and parameters.

No action parameters

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger daemonset_status_enricher name=DAEMONSET_NAME namespace=DAEMONSET_NAMESPACE

Deployment Enrichers

These actions can add context to any deployment-related event, be it from on_prometheus_alert or on_deployment_update.

Deployment status enricher

Playbook Action: deployment_status_enricher

Enrich the finding with deployment status conditions.

Usually these conditions can provide important information regarding possible issues.

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - deployment_status_enricher: {}
  triggers:
  - on_deployment_create: {}

The above is an example. Try customizing the trigger and parameters.

No action parameters

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger deployment_status_enricher name=DEPLOYMENT_NAME namespace=DEPLOYMENT_NAMESPACE

Deployment events enricher

Playbook Action: deployment_events_enricher

Given a deployment, fetch related events in the near past.

Can optionally fetch events for related pods instead.

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - deployment_events_enricher: {}
  triggers:
  - on_deployment_create: {}

The above is an example. Try customizing the trigger and parameters.

optional:
max_events (int) = 8
included_types (str list) = ['Warning', 'Normal']
dependent_pod_mode (bool)

when True, instead of fetching events for the deployment itself, fetch events for pods in the deployment.

max_pods (int) = 1

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger deployment_events_enricher name=DEPLOYMENT_NAME namespace=DEPLOYMENT_NAMESPACE

Job Enrichers

These actions can add context to any job-related event, be it from on_prometheus_alert or on_job_update.

Job events enricher

Playbook Action: job_events_enricher

Given a Kubernetes job, fetch related events in the near past

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - job_events_enricher: {}
  triggers:
  - on_job_failure: {}

The above is an example. Try customizing the trigger and parameters.

optional:
max_events (int) = 8
included_types (str list) = ['Warning', 'Normal']

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger job_events_enricher name=JOB_NAME namespace=JOB_NAMESPACE

Job info enricher

Playbook Action: job_info_enricher

Given a Kubernetes job, add information about the job, from the job spec and status.

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - job_info_enricher: {}
  triggers:
  - on_job_failure: {}

The above is an example. Try customizing the trigger and parameters.

No action parameters

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger job_info_enricher name=JOB_NAME namespace=JOB_NAMESPACE

Job pod enricher

Playbook Action: job_pod_enricher

Given a Kubernetes job, get information about the latest job pod.

Used to get the related pod's events and/or logs

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - job_pod_enricher: {}
  triggers:
  - on_job_failure: {}

The above is an example. Try customizing the trigger and parameters.

optional:
container_name (str)

Specific container to get logs from

warn_on_missing_label (bool)

Send a warning if the alert doesn't have a pod label

regex_replacer_patterns (complex list)

regex patterns to replace text, for example for security reasons (Note: Replacements are executed in the given order)

each entry contains:

required:
regex (str)
optional:
name (str) = Redacted
regex_replacement_style (str)

one of SAME_LENGTH_ASTERISKS or NAMED (See RegexReplacementStyle)

previous (bool)
filter_regex (str)

only shows lines that match the regex

max_events (int) = 8
included_types (str list) = ['Warning', 'Normal']
events (bool) = True

Add the events of the related pod

logs (bool) = True

Add the logs of the related pod

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger job_pod_enricher name=JOB_NAME namespace=JOB_NAMESPACE

Kubernetes Resource Enrichers

These actions can add context to more than one Kubernetes resource type

Playbook Action: related_pods

Return the list of pods related to that k8s resource. For example, return all pods of a given k8s Deployment

Supports Deployments, ReplicaSets, DaemonSets, StatefulSets and Pods

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - related_pods: {}
  triggers:
  - on_pod_crash_loop: {}

The above is an example. Try customizing the trigger and parameters.

optional:
output_format (str) = table

The output format of the action. table or json, table is the default.

include_raw_data (bool)
limit (int)

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger related_pods kind=RESOURCE_KIND name=RESOURCE_NAME

List resource names

Playbook Action: list_resource_names

List the names of the cluster resources for the given kind and namespace

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - list_resource_names:
      kind: string
  triggers:
  - on_pod_create: {}

The above is an example. Try customizing the trigger and parameters.

required:
kind (str)

The k8s resource kind. Must be one of: [node,deployment,statefulset,daemonset,job,persistentvolume,persistentvolumeclaim,service,configmap,networkpolicy].

optional:
namespace (str)

For namespaced k8s resources. List names for the specified namespace. If omitted, all namespaces will be used

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger list_resource_names  kind=KIND

Get resource yaml

Playbook Action: get_resource_yaml

Export Kubernetes resources from the cluster as the yaml file. Expects the kind of resource, its name and namespace.

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - get_resource_yaml: {}
  triggers:
  - on_pod_crash_loop: {}

The above is an example. Try customizing the trigger and parameters.

No action parameters

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger get_resource_yaml kind=RESOURCE_KIND name=RESOURCE_NAME

Event Enrichers

These actions are for use with on_kubernetes_warning_event_create and other Warning Event triggers.

For actions that fetch Warning Events for other triggers, see Job events enricher, Pod events enricher, and Deployment events enricher

Event resource events

Playbook Action: event_resource_events

Given a Kubernetes event, gather all other events on the same resource in the near past

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - event_resource_events: {}
  triggers:
  - on_event_create: {}

The above is an example. Try customizing the trigger and parameters.

No action parameters

Prometheus Enrichers

These actions enrich Prometheus alerts and only support the on_prometheus_alert trigger.

Graph enricher

Playbook Action: graph_enricher

Attach a graph of the Prometheus query that triggered the alert.

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - graph_enricher:
      prometheus_additional_labels: 'cluster: ''cluster-2-test''env: ''prod'''
      prometheus_auth: Basic YWRtaW46cGFzc3dvcmQ=
      prometheus_url: http://prometheus-k8s.monitoring.svc.cluster.local:9090
      prometheus_url_query_string: demo-query=example-data
  triggers:
  - on_prometheus_alert: {}

The above is an example. Try customizing the trigger and parameters.

optional:
prometheus_url (str)

Prometheus url. If omitted, we will try to find a prometheus instance in the same cluster

prometheus_auth (str)

Prometheus auth header to be used in Authorization header. If omitted, we will not add any auth header

prometheus_url_query_string (str)

Additional query string parameters to be appended to the Prometheus connection URL

prometheus_additional_labels (str dict)

A dictionary of additional labels needed for multi-cluster prometheus

add_additional_labels (bool) = True

adds the additional labels (if defined) to the query

prometheus_graphs_overrides (complex list)

each entry contains:

required:
resource_type (str)
item_type (str)
query (str)
optional:
values_format (str)
graph_duration_minutes (int) = 60

the duration of the query in minutes

Custom graph enricher

Playbook Action: custom_graph_enricher

Attach a graph of an arbitrary Prometheus query, specified as a parameter.

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - custom_graph_enricher:
      graph_title: CPU Usage for this nod
      prometheus_additional_labels: 'cluster: ''cluster-2-test''env: ''prod'''
      prometheus_auth: Basic YWRtaW46cGFzc3dvcmQ=
      prometheus_url: http://prometheus-k8s.monitoring.svc.cluster.local:9090
      prometheus_url_query_string: demo-query=example-data
      promql_query: instance:node_cpu_utilisation:rate5m{job="node-exporter", instance=~"$node_internal_ip:[0-9]+",
        cluster=""} != 0
  triggers:
  - on_prometheus_alert: {}

The above is an example. Try customizing the trigger and parameters.

required:
promql_query (str)

Promql query. You can use $pod, $node and $node_internal_ip to template (see example). For more information, see https://prometheus.io/docs/prometheus/latest/querying/basics/

optional:
prometheus_url (str)

Prometheus url. If omitted, we will try to find a prometheus instance in the same cluster

prometheus_auth (str)

Prometheus auth header to be used in Authorization header. If omitted, we will not add any auth header

prometheus_url_query_string (str)

Additional query string parameters to be appended to the Prometheus connection URL

prometheus_additional_labels (str dict)

A dictionary of additional labels needed for multi-cluster prometheus

add_additional_labels (bool) = True

adds the additional labels (if defined) to the query

prometheus_graphs_overrides (complex list)

each entry contains:

required:
resource_type (str)
item_type (str)
query (str)
optional:
values_format (str)
graph_title (str)

A nicer name for the Prometheus query. The graph_title may include template variables like $name, $namespace, $node, $container etc...

graph_duration_minutes (int) = 60

Graph duration is minutes.

chart_values_format (str) = Plain

Customize the y-axis labels with one of: Plain, Bytes, Percentage (see ChartValuesFormat)

hide_legends (bool)

Alert graph enricher

Playbook Action: alert_graph_enricher

Attach a resource-usage graph. The graph is automatically fetched for the Pod/Node that triggered this action.

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - alert_graph_enricher:
      item_type: Pod
      prometheus_additional_labels: 'cluster: ''cluster-2-test''env: ''prod'''
      prometheus_auth: Basic YWRtaW46cGFzc3dvcmQ=
      prometheus_url: http://prometheus-k8s.monitoring.svc.cluster.local:9090
      prometheus_url_query_string: demo-query=example-data
      resource_type: Memory
  triggers:
  - on_prometheus_alert: {}

The above is an example. Try customizing the trigger and parameters.

required:
resource_type (str)

one of: CPU, Memory, Disk (see ResourceChartResourceType)

item_type (str)

one of: Pod, Node (see ResourceChartItemType)

optional:
prometheus_url (str)

Prometheus url. If omitted, we will try to find a prometheus instance in the same cluster

prometheus_auth (str)

Prometheus auth header to be used in Authorization header. If omitted, we will not add any auth header

prometheus_url_query_string (str)

Additional query string parameters to be appended to the Prometheus connection URL

prometheus_additional_labels (str dict)

A dictionary of additional labels needed for multi-cluster prometheus

add_additional_labels (bool) = True

adds the additional labels (if defined) to the query

prometheus_graphs_overrides (complex list)

each entry contains:

required:
resource_type (str)
item_type (str)
query (str)
optional:
values_format (str)
graph_duration_minutes (int) = 60

Graph duration is minutes. Default is 60.

Template enricher

Playbook Action: template_enricher

Attach a paragraph containing templated markdown. You can inject the k8s subject info and additionally on Prometheus alerts, any of the alert’s Prometheus labels.

Common variables to use are ${name}, ${kind}, ${namespace}, and ${node}

A variable like ${foo} will be replaced by the value of info/label foo. If it isn’t present then the text “<missing>” will be used instead.

Check example for adding a template link.

The template can include all markdown directives supported by Slack. Note that Slack markdown links use a different format than GitHub.

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - template_enricher:
      template: <https://platform.robusta.dev/?namespace="${namespace}"&type="${kind}"&name="${name}"|my-link>
  triggers:
  - on_pod_crash_loop: {}

The above is an example. Try customizing the trigger and parameters.

optional:
template (str)

The enrichment templated markdown text

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger template_enricher kind=RESOURCE_KIND name=RESOURCE_NAME

Stack overflow enricher

Playbook Action: stack_overflow_enricher

Add a button to the alert - clicking it will show the top StackOverflow search results on this alert name.

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - stack_overflow_enricher: {}
  triggers:
  - on_prometheus_alert: {}

The above is an example. Try customizing the trigger and parameters.

No action parameters

Default enricher

Playbook Action: default_enricher

Enrich an alert with the original message and labels.

By default, this enricher is last in the processing order, so it will be added to all alerts, that aren't silenced.

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - default_enricher: {}
  triggers:
  - on_prometheus_alert: {}

The above is an example. Try customizing the trigger and parameters.

optional:
alert_annotations_enrichment (bool)

will add the alert annotations to the default alerts if true

Foreign logs enricher

Playbook Action: foreign_logs_enricher

Generic enricher to fetch and attach pod logs.

The logs are fetched for the pod determined by the label selector field in the parameters.

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - foreign_logs_enricher:
      label_selectors:
      - string
      - string
  triggers:
  - on_pod_create: {}

The above is an example. Try customizing the trigger and parameters.

required:
label_selectors (str list)

List of specific label selectors to retrieve logs from

optional:
container_name (str)

Specific container to get logs from

warn_on_missing_label (bool)

Send a warning if the alert doesn't have a pod label

regex_replacer_patterns (complex list)

regex patterns to replace text, for example for security reasons (Note: Replacements are executed in the given order)

each entry contains:

required:
regex (str)
optional:
name (str) = Redacted
regex_replacement_style (str)

one of SAME_LENGTH_ASTERISKS or NAMED (See RegexReplacementStyle)

previous (bool)
filter_regex (str)

only shows lines that match the regex

title_override (str)

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger foreign_logs_enricher  label_selectors=LABEL_SELECTORS

Alert foreign logs enricher

Playbook Action: alert_foreign_logs_enricher

Prometheus alert enricher to fetch and attach pod logs.

This action behaves the same as the foreign_logs_enricher. The logs are fetched for the pod determined by the label selector field in the parameters. The label selector field can use the format ${labels.XYZ} to reference any XYZ label present in the Prometheus alert.

Here is an example using a dynamic selector that depends on the alert's labels

customPlaybooks:
- actions:
    - alert_foreign_logs_enricher:
        label_selectors:
        - "app=${labels.service}"
        - "env=production"
triggers:
  - on_prometheus_alert: {}

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - alert_foreign_logs_enricher:
      label_selectors:
      - string
      - string
  triggers:
  - on_prometheus_alert: {}

The above is an example. Try customizing the trigger and parameters.

required:
label_selectors (str list)

List of specific label selectors to retrieve logs from

optional:
container_name (str)

Specific container to get logs from

warn_on_missing_label (bool)

Send a warning if the alert doesn't have a pod label

regex_replacer_patterns (complex list)

regex patterns to replace text, for example for security reasons (Note: Replacements are executed in the given order)

each entry contains:

required:
regex (str)
optional:
name (str) = Redacted
regex_replacement_style (str)

one of SAME_LENGTH_ASTERISKS or NAMED (See RegexReplacementStyle)

previous (bool)
filter_regex (str)

only shows lines that match the regex

title_override (str)

Alert definition enricher

Playbook Action: alert_definition_enricher

Enrich an alert with the Prometheus query that triggered the alert.

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - alert_definition_enricher: {}
  triggers:
  - on_prometheus_alert: {}

The above is an example. Try customizing the trigger and parameters.

No action parameters

Mention enricher

Playbook Action: mention_enricher

You can define who to mention using a static mentions configuration, Or, you can define it using a label or annotation, that exists either on the Kubernetes resource, or the alert

Order: 1. Resource annotations (For alert, get from FindingSubject. For other resources, get from obj metadata) 2. Resource labels (For alert, get from FindingSubject. For other resources, get from obj metadata) 3. Alert annotations (only for alert) 4. Alert labels (only for alert)

Note this enricher only works with the Slack sink

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - mention_enricher:
      static_mentions:
      - U44V9P1JJ1Z
      - S22H3Q3Q111
  triggers:
  - on_pod_crash_loop: {}

The above is an example. Try customizing the trigger and parameters.

optional:
static_mentions (str list)

List of Slack user ids/subteam ids to be mentioned

mentions_label (str)

An alert label, or Kubernetes resource label, in which the value contains a dot separated ids to mention

message_template (str) = Hey: $mentions

Optional. Custom mention message. Default: "Hey: $mentions"

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger mention_enricher kind=RESOURCE_KIND name=RESOURCE_NAME

Prometheus Silencers

These actions can selectively silence Prometheus alerts. They only work with the on_prometheus_alert trigger:

Node restart silencer

Playbook Action: node_restart_silencer

Silence alerts for pods on a node that recently restarted.

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - node_restart_silencer: {}
  triggers:
  - on_prometheus_alert: {}

The above is an example. Try customizing the trigger and parameters.

optional:
post_restart_silence (int) = 300

Period after restart to silence alerts. Seconds.

Severity silencer

Playbook Action: severity_silencer

Silence alerts with the specified severity level.

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - severity_silencer:
      severity: warning
  triggers:
  - on_prometheus_alert: {}

The above is an example. Try customizing the trigger and parameters.

optional:
severity (str) = none

severity level that should be silenced.

Name silencer

Playbook Action: name_silencer

Silence named alerts.

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - name_silencer:
      names:
      - string
      - string
  triggers:
  - on_prometheus_alert: {}

The above is an example. Try customizing the trigger and parameters.

required:
names (str list)

List of alert names that should be silenced.

Silence alert

Playbook Action: silence_alert

Silence received alert.

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - silence_alert: {}
  triggers:
  - on_prometheus_alert: {}

The above is an example. Try customizing the trigger and parameters.

optional:
log_silence (bool)

prints a log in robusta logs of the silence.

Pod status silencer

Playbook Action: pod_status_silencer

Stop execution based on pod statuses.

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - pod_status_silencer:
      exclude:
      - Unknown
      include:
      - Pending
  triggers:
  - on_pod_all_changes: {}

The above is an example. Try customizing the trigger and parameters.

optional:
include (str list)

If available, will stop processing unless the pod status is in the include list

exclude (str list)

If available, will stop processing if the pod status is in the exclude list

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger pod_status_silencer name=POD_NAME namespace=POD_NAMESPACE