Event Enrichment¶
The actions are used to gather extra data on errors, alerts, and other cluster events.
Use them as building blocks in your own automations.
Node Enrichers¶
These actions can add context to any node-related event, be it from on_prometheus_alert or on_node_update.
Node bash enricher¶
Playbook Action: node_bash_enricher
Execute the specified bash command on the target node. Enrich the finding with the command results.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- node_bash_enricher:
bash_command: ls -l /etc/data/db
triggers:
- on_node_all_changes: {}
The above is an example. Try customizing the trigger and parameters.
- bash_command (str)
Bash command to execute on the target.
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger node_bash_enricher name=NODE_NAME bash_command=BASH_COMMAND
Node status enricher¶
Playbook Action: node_status_enricher
Enrich the finding with the node's status conditions.
Can help troubleshooting Node issues.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- node_status_enricher: {}
triggers:
- on_node_all_changes: {}
The above is an example. Try customizing the trigger and parameters.
No action parameters
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger node_status_enricher name=NODE_NAME
Node running pods enricher¶
Playbook Action: node_running_pods_enricher
Enrich the finding with pods running on this node, along with the 'Ready' status of each pod.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- node_running_pods_enricher: {}
triggers:
- on_node_all_changes: {}
The above is an example. Try customizing the trigger and parameters.
No action parameters
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger node_running_pods_enricher name=NODE_NAME
Node allocatable resources enricher¶
Playbook Action: node_allocatable_resources_enricher
Enrich the finding with the node resources available for allocation.
Can help troubleshooting node issues.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- node_allocatable_resources_enricher: {}
triggers:
- on_node_all_changes: {}
The above is an example. Try customizing the trigger and parameters.
No action parameters
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger node_allocatable_resources_enricher name=NODE_NAME
Node graph enricher¶
Playbook Action: node_graph_enricher
Get a graph of a specific resource for this node.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- node_graph_enricher:
prometheus_url: http://prometheus-k8s.monitoring.svc.cluster.local:9090
resource_type: Memory
triggers:
- on_node_all_changes: {}
The above is an example. Try customizing the trigger and parameters.
- resource_type (str)
one of: CPU, Memory, Disk (see ResourceChartResourceType)
- prometheus_url (str)
Prometheus url. If omitted, we will try to find a prometheus instance in the same cluster
- graph_duration_minutes (int) = 60
Graph duration is minutes. Default is 60.
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger node_graph_enricher name=NODE_NAME resource_type=RESOURCE_TYPE
Node cpu enricher¶
Playbook Action: node_cpu_enricher
Enrich the finding with analysis of the node's CPU usage. Collect information about pods running on this node, their CPU request configuration, their actual cpu usage etc. Provides insightful information regarding node high CPU usage.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- node_cpu_enricher:
prometheus_url: http://prometheus-k8s.monitoring.svc.cluster.local:9090
triggers:
- on_node_all_changes: {}
The above is an example. Try customizing the trigger and parameters.
- prometheus_url (str)
Prometheus url. If omitted, we will try to find a prometheus instance in the same cluster
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger node_cpu_enricher name=NODE_NAME
Pod Enrichers¶
These actions can add context to any pod-related event, be it from on_prometheus_alert or on_pod_update.
Logs enricher¶
Playbook Action: logs_enricher
Enrich the alert with pod logs The pod to fetch logs for is determined by the alert’s pod label from Prometheus.
By default, if the alert has no pod this enricher will silently do nothing.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- logs_enricher: {}
triggers:
- on_pod_crash_loop: {}
The above is an example. Try customizing the trigger and parameters.
- warn_on_missing_label (bool)
Send a warning if the alert doesn't have a pod label
- regex_replacer_patterns (complex list)
regex patterns to replace text, for example for security reasons (Note: Replacements are executed in the given order)
each entry contains:
required:- regex (str)
- name (str) = Redacted
- regex_replacement_style (str)
one of SAME_LENGTH_ASTERISKS or NAMED (See RegexReplacementStyle)
- previous (bool)
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger logs_enricher name=POD_NAME namespace=POD_NAMESPACE
Pod events enricher¶
Playbook Action: pod_events_enricher
Given a Kubernetes pod, fetch related events in the near past
Add this to your Robusta configuration (Helm values.yaml):
actions:
- pod_events_enricher: {}
triggers:
- on_pod_all_changes: {}
The above is an example. Try customizing the trigger and parameters.
- max_events (int) = 8
- included_types (str list) = ['Warning', 'Normal']
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger pod_events_enricher name=POD_NAME namespace=POD_NAMESPACE
Pod bash enricher¶
Playbook Action: pod_bash_enricher
Execute the specified bash command on the target pod. Enrich the finding with the command results.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- pod_bash_enricher:
bash_command: ls -l /etc/data/db
triggers:
- on_pod_all_changes:
alert_name: ExampleLowDiskAlert
The above is an example. Try customizing the trigger and parameters.
- bash_command (str)
Bash command to execute on the target.
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger pod_bash_enricher name=POD_NAME namespace=POD_NAMESPACE bash_command=BASH_COMMAND
Pod graph enricher¶
Playbook Action: pod_graph_enricher
Get a graph of a specific resource for this pod. Note: "Disk" Resource is not supported.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- pod_graph_enricher:
prometheus_url: http://prometheus-k8s.monitoring.svc.cluster.local:9090
resource_type: Memory
triggers:
- on_pod_all_changes: {}
The above is an example. Try customizing the trigger and parameters.
- resource_type (str)
one of: CPU, Memory, Disk (see ResourceChartResourceType)
- prometheus_url (str)
Prometheus url. If omitted, we will try to find a prometheus instance in the same cluster
- graph_duration_minutes (int) = 60
Graph duration is minutes. Default is 60.
- display_limits (bool)
displays on the graph the pod limit for the resource if true (only CPU and Memory are supported)
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger pod_graph_enricher name=POD_NAME namespace=POD_NAMESPACE resource_type=RESOURCE_TYPE
Pod node graph enricher¶
Playbook Action: pod_node_graph_enricher
Get a graph of a specific resource for the node the pod resides on.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- pod_node_graph_enricher:
prometheus_url: http://prometheus-k8s.monitoring.svc.cluster.local:9090
resource_type: Memory
triggers:
- on_pod_all_changes: {}
The above is an example. Try customizing the trigger and parameters.
- resource_type (str)
one of: CPU, Memory, Disk (see ResourceChartResourceType)
- prometheus_url (str)
Prometheus url. If omitted, we will try to find a prometheus instance in the same cluster
- graph_duration_minutes (int) = 60
Graph duration is minutes. Default is 60.
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger pod_node_graph_enricher name=POD_NAME namespace=POD_NAMESPACE resource_type=RESOURCE_TYPE
Pod ps¶
Playbook Action: pod_ps
Fetch the list of running processes in a pod.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- pod_ps: {}
triggers:
- on_pod_all_changes: {}
The above is an example. Try customizing the trigger and parameters.
No action parameters
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger pod_ps name=POD_NAME namespace=POD_NAMESPACE
Pod oom killer enricher¶
Playbook Action: pod_oom_killer_enricher
Retrieves pod and node information for an OOMKilled pod
Add this to your Robusta configuration (Helm values.yaml):
actions:
- pod_oom_killer_enricher: {}
triggers:
- on_pod_all_changes: {}
The above is an example. Try customizing the trigger and parameters.
No action parameters
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger pod_oom_killer_enricher name=POD_NAME namespace=POD_NAMESPACE
Daemonset Enrichers¶
These actions can add context to any daemonset-related event, be it from on_prometheus_alert or on_daemonset_update.
Daemonset status enricher¶
Playbook Action: daemonset_status_enricher
Enrich the finding with daemon set stats.
Includes recommendations for the identified cause.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- daemonset_status_enricher: {}
triggers:
- on_daemonset_delete: {}
The above is an example. Try customizing the trigger and parameters.
No action parameters
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger daemonset_status_enricher name=DAEMONSET_NAME namespace=DAEMONSET_NAMESPACE
Deployment Enrichers¶
These actions can add context to any deployment-related event, be it from on_prometheus_alert or on_deployment_update.
Deployment status enricher¶
Playbook Action: deployment_status_enricher
Enrich the finding with deployment status conditions.
Usually these conditions can provide important information regarding possible issues.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- deployment_status_enricher: {}
triggers:
- on_deployment_delete: {}
The above is an example. Try customizing the trigger and parameters.
No action parameters
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger deployment_status_enricher name=DEPLOYMENT_NAME namespace=DEPLOYMENT_NAMESPACE
Deployment events enricher¶
Playbook Action: deployment_events_enricher
Given a deployment, fetch related events in the near past.
Can optionally fetch events for related pods instead.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- deployment_events_enricher: {}
triggers:
- on_deployment_delete: {}
The above is an example. Try customizing the trigger and parameters.
- max_events (int) = 8
- included_types (str list) = ['Warning', 'Normal']
- dependent_pod_mode (bool)
when True, instead of fetching events for the deployment itself, fetch events for pods in the deployment.
- max_pods (int) = 1
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger deployment_events_enricher name=DEPLOYMENT_NAME namespace=DEPLOYMENT_NAMESPACE
Job Enrichers¶
These actions can add context to any job-related event, be it from on_prometheus_alert or on_job_update.
Job events enricher¶
Playbook Action: job_events_enricher
Given a Kubernetes job, fetch related events in the near past
Add this to your Robusta configuration (Helm values.yaml):
actions:
- job_events_enricher: {}
triggers:
- on_job_update: {}
The above is an example. Try customizing the trigger and parameters.
- max_events (int) = 8
- included_types (str list) = ['Warning', 'Normal']
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger job_events_enricher name=JOB_NAME namespace=JOB_NAMESPACE
Job info enricher¶
Playbook Action: job_info_enricher
Given a Kubernetes job, add information about the job, from the job spec and status.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- job_info_enricher: {}
triggers:
- on_job_update: {}
The above is an example. Try customizing the trigger and parameters.
No action parameters
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger job_info_enricher name=JOB_NAME namespace=JOB_NAMESPACE
Job pod enricher¶
Playbook Action: job_pod_enricher
Given a Kubernetes job, get information about the latest job pod.
Used to get the related pod's events and/or logs
Add this to your Robusta configuration (Helm values.yaml):
actions:
- job_pod_enricher: {}
triggers:
- on_job_update: {}
The above is an example. Try customizing the trigger and parameters.
- max_events (int) = 8
- included_types (str list) = ['Warning', 'Normal']
- events (bool) = True
Add the events of the related pod
- logs (bool) = True
Add the logs of the related pod
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger job_pod_enricher name=JOB_NAME namespace=JOB_NAMESPACE
Kubernetes Resource Enrichers¶
These actions can add context to more than one Kubernetes resource type
Related pods¶
Playbook Action: related_pods
Return the list of pods related to that k8s resource. For example, return all pods of a given k8s Deployment
Supports Deployments, ReplicaSets, DaemonSets, StatefulSets and Pods
Add this to your Robusta configuration (Helm values.yaml):
actions:
- related_pods: {}
triggers:
- on_clusterrolebinding_create: {}
The above is an example. Try customizing the trigger and parameters.
No action parameters
Or any other inheriting trigger. See Trigger-Action Compatibility for details
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger related_pods kind=RESOURCE_KIND name=RESOURCE_NAME
Event Enrichers¶
Event resource events¶
Playbook Action: event_resource_events
Given a Kubernetes event, gather all other events on the same resource in the near past
Add this to your Robusta configuration (Helm values.yaml):
actions:
- event_resource_events: {}
triggers:
- on_kubernetes_warning_event_update: {}
The above is an example. Try customizing the trigger and parameters.
No action parameters
Deployment events enricher¶
Playbook Action: deployment_events_enricher
Given a deployment, fetch related events in the near past.
Can optionally fetch events for related pods instead.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- deployment_events_enricher: {}
triggers:
- on_deployment_delete: {}
The above is an example. Try customizing the trigger and parameters.
- max_events (int) = 8
- included_types (str list) = ['Warning', 'Normal']
- dependent_pod_mode (bool)
when True, instead of fetching events for the deployment itself, fetch events for pods in the deployment.
- max_pods (int) = 1
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger deployment_events_enricher name=DEPLOYMENT_NAME namespace=DEPLOYMENT_NAMESPACE
Pod events enricher¶
Playbook Action: pod_events_enricher
Given a Kubernetes pod, fetch related events in the near past
Add this to your Robusta configuration (Helm values.yaml):
actions:
- pod_events_enricher: {}
triggers:
- on_pod_all_changes: {}
The above is an example. Try customizing the trigger and parameters.
- max_events (int) = 8
- included_types (str list) = ['Warning', 'Normal']
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger pod_events_enricher name=POD_NAME namespace=POD_NAMESPACE
Prometheus Enrichers¶
These actions enrich Prometheus alerts. They only work with the on_prometheus_alert trigger:
Graph enricher¶
Playbook Action: graph_enricher
Enrich the alert with a graph of the Prometheus query which triggered the alert.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- graph_enricher:
prometheus_url: http://prometheus-k8s.monitoring.svc.cluster.local:9090
triggers:
- on_prometheus_alert: {}
The above is an example. Try customizing the trigger and parameters.
- prometheus_url (str)
Prometheus url. If omitted, we will try to find a prometheus instance in the same cluster
Custom graph enricher¶
Playbook Action: custom_graph_enricher
Enrich the alert with a graph of a custom Prometheus query
Add this to your Robusta configuration (Helm values.yaml):
actions:
- custom_graph_enricher:
graph_title: CPU Usage for this nod
prometheus_url: http://prometheus-k8s.monitoring.svc.cluster.local:9090
promql_query: instance:node_cpu_utilisation:rate5m{job="node-exporter", instance=~"$node_internal_ip:[0-9]+",
cluster=""} != 0
triggers:
- on_prometheus_alert: {}
The above is an example. Try customizing the trigger and parameters.
- promql_query (str)
Promql query. You can use $pod, $node and $node_internal_ip to template (see example). For more information, see https://prometheus.io/docs/prometheus/latest/querying/basics/
- prometheus_url (str)
Prometheus url. If omitted, we will try to find a prometheus instance in the same cluster
- graph_title (str)
A nicer name for the Prometheus query.
- graph_duration_minutes (int) = 60
Graph duration is minutes.
- chart_values_format (str) = Plain
Customize the y-axis labels with one of: Plain, Bytes, Percentage (see ChartValuesFormat)
Alert graph enricher¶
Playbook Action: alert_graph_enricher
Enrich the alert with a graph of a relevant resource (Pod or Node).
Add this to your Robusta configuration (Helm values.yaml):
actions:
- alert_graph_enricher:
item_type: Pod
prometheus_url: http://prometheus-k8s.monitoring.svc.cluster.local:9090
resource_type: Memory
triggers:
- on_prometheus_alert: {}
The above is an example. Try customizing the trigger and parameters.
- resource_type (str)
one of: CPU, Memory, Disk (see ResourceChartResourceType)
- item_type (str)
one of: Pod, Node (see ResourceChartItemType)
- prometheus_url (str)
Prometheus url. If omitted, we will try to find a prometheus instance in the same cluster
- graph_duration_minutes (int) = 60
Graph duration is minutes. Default is 60.
Template enricher¶
Playbook Action: template_enricher
Enrich an alert with a paragraph to the alert’s description containing templated markdown. You can inject any of the alert’s Prometheus labels into the markdown.
A variable like $foo will be replaced by the value of the Prometheus label foo. If a label isn’t present then the text “<missing>” will be used instead.
Common variables to use are $alertname, $deployment, $namespace, and $node
The template can include all markdown directives supported by Slack. Note that Slack markdown links use a different format than GitHub.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- template_enricher:
template: The alertname is $alertname and the pod is $pod
triggers:
- on_prometheus_alert: {}
The above is an example. Try customizing the trigger and parameters.
- template (str)
The enrichment templated markdown text
Stack overflow enricher¶
Playbook Action: stack_overflow_enricher
Add a button to the alert - clicking it will show the top StackOverflow search results on this alert name.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- stack_overflow_enricher: {}
triggers:
- on_prometheus_alert: {}
The above is an example. Try customizing the trigger and parameters.
No action parameters
Default enricher¶
Playbook Action: default_enricher
Enrich an alert with the original message and labels.
By default, this enricher is last in the processing order, so it will be added to all alerts, that aren't silenced.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- default_enricher: {}
triggers:
- on_prometheus_alert: {}
The above is an example. Try customizing the trigger and parameters.
No action parameters
Alert definition enricher¶
Playbook Action: alert_definition_enricher
Enrich an alert with the Prometheus query that triggered the alert.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- alert_definition_enricher: {}
triggers:
- on_prometheus_alert: {}
The above is an example. Try customizing the trigger and parameters.
No action parameters
Prometheus Silencers¶
These actions can selectively silence Prometheus alerts. They only work with the on_prometheus_alert trigger:
Node restart silencer¶
Playbook Action: node_restart_silencer
Silence alerts for pods on a node that recently restarted.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- node_restart_silencer: {}
triggers:
- on_prometheus_alert: {}
The above is an example. Try customizing the trigger and parameters.
- post_restart_silence (int) = 300
Period after restart to silence alerts. Seconds.
Severity silencer¶
Playbook Action: severity_silencer
Silence alerts with the specified severity level.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- severity_silencer:
severity: warning
triggers:
- on_prometheus_alert: {}
The above is an example. Try customizing the trigger and parameters.
- severity (str) = none
severity level that should be silenced.
Name silencer¶
Playbook Action: name_silencer
Silence named alerts.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- name_silencer:
names:
- string
- string
triggers:
- on_prometheus_alert: {}
The above is an example. Try customizing the trigger and parameters.
- names (str list)
List of alert names that should be silenced.