Event Enrichment¶
The actions are used to gather extra data on errors, alerts, and other cluster events.
Use them as building blocks in your own automations.
Node Enrichers¶
These actions can add context to any node-related event, be it from on_prometheus_alert or on_node_update.
Node bash enricher¶
Playbook Action
Execute the specified bash command on the target node. Enrich the finding with the command results.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- node_bash_enricher:
bash_command: ls -l /etc/data/db
triggers:
- on_node_create: {}
The above is an example. Try customizing the trigger and parameters.
- bash_command (str)
Bash command to execute on the target.
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger node_bash_enricher name=NODE_NAME bash_command=BASH_COMMAND
Node status enricher¶
Playbook Action
Enrich the finding with the node's status conditions.
Can help troubleshooting Node issues.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- node_status_enricher: {}
triggers:
- on_node_create: {}
The above is an example. Try customizing the trigger and parameters.
No action parameters
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger node_status_enricher name=NODE_NAME
Node running pods enricher¶
Playbook Action
Enrich the finding with pods running on this node, along with the 'Ready' status of each pod.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- node_running_pods_enricher: {}
triggers:
- on_node_create: {}
The above is an example. Try customizing the trigger and parameters.
No action parameters
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger node_running_pods_enricher name=NODE_NAME
Node allocatable resources enricher¶
Playbook Action
Enrich the finding with the node resources available for allocation.
Can help troubleshooting node issues.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- node_allocatable_resources_enricher: {}
triggers:
- on_node_create: {}
The above is an example. Try customizing the trigger and parameters.
No action parameters
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger node_allocatable_resources_enricher name=NODE_NAME
Node graph enricher¶
Playbook Action
Get a graph of a specific resource for this node.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- node_graph_enricher:
prometheus_url: http://prometheus-k8s.monitoring.svc.cluster.local:9090
resource_type: Memory
triggers:
- on_node_create: {}
The above is an example. Try customizing the trigger and parameters.
- resource_type (str)
one of: CPU, Memory, Disk (see ResourceChartResourceType)
- prometheus_url (str)
Prometheus url. If omitted, we will try to find a prometheus instance in the same cluster
- graph_duration_minutes (int) = 60
Graph duration is minutes. Default is 60.
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger node_graph_enricher name=NODE_NAME resource_type=RESOURCE_TYPE
Node cpu enricher¶
Playbook Action
Enrich the finding with analysis of the node's CPU usage. Collect information about pods running on this node, their CPU request configuration, their actual cpu usage etc. Provides insightful information regarding node high CPU usage.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- node_cpu_enricher:
prometheus_url: http://prometheus-k8s.monitoring.svc.cluster.local:9090
triggers:
- on_node_create: {}
The above is an example. Try customizing the trigger and parameters.
- prometheus_url (str)
Prometheus url. If omitted, we will try to find a prometheus instance in the same cluster
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger node_cpu_enricher name=NODE_NAME
Pod Enrichers¶
These actions can add context to any pod-related event, be it from on_prometheus_alert or on_pod_update.
Logs enricher¶
Playbook Action
Enrich the alert with pod logs The pod to fetch logs for is determined by the alert’s pod label from Prometheus.
By default, if the alert has no pod this enricher will silently do nothing.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- logs_enricher: {}
triggers:
- on_pod_all_changes: {}
The above is an example. Try customizing the trigger and parameters.
- warn_on_missing_label (bool)
Send a warning if the alert doesn't have a pod label
- regex_replacer_patterns (complex list)
regex patterns to replace text, for example for security reasons (Note: Replacements are executed in the given order)
each entry contains:
required:- regex (str)
- name (str) = Redacted
- regex_replacement_style (str)
one of SAME_LENGTH_ASTERISKS or NAMED (See RegexReplacementStyle)
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger logs_enricher name=POD_NAME namespace=POD_NAMESPACE
Pod bash enricher¶
Playbook Action
Execute the specified bash command on the target pod. Enrich the finding with the command results.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- pod_bash_enricher:
bash_command: ls -l /etc/data/db
triggers:
- on_pod_all_changes: {}
The above is an example. Try customizing the trigger and parameters.
- bash_command (str)
Bash command to execute on the target.
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger pod_bash_enricher name=POD_NAME namespace=POD_NAMESPACE bash_command=BASH_COMMAND
Pod events enricher¶
Playbook Action
Enrich the finding with the pod events.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- pod_events_enricher: {}
triggers:
- on_pod_all_changes: {}
The above is an example. Try customizing the trigger and parameters.
No action parameters
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger pod_events_enricher name=POD_NAME namespace=POD_NAMESPACE
Pod graph enricher¶
Playbook Action
Get a graph of a specific resource for this pod. Note: "Disk" Resource is not supported.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- pod_graph_enricher:
prometheus_url: http://prometheus-k8s.monitoring.svc.cluster.local:9090
resource_type: Memory
triggers:
- on_pod_all_changes: {}
The above is an example. Try customizing the trigger and parameters.
- resource_type (str)
one of: CPU, Memory, Disk (see ResourceChartResourceType)
- prometheus_url (str)
Prometheus url. If omitted, we will try to find a prometheus instance in the same cluster
- graph_duration_minutes (int) = 60
Graph duration is minutes. Default is 60.
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger pod_graph_enricher name=POD_NAME namespace=POD_NAMESPACE resource_type=RESOURCE_TYPE
Pod ps¶
Playbook Action
Fetch the list of running processes in a pod.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- pod_ps: {}
triggers:
- on_pod_all_changes: {}
The above is an example. Try customizing the trigger and parameters.
No action parameters
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger pod_ps name=POD_NAME namespace=POD_NAMESPACE
Daemonset Enrichers¶
These actions can add context to any daemonset-related event, be it from on_prometheus_alert or on_daemonset_update.
Daemonset status enricher¶
Playbook Action
Enrich the finding with daemon set stats.
Includes recommendations for the identified cause.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- daemonset_status_enricher: {}
triggers:
- on_daemonset_all_changes: {}
The above is an example. Try customizing the trigger and parameters.
No action parameters
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger daemonset_status_enricher name=DAEMONSET_NAME namespace=DAEMONSET_NAMESPACE
Deployment Enrichers¶
These actions can add context to any deployment-related event, be it from on_prometheus_alert or on_deployment_update.
Deployment status enricher¶
Playbook Action
Enrich the finding with deployment status conditions.
Usually these conditions can provide important information regarding possible issues.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- deployment_status_enricher: {}
triggers:
- on_deployment_all_changes: {}
The above is an example. Try customizing the trigger and parameters.
No action parameters
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger deployment_status_enricher name=DEPLOYMENT_NAME namespace=DEPLOYMENT_NAMESPACE
Kubernetes Resource Enrichers¶
These actions can add context to more than one Kubernetes resource type
Related pods¶
Playbook Action
Return the list of pods related to that k8s resource. For example, return all pods of a given k8s Deployment
Supports Deployments, ReplicaSets, DaemonSets, StatefulSets and Pods
Add this to your Robusta configuration (Helm values.yaml):
actions:
- related_pods: {}
triggers:
- on_clusterrolebinding_all_changes: {}
The above is an example. Try customizing the trigger and parameters.
No action parameters
Or any other inheriting trigger. See Trigger-Action Compatibility for details
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger related_pods kind=RESOURCE_KIND name=RESOURCE_NAME
Event Enrichers¶
Event resource events¶
Playbook Action
Given a Kubernetes event, gather all other events on the same resource in the near past
Add this to your Robusta configuration (Helm values.yaml):
actions:
- event_resource_events: {}
triggers:
- on_kubernetes_warning_event_update: {}
The above is an example. Try customizing the trigger and parameters.
- finding_key (str) = DEFAULT
Specify the finding identifier, to reference it in other actions.
Prometheus Enrichers¶
These actions enrich Prometheus alerts. They only work with the on_prometheus_alert trigger:
Graph enricher¶
Playbook Action
Enrich the alert with a graph of the Prometheus query which triggered the alert.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- graph_enricher:
prometheus_url: http://prometheus-k8s.monitoring.svc.cluster.local:9090
triggers:
- on_prometheus_alert: {}
The above is an example. Try customizing the trigger and parameters.
- prometheus_url (str)
Prometheus url. If omitted, we will try to find a prometheus instance in the same cluster
Custom graph enricher¶
Playbook Action
Enrich the alert with a graph of a custom Prometheus query
Add this to your Robusta configuration (Helm values.yaml):
actions:
- custom_graph_enricher:
graph_title: CPU Usage for this nod
prometheus_url: http://prometheus-k8s.monitoring.svc.cluster.local:9090
promql_query: instance:node_cpu_utilisation:rate5m{job="node-exporter", instance=~"$node_internal_ip:[0-9]+",
cluster=""} != 0
triggers:
- on_prometheus_alert: {}
The above is an example. Try customizing the trigger and parameters.
- promql_query (str)
Promql query. You can use $pod, $node and $node_internal_ip to template (see example). For more information, see https://prometheus.io/docs/prometheus/latest/querying/basics/
- prometheus_url (str)
Prometheus url. If omitted, we will try to find a prometheus instance in the same cluster
- graph_title (str)
A nicer name for the Prometheus query.
- graph_duration_minutes (int) = 60
Graph duration is minutes.
- chart_values_format (str) = Plain
Customize the y-axis labels with one of: Plain, Bytes, Percentage (see ChartValuesFormat)
Alert graph enricher¶
Playbook Action
Enrich the alert with a graph of a relevant resource (Pod or Node).
Add this to your Robusta configuration (Helm values.yaml):
actions:
- alert_graph_enricher:
item_type: Pod
prometheus_url: http://prometheus-k8s.monitoring.svc.cluster.local:9090
resource_type: Memory
triggers:
- on_prometheus_alert: {}
The above is an example. Try customizing the trigger and parameters.
- resource_type (str)
one of: CPU, Memory, Disk (see ResourceChartResourceType)
- item_type (str)
one of: Pod, Node (see ResourceChartItemType)
- prometheus_url (str)
Prometheus url. If omitted, we will try to find a prometheus instance in the same cluster
- graph_duration_minutes (int) = 60
Graph duration is minutes. Default is 60.
Template enricher¶
Playbook Action
Enrich an alert with a paragraph to the alert’s description containing templated markdown. You can inject any of the alert’s Prometheus labels into the markdown.
A variable like $foo will be replaced by the value of the Prometheus label foo. If a label isn’t present then the text “<missing>” will be used instead.
Common variables to use are $alertname, $deployment, $namespace, and $node
The template can include all markdown directives supported by Slack. Note that Slack markdown links use a different format than GitHub.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- template_enricher:
template: The alertname is $alertname and the pod is $pod
triggers:
- on_prometheus_alert: {}
The above is an example. Try customizing the trigger and parameters.
- template (str)
The enrichment templated markdown text
Stack overflow enricher¶
Playbook Action
Add a button to the alert - clicking it will show the top StackOverflow search results on this alert name.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- stack_overflow_enricher: {}
triggers:
- on_prometheus_alert: {}
The above is an example. Try customizing the trigger and parameters.
No action parameters
Default enricher¶
Playbook Action
Enrich an alert with the original message and labels.
By default, this enricher is last in the processing order, so it will be added to all alerts, that aren't silenced.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- default_enricher: {}
triggers:
- on_prometheus_alert: {}
The above is an example. Try customizing the trigger and parameters.
No action parameters
Alert definition enricher¶
Playbook Action
Enrich an alert with the Prometheus query that triggered the alert.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- alert_definition_enricher: {}
triggers:
- on_prometheus_alert: {}
The above is an example. Try customizing the trigger and parameters.
No action parameters
Prometheus Silencers¶
These actions can selectively silence Prometheus alerts. They only work with the on_prometheus_alert trigger:
Node restart silencer¶
Playbook Action
Silence alerts for pods on a node that recently restarted.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- node_restart_silencer: {}
triggers:
- on_prometheus_alert: {}
The above is an example. Try customizing the trigger and parameters.
- post_restart_silence (int) = 300
Period after restart to silence alerts. Seconds.
Severity silencer¶
Playbook Action
Silence alerts with the specified severity level.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- severity_silencer:
severity: warning
triggers:
- on_prometheus_alert: {}
The above is an example. Try customizing the trigger and parameters.
- severity (str) = none
severity level that should be silenced.
Name silencer¶
Playbook Action
Silence named alerts.
Add this to your Robusta configuration (Helm values.yaml):
actions:
- name_silencer:
names:
- string
- string
triggers:
- on_prometheus_alert: {}
The above is an example. Try customizing the trigger and parameters.
- names (str list)
List of alert names that should be silenced.