MiscellaneousΒΆ
These actions are useful for general Kubernetes troubleshooting and everything else.
ArgoCDΒΆ
Argo app syncΒΆ
Playbook Action: argo_app_sync
Add this to your Robusta configuration (Helm values.yaml):
customPlaybooks:
- actions:
- argo_app_sync:
argo_app_name: string
argo_token: '********'
argo_url: https://my-argo-cd.com
triggers:
- on_pod_create: {}
The above is an example. Try customizing the trigger and parameters.
- argo_url (str)
http(s) Argo CD server url.
- argo_token (str)
Argo CD authentication token.
- argo_app_name (str)
Argo CD application that needs syncing.
- argo_verify_server_cert (bool) = True
verify Argo CD server certificate. Defaults to True.
- rate_limit_seconds (int) = 1800
this playbook is rate limited. Defaults to 1800 seconds.
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger argo_app_sync argo_url=ARGO_URL argo_token=ARGO_TOKEN argo_app_name=ARGO_APP_NAME
Kubernetes OptimizationΒΆ
Config ab testingΒΆ
Playbook Action: config_ab_testing
Apply YAML configurations to Kubernetes resources for limited periods of time.
Adds adds grafana annotations showing when each configuration was applied.
The execution schedule is defined by the playbook trigger. (every X seconds)
- Commonly used for:
Troubleshooting - Finding the first version a production bug appeared by iterating over image tags Cost/performance optimization - Comparing the cost or performance of different deployment configurations
- Note:
Only changing attributes that already exists in the active configuration is supported.
For example, you can change resources.requests.cpu, if that attribute already exists in the deployment.
Add this to your Robusta configuration (Helm values.yaml):
customPlaybooks:
- actions:
- config_ab_testing:
configuration_sets:
- config_items: '"spec.template.spec.containers[0].resources.requests.cpu":
250m,
"spec.template.spec.containers[0].resources.requests.memory": 128Mi'
config_set_name: string
- config_items: '"spec.template.spec.containers[0].resources.requests.cpu":
250m,
"spec.template.spec.containers[0].resources.requests.memory": 128Mi'
config_set_name: string
grafana_api_key: '********'
grafana_dashboard_uid: 09ec8aa1e996d6ffcd6817bbaff4db1b
grafana_url: http://grafana.namespace.svc
kind: string
name: string
triggers:
- on_schedule: {}
The above is an example. Try customizing the trigger and parameters.
- grafana_api_key (str)
grafana key with write permissions.
- grafana_dashboard_uid (str)
dashboard ID as it appears in the dashboard's url
- kind (str)
The kind of the tested resource. Kind can be 'Deployment'/'StatefulSet' etc
- name (str)
The name of the tested resource.
- configuration_sets (complex list)
List of test configurations.
each entry contains:
required:- config_set_name (str)
The name of this configuration set. .
- config_items (str dict)
The yaml attributes values for this configuration set.
- grafana_url (str)
http(s) url of grafana or None for autodetection of an in-cluster grafana
- api_version (str) = v1
The api version of the tested resource.
- namespace (str) = default
The namespace of the tested resource.
Disk benchmarkΒΆ
Playbook Action: disk_benchmark
Run disk benchmark in your cluster. The benchmark creates a PVC, using the configured storage class, and runs the benchmark using fio. For more details: https://fio.readthedocs.io/en/latest/
Add this to your Robusta configuration (Helm values.yaml):
customPlaybooks:
- actions:
- disk_benchmark:
storage_class_name: string
triggers:
- on_pod_create: {}
The above is an example. Try customizing the trigger and parameters.
- storage_class_name (str)
Pvc storage class, From the available cluster storage classes. standard/fast/etc.
- custom_annotations (str dict)
custom annotations to be used for the running pod/job
- pvc_name (str) = robusta-disk-benchmark
Name of the pvc created for the benchmark.
- test_seconds (int) = 20
The benchmark duration.
- namespace (str) = robusta
Namespace used for the benchmark.
- disk_size (str) = 10Gi
The size of pvc used for the benchmark.
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger disk_benchmark storage_class_name=STORAGE_CLASS_NAME
BackupsΒΆ
Create pvc snapshotΒΆ
Playbook Action: create_pvc_snapshot
Backup a PersistentVolume by creating a VolumeSnapshot
Add this to your Robusta configuration (Helm values.yaml):
customPlaybooks:
- actions:
- create_pvc_snapshot:
name: some_pvc_name
triggers:
- on_schedule: {}
The above is an example. Try customizing the trigger and parameters.
- name (str)
name of the PersistentVolumeClaim to backup
- namespace (str) = default
namespace of the PersistentVolumeClaim to backup
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger create_pvc_snapshot name=NAME
Stress Testing and Chaos EngineeringΒΆ
Generate high cpuΒΆ
Playbook Action: generate_high_cpu
Create a pod with high CPU on the cluster for 60 seconds. Can be used to simulate alerts or other high CPU load scenarios.
Add this to your Robusta configuration (Helm values.yaml):
customPlaybooks:
- actions:
- generate_high_cpu: {}
triggers:
- on_pod_create: {}
The above is an example. Try customizing the trigger and parameters.
No action parameters
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger generate_high_cpu
Http stress testΒΆ
Playbook Action: http_stress_test
Add this to your Robusta configuration (Helm values.yaml):
customPlaybooks:
- actions:
- http_stress_test:
url: string
triggers:
- on_pod_create: {}
The above is an example. Try customizing the trigger and parameters.
- url (str)
In cluster target url.
- n (int) = 1000
Number of requests to run.
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger http_stress_test url=URL
Prometheus alertΒΆ
Playbook Action: prometheus_alert
Simulate Prometheus alert sent to the Robusta runner. Can be used for testing, when implementing actions triggered by Prometheus alerts.
See the full parameters if you need to simulate an alert on a Pod or Node.
This action is typically used via manual triggers and not predefined YAML triggers. An example config is not shown because it is not relevant for this action.
- alert_name (str)
Simulated alert name.
- pod_name (str)
Pod name, for a simulated pod alert.
- node_name (str)
Node name, for a simulated node alert.
- deployment_name (str)
Deployment name, for a simulated deployment alert.
- container_name (str)
Container name, for adding a label on container.
- service (str)
service name, for additional prometheus labels.
- job_name (str)
Job name, for a simulated Job alert.
- name (str)
- hpa (str)
HPA name, for a simulated HorizontalPodAutoscaler alert.
- statefulset_name (str)
- daemonset_name (str)
- namespace (str) = default
Pod namespace, for a simulated pod alert.
- status (str) = firing
Simulated alert status. firing/resolved.
- severity (str) = error
Simulated alert severity.
- description (str) = simulated prometheus alert
Simulated alert description.
- summary (str)
- runbook_url (str)
Simulated alert runbook_url. For example: "https://my-runbook-url.dev"
- fingerprint (str)
Simulated alert fingerprint. For example: "my-unique-fingerprint"
- labels (str)
Additional alert labels. For example: "key1: val1, key2: val2"
- annotations (str)
Additional alert labels. For example: "key1=val1, key2=val2". Note that annotation separator is =
- starts_at (datetime)
- ends_at (datetime)
- generator_url (str) = http://localhost:9090/graph?g0.expr=up%7Bjob%3D%22apiserver%22%7D&g0.tab=0&g0.stacked=0&g0.show_exemplars=0&g0.range_input=1h
Prometheus generator_url. Some enrichers, use this attribute to query Prometheus.
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger prometheus_alert alert_name=ALERT_NAME
Adding Slack user/group mentions to k8s/Prometheus eventsΒΆ
Mention enricherΒΆ
Playbook Action: mention_enricher
You can define who to mention using a static mentions configuration, Or, you can define it using a label or annotation, that exists either on the Kubernetes resource, or the alert
Order: 1. Resource annotations (For alert, get from FindingSubject. For other resources, get from obj metadata) 2. Resource labels (For alert, get from FindingSubject. For other resources, get from obj metadata) 3. Alert annotations (only for alert) 4. Alert labels (only for alert)
Note this enricher only works with the Slack sink
Add this to your Robusta configuration (Helm values.yaml):
customPlaybooks:
- actions:
- mention_enricher:
static_mentions:
- U44V9P1JJ1Z
- S22H3Q3Q111
triggers:
- on_pod_all_changes: {}
The above is an example. Try customizing the trigger and parameters.
- static_mentions (str list)
List of Slack user ids/subteam ids to be mentioned
- mentions_label (str)
An alert label, or Kubernetes resource label, in which the value contains a dot separated ids to mention
- message_template (str) = Hey: $mentions
Optional. Custom mention message. Default: "Hey: $mentions"
on_kubernetes_warning_event_delete
This action can be manually triggered using the Robusta CLI:
robusta playbooks trigger mention_enricher kind=RESOURCE_KIND name=RESOURCE_NAME