MiscellaneousΒΆ

These actions are useful for general Kubernetes troubleshooting and everything else.

ArgoCDΒΆ

Argo app syncΒΆ

Playbook Action: argo_app_sync

Sync a specified Argo CD application. Send a finding notifying the sync was performed

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - argo_app_sync:
      argo_app_name: string
      argo_token: '********'
      argo_url: https://my-argo-cd.com
  triggers:
  - on_pod_create: {}

The above is an example. Try customizing the trigger and parameters.

required:
argo_url (str)

http(s) Argo CD server url.

argo_token (str)

Argo CD authentication token.

argo_app_name (str)

Argo CD application that needs syncing.

optional:
argo_verify_server_cert (bool) = True

verify Argo CD server certificate. Defaults to True.

rate_limit_seconds (int) = 1800

this playbook is rate limited. Defaults to 1800 seconds.

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger argo_app_sync  argo_url=ARGO_URL argo_token=ARGO_TOKEN argo_app_name=ARGO_APP_NAME

Kubernetes OptimizationΒΆ

Config ab testingΒΆ

Playbook Action: config_ab_testing

Apply YAML configurations to Kubernetes resources for limited periods of time.

Adds adds grafana annotations showing when each configuration was applied.

The execution schedule is defined by the playbook trigger. (every X seconds)

Commonly used for:

Troubleshooting - Finding the first version a production bug appeared by iterating over image tags Cost/performance optimization - Comparing the cost or performance of different deployment configurations

Note:

Only changing attributes that already exists in the active configuration is supported.

For example, you can change resources.requests.cpu, if that attribute already exists in the deployment.

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - config_ab_testing:
      configuration_sets:
      - config_items: '"spec.template.spec.containers[0].resources.requests.cpu":
          250m,

          "spec.template.spec.containers[0].resources.requests.memory": 128Mi'
        config_set_name: string
      - config_items: '"spec.template.spec.containers[0].resources.requests.cpu":
          250m,

          "spec.template.spec.containers[0].resources.requests.memory": 128Mi'
        config_set_name: string
      grafana_api_key: '********'
      grafana_dashboard_uid: 09ec8aa1e996d6ffcd6817bbaff4db1b
      grafana_url: http://grafana.namespace.svc
      kind: string
      name: string
  triggers:
  - on_schedule: {}

The above is an example. Try customizing the trigger and parameters.

required:
grafana_api_key (str)

grafana key with write permissions.

grafana_dashboard_uid (str)

dashboard ID as it appears in the dashboard's url

kind (str)

The kind of the tested resource. Kind can be 'Deployment'/'StatefulSet' etc

name (str)

The name of the tested resource.

configuration_sets (complex list)

List of test configurations.

each entry contains:

required:
config_set_name (str)

The name of this configuration set. .

optional:
config_items (str dict)

The yaml attributes values for this configuration set.

optional:
grafana_url (str)

http(s) url of grafana or None for autodetection of an in-cluster grafana

api_version (str) = v1

The api version of the tested resource.

namespace (str) = default

The namespace of the tested resource.

Disk benchmarkΒΆ

Playbook Action: disk_benchmark

Run disk benchmark in your cluster. The benchmark creates a PVC, using the configured storage class, and runs the benchmark using fio. For more details: https://fio.readthedocs.io/en/latest/

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - disk_benchmark:
      storage_class_name: string
  triggers:
  - on_pod_create: {}

The above is an example. Try customizing the trigger and parameters.

required:
storage_class_name (str)

Pvc storage class, From the available cluster storage classes. standard/fast/etc.

optional:
custom_annotations (str dict)

custom annotations to be used for the running pod/job

pvc_name (str) = robusta-disk-benchmark

Name of the pvc created for the benchmark.

test_seconds (int) = 20

The benchmark duration.

namespace (str) = robusta

Namespace used for the benchmark.

disk_size (str) = 10Gi

The size of pvc used for the benchmark.

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger disk_benchmark  storage_class_name=STORAGE_CLASS_NAME

BackupsΒΆ

Create pvc snapshotΒΆ

Playbook Action: create_pvc_snapshot

Backup a PersistentVolume by creating a VolumeSnapshot

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - create_pvc_snapshot:
      name: some_pvc_name
  triggers:
  - on_schedule: {}

The above is an example. Try customizing the trigger and parameters.

required:
name (str)

name of the PersistentVolumeClaim to backup

optional:
namespace (str) = default

namespace of the PersistentVolumeClaim to backup

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger create_pvc_snapshot  name=NAME

Stress Testing and Chaos EngineeringΒΆ

Generate high cpuΒΆ

Playbook Action: generate_high_cpu

Create a pod with high CPU on the cluster for 60 seconds. Can be used to simulate alerts or other high CPU load scenarios.

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - generate_high_cpu: {}
  triggers:
  - on_pod_create: {}

The above is an example. Try customizing the trigger and parameters.

No action parameters

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger generate_high_cpu 

Http stress testΒΆ

Playbook Action: http_stress_test

Run an http stress test and send the results

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - http_stress_test:
      url: string
  triggers:
  - on_pod_create: {}

The above is an example. Try customizing the trigger and parameters.

required:
url (str)

In cluster target url.

optional:
n (int) = 1000

Number of requests to run.

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger http_stress_test  url=URL

Prometheus alertΒΆ

Playbook Action: prometheus_alert

Simulate Prometheus alert sent to the Robusta runner. Can be used for testing, when implementing actions triggered by Prometheus alerts.

See the full parameters if you need to simulate an alert on a Pod or Node.

This action is typically used via manual triggers and not predefined YAML triggers. An example config is not shown because it is not relevant for this action.

required:
alert_name (str)

Simulated alert name.

optional:
pod_name (str)

Pod name, for a simulated pod alert.

node_name (str)

Node name, for a simulated node alert.

deployment_name (str)

Deployment name, for a simulated deployment alert.

container_name (str)

Container name, for adding a label on container.

service (str)

service name, for additional prometheus labels.

job_name (str)

Job name, for a simulated Job alert.

name (str)
hpa (str)

HPA name, for a simulated HorizontalPodAutoscaler alert.

statefulset_name (str)
daemonset_name (str)
namespace (str) = default

Pod namespace, for a simulated pod alert.

status (str) = firing

Simulated alert status. firing/resolved.

severity (str) = error

Simulated alert severity.

description (str) = simulated prometheus alert

Simulated alert description.

summary (str)
runbook_url (str)

Simulated alert runbook_url. For example: "https://my-runbook-url.dev"

fingerprint (str)

Simulated alert fingerprint. For example: "my-unique-fingerprint"

labels (str)

Additional alert labels. For example: "key1: val1, key2: val2"

annotations (str)

Additional alert labels. For example: "key1=val1, key2=val2". Note that annotation separator is =

starts_at (datetime)
ends_at (datetime)
generator_url (str) = http://localhost:9090/graph?g0.expr=up%7Bjob%3D%22apiserver%22%7D&g0.tab=0&g0.stacked=0&g0.show_exemplars=0&g0.range_input=1h

Prometheus generator_url. Some enrichers, use this attribute to query Prometheus.

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger prometheus_alert  alert_name=ALERT_NAME

Adding Slack user/group mentions to k8s/Prometheus eventsΒΆ

Mention enricherΒΆ

Playbook Action: mention_enricher

You can define who to mention using a static mentions configuration, Or, you can define it using a label or annotation, that exists either on the Kubernetes resource, or the alert

Order: 1. Resource annotations (For alert, get from FindingSubject. For other resources, get from obj metadata) 2. Resource labels (For alert, get from FindingSubject. For other resources, get from obj metadata) 3. Alert annotations (only for alert) 4. Alert labels (only for alert)

Note this enricher only works with the Slack sink

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - mention_enricher:
      static_mentions:
      - U44V9P1JJ1Z
      - S22H3Q3Q111
  triggers:
  - on_pod_all_changes: {}

The above is an example. Try customizing the trigger and parameters.

optional:
static_mentions (str list)

List of Slack user ids/subteam ids to be mentioned

mentions_label (str)

An alert label, or Kubernetes resource label, in which the value contains a dot separated ids to mention

message_template (str) = Hey: $mentions

Optional. Custom mention message. Default: "Hey: $mentions"

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger mention_enricher kind=RESOURCE_KIND name=RESOURCE_NAME