Remediation¶

Create a kubernetes job with the specified parameters

In addition, the job pod receives the following alert parameters as environment variables

ALERT_NAME

ALERT_STATUS

ALERT_OBJ_KIND - oneof pod/deployment/node/job/daemonset or None in case it's unknown

ALERT_OBJ_NAME

ALERT_OBJ_NAMESPACE (If present)

ALERT_OBJ_NODE (If present)

ALERT_LABEL_{LABEL_NAME} for every label on the alert. For example a label named foo becomes ALERT_LABEL_FOO

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - alert_handling_job:
      command:
      - perl
      - -Mbignum=bpi
      - -wle
      - print bpi(2000)
      image: string
  triggers:
  - on_prometheus_alert: {}

The above is an example. Try customizing the trigger and parameters.

image (str)

The job image.

command (str list)

The job command as array of strings

name (str) = robusta-action-job

Custom name for the job and job container.

namespace (str) = default

The created job namespace.

service_account (str)

Job pod service account. If omitted, default is used.

restart_policy (str) = OnFailure

Job container restart policy

job_ttl_after_finished (int) = 120

Delete finished job ttl (seconds). If omitted, jobs will not be deleted automatically.

notify (bool)

Add a notification for creating the job.

wait_for_completion (bool) = True

Wait for the job to complete and attach it's output. Only relevant when notify=true.

completion_timeout (int) = 300

Maximum seconds to wait for job to complete. Only relevant when wait_for_completion=true.

backoff_limit (int)

Specifies the number of retries before marking this job failed. Defaults to 6

active_deadline_seconds (int)

Specifies the duration in seconds relative to the startTime

that the job may be active before the system tries to terminate it; value must be

positive integer

env (envvar list)

Inject environment variables and secrets just like you do with a Kubernetes Job.

• on_prometheus_alert

Deletes a pod

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - delete_pod: {}
  triggers:
  - on_pod_oom_killed: {}

The above is an example. Try customizing the trigger and parameters.

No action parameters

• on_container_oom_killed

• on_image_pull_backoff

• on_pod_all_changes

• on_pod_crash_loop

• on_pod_create

• on_pod_delete

• on_pod_oom_killed

• on_pod_update

• on_prometheus_alert

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger delete_pod name=POD_NAME namespace=POD_NAMESPACE 

Delete the job from the cluster

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - delete_job: {}
  triggers:
  - on_job_failure: {}

The above is an example. Try customizing the trigger and parameters.

No action parameters

• on_job_all_changes

• on_job_create

• on_job_delete

• on_job_failure

• on_job_update

• on_prometheus_alert

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger delete_job name=JOB_NAME namespace=JOB_NAMESPACE 

Notify when the HPA reaches its maximum replicas and allow fixing it.

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - alert_on_hpa_reached_limit: {}
  triggers:
  - on_horizontalpodautoscaler_update: {}

The above is an example. Try customizing the trigger and parameters.

increase_pct (int) = 20

Increase the HPA max_replicas by this percentage.

• on_horizontalpodautoscaler_all_changes

• on_horizontalpodautoscaler_create

• on_horizontalpodautoscaler_delete

• on_horizontalpodautoscaler_update

Performs rollout restart on a kubernetes workload. Supports deployments, deploymentconfig, daemonsets and statefulsets related events.

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - rollout_restart: {}
  triggers:
  - on_prometheus_alert: {}

The above is an example. Try customizing the trigger and parameters.

No action parameters

• on_clusterrole_all_changes

• on_clusterrole_create

• on_clusterrole_delete

• on_clusterrole_update

• on_clusterrolebinding_all_changes

• on_clusterrolebinding_create

• on_clusterrolebinding_delete

• on_clusterrolebinding_update

• on_configmap_all_changes

• on_configmap_create

• on_configmap_delete

• on_configmap_update

• on_container_oom_killed

• on_daemonset_all_changes

• on_daemonset_create

• on_daemonset_delete

• on_daemonset_update

• on_deployment_all_changes

• on_deployment_create

• on_deployment_delete

• on_deployment_update

• on_event_all_changes

• on_event_create

• on_event_delete

• on_event_update

• on_horizontalpodautoscaler_all_changes

• on_horizontalpodautoscaler_create

• on_horizontalpodautoscaler_delete

• on_horizontalpodautoscaler_update

• on_image_pull_backoff

• on_ingress_all_changes

• on_ingress_create

• on_ingress_delete

• on_ingress_update

• on_job_all_changes

• on_job_create

• on_job_delete

• on_job_failure

• on_job_update

• on_kubernetes_warning_event

• on_kubernetes_warning_event_create

• on_kubernetes_warning_event_delete

• on_kubernetes_warning_event_update

• on_namespace_all_changes

• on_namespace_create

• on_namespace_delete

• on_namespace_update

• on_networkpolicy_all_changes

• on_networkpolicy_create

• on_networkpolicy_delete

• on_networkpolicy_update

• on_node_all_changes

• on_node_create

• on_node_delete

• on_node_update

• on_persistentvolume_all_changes

• on_persistentvolume_create

• on_persistentvolume_delete

• on_persistentvolume_update

• on_persistentvolumeclaim_all_changes

• on_persistentvolumeclaim_create

• on_persistentvolumeclaim_delete

• on_persistentvolumeclaim_update

• on_pod_all_changes

• on_pod_crash_loop

• on_pod_create

• on_pod_delete

• on_pod_oom_killed

• on_pod_update

• on_prometheus_alert

• on_replicaset_all_changes

• on_replicaset_create

• on_replicaset_delete

• on_replicaset_update

• on_service_all_changes

• on_service_create

• on_service_delete

• on_service_update

• on_serviceaccount_all_changes

• on_serviceaccount_create

• on_serviceaccount_delete

• on_serviceaccount_update

• on_statefulset_all_changes

• on_statefulset_create

• on_statefulset_delete

• on_statefulset_update

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger rollout_restart kind=RESOURCE_KIND name=RESOURCE_NAME 

Cordon, Taints a node as unschedulable.

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - cordon: {}
  triggers:
  - on_node_create: {}

The above is an example. Try customizing the trigger and parameters.

No action parameters

• on_node_all_changes

• on_node_create

• on_node_delete

• on_node_update

• on_prometheus_alert

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger cordon name=NODE_NAME 

Unordon, Taints a node as schedulable.

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - uncordon: {}
  triggers:
  - on_node_create: {}

The above is an example. Try customizing the trigger and parameters.

No action parameters

• on_node_all_changes

• on_node_create

• on_node_delete

• on_node_update

• on_prometheus_alert

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger uncordon name=NODE_NAME 

Drain, taints a node as unschedulable, and evicts all pods from the node. DaemonSets pods are skipped, as they tolerant unschedulable nodes by default.

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - drain: {}
  triggers:
  - on_node_create: {}

The above is an example. Try customizing the trigger and parameters.

No action parameters

• on_node_all_changes

• on_node_create

• on_node_delete

• on_node_update

• on_prometheus_alert

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger drain name=NODE_NAME