Track Failed Liveness Probes¶

Lets track failed Liveness Probes and notify the user. Notifications will be sent to all configured Sinks like Slack, MSTeams, or DataDog. It is also possible to route notifications to specific sinks.

Defining a Playbook¶

Add the following YAML to the customPlaybooks Helm value:

customPlaybooks:
- triggers:
    - on_kubernetes_warning_event_create:
        include: ["Liveness"]   # fires on failed Liveness probes
  actions:
    - create_finding:
        aggregation_key: "Failed Liveness Probe"
        severity: HIGH
        title: "Failed liveness probe: $name"
    - event_resource_events: {}

Then do a Helm Upgrade.

Testing Your Playbook¶

Apply the following command the create a failing liveness probe.

kubectl apply -f https://raw.githubusercontent.com/robusta-dev/kubernetes-demos/main/liveness_probe_fail/failing_liveness_probe.yaml

Output

Failed liveness probe notification on Slack

How it Works¶

This playbook uses the on_kubernetes_warning_event_create trigger, that fires once for each Liveness probe failure.

It uses the create_finding action to generate a notification message, and event_resource_events action to gather all other events on the same resource in the near past.