Track Failed Liveness Probes¶

Lets track failed Liveness Probes and notify the user. Notifications will be sent to configured Sinks like Slack or MSTeams. It is also possible to route notifications to specific sinks.

Define a Playbook to Track Liveness Probes¶

Add the following YAML to the customPlaybooks Helm value:

customPlaybooks:
- triggers:
    - on_kubernetes_warning_event_create:
        include: ["Liveness"]   # fires on failed Liveness probes
  actions:
    - create_finding:
        aggregation_key: "Failed Liveness Probe"
        severity: HIGH
        title: "Failed liveness probe: $name"
    - event_resource_events: {}

Then do a Helm Upgrade.

Testing Your Playbook¶

Apply the following command the create a failing liveness probe.

kubectl apply -f https://raw.githubusercontent.com/robusta-dev/kubernetes-demos/main/liveness_probe_fail/failing_liveness_probe.yaml

You should get a notification in your configured sink.

Example Slack Notification

Failed liveness probe notification on Slack

How it Works¶

This playbook uses the on_kubernetes_warning_event_create trigger, which fires for Liveness probe failures in your cluster.

It uses the create_finding action to generate a notification message, and event_resource_events action to gather all other events on the same resource in the near past.