Track Kubernetes ChangesΒΆ

Let's track changes to Kubernetes objects using Robusta. Notifications will be sent to a Sinks, like Slack or MSTeams.

In this tutorial you will:

  • Specify which Kubernetes object to track

  • Filter out noisy changes and only track certain YAML fields

  • Send a diff of exactly what changed

Why Track Kubernetes Changes?

Change tracking is useful in organizations where multiple teams deploy to the same cluster. Some use cases:

  • DevOps and Platform Teams: Track all changes to Ingresses and other sensitive cluster resources.

  • Developers: Get notified each time your application is deployed to production.

  • Security and DevSecOps: Assorted use cases.

Defining a PlaybookΒΆ

Add the following YAML to the customPlaybooks Helm value:

customPlaybooks:
- triggers:
    - on_deployment_update: {}
  actions:
    - resource_babysitter:
        omitted_fields: []
        fields_to_monitor: ["spec.replicas"]

Then perform a Helm Upgrade.

Testing Your PlaybookΒΆ

Scale a deployment that exists in your cluster:

Run the following YAML files to simulate a deployment change

kubectl apply -f https://raw.githubusercontent.com/robusta-dev/kubernetes-demos/main/crashpod/healthy.yaml
kubectl apply -f https://raw.githubusercontent.com/robusta-dev/kubernetes-demos/main/crashpod/broken.yaml

A Robusta notification will arrive in your configured sinks, showing exactly what changed in the deployment:

How it WorksΒΆ

We configured a custom playbook with the trigger on_deployment_update. This trigger fires whenever Kubernetes Deployments are updated.

The trigger fires on all Deployment changes, even uninteresting changes to the Deployment's status performed by Kubernetes itself on static clusters.

The action is resource_babysitter action, which itself performs further filtering and ignores uninteresting changes. This action is a little unusual - most of the time triggers perform all the filtering and actions act on everything that reaches them.

In the future we're planning to improve the trigger mechanism. Filters like fields_to_monitor will move from the resource_babysitter into triggers like on_deployment_update <on_deployment_update>.

Adding Change RoutingΒΆ

To send change notifications to a specific sink instead of all sinks, you can choose between two methods:

  1. Use Sink Matchers

  2. Explicitly specify a sink in the playbook

Here is the latter method:

customPlaybooks:
- triggers:
  - on_deployment_update: {}
  actions:
  - resource_babysitter:
      omitted_fields: []
      fields_to_monitor: ["spec.replicas"]
  sinks:
  - some_sink_name

CleanupΒΆ

Remove this playbook from customPlaybooks and perform a Helm Upgrade.