Kubernetes Misconfigurations (Popeye)

Popeye is a utility that scans live Kubernetes clusters and reports potential issues with resources and configurations.

By optionally integrating Popeye with Robusta you can:

  1. Get weekly Popeye scan reports in Slack via Robusta OSS (disabled by default, see below to configure)

  2. View Popeye scans from all your clusters in the Robusta UI (enabled by default for UI users)

Sending Weekly Popeye Scan Reports to Slack

With or without the UI, you can configure additional scans on a schedule as shown below. The results can be sent as a PDF to Slack or to the Robusta UI.

customPlaybooks:
- triggers:
  - on_schedule:
      fixed_delay_repeat:
        repeat: 1 # number of times to run or -1 to run forever
        seconds_delay: 604800 # 1 week
  actions:
  - popeye_scan:
      spinach: |
        popeye:
          excludes:
              v1/pods:
              - name: rx:kube-system
  sinks:
  - "robusta_ui_sink"

Note

Other sinks like MSTeams are not supported yet.

Taints, Tolerations and NodeSelectors

To run Popeye on a GPU enabled cluster or on specific nodes you can set custom tolerations or a nodeSelector in your generated_values.yaml file as follows:

globalConfig:
  popeye_job_spec:
    tolerations:
    - key: "key1"
      operator: "Exists"
      effect: "NoSchedule"
    nodeSelector:
      kubernetes.io/arch: "amd64"
      nodeName: "your-selector"

Note

Popeye does not support arm nodes yet. If your cluster has both Arm and x64 nodes add kubernetes.io/arch: "amd64" as a node selector to schedule Popeye jobs on the x64 nodes.

Troubleshooting Popeye

Popeye scans run as Jobs in your cluster. If there are issues with a scan, troubleshoot as follows:

Events

  • To find errors with the Popeye job run:

kubectl get events --all-namespaces --field-selector=type!=Normal | grep popeye-job

Logs

  • Additional errors can sometimes be found in the Robusta runner logs:

robusta logs

Known issues

couldn't get resource list for external.metrics.k8s.io/v1beta1

This is a known issue, there is a working workaround, which involves deploying a dummy workload. Read more about it here.

exec /bin/sh: exec format error

At the moment, Popeye docker images are only compiled for linux/amd64 os/arch. This error suggests you are running the Popeye image on a different os/arch node.

Reference

Popeye scan

Playbook Action: popeye_scan

Displays a popeye scan report.

You can trigger a Popeye scan at any time, by running the following command:

robusta playbooks trigger popeye_scan

Add this to your Robusta configuration (Helm values.yaml):

customPlaybooks:
- actions:
  - popeye_scan: {}
  triggers:
  - on_schedule: {}

The above is an example. Try customizing the trigger and parameters.

optional:
custom_annotations (str dict)

custom annotations to be used for the running pod/job

service_account_name (str) = robusta-runner-service-account

The account name to use for the Popeye scan job.

args (str)

Deprecated - Popeye cli arguments.

popeye_args (str) = -s no,ns,po,svc,sa,cm,dp,sts,ds,pv,pvc,hpa,pdb,cr,crb,ro,rb,ing,np,psp

Popeye cli arguments.

spinach (str) = popeye:     excludes:         apps/v1/daemonsets:         - name: rx:kube-system         apps/v1/deployments:         - name: rx:kube-system         v1/configmaps:         - name: rx:kube-system         v1/pods:         - name: rx:.*           codes:           - 106           - 107         - name: rx:kube-system         v1/services:         - name: rx:kube-system         v1/namespaces:         - name: kube-system

Spinach.yaml config file to supply to the scan.

timeout (int) = 300

Time span for yielding the scan.

popeye_job_spec (dict)

A dictionary for passing spec params such as tolerations and nodeSelector.

This action can be manually triggered using the Robusta CLI:

robusta playbooks trigger popeye_scan