Centralized Prometheus

Follow this guide to connect Robusta to a central Prometheus (e.g. Thanos/Mimir), running outside the cluster monitored by Robusta.

You will need to configure two integrations: one to send alerts to Robusta and another to let Robusta query metrics and create silences.

Send Alerts to Robusta

This integration lets your central Prometheus send alerts to Robusta, as if they were in the same cluster:

  1. Enable cloud-routing of alerts by setting disableCloudRouting: false in generated_values.yaml.

  2. Verify that all alerts contain a label named cluster_name or cluster, matching the cluster_name defined in Robusta's configuration. This is necessary to identify which robusta-runner should receive alerts.

  3. Edit the configuration for your centralized AlertManager:

alertmanager.yaml

receivers:
  - name: 'robusta'
    webhook_configs:
      - url: 'https://api.robusta.dev/integrations/generic/alertmanager'
        http_config:
          authorization:
            # Replace <TOKEN> with a string in the format `<ACCOUNT_ID> <SIGNING_KEY>`
            credentials: <TOKEN>
        send_resolved: true # (3)

route: # (1)
  routes:
  - receiver: 'robusta'
    group_by: [ '...' ]
    group_wait: 1s
    group_interval: 1s
    matchers:
      - severity =~ ".*"
    repeat_interval: 4h
    continue: true # (2)

  1. Make sure the Robusta route is the first route defined. If it isn't the first route, it might not receive alerts. When a route is matched, the alert will not be sent to following routes, unless the route is configured with continue: true.

  2. Ensures that alerts continue to be sent even after a match is found

  3. Enables sending resolved alerts to Robusta

Verify it Works

Send a dummy alert to AlertManager:

robusta demo-alert

If everything is setup properly, this alert will reach Robusta. It will show up in the Robusta UI, Slack, and other configured sinks.

I configured AlertManager, but I'm not receiving alerts?

Try sending a demo-alert as described above. If nothing arrives, check:

  1. AlertManager UI status page - verify that your config was picked up

  2. kube-prometheus-operator logs (if relevant)

  3. AlertManager logs

Reach out on Slack for assistance.

Robusta isn't mapping alerts to Kubernetes resources

Robusta enriches alerts with Kubernetes and log data using Prometheus labels for mapping. Standard label names are used by default. If your setup differs, you can customize this mapping to fit your environment.

Configure Metric Querying

Metrics querying lets Robusta pull metrics and create silences.

If Robusta fails to auto-detect the Prometheus and Alertmanager urls - and you see related connection errors in the logs - configure the prometheus_url and alertmanager_url in your Helm values and update Robusta

globalConfig: # This line should already exist
    # Add the lines below
    alertmanager_url: "http://ALERT_MANAGER_SERVICE_NAME.NAMESPACE.svc.cluster.local:9093" # (1)
    prometheus_url: "http://PROMETHEUS_SERVICE_NAME.NAMESPACE.svc.cluster.local:9090" # (2)

    # If Prometheus has data for multiple clusters, tell Robusta how to query data for this cluster only
    # prometheus_additional_labels:
    #   cluster: 'CLUSTER_NAME_HERE'

    # If using Grafana alerts, add this too
    # grafana_api_key: <YOUR GRAFANA EDITOR API KEY> # (3)
    # alertmanager_flavor: grafana

    # If necessary, see docs below
    # prometheus_auth: ...
    # alertmanager_auth: ...

  1. Example: http://alertmanager-Helm_release_name-kube-prometheus-alertmanager.default.svc.cluster.local:9093.

  2. Example: http://Helm_Release_Name-kube-prometheus-prometheus.default.svc.cluster.local:9090

  3. This is necessary for Robusta to create silences when using Grafana Alerts, because of minor API differences in the AlertManager embedded in Grafana.

You can optionally setup authentication, SSL verification, and other parameters described below.

Verify it Works

Open any application in the Robusta UI. If CPU and memory graphs are shown, everything is working.

If you don't use the Robusta UI, trigger a demo OOMKill alert, and verify that Robusta sends a Slack/Teams message with a memory graph included. If so, everything is configured properly.

Optional Settings

Authentication Headers

If Prometheus and/or AlertManager require authentication, add the following to generated_values.yaml:

globalConfig:
  prometheus_auth: Bearer <YOUR TOKEN> # Replace <YOUR TOKEN> with your actual token or use any other auth header as needed
  alertmanager_auth: Basic <USER:PASSWORD base64-encoded> # Replace <USER:PASSWORD base64-encoded> with your actual credentials, base64-encoded, or use any other auth header as needed

These settings may be configured independently.

SSL Verification

By default, Robusta does not verify the SSL certificate of the Prometheus server.

To enable SSL verification, add the following to Robusta's generated_values.yaml:

runner:
  additional_env_vars:
  - name: PROMETHEUS_SSL_ENABLED
    value: "true"

If you have a custom Certificate Authority (CA) certificate, add one more setting:

runner:
  certificate: "<YOUR BASE-64 ENCODED DATA>" # base64-encoded certificate value

Filtering Prometheus Queries by Cluster

If the same centralized Prometheus is used for many clusters, you will want to add a cluster name to all queries.

You can do so with the prometheus_url_query_string parameter, shown below:

globalConfig:
  # Additional query string parameters to be appended to the Prometheus connection URL (optional)
  prometheus_url_query_string: "cluster=prod1&x=y"