Victoria Metrics

This guide walks you through configuring Victoria Metrics with Robusta.

You will need to configure two integrations: one to send alerts to Robusta and another to let Robusta query metrics and create silences.

Send Alerts to Robusta

To configure, edit AlertManager's configuration:

AlertManager config for sending alerts to Robusta

Add the following to your AlertManager's config Secret

Do not apply in other ways, they all have limitations and won't forward all alerts.

receivers:
  - name: 'robusta'
    webhook_configs:
      - url: 'http://<helm-release-name>-runner.<namespace>.svc.cluster.local/api/alerts'
        send_resolved: true # (3)
  - name: 'default-receiver'

route: # (1)
  routes:
    - receiver: 'robusta'
      group_by: [ '...' ]
      group_wait: 1s
      group_interval: 1s
      matchers:
        - severity =~ ".*"
      repeat_interval: 4h
      continue: true # (2)
  receiver: 'default-receiver'

  1. Put Robusta's route as the first route, to guarantee it receives alerts. If you can't do so, you must guarantee all previous routes set continue: true set.

  2. Keep sending alerts to receivers defined after Robusta.

  3. Important, so Robusta knows when alerts are resolved.

Add the following to your AlertManager configuration, wherever it is defined.

receivers:
  - name: 'robusta'
    webhook_configs:
      - url: 'http://<helm-release-name>-runner.<namespace>.svc.cluster.local/api/alerts'
        send_resolved: true # (3)

route: # (1)
  routes:
    - receiver: 'robusta'
      group_by: [ '...' ]
      group_wait: 1s
      group_interval: 1s
      matchers:
        - severity =~ ".*"
      repeat_interval: 4h
      continue: true # (2)

  1. Put Robusta's route as the first route, to guarantee it receives alerts. If you can't do so, you must guarantee all previous routes set continue: true set.

  2. Keep sending alerts to receivers defined after Robusta.

  3. Important, so Robusta knows when alerts are resolved.

Verify it Works

Send a dummy alert to AlertManager:

robusta demo-alert

If everything is setup properly, this alert will reach Robusta. It will show up in the Robusta UI, Slack, and other configured sinks.

I configured AlertManager, but I'm not receiving alerts?

Try sending a demo-alert as described above. If nothing arrives, check:

  1. AlertManager UI status page - verify that your config was picked up

  2. kube-prometheus-operator logs (if relevant)

  3. AlertManager logs

Reach out on Slack for assistance.

Robusta isn't mapping alerts to Kubernetes resources

Robusta enriches alerts with Kubernetes and log data using Prometheus labels for mapping. Standard label names are used by default. If your setup differs, you can customize this mapping to fit your environment.

Configure Metric Querying

Metrics querying lets Robusta pull metrics and create silences.

Add the following to generated_values.yaml and update Robusta.

globalConfig: # this line should already exist
    # add the lines below
    alertmanager_url: "http://ALERT_MANAGER_SERVICE_NAME.NAMESPACE.svc.cluster.local:9093"
    grafana_url: ""
    prometheus_url: "http://VICTORIA_METRICS_SERVICE_NAME.NAMESPACE.svc.cluster.local:8429"
    # Add any labels that are relevant to the specific cluster (optional)
    # prometheus_additional_labels:
    #   cluster: 'CLUSTER_NAME_HERE'

    # Additional query string parameters to be appended to the Prometheus connection URL (optional)
    # prometheus_url_query_string: "demo-query=example-data&another-query=value"

    # Create alert silencing when using Grafana alerts (optional)
    # grafana_api_key: <YOUR GRAFANA EDITOR API KEY> # (1)
    # alertmanager_flavor: grafana

  1. This is necessary for Robusta to create silences when using Grafana Alerts, because of minor API differences in the AlertManager embedded in Grafana.

Optional Settings

Prometheus flags checks

Robusta utilizes the flags API to retrieve data from Prometheus-style metric stores. However, some platforms like Google Managed Prometheus, Azure Managed Prometheus etc, do not implement the flags API.

You can disable the Prometheus flags API check by setting the following option to false.

globalConfig:
  check_prometheus_flags: true/false