Victoria Metrics

This guide walks you through configuring Victoria Metrics with Robusta.

You will need to configure two integrations: both a push integration and a pull integration.

Configure Push Integration

A push integration sends alerts to Robusta. To configure it, edit AlertManager's configuration:

AlertManager config for sending alerts to Robusta

receivers:
  - name: 'robusta'
    webhook_configs:
      - url: 'http://robusta-runner.default.svc.cluster.local/api/alerts' # (2)
        sendResolved: true # (4)

route: # (1)
  routes:
    - receiver: 'robusta'
      group_by: [ '...' ]
      group_wait: 1s
      group_interval: 1s
      matchers:
        - severity =~ ".*"
      repeat_interval: 4h
      continue: true # (3)

  1. Put Robusta's route as the first route, to guarantee it receives alerts. If you can't do so, you must guarantee all previous routes set continue: true set.

  2. This assumes Robusta was installed in the default namespace, using a Helm release named robusta.

    • If the namespace is foobar, replace default with foobar

    • If the Helm release is named robert then replace robusta with robert

  3. Keep sending alerts to receivers defined after Robusta.

  4. Important, so Robusta knows when alerts are resolved.

Verify it Works

Send a dummy alert to AlertManager:

robusta demo-alert

If everything is setup properly, this alert will reach Robusta. It will show up in the Robusta UI, Slack, and other configured sinks.

I configured AlertManager, but I'm not receiving alerts?

Try sending a demo-alert as described above. If nothing arrives, check:

  1. AlertManager logs

  2. kube-prometheus-operator logs (if relevant)

  3. AlertManager UI status page - verify that your config was picked up

Reach out on Slack for assistance.

Configure Pull Integration

A pull integration lets Robusta pull metrics and create silences.

Add the following to generated_values.yaml and update Robusta.

globalConfig: # this line should already exist
    # add the lines below
    alertmanager_url: "http://ALERT_MANAGER_SERVICE_NAME.NAMESPACE.svc.cluster.local:9093"
    grafana_url: ""
    prometheus_url: "http://VICTORIA_METRICS_SERVICE_NAME.NAMESPACE.svc.cluster.local:8429"
    # Add any labels that are relevant to the specific cluster (optional)
    # prometheus_additional_labels:
    #   cluster: 'CLUSTER_NAME_HERE'

    # Additional query string parameters to be appended to the Prometheus connection URL (optional)
    # prometheus_url_query_string: "demo-query=example-data&another-query=value"

    # Create alert silencing when using Grafana alerts (optional)
    # grafana_api_key: <YOUR GRAFANA EDITOR API KEY> # (1)
    # alertmanager_flavor: grafana

  1. This is necessary for Robusta to create silences when using Grafana Alerts, because of minor API differences in the AlertManager embedded in Grafana.