In-cluster Prometheus¶
Here's how to integrate an existing Prometheus with Robusta in the same cluster:
Send alerts to Robusta by adding a receiver to AlertManager
- Point Robusta at Prometheus so it can query metrics and silence alerts
Robusta will attempt auto-detection, so this is not always necessary!
If your Prometheus is in a different cluster, refer to Centralized Prometheus.
Send Alerts to Robusta¶
AlertManager config for sending alerts to Robusta
Add the following to your AlertManager's config Secret
Do not apply in other ways, they all have limitations and won't forward all alerts.
receivers:
- name: 'robusta'
webhook_configs:
- url: 'http://<helm-release-name>-runner.<namespace>.svc.cluster.local/api/alerts'
send_resolved: true # (3)
- name: 'default-receiver'
route: # (1)
routes:
- receiver: 'robusta'
group_by: [ '...' ]
group_wait: 1s
group_interval: 1s
matchers:
- severity =~ ".*"
repeat_interval: 4h
continue: true # (2)
receiver: 'default-receiver'
Put Robusta's route as the first route, to guarantee it receives alerts. If you can't do so, you must guarantee all previous routes set
continue: trueset.Keep sending alerts to receivers defined after Robusta.
Important, so Robusta knows when alerts are resolved.
Add the following to your AlertManager configuration, wherever it is defined.
receivers:
- name: 'robusta'
webhook_configs:
- url: 'http://<helm-release-name>-runner.<namespace>.svc.cluster.local/api/alerts'
send_resolved: true # (3)
route: # (1)
routes:
- receiver: 'robusta'
group_by: [ '...' ]
group_wait: 1s
group_interval: 1s
matchers:
- severity =~ ".*"
repeat_interval: 4h
continue: true # (2)
Put Robusta's route as the first route, to guarantee it receives alerts. If you can't do so, you must guarantee all previous routes set
continue: trueset.Keep sending alerts to receivers defined after Robusta.
Important, so Robusta knows when alerts are resolved.
Verify it Works¶
Send a dummy alert to AlertManager:
If you have the Robusta CLI installed, you can send a test alert using the following command:
robusta demo-alert
In the Robusta UI, go to the "Clusters" tab, choose the right cluster and click "Simulate Alert".
Then
Check Send alert with no resource.
Provide a name for the alert in the Alert name (identifier) field (e.g., "Testing Prod AlertManager").
Select Alert Manager under the "Send alert to" section.
Click the Simulate Alert button to send the test alert.
If everything is setup properly, this alert will reach Robusta. It will show up in the Robusta UI, Slack, and other configured sinks.
Note
It might take a few minutes for the alert to arrive due to AlertManager's group_wait and group_interval settings. More info here.
I configured AlertManager, but I'm not receiving alerts?
Try sending a demo-alert as described above. If nothing arrives, check:
AlertManager UI status page - verify that your config was picked up
kube-prometheus-operator logs (if relevant)
AlertManager logs
Reach out on Slack for assistance.
Robusta isn't mapping alerts to Kubernetes resources
Robusta enriches alerts with Kubernetes and log data using Prometheus labels for mapping. Standard label names are used by default. If your setup differs, you can customize this mapping to fit your environment.
Configure Metric Querying¶
Metrics querying lets Robusta pull metrics and create silences.
If Robusta fails to auto-detect the Prometheus and Alertmanager urls - and you see related connection errors in the logs - configure the prometheus_url and alertmanager_url in your Helm values and update Robusta
globalConfig: # This line should already exist
# Add the lines below
alertmanager_url: "http://ALERT_MANAGER_SERVICE_NAME.NAMESPACE.svc.cluster.local:9093" # (1)
prometheus_url: "http://PROMETHEUS_SERVICE_NAME.NAMESPACE.svc.cluster.local:9090" # (2)
# If Prometheus has data for multiple clusters, tell Robusta how to query data for this cluster only
# prometheus_additional_labels:
# cluster: 'CLUSTER_NAME_HERE'
# If using Grafana alerts, add this too
# grafana_api_key: <YOUR GRAFANA EDITOR API KEY> # (3)
# alertmanager_flavor: grafana
# If necessary, see docs below
# prometheus_auth: ...
# alertmanager_auth: ...
# If using a multi-tenant prometheus or alertmanager, pass the org id to all queries
# prometheus_additional_headers:
# X-Scope-OrgID: <org id>
# alertmanager_additional_headers:
# X-Scope-OrgID: <org id>
Example: http://alertmanager-Helm_release_name-kube-prometheus-alertmanager.default.svc.cluster.local:9093.
Example: http://Helm_Release_Name-kube-prometheus-prometheus.default.svc.cluster.local:9090
This is necessary for Robusta to create silences when using Grafana Alerts, because of minor API differences in the AlertManager embedded in Grafana.
You can optionally setup authentication, SSL verification, and other parameters described below.
Verify it Works¶
Open any application in the Robusta UI. If CPU and memory graphs are shown, everything is working.
If you don't use the Robusta UI, trigger a demo OOMKill alert, and verify that Robusta sends a Slack/Teams message with a memory graph included. If so, everything is configured properly.
Optional Settings¶
Authentication Headers¶
If Prometheus and/or AlertManager require authentication, add the following to generated_values.yaml:
globalConfig:
prometheus_auth: Bearer <YOUR TOKEN> # Replace <YOUR TOKEN> with your actual token or use any other auth header as needed
alertmanager_auth: Basic <USER:PASSWORD base64-encoded> # Replace <USER:PASSWORD base64-encoded> with your actual credentials, base64-encoded, or use any other auth header as needed
These settings may be configured independently.
SSL Verification¶
By default, Robusta does not verify the SSL certificate of the Prometheus server.
To enable SSL verification, add the following to Robusta's generated_values.yaml:
runner:
additional_env_vars:
- name: PROMETHEUS_SSL_ENABLED
value: "true"
If you have a custom Certificate Authority (CA) certificate, add one more setting:
runner:
certificate: "<YOUR BASE-64 ENCODED DATA>" # base64-encoded certificate value