In-cluster PrometheusΒΆ
Are you already running Prometheus in the same cluster as Robusta? Follow this guide to connect your Prometheus to Robusta. If your Prometheus is in a different cluster refer to the Centralized Prometheus documentation to integrate it with Robusta.
You will need to configure two integrations: a push integration and a pull integration. (Both are necessary.)
Configure Push IntegrationΒΆ
A push integration sends alerts to Robusta. Add the config below to the appropriate AlertManager file.
Note
If you're using kube-prometheus-stack, add the below config to AlertManager's config Secret. Using other configuration files like Helm values file will only forward alerts from one namespace.
AlertManager config for sending alerts to Robusta
receivers:
- name: 'robusta'
webhook_configs:
- url: 'http://robusta-runner.default.svc.cluster.local/api/alerts' # (2)
sendResolved: true # (4)
route: # (1)
routes:
- receiver: 'robusta'
group_by: [ '...' ]
group_wait: 1s
group_interval: 1s
matchers:
- severity =~ ".*"
repeat_interval: 4h
continue: true # (3)
Put Robusta's route as the first route, to guarantee it receives alerts. If you can't do so, you must guarantee all previous routes set
continue: true
set.- This assumes Robusta was installed in the
default
namespace, using a Helm release namedrobusta
. If the namespace is
foobar
, replacedefault
withfoobar
If the Helm release is named
robert
then replacerobusta
withrobert
- This assumes Robusta was installed in the
Keep sending alerts to receivers defined after Robusta.
Important, so Robusta knows when alerts are resolved.
Verify it WorksΒΆ
Send a dummy alert to AlertManager:
robusta demo-alert
If everything is setup properly, this alert will reach Robusta. It will show up in the Robusta UI, Slack, and other configured sinks.
I configured AlertManager, but I'm not receiving alerts?
Try sending a demo-alert as described above. If nothing arrives, check:
AlertManager logs
kube-prometheus-operator logs (if relevant)
AlertManager UI status page - verify that your config was picked up
Reach out on Slack for assistance.
Configure Pull IntegrationΒΆ
A pull integration lets Robusta pull metrics and create silences.
To configure it, add the following to generated_values.yaml
and update Robusta.
globalConfig: # this line should already exist
# add the lines below
alertmanager_url: "http://ALERT_MANAGER_SERVICE_NAME.NAMESPACE.svc.cluster.local:9093" # (1)
grafana_url: ""
prometheus_url: "http://PROMETHEUS_SERVICE_NAME.NAMESPACE.svc.cluster.local:9090" # (2)
# Add any labels that are relevant to the specific cluster (optional)
# prometheus_additional_labels:
# cluster: 'CLUSTER_NAME_HERE'
# Create alert silencing when using Grafana alerts (optional)
# grafana_api_key: <YOUR GRAFANA EDITOR API KEY> # (3)
# alertmanager_flavor: grafana
Example: http://alertmanager-Helm_release_name-kube-prometheus-alertmanager.default.svc.cluster.local:9093.
Example: http://Helm_Release_Name-kube-prometheus-prometheus.default.svc.cluster.local:9090
This is necessary for Robusta to create silences when using Grafana Alerts, because of minor API differences in the AlertManager embedded in Grafana.
You can optionally setup authentication, SSL verification, and other parameters described below.
Verify it WorksΒΆ
Open any application in the Robusta UI. If CPU and memory graphs are shown, everything is working.
If you don't use the Robusta UI, trigger a demo OOMKill alert, and verify that Robusta sends a Slack/Teams message with a memory graph included. If so, everything is configured properly.
Optional SettingsΒΆ
Authentication HeadersΒΆ
If Prometheus and/or AlertManager require authentication, add the following to generated_values.yaml
:
globalConfig:
prometheus_auth: Bearer <YOUR TOKEN> # Replace <YOUR TOKEN> with your actual token or use any other auth header as needed
alertmanager_auth: Basic <USER:PASSWORD base64-encoded> # Replace <USER:PASSWORD base64-encoded> with your actual credentials, base64-encoded, or use any other auth header as needed
The two settings may be configured independently.
SSL VerificationΒΆ
By default, Robusta does not verify the SSL certificate of the Prometheus server.
To enable SSL verification, add the following to Robusta's generated_values.yaml
:
runner:
additional_env_vars:
- name: PROMETHEUS_SSL_ENABLED
value: "true"
If you have a custom Certificate Authority (CA) certificate, add one more setting:
runner:
certificate: "<YOUR BASE-64 ENCODED DATA>" # base64-encoded certificate value