Follow this guide to connect Robusta to a central Prometheus, running outside the cluster monitored by Robusta.
You will need to configure two integrations: a push integration and a pull integration. (Both are necessary.)
Configure Push Integration¶
A push integration lets your central Prometheus send alerts to Robusta, as if they were in the same cluster:
Enable cloud-routing of alerts by setting
Verify that all alerts contain a label named
cluster_name, matching the cluster_name defined in Robusta's configuration. This is necessary to identify which robusta-runner should receive alerts.
Edit the configuration for your centralized AlertManager:
receivers: - name: 'robusta' webhook_configs: - url: 'https://api.robusta.dev/integrations/generic/alertmanager' http_config: authorization: # Replace <TOKEN> with a string in the format `<ACCOUNT_ID> <SIGNING_KEY>` credentials: <TOKEN> send_resolved: true # (3) route: # (1) routes: - receiver: 'robusta' group_by: [ '...' ] group_wait: 1s group_interval: 1s matchers: - severity =~ ".*" repeat_interval: 4h continue: true # (2)
Make sure the Robusta
routeis the first
routedefined. If it isn't the first route, it might not receive alerts. When a
routeis matched, the alert will not be sent to following routes, unless the
routeis configured with
Ensures that alerts continue to be sent even after a match is found
Enables sending resolved alerts to Robusta
Verify it Works¶
Send a dummy alert to AlertManager:
If everything is setup properly, this alert will reach Robusta. It will show up in the Robusta UI, Slack, and other configured sinks.
I configured AlertManager, but I'm not receiving alerts?
Try sending a demo-alert as described above. If nothing arrives, check:
kube-prometheus-operator logs (if relevant)
AlertManager UI status page - verify that your config was picked up
Reach out on Slack for assistance.
Configure Pull Integration¶
A pull integration lets Robusta pull metrics and create silences.
To configure it, add the following to
generated_values.yaml and update Robusta.
globalConfig: # this line should already exist # add the lines below alertmanager_url: "http://ALERT_MANAGER_SERVICE_NAME.NAMESPACE.svc.cluster.local:9093" # (1) grafana_url: "" prometheus_url: "http://PROMETHEUS_SERVICE_NAME.NAMESPACE.svc.cluster.local:9090" # (2) # Add any labels that are relevant to the specific cluster (optional) # prometheus_additional_labels: # cluster: 'CLUSTER_NAME_HERE' # Create alert silencing when using Grafana alerts (optional) # grafana_api_key: <YOUR GRAFANA EDITOR API KEY> # (3) # alertmanager_flavor: grafana
This is necessary for Robusta to create silences when using Grafana Alerts, because of minor API differences in the AlertManager embedded in Grafana.
You can optionally setup authentication, SSL verification, and other parameters described below.
Verify it Works¶
Open any application in the Robusta UI. If CPU and memory graphs are shown, everything is working.
If you don't use the Robusta UI, trigger a demo OOMKill alert, and verify that Robusta sends a Slack/Teams message with a memory graph included. If so, everything is configured properly.
If Prometheus and/or AlertManager require authentication, add the following to
globalConfig: prometheus_auth: Bearer <YOUR TOKEN> # Replace <YOUR TOKEN> with your actual token or use any other auth header as needed alertmanager_auth: Basic <USER:PASSWORD base64-encoded> # Replace <USER:PASSWORD base64-encoded> with your actual credentials, base64-encoded, or use any other auth header as needed
The two settings may be configured independently.
By default, Robusta does not verify the SSL certificate of the Prometheus server.
To enable SSL verification, add the following to Robusta's
runner: additional_env_vars: - name: PROMETHEUS_SSL_ENABLED value: "true"
If you have a custom Certificate Authority (CA) certificate, add one more setting:
runner: certificate: "<YOUR BASE-64 ENCODED DATA>" # base64-encoded certificate value
Filtering Prometheus Queries by Cluster¶
If the same centralized Prometheus is used for many clusters, you will want to add a cluster name to all queries.
You can do so with the
prometheus_url_query_string parameter, shown below:
globalConfig: # Additional query string parameters to be appended to the Prometheus connection URL (optional) prometheus_url_query_string: "cluster=prod1&x=y"