Tempo (Grafana)¶

By enabling this toolset, HolmesGPT will be able to fetch trace information from Grafana Tempo to debug performance related issues, like high latency in your application.

Prerequisites¶

A Grafana service account token with the following permissions:

• Basic role -> Viewer

• Data sources -> Reader

Check out this video on creating a Grafana service account token.

Getting Grafana URL¶

You can find the Grafana URL required for Tempo in your Grafana cloud account settings.

Obtaining the datasource UID¶

You may have multiple Tempo data sources setup in Grafana. HolmesGPT uses a single Tempo datasource to fetch the traces and it needs to know the UID of this datasource.

A simple way to get the datasource UID is to access the Grafana API by running the following request:

# port forward if you are using Robusta's grafana from your kubernetes cluster
kubectl port-forward svc/robusta-grafana 3000:80
# List the Tempo data sources
curl -s -u <username>:<password> http://localhost:3000/api/datasources | jq '.[] | select(.type == "tempo")'
{
    "id": 3,
    "uid": "klja8hsa-8a9c-4b35-1230-7baab22b02ee",
    "orgId": 1,
    "name": "Tempo",
    "type": "tempo",
    "typeName": "Tempo",
    "typeLogoUrl": "/public/app/plugins/datasource/tempo/img/tempo_icon.svg",
    "access": "proxy",
    "url": "http://tempo-query-frontend.tempo:3100",
    "user": "",
    "database": "",
    "basicAuth": false,
    "isDefault": false,
    "jsonData": {
        "tlsSkipVerify": true
    },
    "readOnly": false
}
# In this case, there is a single Tempo datasourceUID
# with UID "klja8hsa-8a9c-4b35-1230-7baab22b02ee"

Configuration¶

holmes:
  toolsets:
    grafana/tempo:
      enabled: true
      config:
        api_key: <your grafana API key>
        url: <your grafana url> # e.g. https://acme-corp.grafana.net
        grafana_datasource_uid: <the UID of the tempo data source in Grafana>
        labels:
          pod: "k8s.pod.name"
          namespace: "k8s.namespace.name"
          deployment: "k8s.deployment.name"
          node: "k8s.node.name"
          service: "service.name"

helm upgrade robusta robusta/robusta --values=generated_values.yaml --set clusterName=<YOUR_CLUSTER_NAME>

toolsets:
  grafana/tempo:
    enabled: true
    config:
        api_key: <your grafana service account token>
        url: <your grafana url> # e.g. https://acme-corp.grafana.net
        grafana_datasource_uid: <the UID of the tempo data source in Grafana>
        labels:
          pod: "k8s.pod.name"
          namespace: "k8s.namespace.name"
          deployment: "k8s.deployment.name"
          node: "k8s.node.name"
          service: "service.name"

holmes ask "The payments DB is very slow, check tempo for any trace data"

Search labels

You can tweak the labels used by the toolset to identify kubernetes resources. This is only needed if the trace labels differ from the defaults in the example above.

Capabilities¶

The table below describes the specific capabilities provided by this toolset. HolmesGPT can decide to invoke any of these capabilities when answering questions or investigating issues.