Tempo (Grafana)¶

By enabling this toolset, HolmesGPT will be able to fetch trace information from Grafana Tempo to debug performance related issues, like high latency in your application.

Prerequisites¶

A Grafana service account token with the following permissions:

Basic role -> Viewer
Data sources -> Reader

Check out this video on creating a Grafana service account token.

Getting Grafana URL¶

You can find the Grafana URL required for Loki and Tempo in your Grafana cloud account settings.

Configuration¶

Robusta Helm Chat Holmes CLI

holmes:
  toolsets:
    grafana/tempo:
      enabled: true
      config:
        api_key: <your grafana API key>
        url: https://grafana-url

Update your Helm values (generated_values.yaml) with the above configuration and run a Helm upgrade:

helm upgrade robusta robusta/robusta --values=generated_values.yaml --set clusterName=<YOUR_CLUSTER_NAME>

Add the following to ~/.holmes/config.yaml, creating the file if it doesn't exist:

toolsets:
  grafana/tempo:
    enabled: true
    config:
        api_key: <your grafana service account token>
        url: <your grafana url> # e.g. https://acme-corp.grafana.net

To test, run:

holmes ask "The payments DB is very slow, check tempo for any trace data"

Capabilities¶

The table below describes the specific capabilities provided by this toolset. HolmesGPT can decide to invoke any of these capabilities when answering questions or investigating issues.

Tool Name	Description
list_all_datasources	Fetches All the data sources in Grafana
fetch_tempo_traces_by_min_duration	Lists Tempo traces ids that exceed a specified minimum duration in a given time range
fetch_tempo_trace_by_id	Retrieves detailed information about a Tempo trace using its trace ID. Use this to investigate a trace.