Loki (Grafana)

By enabling this toolset, HolmesGPT will be able to query logs from Loki by proxying through a Grafana instance.

After this toolset is enabled, you can ask Holmes questions like: Look at my cluster with kubectl, pick an arbitrary node, then fetch logs from loki for that node. Summarize all problems you find.

Prerequisites

A Grafana service account token with the following permissions:

  • Basic role -> Viewer

  • Data sources -> Reader

Check out this video on creating a Grafana service account token.

Getting Grafana URL

You can find the Grafana URL required for Loki and Tempo in your Grafana cloud account settings.

Configuration

Add the following to Robusta's Helm values:

holmes:
  toolsets:
    grafana/loki:
      enabled: true
      config:
        api_key: <your grafana service account token>
        url: <your grafana url> # e.g. https://acme-corp.grafana.net

Update your Helm values (generated_values.yaml) with the above configuration and run a Helm upgrade:

helm upgrade robusta robusta/robusta --values=generated_values.yaml --set clusterName=<YOUR_CLUSTER_NAME>

You can optionally tweak the search terms used by the toolset. This is only needed if your Loki logs settings for pod, namespace and node differ from the defaults listed below. To do so, add these search keys to the configuration:

holmes:
  toolsets:
    grafana/loki:
      enabled: true
      config:
        api_key: <your grafana service account token>
        url: <your grafana url> # e.g. https://acme-corp.grafana.net
        pod_name_search_key: "pod"
        namespace_search_key: "namespace"
        node_name_search_key: "node"

Add the following to ~/.holmes/config.yaml, creating the file if it doesn't exist:

toolsets:
  grafana/loki:
    enabled: true
    config:
        api_key: <your grafana service account token>
        url: <your grafana url> # e.g. https://acme-corp.grafana.net

You can optionally tweak the search terms used by the toolset. This is only needed if your Loki logs settings for pod, namespace and node differ from the defaults listed below. To do so, add these search keys to the configuration:

toolsets:
  grafana/loki:
    enabled: true
    config:
        api_key: <your grafana service account token>
        url: <your grafana url> # e.g. https://acme-corp.grafana.net
      pod_name_search_key: "pod"
      namespace_search_key: "namespace"
      node_name_search_key: "node"

To test, run:

holmes ask "Which applications have an issue and check the logs on loki for the reasons"

Capabilities

The table below describes the specific capabilities provided by this toolset. HolmesGPT can decide to invoke any of these capabilities when answering questions or investigating issues.

Tool Name

Description

list_loki_datasources

Fetches the Loki data sources in Grafana

fetch_loki_logs_by_node

Fetches the Loki logs for a given node

fetch_loki_logs_by_label

Fetches the Loki logs for a label and value from a Tempo trace

fetch_loki_logs_by_pod

Fetches the Loki logs for a given pod