LokiΒΆ
By enabling this toolset, HolmesGPT will fetch node and pods logs from Loki. Loki can be accessed directly or by proxying through a Grafana instance.
You should enable this toolset to replace the default kubernetes/logs toolset if all your kubernetes/pod logs are consolidated inside Loki. It will make it easier for HolmesGPT to fetch incident logs, including the ability to precisely consult past logs.
HolmesGPT provides several out-of-the-box alternatives for log access. You can select from these options:
kubernetes/logs: Access logs with
kubectl logs
commands. This is the default toolset.coralogix/logs: Access logs through Coralogix.
grafana/loki: Access Loki logs by proxying through a Grafana instance.
Proxying through GrafanaΒΆ
This is the recommended approach because we intend to add more capabilities to the toolset that are only available with Grafana.
PrerequisitesΒΆ
A Grafana service account token with the following permissions:
Basic role -> Viewer
Data sources -> Reader
Check out this video on creating a Grafana service account token.
Getting Grafana URL
You can find the Grafana URL required for Loki in your Grafana cloud account settings.

Obtaining the datasource UID
You may have multiple Loki data sources setup in Grafana. HolmesGPT uses a single Loki datasource to fetch the logs and it needs to know the UID of this datasource.
A simple way to get the datasource UID is to access the Grafana API by running the following request:
# port forward if you are using Robusta's grafana from your kubernetes cluster
kubectl port-forward svc/robusta-grafana 3000:80
# List the loki data sources
curl -s -u <username>:<password> http://localhost:3000/api/datasources | jq '.[] | select(.type == "loki")'
{
"id": 2,
"uid": "klja8hsa-8a9c-4b35-1230-7baab22b02ee",
"orgId": 1,
"name": "Loki-kubernetes",
"type": "loki",
"typeName": "Loki",
"typeLogoUrl": "/public/app/plugins/datasource/loki/img/loki_icon.svg",
"access": "proxy",
"url": "http://loki.loki:3100",
"user": "",
"database": "",
"basicAuth": false,
"isDefault": false,
"jsonData": {
"httpHeaderName1": "admin",
"httpHeaderName2": "X-Scope-OrgID",
"tlsSkipVerify": true
},
"readOnly": false
}
# In this case, there is a single Loki datasourceUID
# with UID "klja8hsa-8a9c-4b35-1230-7baab22b02ee"
Configuration (grafana proxy)ΒΆ
holmes:
toolsets:
grafana/loki:
enabled: true
config:
api_key: <your grafana API key>
url: https://xxxxxxx.grafana.net # Your Grafana cloud account URL
grafana_datasource_uid: <the UID of the loki data source in Grafana>
kubernetes/logs:
enabled: false # Disable HolmesGPT's default logging mechanism
Update your Helm values (generated_values.yaml) with the above configuration and run a Helm upgrade:
helm upgrade robusta robusta/robusta --values=generated_values.yaml --set clusterName=<YOUR_CLUSTER_NAME>
Add the following to ~/.holmes/config.yaml, creating the file if it doesn't exist:
toolsets:
grafana/loki:
enabled: true
config:
api_key: <your grafana API key>
url: https://xxxxxxx.grafana.net # Your Grafana cloud account URL
grafana_datasource_uid: <the UID of the loki data source in Grafana>
kubernetes/logs:
enabled: false # Disable HolmesGPT's default logging mechanism
Direct connectionΒΆ
The toolset can directly connect to a Loki instance without proxying through a Grafana instance.
This is done by not setting the grafana_datasource_uid
field. Not setting this field makes HolmesGPT
assume that it is directly connecting to Loki.
Configuration (direct connection)ΒΆ
holmes:
toolsets:
grafana/loki:
enabled: true
config:
url: http://loki.logging
headers:
X-Scope-OrgID: "<tenant id>" # Set the X-Scope-OrgID if loki multitenancy is enabled
kubernetes/logs:
enabled: false # Disable HolmesGPT's default logging mechanism
Update your Helm values (generated_values.yaml) with the above configuration and run a Helm upgrade:
helm upgrade robusta robusta/robusta --values=generated_values.yaml --set clusterName=<YOUR_CLUSTER_NAME>
Add the following to ~/.holmes/config.yaml, creating the file if it doesn't exist:
toolsets:
grafana/loki:
enabled: true
config:
url: http://loki.logging
headers:
X-Scope-OrgID: "<tenant id>" # Set the X-Scope-OrgID if loki multitenancy is enabled
kubernetes/logs:
enabled: false # Disable HolmesGPT's default logging mechanism
Advanced configurationΒΆ
Search labels
You can tweak the labels used by the toolset to identify kubernetes resources. This is only needed if your
Loki logs settings for pod
, and namespace
differ from the defaults in the example above.
holmes:
toolsets:
grafana/loki:
enabled: true
config:
url: ...
labels:
pod: "pod"
namespace: "namespace"
Update your Helm values (generated_values.yaml) with the above configuration and run a Helm upgrade:
helm upgrade robusta robusta/robusta --values=generated_values.yaml --set clusterName=<YOUR_CLUSTER_NAME>
Add the following to ~/.holmes/config.yaml, creating the file if it doesn't exist:
toolsets:
grafana/loki:
enabled: true
config:
url: ...
labels:
pod: "pod"
namespace: "namespace"
Use the following commands to list Loki's labels and determine which ones to use:
# Make Loki accessible locally
kubectl port-forward svc/loki 3100:3100
# List all labels. You may have to add the -H 'X-Scope-OrgID:<org id>' option with a valid org id
curl http://localhost:3100/loki/api/v1/labels
Disabling the default toolset
If Loki is your primary datasource for logs, it is advised to disable the default HolmesGPT logging
tool by disabling the kubernetes/logs
toolset. Without this. HolmesGPT may still use kubectl to
fetch logs instead of Loki.
holmes:
toolsets:
kubernetes/logs:
enabled: false
CapabilitiesΒΆ
The table below describes the specific capabilities provided by this toolset. HolmesGPT can decide to invoke any of these capabilities when answering questions or investigating issues.
Tool Name |
Description |
---|---|
fetch_loki_logs_for_resource |
Fetches the Loki logs for a given kubernetes resource |
fetch_loki_logs |
Fetches Loki logs from any query |