Log LLM Requests to Grafana Loki

Coding assistants, agents, and automation pipelines send prompts to large language models all day long, and that traffic is as production critical as any other API call. Without central logging, nobody can answer which model was called, by whom, how many tokens it consumed, or what a failing agent actually sent. Shadow AI usage spreads in that blind spot.

Membrane can write every exchange that passes through the LLM Gateway to Grafana Loki. Requests and responses end up in a log aggregation store that is built for high volume, cheap to operate, and queryable with LogQL from Grafana dashboards.

This page shows how to configure the lokiExchangeStore for prompt logging and how to combine it with the LLM Gateway to get self-hosted LLM observability and a complete audit trail of AI usage.

The lokiExchangeStore requires Membrane 7.3.2 or later.

Why Log LLM Requests and Prompts?

Routing AI traffic through a gateway is the prerequisite for governance. Centralized logging is what makes that governance visible:

• Audit trail, compliance, and reporting. One log of every AI call answers all three at once. It lets you reconstruct which prompt was sent to which provider and model and what came back, which is the evidence regulations such as the EU AI Act expect when an organization has to show how its AI systems are used. The same stream also feeds Grafana dashboards, so the audit log and the usage reporting are the same data.
• Cost attribution and chargeback. Token counts are recorded per exchange, so consumption can be correlated with the teams, applications, and tools that caused it.
• Troubleshooting. When a call fails or a model refuses, the log holds the exact request an agent or IDE produced, including the payload the client itself never shows you.
• Security. Unusual usage patterns, unexpected models, and clients that were never authorized to call an LLM all become visible in the same data.

How the Loki Exchange Store Works

Membrane collects every request and response pair in an Exchange. Membrane ships several exchange stores, for example an in-memory store for the admin console and a file based store for local debugging.

The lokiExchangeStore pushes exchanges to a Loki instance instead. Loki indexes only the stream labels and stores the log lines themselves compressed, which makes it a good fit for the large payloads that LLM calls produce.

The gateway talks to the Loki push API over HTTP directly, so no Promtail agent, log shipper sidecar, or log file to scrape is involved. It also does not matter which provider the request goes to: calls to OpenAI, Anthropic, or Google AI are logged the same way, because what is logged is the exchange itself, not the provider's wire format.

If the store is configured globally, it is not limited to AI traffic. Every API defined in the same configuration is logged through it, whether it serves REST, SOAP, or LLM calls. Configured on a single API, only that API's traffic ends up in Loki.

Configuring the Loki Exchange Store

Declare the store in the components section of the configuration. It needs the URL of the Loki instance and a job name that becomes the stream label of the log entries.

components:
  exchangeStore:
    lokiExchangeStore:
      url: http://localhost:3100
      job: gateway-eu

components:
  exchangeStore:
    lokiExchangeStore:
      url: http://localhost:3100
      job: gateway-eu

Listing 1: Sending exchanges to a local Loki instance

Use a distinct job per gateway instance, stage, or region. The label is what you filter on later, so gateway-eu, gateway-staging, or llm-gateway-prod are more useful than a generic name.

Sample Configuration

The following configuration exposes an OpenAI compatible LLM Gateway and logs all traffic to Loki. Client applications send their requests to the gateway instead of directly to OpenAI. The gateway forwards the request using the centrally managed provider key and writes the exchange to Loki.

api:
  port: 2000
  flow:
    - llmGateway:
        apiKey: <<Replace with your API_KEY>>
        openai: {}
  target:
    url: https://api.openai.com

components:
  exchangeStore:
    lokiExchangeStore:
      url: http://localhost:3100
      job: gateway-eu

api:
  port: 2000
  flow:
    - llmGateway:
        apiKey: <<Replace with your API_KEY>>
        openai: {}
  target:
    url: https://api.openai.com

components:
  exchangeStore:
    lokiExchangeStore:
      url: http://localhost:3100
      job: gateway-eu

Listing 2: LLM Gateway with central logging to Loki

Querying LLM Traffic in Grafana with LogQL

Once the gateway forwards exchanges, the traffic can be explored in a Grafana dashboard with LogQL. Select the stream by its job label:

{job="gateway-eu"}

{job="gateway-eu"}

Listing 3: LogQL query selecting all exchanges of one gateway

From there the usual LogQL toolbox applies: filter for a status code, search the payload for a model name, or aggregate over time to chart AI usage next to the rest of your API metrics. Membrane also exposes runtime statistics for Prometheus, so logs and metrics can be combined in the same Grafana dashboard.

Open Source, Self-Hosted AI Observability

Membrane and Grafana Loki are both open source and run on your own infrastructure. Unlike hosted LLM observability services, no prompt, completion, or API key ever leaves your network.

That matters as soon as prompts contain customer data, source code, or internal documents. Data residency and GDPR requirements are easier to satisfy when the log store is inside your own perimeter, and retention periods are configured in Loki.

It also keeps the stack small. If Grafana and Loki are already running for infrastructure logs, AI traffic becomes one more stream next to them instead of another SaaS subscription.

How to Set It Up

Setting this up takes a few minutes.

1. Download Membrane API Gateway in version 7.3.2 or later.
2. Run a Loki instance, for example with Docker Compose, and note its URL.
3. Explore the examples in the tutorials/ai/llm-gateway folder.
4. Add the lokiExchangeStore component to your configuration and adapt url and job.
5. Add Loki as a data source in Grafana and query the job label.

Logging is one building block of AI governance. See Open Source LLM Gateway for AI Governance for token quotas and model governance, and Sharing AI Accounts Without Sharing the API Keys for centralized credential handling.