Open-source LLM observability for self-hosted inference

Why Self-Hosted Inference Needs a Different Observability Stack

The 2024–2025 wave of LLM observability tools — LangSmith, Humanloop, PromptLayer, Weights & Biases Weave, Braintrust — was built for teams calling OpenAI, Anthropic, and Google APIs. The telemetry surface those tools instrument is the application layer: prompts, completions, token counts, latency from the caller’s perspective. For teams on the API side of the inference boundary, that is the right surface.

But the inference boundary has been moving. As of 2026, a growing share of production LLM workloads — especially at the enterprise, regulated, and cost-conscious end of the market — runs on self-hosted inference. vLLM on H100 clusters. Hugging Face TGI behind an internal API. Ollama for on-device and edge workloads. SGLang for multi-model serving. These deployments have the same application-layer telemetry needs as the API-customer stack, plus a second layer the SaaS tools do not touch: the inference runtime itself. GPU utilization. KV cache efficiency. TTFT (time to first token) and TPOT (time per output token) broken out. Batching decisions. Model swap events. Per-team GPU-hour cost allocation.

This post is the 2026 open-source playbook for LLM observability on self-hosted inference. It covers the tools, the metric taxonomy, the architecture, and the rollout sequence that works for platform-engineering teams standing up internal inference as a service for their company.

The Four Metric Families That Matter

A 2026 self-hosted inference observability program tracks four metric families:

Family 1 — Token economics. Tokens in, tokens out, total per request, per model, per team, per feature. The unit economics of self-hosted inference rest on this data. Without it, cost allocation is guesswork. The 2026 default is to emit per-request token counts as OpenTelemetry GenAI attributes (gen_ai.usage.input_tokens, gen_ai.usage.output_tokens) and aggregate in the metrics pipeline.

Family 2 — Latency. TTFT (time to first token) is the latency metric that matters for interactive UX — it is the 2026 analogue to page-load time. TPOT (time per output token) is the metric that determines perceived streaming speed after the first token arrives. End-to-end latency is a derived metric. Queue time is the time between request arrival and TTFT and is the leading indicator of capacity shortage. Instrument all four.

Family 3 — Quality. Regressions — a new model version, a new system prompt, a new RAG retriever — degrade quality before they are caught by users. The 2026 pattern is continuous eval: a rolling sample of production traffic scored against a reference set by a judge model or a deterministic rubric, with drift alerts when the score crosses a threshold. Langfuse and Arize Phoenix both support this as a first-class workflow.

Family 4 — GPU utilization and runtime health. GPU memory utilization, GPU compute utilization, KV cache hit rate, batch size distribution, model-swap frequency for multi-model serving, request queue depth, and failed-request rate by reason. The 2026 default is to scrape the inference runtime’s Prometheus endpoint (vLLM, TGI, and SGLang all expose one) and join with the application-layer traces via OpenTelemetry.

Instrumenting three of the four families is common; instrumenting all four is the discipline that separates 2026 mature platforms from 2025 experimental ones.

The 2026 Open-Source Stack

A representative 2026 open-source LLM observability stack for self-hosted inference:

• Langfuse — OSS LLM observability with traces, evals, prompt management, and cost tracking. Self-hostable. Supports the OpenTelemetry GenAI semantic conventions natively. The 2026 default for traces and evals.
• Arize Phoenix — OSS LLM observability with strong focus on evaluation, embeddings visualization, and RAG drift detection. Self-hostable. Complements Langfuse for teams with heavy RAG workloads.
• OpenLIT — OSS auto-instrumentation for LLM workloads. Supports OpenTelemetry GenAI semantic conventions. Works well as the telemetry emitter feeding Langfuse or Phoenix.
• OpenTelemetry Collector — The pipeline component. Receives OTLP from the application, transforms, and routes to the downstream backends.
• Prometheus + Grafana — For the GPU-utilization and inference-runtime metrics. vLLM, TGI, and SGLang all expose Prometheus endpoints.
• Loki or OpenSearch — For the log stream. Application logs, inference-runtime logs, and the OTel span logs all converge here.
• Mimir or Thanos — For long-term metric retention if Prometheus alone is insufficient.

The typical 2026 deployment runs Langfuse as the LLM-observability backend, Prometheus + Grafana for the runtime metrics, and an OpenTelemetry Collector tying the two surfaces together via trace-to-metric exemplars. Arize Phoenix is added for teams with significant RAG complexity; OpenLIT is added for teams who want zero-code instrumentation.

---

See the IAN team run on your cloud. We connect to your AWS account via a scoped read-only role, run the Observe-tier agents, and leave you with a concrete audit report — cost waste, security exposure, compliance gaps, and a labor-offset estimate. You keep the findings regardless of next steps. Get a free infrastructure audit →

---

The OpenTelemetry GenAI Semantic Conventions

OpenTelemetry published the GenAI semantic conventions in 2024 and matured them through 2025. As of 2026, the conventions are stable enough to be the default schema. The core attributes:

• gen_ai.system — the inference system (vllm, tgi, ollama, sglang).
• gen_ai.request.model — the model name (meta-llama/Llama-3.3-70B, mistralai/Mistral-Large-2).
• gen_ai.request.temperature, gen_ai.request.max_tokens, gen_ai.request.top_p — the sampling parameters.
• gen_ai.response.model — the actual served model (may differ from requested in a multi-model serving setup).
• gen_ai.response.finish_reasons — why generation stopped.
• gen_ai.usage.input_tokens, gen_ai.usage.output_tokens — the token counts.
• gen_ai.operation.name — the operation (chat, text_completion, embeddings).

Emit these from the application layer, from the inference runtime, or — ideally — from both, with span linkage. The 2026 platform advantage is that every downstream tool (Langfuse, Phoenix, OpenLIT, Grafana Tempo) understands this schema natively. Custom attributes still work but require per-tool mapping.

Architecture: The Three-Surface Pattern

A 2026 self-hosted inference observability architecture converges three telemetry surfaces into a single query plane:

Surface 1 — Application. Your app (chat, agent, RAG) emits OTLP traces with GenAI attributes via OpenLIT or manual instrumentation. Each trace includes the user request, the prompt, the retrieval calls, and the completion. Sent to the OTel Collector.

Surface 2 — Inference runtime. vLLM / TGI / SGLang emit Prometheus metrics (GPU utilization, batching, KV cache, request queue) on a :8000/metrics endpoint. Scraped by Prometheus.

Surface 3 — GPU host. Node-exporter + DCGM-exporter (NVIDIA’s Data Center GPU Manager exporter) emit host-level and GPU-level hardware metrics. Scraped by Prometheus.

The OTel Collector routes:

• Traces → Langfuse (for LLM-specific analysis) + Grafana Tempo (for cross-service distributed tracing)
• Metrics → Prometheus / Mimir
• Logs → Loki / OpenSearch
• Evals (separately triggered) → Langfuse or Arize Phoenix

Grafana then queries all three — PromQL for runtime metrics, TraceQL for trace correlation, LogQL for logs — with exemplar linking so a spike in TTFT clicks through to the offending trace, which clicks through to the GPU utilization profile at that timestamp. This is the 2026 pattern for deep debugging of self-hosted inference.

Cost Allocation: The Killer Feature for Platform Teams

Once the token-economics data is flowing, per-team cost allocation becomes the feature that platform teams use to justify the inference platform to finance. The 2026 pattern:

1. Tag every request at the application layer with team, product, and feature attributes.
2. In the metrics pipeline, compute a per-team cost using a simple cost model: (input_tokens * input_price) + (output_tokens * output_price) + (gpu_seconds * gpu_hourly_rate / 3600).
3. Choose input and output prices that reflect the marginal cost of self-hosted inference on your hardware — roughly 15–40% of API provider list prices for the same model, depending on utilization.
4. Publish a per-team, per-month cost dashboard. Break out by product and feature.
5. For teams pushing the utilization to the point of needing additional GPU capacity, the cost dashboard becomes the justification artifact.

This is the 2026 killer feature for internal inference platforms. Without cost allocation, the platform is a free good and will be consumed accordingly. With cost allocation, the platform is a measurable service and teams can make informed decisions about when to prompt-engineer-for-fewer-tokens versus scale-up-the-cluster.

Quality Drift Detection

The third metric family — quality — is the one most teams under-instrument. A 2026 drift-detection workflow:

1. Define a reference eval set per product use case — 50–200 inputs with known good outputs.
2. Run the eval set nightly against production traffic’s current configuration (current model, current system prompt, current RAG configuration).
3. Score with a judge model (Claude or GPT-4 class) or a deterministic rubric.
4. Record the aggregate score in Langfuse or Phoenix.
5. Alert when the aggregate score drops below a threshold.

Separately, run a rolling sample of production traffic (typically 1–5% of requests) through the same judge-model scoring to detect real-world drift that the fixed eval set may not catch. This is the 2026 continuous-eval pattern, and both Langfuse and Phoenix support it as a built-in workflow.

Three-Phase Rollout

Phase 1 — Traces and token economics (weeks 1–3). Deploy Langfuse self-hosted. Instrument the application layer with OpenLIT or manual OTel GenAI spans. Validate traces arriving in Langfuse with full token counts. Stand up the OTel Collector. Wire the cost-allocation dashboard.

Phase 2 — Runtime and GPU metrics (weeks 4–6). Scrape vLLM / TGI Prometheus endpoints. Deploy DCGM-exporter on the GPU hosts. Stand up Grafana dashboards for GPU utilization, KV cache hit rate, request queue depth, and batching distribution. Link traces to runtime metrics via exemplars.

Phase 3 — Evals and drift detection (weeks 7–10). Define the reference eval set per product use case. Wire the nightly eval run in Langfuse or Phoenix. Set the drift-alert thresholds. Begin the continuous-eval sampling pipeline for production traffic.

By the end of Phase 3, the platform has token-economics cost allocation, runtime performance visibility with trace correlation, and quality-drift detection. The observability posture supports both day-to-day platform operations and the finance and product conversations that determine whether the self-hosted inference investment is justified.

How IAN Helps

IAN surfaces the inference-platform cost and utilization data alongside the rest of the cloud-cost picture — GPU-hours consumed, per-team allocation, idle-capacity windows, and the specific instances and model loads that are driving the cost curve. For platform-engineering teams running internal inference as a service, IAN closes the loop between the Langfuse / Grafana observability view and the cloud-bill reality, so the decisions to right-size the cluster, swap in quantized models, or consolidate onto fewer nodes are made against a full cost model rather than a partial one.

The 2026 self-hosted inference platform ships with observability from day one. IAN makes the platform’s unit economics legible to the engineering and finance leaders who have to justify the next expansion.

Get a free infrastructure audit → | See pricing →