Self-hosted LLM inference cost in 2026: vLLM, SGLang, LMDeploy, and a gateway

Self-Hosted Inference Is Now an Engine Decision, Not a Build Decision

In 2024, “self-host an LLM” usually meant “stand up a vLLM container, point it at a model, and call it done.” In 2026 the engine layer has matured to the point where the engine choice — vLLM, SGLang, LMDeploy, TensorRT-LLM, TGI — meaningfully changes both throughput and total cost of ownership for the same model on the same hardware. The right answer is workload-class-specific, and the teams that get it right are running multiple engines behind a single gateway.

Three numbers anchor the 2026 picture:

• SGLang and LMDeploy benchmark at roughly 16,200 tokens per second on an H100 for the standard inference workload mix.
• vLLM lands at roughly 12,500 tokens per second on the same hardware — a 29% throughput gap.
• That 29% gap translates to roughly $15,000 per month in GPU savings when serving one million requests per day, before counting the secondary savings (smaller fleet → smaller egress, smaller storage, smaller observability bill).

The gap is not because vLLM is poorly engineered. vLLM is the most mature production-ready open-source inference engine and ships the largest ecosystem (the broadest model support, the cleanest OpenAI-compatible API, the deepest community of production deployments). The gap is because the workload-class fit matters more than the headline maturity number.

Workload-Class Engine Fit

Five workload classes drive the engine decision in 2026:

• Multi-turn conversations (chat, agentic loops, multi-step reasoning). SGLang wins via RadixAttention, a memory manager designed for the cache-reuse patterns that multi-turn workloads create. The KV-cache overlap across turns is what closes the throughput gap on the most common 2026 workload.
• Quantized model serving (FP8, INT8, AWQ, GPTQ). LMDeploy wins via TurboMind, a custom C++ backend that eliminates Python overhead and produces higher tokens-per-second on quantized weights than the Python-managed alternatives.
• General-purpose production serving with the broadest model coverage. vLLM wins on ecosystem maturity, model support, and the cleanest OpenAI-compatible API — which matters more in production than the headline tokens-per-second metric when the team is debugging the seventh edge case at 2 AM.
• High-throughput batch inference with the largest possible context windows. TensorRT-LLM still wins on raw throughput on Blackwell hardware where the NVIDIA-specific kernels are tuned, but the operational complexity is materially higher than the open-source alternatives.
• Long-context, low-traffic serving. TGI (Text Generation Inference, from Hugging Face) is the easiest to operate and integrates cleanly with the Hugging Face model registry; the throughput is lower than the alternatives but the operational simplicity is real.

The engine-per-workload-class pattern is not theoretical. The 2026 production deployments at scale that report the largest cost savings are the ones running multiple engines behind a single routing layer — and the routing layer is the second half of this story.

The LiteLLM Gateway Pattern

A multi-engine inference architecture only works if the consumers — application code, agentic loops, the in-IDE agents — don’t have to know which engine is serving the request. LiteLLM is the open-source gateway that makes that abstraction work in 2026.

The pattern:

• OpenAI-compatible API at the edge. Every consumer speaks the OpenAI Chat Completions API. The gateway translates to whatever the underlying engine speaks.
• Virtual keys for per-tenant cost tracking. The gateway issues per-team or per-application keys, tracks token usage per key, and emits cost data into the FinOps pipeline. This is the per-tenant chargeback layer for self-hosted inference.
• Fallback routing across 100+ providers. When the self-hosted fleet is over capacity, the gateway falls back to a hosted provider (Anthropic, OpenAI, Together, Anyscale, Fireworks). When the hosted provider has degraded latency, the gateway shifts traffic back to the self-hosted fleet.
• Workload-class routing. The gateway routes the multi-turn conversation traffic to the SGLang fleet, the quantized-model traffic to LMDeploy, the general-purpose traffic to vLLM, and the batch inference traffic to TensorRT-LLM — driven by request metadata or by a header set on the consumer side.

Envoy AI Gateway is the Envoy-native alternative for teams already running Envoy as a service mesh. The functional surface is similar; the choice is mostly about where the gateway fits in the existing infrastructure.

The LiteLLM (or Envoy AI Gateway) pattern is what turns a multi-engine inference architecture into something the application teams don’t have to think about. The cost lever lives in the gateway layer, not in the application layer.

Self-Hosted vs Hosted vs Hybrid

The 2026 cost-versus-latency picture is not binary. The three architectures and what each is good for:

• Pure self-hosted. Lowest per-token cost when the engine and workload-class are matched well, the GPU fleet is sized for steady-state utilization, and the team has the operational capacity to run the inference engines in production. The break-even point is workload-specific but typically lands around 200-500 million tokens per day.
• Pure hosted (Anthropic, OpenAI, Together, Fireworks, Anyscale). Lowest operational cost, fastest time-to-launch, the model latency / quality is the vendor’s problem. The break-even point is reversed: when token volume is below the self-hosted break-even, the hosted price is the right price.
• Hybrid. Self-host the steady-state base load, fall back to hosted for traffic spikes and for the long tail of model variants that don’t justify dedicated capacity. The LiteLLM gateway is what makes this clean.

Most teams running material AI workloads in 2026 land on hybrid. The question is the steady-state-to-spike ratio, which determines the size of the self-hosted fleet.

See the IAN team run on your cloud. We connect to your AWS account via a scoped read-only role, run the Observe-tier agents, and leave you with a concrete audit report — cost waste, security exposure, compliance gaps, and a labor-offset estimate. You keep the findings regardless of next steps. Get a free infrastructure audit →

What Tooling Helps and Where It Falls Short

The 2026 inference-engine tooling landscape covers parts of the picture well. The engine projects (vLLM, SGLang, LMDeploy, TensorRT-LLM, TGI) ship benchmarks and tuning guides. LiteLLM and Envoy AI Gateway cover the routing and per-tenant cost tracking. Hugging Face’s TGI and the model registry cover the model-supply side. Provider-side tooling (NVIDIA NIM, AWS SageMaker JumpStart, GCP Vertex AI inference) covers the managed-stack option for teams that don’t want to operate the engine themselves.

Where the tooling is thin is on the cross-engine routing decision itself. Most teams pick one engine, ship it, and live with the workload-fit mismatch. The cost-saving lever — running multiple engines behind a gateway with workload-class routing — exists in the open-source stack but is not packaged as a turnkey product.

Three things in particular are hard to wire up with off-the-shelf tooling:

• Continuous workload-class classification of every inference request against the five-class matrix.
• Continuous engine benchmarking on the customer’s own GPU fleet, not on the published benchmark results which are workload-mix and hardware-mix specific.
• Reversible routing changes that promote a workload class from one engine to another when the benchmark data justifies it.

Those three live in the gap between point-tool inference and operational execution. They are the work of an active operational layer, not a static gateway config.

Capability Tiers Mapped to Inference-Engine Operations

The Observe / Operate / Administer capability-tier model applies cleanly to self-hosted inference:

• Observe. Continuous workload-class classification across every inference request, per-engine benchmarking on the customer’s own fleet, per-tenant cost tracking via the LiteLLM virtual keys, and waste flagging (idle inference replicas, mis-routed workloads, model-fleet drift). Auto-execute, fully audited.
• Operate. Reversible deployment actions: scaling inference replicas up or down, draining replicas for engine version upgrades, shifting traffic between engines based on benchmark data, falling back to hosted providers during over-capacity events. Auto-execute when in policy and reversible; gated when out-of-policy.
• Administer. Engine selection for new model fleets, model-license decisions, multi-year capacity commitments at specialized providers, and changes to the fallback-provider list itself. Always requires explicit approval; separation-of-duties enforced.

The mapping is what makes the multi-engine pattern deployable inside a real organization. The engineering team owns the engine selection; the deployment agent owns the continuous routing and scaling decisions; the audit trail owns the evidence.

How IAN Helps: The Deployment-Agent on the Active Operational Layer

IAN is the AI DevOps team for cloud infrastructure, delivered as a coordinated team of specialized agents on the active operational layer. The deployment agent is one of those agents, and self-hosted inference is one of its primary specializations.

The deployment agent runs continuous workload-class classification across every inference request and continuous engine benchmarking on the customer’s own GPU fleet. The classification and benchmark data land in the audit trail with routing recommendations attached. When a workload class would benefit from a different engine, the agent surfaces the recommendation in the team’s chat ops surface with the migration plan, the projected throughput delta, and the rollback path.

Reversible Operate-tier actions — scaling inference replicas, draining replicas for engine upgrades, shifting traffic between engines, falling back to hosted providers — execute automatically when policy permits. Engine selection for new model fleets and changes to the fallback-provider list escalate to the Administer tier with separation-of-duties enforced. Every action lands in the immutable audit trail, which doubles as the evidence pack for the platform and FinOps teams.

Pricing is BYOK and usage-based with a monthly minimum. Customers bring their own model keys (Claude, OpenAI, or another provider) and pay inference cost directly to their model vendor for the hosted-fallback path. For the self-hosted path, the BYOK model means customers pay the GPU bill directly to their cloud provider. IAN charges for the orchestration layer, per agent action, per cloud account, per operation class.

The Three-Phase Rollout

Phase 1 — Stand up the inference Observe layer. Wire workload-class classification, per-engine benchmarking, and per-tenant cost tracking into the deployment-agent’s read path across the inference fleet. Two-to-four weeks for a team running a single-engine deployment.

Phase 2 — Promote scoped Operate-tier actions. Let the deployment agent scale inference replicas, drain replicas for upgrades, and shift traffic between engines under capability-tier governance. Measure the throughput-and-cost delta against the Phase 1 baseline. Two-to-three months.

Phase 3 — Run the multi-engine routing loop in production. Promote the deployment agent to recommend engine selection for new model fleets and to manage the hybrid self-hosted-plus-hosted fallback policy. Keep model-license decisions and capacity commitments gated to Administer-tier approval. Continue running the reversible Operate-tier actions in parallel.

The pattern compounds. By the time Phase 3 lands, the platform team is doing different work — fewer minutes tuning a single engine to handle every workload class, more minutes setting policy on which workload classes should run on which engine — and the deployment agent has accumulated enough audit data to make the next round of policy tuning a data exercise rather than a debate.

Get a free infrastructure audit → | See pricing →