Cut LLM Costs 40–80% for Engineers with Routing, Caching and Batching

The three highest-return levers for cutting large language model spend are model routing and selection, context compaction with prompt caching, and batching. Together, published studies show workload-dependent savings ranging from roughly 40% to over 80%, though results vary sharply by task and traffic pattern. Model routing and quality-aware selection typically deliver the fastest wins, prompt caching and compaction compound on top of that, and batching adds a further discount on anything that can tolerate delay. Measure cost per successful task before and after every change; if you skip that step, you’re optimising blind. Self-hosted teams running GPU fleets can layer advanced routing on top for additional gains.

TL;DR:

• Effective model routing and selection can save between 40% and over 80% of LLM costs, depending on workload and traffic pattern.
• Monitoring metrics such as cost per successful task and cache hit rate is crucial to accurately measure and optimize spending.
• Context length and retries significantly impact cost, especially with output-heavy tasks or high failure rates, requiring careful infrastructure management.
• Implementing workload-aware routing, prompt caching, and context compaction can lead to substantial savings, especially when combined with quality-aware predictions.
• Self-hosted fleets must address the context length cliff and capacity planning, while managed platforms offer faster deployment but less control.

Table of Contents

• Where do LLM costs actually come from?
• What should you measure to control LLM spend?
• Which levers cut LLM costs the most?
• How does advanced routing translate into production savings?
• How do you roll this out without breaking production?
• What do the research papers actually prove about LLM savings?
• What I’d tell any team starting this today
• Want a managed route instead of building this yourself?
• Where to go deeper on the research and provider docs
• Sources
• FAQ

Where do LLM costs actually come from?

Every dollar you spend on inference traces back to four variables: token volume, model choice, concurrency, and how efficiently you use context. Get the mental model right and the optimisation levers later in this guide make a lot more sense.

Input and output tokens are priced separately by every major provider, and output tokens usually cost two to five times more than input tokens. That asymmetry matters more than most teams realise. A summarisation task that ingests 4,000 tokens and returns 200 is cheap. A code-generation task that ingests 500 tokens and returns 3,000 can cost far more per call, even though the raw token count looks smaller.

Context length is the second multiplier. As a conversation or retrieval-augmented prompt grows, the model has to process the entire context window on every call, and the key-value cache backing that context grows with it. On self-hosted infrastructure, this creates what researchers call a “cost cliff”: once a request’s context crosses a certain length, GPU memory pressure forces smaller batch sizes, and throughput drops sharply. The FleetOpt research on compress‑and‑route co‑design documents this cliff directly and shows it’s a major reason homogeneous GPU fleets waste capacity.

Concurrency and traffic spikes are the third multiplier, and they hit self-hosted and API-based deployments differently:

• API-based systems pay per token regardless of concurrency, but rate limits and retries during spikes inflate effective cost per successful task.
• Self-hosted fleets must provision for peak concurrency, meaning idle GPU capacity during off-peak hours is money burned for nothing.
• Hybrid setups route overflow traffic to API providers during spikes, trading a cost premium for elasticity.

Here’s a worked example. Say a support-triage endpoint sends 1,500 input tokens and receives 300 output tokens per call, running 50,000 calls a month. At a blended rate where output costs four times input, that endpoint’s real cost per request is dominated by the output side even though input tokens are five times more numerous. If 8% of calls fail and get retried once, your cost per successful task is meaningfully higher than cost per call. Teams that only track cost per call miss this gap entirely and underestimate their real spend by a wide margin.

What should you measure to control LLM spend?

You can’t optimise what you don’t measure, and LLM cost sprawls fast when nobody owns the dashboard. Instrumentation has to happen before you touch a single routing rule, or you won’t know if a change actually saved money or just moved the cost somewhere less visible.

Six metrics matter more than the rest:

1. Tokens in and tokens out, tracked separately, per endpoint and per feature.
2. Cost per successful task, not cost per call. A retried or failed call still burned tokens.
3. Retries per task, because retry rates quietly double effective spend on flaky prompts.
4. Cache hit rate, for both prompt caching and any semantic or embedding cache in front of the model.
5. Model-route split, showing what percentage of traffic lands on each model tier.
6. P99 time-to-first-token (TTFT), because latency regressions often accompany botched routing changes.

Attribute every one of these by feature, endpoint, or user flow rather than aggregating across the whole product. A single “chatbot” cost line hides the fact that your onboarding flow burns 60% of the budget on a task that could run on a cheaper model. Tag requests at the API gateway with a feature identifier and you get that breakdown almost for free.

Before rolling any change to production, build a small evaluation set, somewhere between 100 and 300 real examples pulled from production logs, and score both cost and quality against it. Set a quality floor (an accuracy or human-graded score you refuse to go below) and treat any experiment that breaches it as a failure, no matter how much it saves.

Pro Tip: Set an automated alert on cache hit rate and cost-per-successful-task, not just raw spend. A silent quality regression often shows up first as a spike in retries, which raw dollar tracking won’t catch until the invoice arrives.

Which levers cut LLM costs the most?

Once you can measure cost per successful task, the actual levers fall into five categories: model selection, routing, context compaction, caching, and batching. Apply them roughly in that order, because each one makes the next one more effective.

Model selection by workload shape. Don’t pick a model by benchmark leaderboard position. Compute your input-to-output token ratio and your accuracy floor first, then find the cheapest model that clears that floor. A workload-shape analysis that ignores token ratios and compares only sticker prices per million tokens routinely picks the wrong model, because a model that’s cheap per input token can be expensive once your workload is output-heavy.

Router patterns. Three patterns cover most production cases:

• Cheap-first routing sends every request to the smallest capable model first, escalating only on a confidence or quality signal.
• Fallback routing sends requests to a strong model by default but degrades to a cheaper one under rate limits or cost caps.
• Batch-aware routing groups requests by expected latency tolerance before choosing a model, so async jobs never compete for premium-tier capacity that live traffic needs.

Quality-aware, budget-constrained routing is where the research gets genuinely useful. A 2024 study on document-processing workloads used quality prediction to pick the cheapest model likely to meet a task’s quality bar, and reported 40% to 90% cost reductions with 4% to 7% quality improvements on the workloads it tested. That’s not a universal number, it’s workload-dependent and the study was experimental, but the direction is consistent: predicting quality before you spend on a big model beats routing by static rules.

Context compaction. Three techniques do most of the work:

1. Extractive compression, summarising or trimming conversation history before it re-enters the prompt.
2. Prefix reuse, structuring prompts so the stable portion (system instructions, few-shot examples) sits first and rarely changes.
3. Hard token budgets per turn, forcing upstream code to truncate rather than letting context grow unbounded.

Prompt caching. Provider-side caching can be dramatic when it works. Some providers document cache-read pricing as low as 0.1 times the standard input price for cached tokens, but that only pays off if your prefix is genuinely stable across calls. Set realistic expectations: a support bot with a fixed system prompt might see cache hit rates above 70%, while a highly personalised agent with per-user context injected early in the prompt might see under 20%. Measure it before you assume caching will save anything.

Batching for async work. Anything that doesn’t need a response inside a few seconds should run through a batch API. OpenAI’s Batch API accepts JSONL batch input files and processes them within a completion window, and providers commonly document roughly 50% discounts for this kind of asynchronous processing. Nightly enrichment jobs, bulk summarisation, and offline evaluation runs are the obvious candidates.

Finally, lock in operational guards: hard output token limits, explicit stop sequences, deterministic seeding where reproducibility matters, and a retry policy that caps attempts rather than retrying indefinitely on transient errors.

Pro Tip: Cap max output tokens on every endpoint, even ones you think never hit the limit. A single malformed prompt that triggers a runaway generation can cost more than a week of normal traffic on that endpoint.

How does advanced routing translate into production savings?

Adaptive, quality-aware routing goes a step further than static rules by estimating, before the call is made, whether a cheaper model will clear the quality bar for that specific request. An ACM-published study on adaptive model and strategy routing reported up to 60% lower API cost in its benchmarks while holding accuracy comparable or better. The catch is that this depends heavily on how well your quality estimator matches your actual traffic. A router tuned on customer-support tickets won’t transfer cleanly to legal document review.

For self-hosted fleets, the more interesting design pattern is the two-pool architecture. Instead of running one homogeneous fleet sized for your worst-case context length, you split GPU capacity into a short-context pool and a long-context pool, then route requests based on estimated context size. FleetOpt’s compress-and-route (C&R) approach adds a compression step at the gateway that shrinks borderline requests just enough to avoid crossing the cliff into the expensive pool. Combined, this design reported 6% to 82% lower total GPU cost versus a homogeneous fleet, with the wide range reflecting how much a given workload actually suffers from the cliff in the first place.

Batch-level, capacity-aware routing is a third pattern worth piloting, especially for teams running large offline jobs. Rather than routing each request independently, the router considers the whole batch’s composition and available capacity together. Research on batch-level routing under adversarial conditions found this can outperform per-query routing when batches are skewed or intentionally structured to stress the router, but it introduces real planning overhead: you need to size batches carefully and monitor for adversarial patterns in submitted work.

Before piloting any of these, you need three things in place:

• Request-level telemetry on context length, model route, and outcome quality.
• A held-out evaluation set that reflects your actual traffic distribution, not a public benchmark.
• Clear SLO trade-offs agreed with stakeholders, since two-pool and batch-level routing can add latency variance even as they cut cost.

How do you roll this out without breaking production?

Treat cost optimisation as an experiment programme, not a one-off migration. An eight-week timeline gives you room to measure, ship quick wins, and only then attempt anything structurally risky.

1. Weeks 0 to 1: baseline. Instrument cost per successful task, tokens in/out, and cache hit rate. Pull 100 to 300 real production examples into an evaluation set before touching anything.
2. Weeks 1 to 3: quick wins. Set hard output token limits, add stop sequences, and move any offline-tolerant job to a batch API.
3. Weeks 3 to 6: medium effort. Turn on prompt caching where prefixes are stable, add cheap-first routing for low-risk tasks, and apply targeted context compaction to your longest-running conversations.
4. Weeks 6 to 8: advanced pilot. Trial a two-pool fleet or batch-level routing on a single high-volume workload, with a rollback plan defined before launch.

At every stage, evaluate against cost per successful task, not raw spend, and set a quality floor you won’t breach regardless of savings. If a change drops quality below that floor, or cache hit rate collapses unexpectedly, roll it back immediately rather than trying to tune around it in production.

Pro Tip: Run the advanced pilot on your second or third highest-volume workload, not your highest. You want a big enough sample to trust the numbers without betting your most critical traffic on an unproven fleet design.

What do the research papers actually prove about LLM savings?

Published results are directionally consistent but the magnitudes swing widely, and reading past the headline percentage matters more than quoting it. The document-processing study reporting 40% to 90% cost reductions tested specific summarisation workloads, not general-purpose chat. The adaptive routing paper’s 60% API cost reduction reflects its own benchmark suite. FleetOpt’s 6% to 82% GPU cost reduction depends entirely on how much your workload actually suffers from the context-length cost cliff, since a fleet running uniformly short requests won’t see anywhere near the top of that range.

Three operational rules of thumb hold up across this research:

• Quality-aware routing beats static model assignment whenever you can build even a rough quality estimator for your task.
• Two-pool fleet designs only pay off when a meaningful share of your traffic sits near the context-length cliff. If your requests are uniformly short, skip the complexity.
• Batch-level routing helps most when your traffic is genuinely batchable and you can tolerate the added planning overhead of capacity-aware scheduling.

Self-hosted teams should also weigh routing decisions against live instance state. Research on self-hosted inference performance found that routing decisions ignoring context length, cache reuse, and current batching state underperform a router that accounts for all three, even when the underlying models are identical.

The most common failure mode isn’t picking the wrong lever, it’s measuring the wrong metric. Teams that track raw dollar spend instead of cost per successful task routinely declare victory on changes that quietly increased retries or degraded quality. The second most common failure is testing on a static benchmark and assuming it transfers to production traffic that spikes, drifts, and includes edge cases no benchmark captures.

What I’d tell any team starting this today

Most LLM cost blowouts trace back to skipping measurement, not picking the wrong model. Teams chase a routing pattern that saved someone else 80% and roll it into production without a baseline, then can’t tell if the eventual saving is real or just a rounding error against normal traffic drift.

Start with cost per successful task, run every change as a controlled experiment, and treat quality floors as non-negotiable. We’ve watched fast-moving teams deploy agent workloads across channels in under a week, and the ones that stay disciplined about measurement are the ones whose savings actually hold up three months later. If you’re running these experiments yourself, we’d genuinely like to hear what worked.

— Agent

Want a managed route instead of building this yourself?

If your team would rather skip building routing infrastructure, caching layers, and observability dashboards from scratch, Agentrelease offers a different path: a white-label platform for launching AI agents across iMessage, WhatsApp, Instagram DMs, Messenger, SMS, email, web chat, and voice, all under your own brand.

Instead of engineering your own model-routing and caching stack, you get server-side revenue attribution, enterprise-grade security, unlimited tenant creation, and GPT-5-powered conversations tuned to your niche, offer, and tone, typically deployed in under a week. That speed matters if your priority is getting agents into production and generating revenue rather than running a lengthy cost-optimisation pilot on infrastructure you’d otherwise have to build and maintain yourselves. Agentrelease runs at a flat $497 per month with unlimited agents and channels, and no setup, per-message, or revenue-share fees on top, detailed on the Agentrelease pricing page. Agencies and resellers can layer on the White-Label Program for full branding and domain control. If you’re weighing an in-house optimisation build against a managed platform, compare the two directly, check the pricing details and see which route gets your agents live faster.

Where to go deeper on the research and provider docs

The claims above draw on peer-reviewed and preprint research plus provider documentation you can check directly:

• Quality‑aware model selection for document processing
• Adaptive model–strategy routing (ACM / arXiv)
• FleetOpt: compress‑and‑route fleet co‑design
• OpenAI Batch API definition
• Cloud OS workstation infrastructure, for teams managing their own GPU clusters

Sources

• Quality‑aware model selection for document processing (2024)
• FleetOpt: fleet co‑design with compress‑and‑route (2026)

FAQ

How do you optimise LLM cost?

Start by measuring cost per successful task, not raw spend, then apply the highest-return levers in order: match model choice to your workload’s input/output token ratio, add prompt caching and context compaction, and move anything delay-tolerant to a batch API. Studies on quality-aware routing report 40% to 90% cost reductions on the document-processing workloads they tested, though results vary by task.

How much does 1 million tokens cost with an LLM?

Pricing varies significantly by provider and model tier, and input tokens are typically priced separately from, and cheaper than, output tokens. Rather than quoting a single figure, check your provider’s current pricing page and calculate your specific cost using your actual input-to-output token ratio, since that ratio changes which model is cheapest for your workload.

How much does it cost to run your own LLM?

Self-hosting costs depend on GPU fleet size, model size, concurrency requirements, and how well your routing handles the context-length cost cliff described in FleetOpt’s fleet co-design research. Teams considering self-hosting should budget for both compute and the engineering time to build routing, caching, and observability, or evaluate a managed platform like Agentrelease if speed to deployment matters more than infrastructure control.

Which LLM is the cheapest to run?

There’s no single cheapest model. The right choice depends on your input/output token ratio, your quality floor, and whether your workload can use caching and batching effectively. A workload-shape analysis comparing sticker prices without modelling your actual token mix will routinely point you to the wrong model.

Does batching really cut LLM costs in half?

OpenAI’s Batch API supports large JSONL batch inputs with a completion window, making it well suited to enrichment, summarisation, and offline evaluation jobs, though live, latency-sensitive traffic can’t use this discount.