LLM costs rarely become a problem because of one spectacular request. They accumulate quietly: repeated questions phrased in slightly different ways, long prompts carrying unnecessary context, retries after timeouts, and routine tasks sent to a model that is far more capable—and expensive—than the work requires.
The corrective pattern is not a single cache bolted onto an API call. It is a decision layer that knows when an earlier answer is safe to reuse, when a smaller model is sufficient, when a frontier model is justified, and what the application should do when its preferred path is unavailable. The diagram above shows that control flow in its simplest useful form.
Start with the cost of a completed task
Token price is an input, not the operating metric. A cheaper call that produces more retries, manual review, or abandoned sessions can cost more than a larger model used once. Measure cost per accepted answer or completed workflow, then break it into prompt tokens, output tokens, embedding calls, cache operations, retries, and human intervention.
Segment the baseline by task family. Classification, extraction, search assistance, drafting, and multi-step reasoning have different quality and latency requirements. This workload map becomes the evidence for routing rules; without it, “use the smaller model first” is simply a guess.
Use exact and semantic caches for different jobs
An exact cache is appropriate when the normalized request, authorized context, model configuration, and policy version match. It is deterministic and easy to invalidate. A semantic cache looks for a previously answered request with a sufficiently similar meaning. It can capture more reuse, but it introduces a judgment: two prompts may look close in embedding space while requiring different answers.
Keep the cache key wider than the user’s text. Include tenant, locale, data or knowledge-base version, authorization scope, prompt template, model family, tool availability, and response schema. A hit is acceptable only if the stored answer is fresh for that task and was produced under a compatible policy. This prevents a fast cache from becoming a fast path around security or change control.
Confidence needs two thresholds
Treat similarity as one signal rather than a verdict. Use a high threshold for automatic reuse, a lower review band for tasks where verification is inexpensive, and a miss below that. Tune thresholds with real query pairs and measure accepted hits, incorrect hits, and missed reuse separately. A high hit rate is not a success if it hides the wrong answer.
Route by difficulty, risk, and available budget
Routing should begin with observable features: prompt length, requested operation, number of documents, tool requirements, language, expected output structure, and the business consequence of an error. Straightforward classification or rewriting can often use a smaller model. Ambiguous analysis, unfamiliar inputs, or high-impact recommendations may justify a stronger model or mandatory review.
Do not ask the model to be the sole judge of whether it needs a better model. Combine static policy with measured signals and controlled escalation. A small model can attempt the task, but schema validation, groundedness checks, confidence rules, or task-specific evaluators should decide whether to accept the result. Record the reason for every escalation so the policy can be improved instead of accumulating folklore.
Design fallback as a product behavior
Fallback is often reduced to “call a second provider.” In production, the useful question is what level of service the user still needs. A timeout might trigger a smaller model, a retrieval-only answer, a queued response, a read-only mode, or a request for human review. The right fallback depends on the task’s deadline and the cost of a plausible but weak answer.
Set a total latency budget and allocate it across cache lookup, retrieval, model inference, validation, and retry. Limit the number of attempts. Use circuit breakers when a dependency is failing, and ensure retries are idempotent when tools or business actions are involved. A chain of invisible retries can turn a brief provider issue into an expensive application-wide queue.
Protect the cache from leakage and poisoning
A semantic cache stores meaning-rich representations and generated answers, so it belongs inside the application’s trust boundary. Partition by tenant or enforce an equally strong row-level policy. Encrypt stored content, apply retention rules, and avoid caching secrets, regulated data, or responses whose authorization cannot be reconstructed at read time.
Only validated responses should enter the shared cache. Store provenance, policy version, source freshness, and evaluation status with each entry. Invalidate on knowledge changes and security-policy changes, not only on time-to-live. Test prompt injection and cache poisoning explicitly: an attacker should not be able to create a reusable answer that later appears trustworthy to another user.
Operate the routing policy with evidence
Track semantic and exact hit rates, accepted-hit rate, false reuse, cost per completed task, p50 and p95 latency, model escalation, fallback activation, and human override. Break these metrics down by task and tenant. Aggregate averages can conceal one expensive workflow or a language where the similarity threshold performs poorly.
A safe rollout begins in observation mode: calculate the route and cache decision while the existing path still serves users. Compare the proposed decision with the actual outcome. Then enable exact caching, introduce semantic reuse for one low-risk task, and add model routing only after the evaluation set can detect regressions. The architecture earns savings by making decisions repeatable—not by assuming every request is interchangeable.