Inference Economics: Caching, Small-Model Routing & Token Optimization

Deploying a single chatbot is relatively cheap. Deploying a multi-agent swarm that executes 8 reasoning steps per user request, calls 4 search tools, and processes 50,000 tokens of context per run can rapidly inflate your monthly cloud AI bill into five or six figures.

High costs also correlate directly with high latency: waiting for a massive 400B model to compute 5 intermediate tool routing calls creates a sluggish, unusable customer experience.

In this post, we explore Inference Economics for Agents: implementing semantic caching, dynamic model cascading (routing cheap 8B models for tools and reserving flagship models for reasoning), and leveraging prompt prefix caching.

1. The Cost Pyramid: Where Agent Tokens Go

┌─────────────────────────────────────────────────────────────────────────┐
│                      AGENT TOKEN CONSUMPTION BREAKDOWN                  │
│                                                                         │
│   45%  System Instructions & Re-Sent History across Multiple Steps      │
│   35%  Uncompressed Tool Observations & Vector Documents                │
│   15%  Intermediate Chain-of-Thought Scratchpad Reasoning               │
│    5%  Actual Final Answer Displayed to the User                        │
└─────────────────────────────────────────────────────────────────────────┘

Notice that 95% of your token costs are paid for internal plumbing and intermediate steps, not the final answer. Optimizing internal steps yields dramatic financial returns.

2. Dynamic Model Cascading (Small-to-Large Routing)

Not every node in a LangGraph workflow requires an expensive flagship model like GPT-4o or Claude 3.5 Sonnet.

• Trivial Tasks: Categorization, routing decisions, tool selection with simple arguments, and factual extraction can be executed flawlessly by modern compact models (e.g., Llama 3 8B, GPT-4o-mini, Claude 3.5 Haiku) at 1/20th the cost and 1/4th the latency.
• Complex Tasks: Multi-step mathematical reasoning, ambiguous code synthesis, and final synthesis warrant flagship models.

from typing import Dict, Any

class CascadingModelRouter:
    def __init__(self, cheap_client, expensive_client):
        self.fast_model = cheap_client       # e.g., gpt-4o-mini ($0.15 / 1M tokens)
        self.heavy_model = expensive_client   # e.g., gpt-4o ($2.50 / 1M tokens)

    def route_and_execute(self, task_type: str, messages: list) -> str:
        """Dynamically assigns model tier based on task complexity."""
        if task_type in ["tool_routing", "document_grading", "intent_classification"]:
            # Route to fast/cheap model tier
            return self.fast_model.invoke(messages)
        else:
            # Route high-consequence synthesis to flagship tier
            return self.heavy_model.invoke(messages)

3. Semantic Caching with Redis

Traditional key-value caches fail for LLMs because natural language queries vary slightly ("What is our refund window?" vs. "How many days do I have to return an item?").

Semantic Caching embeds the user query and searches past cache records using vector cosine distance. If similarity exceeds a threshold (e.g., $\ge 0.96$), the cached agent trajectory is returned instantly:

import redis
import json
import numpy as np

class SemanticAgentCache:
    def __init__(self, redis_client, embed_fn, similarity_threshold: float = 0.96):
        self.redis = redis_client
        self.embed_fn = embed_fn
        self.threshold = similarity_threshold

    def get(self, query: str) -> dict:
        query_vector = self.embed_fn(query)

        # In production: Query Redis Vector Similarity Search (HNSW index)
        # If nearest neighbor cosine similarity >= threshold, return cached response
        # Latency: ~5ms | Cost: $0.00
        return None  # Cache miss fallback

    def set(self, query: str, final_response: str, tools_used: list):
        query_vector = self.embed_fn(query)
        # Store vector and payload with TTL (e.g. 24 hours)

4. Exploiting Prompt Prefix Caching

Modern frontier API providers (Anthropic, OpenAI, DeepSeek) provide automatic or explicit Prompt Caching.

When you send an identical prefix (e.g., 5,000 tokens of system instructions and tool schemas) across consecutive steps, the provider charges up to 90% less for the cached input tokens and computes them with near-zero latency.

The Immutable Prefix Rule:

To maximize prompt cache hits, never inject dynamic variables into the top of your system prompt.

❌ BAD (Breaks Cache every call):
System Prompt: "The current timestamp is {timestamp}. The user is {user_id}. You are an auditor..."

✅ GOOD (100% Cache Hit Rate):
System Prompt: "You are an auditor. [Follow standard instructions]..."
User Message: "<session_metadata timestamp='...' user_id='...' />"

Summary & Next Steps

High-performance AI engineering is as much about economics as it is about algorithms. By cascading cheap models for routing, implementing semantic caches, and structuring prompts for prefix caching, you can run high-capability agent swarms at enterprise margins.

Now, it is time to bring every concept from this entire series together into a unified, functional system.

In [Post 17: End-to-End Capstone: The Autonomous Financial Analyst], we build an enterprise agent incorporating tools, hybrid RAG, LangGraph state machines, HITL approval, and LangSmith tracing into one comprehensive codebase.