If You’re Not Doing These 4 Types of Caching, You’re Wasting Tokens

I see a lot of AI builders spending enormous amounts of time optimizing prompts and skills while leaving money on the table when it comes to token efficiency. We shorten instructions, remove examples, reduce tool descriptions, and obsess over context windows. All of that can help, but a bigger question is whether the model should be processing those tokens again in the first place.

That starts with your caching strategy. For a reasonably sophisticated agent system, there are at least four kinds of caching you should be thinking about: prompt caching, tool response caching, output caching, and structured data caching.

They operate at different layers, but they all follow the same principle: do not pay the model to repeatedly process information or perform work that you already have.

1. Prompt Caching

This is the one people are increasingly familiar with because the model providers themselves offer it. The basic idea is simple: if a large portion of your input is identical to something the provider has processed recently, it may be able to reuse some of that computation instead of starting over. OpenAI, Anthropic, and others each have their own mechanics, pricing, and retention rules, but the architectural lesson is the same.

Your prompts need stable regions. For prefix-based caching in particular, static content should come first and dynamic content later. That means your prompt builder should also be deterministic. If 30 tool definitions have not changed, do not arbitrarily reorder them. If ten pieces of system context are identical, do not let retrieval order, timestamps, or implementation details move them around between requests. A semantically identical prompt is not necessarily a cache-identical prompt, and one small change early in the prefix can wipe out the savings for everything that follows.

You can spend hours shaving a few hundred tokens off a system prompt while accidentally destroying cache reuse across thousands of requests. Prompt optimization should therefore include prompt stability, not just prompt length.

2. Tool Response Caching

Provider prompt caching does not solve another common source of waste: agents repeatedly rediscovering information they already retrieved. Start a new conversation or agent session and the agent often checks the repository again, reloads project configuration, reads the same wiki page, calls the same customer API, or queries the same database just to rebuild enough context to continue.

If nothing changed, your harness should not force the agent to rediscover it. There is an obvious savings from avoiding the external API or tool call, but there is also a second savings that may be larger: eliminating an LLM round trip. A typical agent loop goes model → tool call → tool execution → model again with the result. That second model invocation usually means replaying the context plus the tool response, so avoiding the round trip can save a surprising number of tokens.

I increasingly think this should be the harness’s problem. The harness should know what information has already been materialized, what it depended on, how fresh it is, and whether it can still be reused. When the agent needs that information, the harness can inject it just in time rather than requiring another tool call. A wiki page might be keyed by revision, a repository file by commit SHA, a database result by a short TTL, and a customer profile by the events that invalidate it.

Think of the harness as a just-in-time context materializer. The interesting engineering problem is not storing the response. It is knowing when that response is still valid.

3. Output Caching / Artifact Caching

Now take the same idea one level higher. If you give a model the same instructions, the same source material, the same supporting context, and the same model, should you perform that work again?

My rule of thumb is: if the model has not changed and the inputs have not changed, the output should be roughly the same, or at least close enough that you should seriously question why you are recomputing it. I am using inputs loosely here to mean the full effective context of the operation: instructions, source material, selected knowledge, tool results, upstream artifacts, configuration, and anything else that meaningfully affects the result.

The hard part is that agent systems are often terrible at defining those inputs. Context gets assembled dynamically, retrieval results change, conversations accumulate state, and hidden parts of the harness alter prompts. Two runs that appear identical at the application layer may actually be different model calls. The better you get at identifying and reproducing the effective context for a unit of work, the more model output starts to look cacheable.

This is one of the architectural ideas behind ThruWire. Model work produces artifacts with explicit dependencies, and the runtime has a structured representation of the work and the context that contributed to it. That makes it possible to determine that an intermediate artifact is still valid for a particular set of inputs and reuse it instead of blindly rerunning the model.

You can think of this as memoization for agentic work. It also changes how you structure the work itself. Instead of one giant agent invocation that does ten things, smaller units of model work can have understandable inputs and reusable outputs. If step seven changes, you should not necessarily have to pay to rerun steps one through six.

4. Structured Data Caching

There is one more kind of caching that I think is badly underused in agent systems: structured data like SQL query results. Client memory is cheaper than tokens. Be liberal about keeping useful data around and frugal about actually dragging that data into the model’s context.

Imagine an agent retrieves 200,000 customer records. You could serialize a giant subset to JSON and stuff it into the prompt because the model might need it. A better approach is to keep that dataset in a locally queryable format and give the agent a tool for asking precise questions about it. Parquet or Iceberg can work well here, with @duckdb providing a lightweight query layer over the data.

Instead of putting the whole dataset into context, give the agent a local query tool and let it request only what it needs. The model might ask for the top 20 high-revenue customers with elevated churn risk, and only those rows cross the model boundary. You keep a rich representation of potentially useful information close to the agent while delaying tokenization until the last possible moment.

That distinction is important. Agent memory should not mean putting everything the agent might remember into its prompt. Memory should mean making information cheaply addressable when the agent needs it. The context window is a working set, not a database.

Cache the Work, Not Just the Prompt

Once you look at agents this way, token efficiency becomes a systems problem. A good agent harness should constantly be asking whether it has already processed something, already retrieved it, already produced it, or actually needs to put it into context yet.

Prompt engineering and skill design still matter, but there is a more fundamental optimization underneath both of them.

Be caching rich and token poor.