How Much of Your Architecture Is a Temporary Model Workaround?

AI builders are spending enormous amounts of time building around limitations in the models that may disappear in the coming years. Which ones are safe to build on? And how much of what we’re building today will turn out to have been architectural workarounds?

Some of the hardest engineering problems are caused by limitations in the models themselves. Context is scarce and inference is expensive. Agents forget things. If you give them a long-running task, they tend to go off the rails without a very smart harness. Even getting a model to reliably use tools requires a surprising amount of machinery around it.

We are all building around these limitations even if we don’t realize it. We retrieve information because models can’t hold everything in context, and we give agents memory because they otherwise forget what happened before. We break large jobs into smaller ones because models struggle to stay coherent over long stretches of work. If one agent can’t reliably handle the whole job, we divide the work among several.

There is nothing wrong with doing any of this. These are real problems today. But they create an unusual architectural risk. We may be building increasingly elaborate infrastructure around constraints that are moving underneath us.

The key question is which limitations are temporary properties of today’s models and which represent real boundaries that AI systems will still need years from now.

Practically Unlimited Context

Context windows are an obvious place to start because their trajectory is clear. We went from a few thousand tokens to hundreds of thousands and then millions remarkably quickly. At the same time, improvements in attention and inference are making those larger windows more practical to use.

No one knows exactly when we'll get to hundreds of millions or a billion tokens, but are headed in that direction. At that point, context becomes abundant enough that application architecture changes. An agent might eventually be able to work with an entire repository alongside a large amount of its project history all at once. It could have access to previous work without someone carefully selecting a handful of fragments to place in its prompt.

There will always be more information somewhere, and attention itself may remain scarce even if tokens aren’t. But the role of retrieval could change considerably. At that point, we may start investing more in helping the model navigate a working set it can already access rather than in retrieval algorithms.

Agents That Don't Forget

Memory has become one of the most active layers of the agent stack. There are now companies building entire products around giving agents persistence. There is an entire ecosystem of technology emerging for turning what happened during previous runs into reusable knowledge and skills.

Some of this is clearly compensating for the fact that models are effectively born again every time they are invoked. Whatever happened during the last run has to be reconstructed and supplied again. That makes external memory essential.

But that boundary isn’t guaranteed to remain where it is. Models and model systems are getting better at maintaining long-running context and preserving useful state across interactions. Some experimental models can already compact their histories as they work. Research into test-time learning goes further by exploring whether "experience" can actually change future model behavior.

That doesn’t mean memory infrastructure goes away. But there is an important difference between memory that exists because the model forgets and memory that exists because the organization needs to remember.

I would be much more comfortable building around the second.

Dependable Long-horizon Orchestration

A lot of agent architecture assumes that models can’t be trusted with too much work at once. So we decompose tasks and build graphs of work. We create specialized agents for pieces of the job and retry them individually when they fail. Increasingly sophisticated orchestration layers have grown around this basic constraint.

These techniques are needed because they address a real weakness in the models. Agents can perform amazingly well on bounded problems. Give them a long, open-ended one and all bets are off. Sometimes it produces amazing results and other times it just wastes hours of tokens.

But reliable task horizon is improving. If that trajectory continues, the natural unit of work we give a model gets larger. Work that currently needs to be divided into twenty carefully orchestrated steps may eventually be something we hand to a model as a single assignment and let it decompose for itself.

This doesn’t necessarily eliminate orchestration. A software factory, for example, still has good reasons to separate specification from implementation and verification. But software factories will need to focus more on encoding how organizations work rather than compensating for deficits in the model.

Cheap and Commoditized Inference

Another architectural assumption hiding everywhere is that inference is expensive. We design around that cost in ways that have become almost automatic. We cache aggressively and route work between models tiers. Or sometimes we build deterministic machinery for a task largely because asking a capable model every time would be too expensive.

This assumption may not hold up. Hardware is improving and model architectures are getting more efficient. And the open weight models are competing here in a way that will drive costs down.

Imagine capable inference becoming another hundred times cheaper. Suddenly it may be perfectly reasonable to ask a model to interpret an API response rather than maintaining brittle transformation logic. A workflow might make dozens of small semantic decisions along the way instead of trying to minimize every model call.

The economic boundary between deterministic code and inference could move substantially, and architecture built primarily around minimizing model calls may age surprisingly quickly.

Models With Built-In Harnesses

Today’s models live inside increasingly sophisticated harnesses. The harness is responsible for much of what makes an agent feel like an agent in the first place. It assembles context, exposes tools and keeps the execution loop moving. As agents have become more ambitious, harnesses have taken on more responsibility for keeping them on track.

There is already pressure on that boundary from both directions. Agent frameworks are becoming more sophisticated, but model providers are also pushing more agent behavior closer to the model and its APIs. Models are increasingly trained to use tools and operate computers. They are also getting better at recovering from the consequences of their own actions over longer interactions.

Over time, capabilities we currently consider features of the harness may become ordinary properties of the model interface. A future model might manage much of its own working context and sustain extremely long execution loops without the application implementing those capabilities independently. Parallel work and tool use could move further down the stack as well.

The harness isn’t going away, but it's a moving target.

Work Risks Becoming Opaque

There is a downside to models absorbing more of the machinery we currently build around them. The more context management, memory, planning, and decomposition happen inside the model, the less of the work may be visible to the systems around it.

Today, a lot of this work is externalized almost by accident when an orchestrator creates explicit steps or a memory system records what information was brought into context. Or we can see the work by inspecting the outputs that our subagents produced. These systems exist partly because of model limitations, but they also leave behind useful evidence of how the work happened.

But if all this is happening inside the model in hidden state, we also lose something. Soon we can give the model a huge working context and an objective, and it might spend hours navigating information, decomposing the problem, revising its approach, and carrying forward what it learns. The result could be a much simpler architecture, but also one where far more of the valuable work happens inside inference and disappears when the run is over.

This creates an interesting tension between frontier model innovation and what AI builders may actually need. Frontier labs have every reason to push more context management, planning, memory, and decomposition into the model. But as more of that work moves inside inference, application builders risk losing access to the intermediate state that makes the work understandable and reusable.

As models become capable of doing more internally, we may need to become much more deliberate about deciding what they are required to leave behind.

Build for a Moving Target

The improvements we can see coming are probably only part of the story. Some of the model capabilities that change how we build over the next few years may be things we aren’t even thinking about yet. A few could make entire categories of infrastructure less important (or more important) almost overnight.

That is part of what makes building in this moment so interesting and so difficult. We are going to have to keep adapting, throwing away some of what we build, and inventing new layers around capabilities we don’t have yet.