In the Token Stream: Architectural Options for Monetizing AI

Model providers have a business model tied directly to AI usage. As customers use more inference, they make more money. For those of us building on top of those models, that relationship is not always obvious. But look across the AI landscape and you’ll see legions of startups designing their architectures to create similar economics. Here’s how they’re doing it.

You can think of this as getting in the token stream. The most literal version is reselling inference and putting a margin on every token, but there are many ways to tie your revenue to inference without becoming a model provider yourself.

To make this work, you have to decide where your product sits relative to the model, the agent runtime, and the rest of the application stack. You can own the path through which inference runs, own the runtime that creates it, or integrate into a stack controlled by someone else. Here are the primary options.

Put Inference Inside the Product

The most common architecture is also the simplest. You build an application or agent that uses inference, and customers pay to use the product. Anyone in the AI space is familiar with this approach.

Cursor and Devin are straightforward examples of this architecture. Customers pay for an AI product that performs work, while the product consumes model inference underneath.

The customer might pay per seat, per task, through usage-based pricing, or as part of a subscription. In practice, heavier users often move into higher-priced plans or usage tiers, so revenue can still rise with the amount of inference the product supports. Underneath that pricing, the product calls models to do the work. More customer usage creates more inference, and the inference becomes part of the cost of delivering the product.

The customer does not need to know how many tokens or model calls were involved. They are paying for the product and the work it performs.

This is the baseline architecture. It creates a direct way to get into the token stream without selling infrastructure at all. The application owns the customer relationship and decides how much inference to use to deliver the product. If a task can be completed with a cheaper model, fewer tokens, better caching, or a more efficient agent loop, the cost of delivering the same product goes down.

Put a Gateway in Front of the Models

One of the most direct architectures is to put a gateway in front of the model provider, and many are trying to do this. Instead of an application calling OpenAI, Anthropic, Google, or another provider directly, it calls the gateway, which forwards the request and returns the result.

Portkey is a direct example of this architecture. Its AI Gateway sits between applications and model providers and provides routing, fallbacks, load balancing, retries, caching, and other controls across providers.

Cloudflare AI Gateway takes a similar position in the request path. Applications route AI requests through Cloudflare, which can then provide analytics, caching, rate limiting, retries, model fallback, and other controls without owning the underlying models.

A thin version of this architecture allows customers to bring their own model credentials and pay the gateway for usage. Alternatively, the gateway can resell the underlying inference and include its margin in the price.

A fatter version may be the enterprise policy gateway. Microsoft Azure API Management’s AI gateway applies policy controls to AI endpoints and supports token limits, load balancing, logging, content-safety policies, and governance across model backends.

An enterprise can make a gateway like this the controlled route between its applications and external models. The gateway is then doing more than transporting tokens. It’s an enforcement point for enterprise inference, while its usage still grows naturally as more inference crosses that boundary.

Own the Runtime That Creates the Inference

This architecture is one level above individual model calls. Instead of sitting between the application and the model, the infrastructure company owns the runtime that decides when and how those calls happen. The opportunity for margin lives inside those decisions.

Amazon Bedrock AgentCore Runtime is one example of this. Developers deploy agents into the runtime, and AgentCore provides the execution environment in which those agents run rather than simply proxying an individual model request.

LangSmith Deployment provides another version through managed infrastructure for deploying and running agents built around LangGraph. The runtime is responsible for executing longer-lived agent work around those model calls.

The customer asks the runtime to execute an agent, complete a task, or run a workflow, and the runtime determines how to do the work. One request might require a single model call while another requires twenty, with the runtime choosing models and executing tools along the way. It owns the execution until the work is complete.

This gives the runtime provider an opportunity to control more of the economics. The customer pays for execution performed by the runtime, while the underlying inference may be billed separately or bundled into the runtime’s higher-level unit of work.

Bundling inference opens up another source of margin. Better model selection, caching, compilation, and other execution improvements can lower the cost of producing the same work. A runtime that sells a higher-level unit of work can therefore improve its margin as the underlying execution gets more efficient.

Insert Middleware Into Someone Else’s Agent Loop

Owning the runtime is powerful, but it is not required. Agent frameworks are exposing more places for outside infrastructure to participate in their loops.

An agent runtime natively calls models, executes tools, updates state, and decides what happens next. Frameworks such as the OpenAI Agents SDK, Google ADK, Semantic Kernel, and LangChain middleware now expose callbacks, filters, wrappers, and middleware around these operations, giving outside products explicit places to integrate into the loop.

OpenAI, for example, exposes lifecycle hooks around model and tool calls, while LangChain includes hooks such as 'before_model', 'after_model', 'wrap_model_call,' and 'wrap_tool_call.'

That allows an external product to get installed into a loop owned by somebody else. A security product might inspect a model request before it is sent. Or an evaluation system might judge an output before the agent continues. You could build an external policy system that decides whether a proposed tool call is allowed.

Some frameworks now expose these controls quite explicitly. OpenAI’s Agents SDK has a 'callModelInputFilter' that can modify model input immediately before invocation, Semantic Kernel has function and prompt filters, and Vercel’s AI SDK language-model middleware can wrap a model and intercept or modify calls before they reach the underlying provider.

The economics can still track inference closely. A third party can participate in thousands of agent loops without building the harness that owns any of them, provided the frameworks make enough of the loop extensible.

Put Inference Behind a Tool

We often say the model is the brain and tools are the hands, with the model doing the thinking while tools perform deterministic actions in the outside world. That analogy has been slowly breaking down with what some might call “hands with brains.”

Granola’s MCP server provides a good example. It exposes conventional tools for listing meetings, retrieving meeting details, and fetching transcripts, but it also exposes query_granola_meetings, which Granola describes as “Chat with Granola” and uses to answer questions based on meeting history.

The outer agent may be Claude, ChatGPT, Codex, or another MCP client. It decides that it needs something from Granola, calls the tool, and Granola performs whatever retrieval and inference are required before returning the result.

Firecrawl’s MCP server provides an even clearer version of the architecture. Its 'firecrawl_agent' tool can take a research request, search and navigate the web, read pages, and assemble structured results across multiple sources.

In some ways, the original brains-and-hands analogy was convenient for frontier model providers. It presented an architecture where their model was the only intelligent thing in the block diagram, and everything around it was a capability to be called. But the reality is that intelligence is increasingly distributed.

The outer model is no longer simply using a hand. It is delegating part of the thinking to another system that has its own models, context, data, and execution logic. MCP gives those services a standard way to present themselves to many different agent environments.

This architecture also has different economics from middleware. The vendor does not need to touch every model call made by the outer agent because it creates its own inference whenever the agent decides to delegate work to it.

The Economics Are Still Unclear

More value is being added on top of models than ever before. That value should rightly be monetized by the companies building it.

We are still figuring out what those economics should look like. I expect we will discover even more architectures for tying revenue to AI usage as the stack matures.

There is also a risk in how many companies are chasing the same pool of inference spend. If every layer wants to add its own margin to the token stream, the combined markup can get large quickly.

Getting into the token stream is only part of the job. Your product also has to add enough value to justify the additional spend. Customers may accept a substantial premium for infrastructure that materially improves the product, but there is probably a limit to how many tolls they will tolerate on the way to the model.