I’ve Seen This Movie Before: What AI Builders Can Learn From Data Engineering

AI feels new, but many of the problems surrounding it are familiar. We are coordinating complex work, moving information between systems, managing change, and trying to make powerful technology useful to people with very different levels of technical expertise.

Data engineering has spent decades working through versions of these problems. Probabilistic models introduce genuinely new challenges, but they do not make every architectural question new. We should carry forward the patterns that already work rather than rediscovering them through avoidable failures.

After decades working with data systems and now building with AI, I keep seeing the same architectural lessons reappear in a new form. Four of them continue to shape how I think about AI systems.

1. Graphs Are Your Friend

Data engineering has used graphs for a long time. Query planners turn SQL into execution graphs, orchestrators represent jobs as directed acyclic graphs, and lineage systems track how changes in one dataset affect downstream models and dashboards. Tools such as dbt define transformed datasets as nodes connected by explicit dependencies.

Graphs are useful because complex work is rarely a simple sequence, nor is it a simple loop. Some operations depend on others, some can run in parallel, and some should execute only under specific conditions. A graph makes those relationships visible and executable.

AI applications often have the same shape. A system might retrieve information, classify a request, generate several candidate outputs, evaluate them, request approval, and produce a final artifact. That work already forms a graph, even when it is presented as a single agent loop.

The lesson goes beyond choosing a graph-based orchestration framework. The orchestration itself should usually be deterministic, even when the work inside a node is probabilistic. The model can interpret, synthesize, and generate, while the system defines which steps exist, what they receive, what they produce, and what depends on them.

This creates a practical boundary between flexibility and control. Agent loops remain useful when a task requires exploration or flexible tool use, but they should not automatically become the architecture for the whole system. Data engineering already learned the cost of hiding important dependencies inside procedural logic, and AI does not need to repeat that process.

2. Extract, Transform, and Load Are Different Jobs

Data engineering separated the movement of information into distinct phases: extract, transform, and load. The order has evolved, with many modern platforms favoring ELT, but the core lesson remains that retrieving information, changing it, and persisting the result are different responsibilities.

There is a direct analogy in AI. Extraction brings new context into the system through tool calls, APIs, databases, document retrieval, search, or other services. It determines what information enters the process and where it came from.

Transformation is where the model does its work. It may classify, summarize, compare, interpret, or synthesize. This phase receives the most attention because the model is the novel component, but its quality depends heavily on what was extracted and how clearly that context was represented.

Loading means persisting the useful result somewhere durable. The output might be a structured record, an intermediate artifact, a decision, an annotation, or a finished deliverable. Once stored, that result can become input for later AI calls, so what is loaded today may become context extracted tomorrow.

Many current AI applications collapse all three jobs into one chat session. Information is retrieved into a prompt, transformed by the model, and returned in a conversation, but the useful result may never be persisted outside the transcript. The boundaries between source material, model interpretation, and authoritative context remain unclear.

That approach works for isolated interactions, but it becomes harder to manage when work continues across users, sessions, and systems. AI needs a durable context layer that can hold sources, intermediate artifacts, decisions, and finished outputs in an organized, queryable form.

Separating these jobs also makes failures easier to diagnose. We can ask whether the correct information was retrieved, whether the model transformed it well, and whether the result was stored appropriately. When everything is combined inside one opaque loop, every failure starts to look like a model failure.

3. Change Detection Is Key

Data systems are constantly changing. Records are added or updated, schemas evolve, and business logic changes. Data engineering developed techniques such as change data capture, incremental processing, materialized views, lineage tracking, and selective recomputation to keep downstream systems consistent.

All of those techniques address the same question: what changed, and what does that change affect? AI systems need to answer it for context, artifacts, decisions, and generated outputs.

Suppose a company policy changes. Which procedures relied on the old version, which summaries are now stale, and which decisions should be reviewed? Which cached transformations remain valid, and how far should the update propagate?

The same problem appears when a customer record changes, a document is corrected, a human revises an intermediate artifact, or a model produces a better version of an earlier result. The system needs to understand both the change and its downstream consequences.

Without explicit change detection and dependency tracking, AI systems tend toward two poor outcomes. They either trust stale context because they do not know what became invalid, or they rerun everything because they cannot identify what remains valid.

Caching, replay, and propagation are all parts of this larger problem. A cache is useful only when the system knows whether its result still applies. Replay is efficient only when it can begin from the affected point, and propagation is safe only when dependencies are explicit.

AI artifacts therefore need identity, history, and lineage. The system should know which inputs shaped an output, which versions were used, and which downstream artifacts depend on it. A generated result should be treated as part of a connected system of work rather than an isolated block of text.

Data engineering moved beyond full refreshes because they were expensive and unnecessary. AI systems should not have to learn the same lesson through large bills, slow execution, and stale outputs.

4. Democratization Creates New Interfaces and New Roles

Data became more useful as the industry created different interfaces for different participants. Data engineers built pipelines, analytics engineers created governed transformations, analysts explored data, and business users consumed dashboards, reports, and self-service tools.

Not everyone interacted with the data platform in the same way. The ecosystem developed layers of abstraction that allowed more people to participate without requiring everyone to understand the underlying infrastructure.

AI is currently centered on chat. Chat is flexible and broadly accessible, but it is unlikely to remain the only interface through which people participate in AI-enabled work. Some users will want conversation, while others will need structured forms, workflows, studios, workbenches, dashboards, editors, or approval interfaces.

A person approving a financial decision needs a different experience from someone brainstorming ideas. A domain expert curating shared knowledge needs different controls from an end user requesting a summary. Democratization means designing interfaces that match each participant’s responsibilities and expertise, not giving everyone the same blank text box.

The data industry did not reach broad adoption by forcing every participant into SQL or asking every business user to operate a pipeline. It created tools and abstractions for different kinds of work, and AI should expect a similar progression.

New roles may also emerge. The rise of the analytics engineer created a category between traditional data engineering and business analysis. Analytics engineers organized transformations, defined reusable models, applied software engineering practices to analytical logic, and helped make data trustworthy across an organization.

AI may produce a similar role around context. A context engineer might organize the information AI systems depend on, define relationships between sources and derived artifacts, establish what is authoritative, and determine how changes should propagate.

The title may change, but the responsibility is likely to remain. As AI becomes embedded in organizational work, someone will need to take ownership of the structure and quality of the context surrounding the model. Prompting alone does not cover that responsibility, and existing engineering or product roles may not cover it completely either.

The Patterns Are Familiar

AI introduces capabilities that data engineering never had to account for. Models can interpret ambiguous requests, generate new material, work across unstructured information, and perform transformations that cannot be reduced to conventional rules.

Those capabilities make the surrounding architecture more important. We still need to represent dependencies, separate retrieval from transformation and persistence, understand what changed, and design interfaces for different kinds of participants.

There will be plenty of genuinely new AI problems to solve, so we should avoid spending unnecessary time relearning the systems lessons that already have good answers.