“Distribution” error: Why AI fails at engineering

For example, even if you use the strongest model and ask it something about engineering, and you’re like me pretending to be an engineer, you might look at what comes back and think, ‘wow, that’s amazing.'

But if you’re an actual expert engineer, that’s not the case. You’ll find a number of things you would never write or do. So models ‘kind of’ know engineering, but don’t actually know engineering.

If as a user you don’t know the domain, you’re likely to think it’s correct. And that’s very dangerous.

From the beginning, the idea was that if we had a model that was superhuman at coding and maths, it would just generalise beyond the world of coding and into other areas like law, architecture and engineering. It was a solid idea, but it turns out that’s not happening yet.

Coding is the only task in the world right now that’s in distribution. The singular focus on coding has been a real blind spot in AI.

When it comes to engineering, for instance, a calculation is not just a calculation. It belongs to a drawing set, a schedule, a climate zone, a standard, a collection of assumptions, and a downstream chain of decisions that may depend on it being right for this specific case and not some nearby case.

In that environment, generic competence is not enough. It requires understanding -- in both a visual and multimodal way -- of why things are not in the right place. 

These kinds of tasks are very simple for humans, but very hard for current AI models.

That’s why you need to embed human expertise directly. Tell the model what not to do, define the boundaries of what it doesn’t know, and make explicit the order of operations, checks, guidelines and logic an expert already follows.

It’s tempting to treat engineering as one more expert domain that large models will gradually absorb as they become smarter and better trained. There is some truth in that. Engineering work is full of technical language, quantitative reasoning, standards, and procedural knowledge, all of which are at least partially representable in text, math, and code. But that description misses the thing that makes the domain difficult in practice.

We need task-specific data. Think of a Word document. Fifty people collaborate. Comments everywhere. What do we keep? The final version. All that history of collaboration is gone.

From a model’s point of view, that missing history is everything — understanding both what to do and what not to do, what changed, what failed, and why something was rejected.

I think this concept of ‘humans in the loop’ that we’re hearing more and more often really reveals our backwards thinking about human–AI collaboration. It implies humans are passive validators, like quality checkers on an AI assembly line.

“What we actually want is humans driving the loop.

If you design the workflow properly, you embed that expertise directly. This approach also helps ensure expertise and skills don’t atrophy over time. Instead, people can sharpen their thinking and problem solving through this process.

If engineering and AEC are going to benefit meaningfully from AI systems, it needs people who understand the domains, the artifacts, the review practices, and the economics of real workflows to help define what good looks like.

Before this becomes a field of grand claims about what AI can do, it should become a field of good experiments, good environments, and good judgment about where these systems can actually be trusted to deliver.