Why AI-generated apps compile but do not work
· updated
AI-generated apps fail while every signal says success because compiling only proves the code parses — it proves nothing about whether the code does what was asked. A file containing a heading and a comment saying the rest goes here compiles perfectly. So does an app with a filter wired to nothing.
If you have used an AI builder for anything real, you have seen it: the code generates, it type-checks, the preview loads, and the thing does not do what you asked. Nothing errored. That gap is the whole problem, and it is worth being precise about why it happens.
Compiling only proves it parses
A compile check answers one question: is this valid code? A type checker answers a slightly better one: are the types consistent? Neither can answer whether the feature exists. Both are necessary and neither is close to sufficient, and the reason this catches people out is that they are the only checks most tools run.
This is why a stub is the worst failure mode available. An app that crashes tells you something is wrong. An app that compiles, loads, and quietly omits half the request tells you nothing at all — and the omission is usually the part you cared about, because the hard part is the part a model is most likely to skip.
How they actually fail
- Scaffold text left in place — a comment where a feature should be, shipped as if it were finished.
- A requested capability with no trace in the code at all. You asked it to save your data; nothing ever writes it.
- A component that builds an interface and never returns it, so the screen is blank.
- State nothing can change: a filter exists, the list is correctly filtered by it, and no control anywhere can set it. The feature is written and permanently unreachable.
- An input you cannot type into, because what it displays and what it updates are two different things.
- Imports of packages the project does not have, which is a compile error if you are lucky and a runtime failure if the name happens to resolve to something else.
Every one of those compiles. Most of them look right in a screenshot, which is why demos of AI builders are so much more convincing than using one.
Why models do this
It is not laziness, and it is not a bug that a better model fixes. A language model is producing the most plausible continuation of a prompt, and a scaffolded component with a TODO where the hard part goes is an extremely plausible piece of code — it appears constantly in real repositories, which is exactly where the model learned it.
The same logic explains the other failures. Wiring a filter to a control is more tokens and more chances to be wrong than declaring the filter; a plausible-looking component is easier to produce than a correct one. Left unchecked, generation optimises for looking finished. Something outside the model has to insist on the difference.
What Adrian checks beyond compiling
The compile gate runs first, and it is a real one: dependencies are installed and the project is type-checked before you see anything. If it fails, sources are repaired and it retries rather than handing you the error.
After that, Adrian reads the code it just wrote and checks it against the request. The checks are deterministic — they report what can be proven from the source, not a second model's opinion, because a judge that can hallucinate cannot be the thing that catches hallucination. That constraint is the design, not a limitation of it.
It looks for the failures above: leftover scaffold text, requested capabilities with no implementation, components that render nothing, state with no way to change it, controls that cannot be operated. It also checks the visual result for the tells that mark a generic template — default font stacks, untinted greys, elements left completely unstyled.
Checking for what is missing, not just what is wrong
The harder question is absence. A landing page with no footer is not an error anywhere — nothing is broken, something simply is not there — so Adrian scores the generated source against what the kind of app you asked for is expected to contain, and refines when the result falls short of the threshold.
What counts as expected depends on the archetype. A landing page is checked for navigation, a hero, content sections and a footer; a dashboard is checked for metric cards, a visualisation and a data table. Asking a dashboard for a footer would be noise, and a check that produces noise is a check people turn off.
What happens when it finds something
Findings are not a report you have to read. They are handed back to the coder as specific instructions — which state, which setter, which control is missing — and the build repairs itself before you see it. Then the same checks run again, so whether the repair worked is measured rather than assumed.
This is the part that matters most and gets described least. A tool that detects a problem and shows you a warning has moved the work to you. The loop only pays for itself if the finding goes back into the pipeline automatically, and if the re-check is the same check rather than a softer one.
What this does not claim
This is a floor, not a guarantee. Passing every check means nothing provable is wrong — it does not mean the app is good. A check that cried wolf would get switched off and then catch nothing, so each one stays silent unless it can prove the problem from the code itself. That is a deliberate trade: fewer findings, all of them real.
The point is not that generated code is now perfect. It is that the obvious, checkable failures should never reach you with a success message attached. The desktop app is free if you want to see the gate run on something of your own, and this walks through the whole pipeline.