Brainless Geniuses

LLMs can do almost anything now. Write code, break down a contract, discuss philosophy, take an exam for you. And they do it well enough that, from a layperson’s view, it’s natural to feel we have finally gotten artificial intelligence.

Whatever it looks like, it is just an algorithm of staggering scale — one that convinces us it’s something it isn’t.

Take it down to the simplest case. Picture the equation 2 * x = 4. A human solves it like this: knows what multiplication is, knows what equality is, deduces that x = 2. The human has concepts — number, operation, equality.

A neuron inside an LLM doesn’t work that way. It has no concepts at all. During training it has x (a weight, some number) and pairs of "input → expected output" pulled from terabytes of text. The training algorithm nudges that x back and forth, millions of times, until its value produces an output close enough to the expected one. The neuron does not learn that x = 2. It arrives by brute fitting at a number that statistically works.

Now multiply that by hundreds of billions of neurons, each with its own x, wired into a giant graph. Individually, none of them knows anything. But the whole network in aggregate is tuned so that the input "the capital of France" produces "Paris" — because in the training data, that's what worked millions of times.

So when you ask an LLM 2 + 3, it doesn't know what 2 is, what 3 is, or what plus is. It isn't computing. It's generating an output that statistically fits inputs of that shape. It usually returns 5, because in the training data "2 + 3 =" was followed by 5 most of the time. On slightly harder arithmetic, it confidently gets things wrong — and that's not a bug, that's the architecture.

The most expensive mistake of the past three years: confusing coherent talk about the world with knowledge of the world. LLMs do the first phenomenally. The second they cannot do, and architecturally cannot.

To talk about a system’s limits, you need to understand what the system does.

An LLM is a neural network trained on one task: predict the next token in a sequence. A token is a fragment of a word, usually 3–4 characters. When you write “The capital of France is “, the model is not “recalling” the answer. It computes a probability distribution over all possible next tokens, and the token “Paris” comes out with the highest probability — because in the training corpus, that token followed the phrase “the capital of France” more often than any other.

That’s all. No logic, no reasoning, no understanding. Just a statistical approximation of how text is distributed.

And here is where it gets interesting. If the corpus contains enough mathematical proofs, legal arguments, source code, philosophical essays — the model learns to reproduce the structure of those texts. Not the subject, the form. The result is a strange hybrid creature: it speaks correctly about mathematics because it learned to imitate mathematicians’ texts, but it does not actually possess mathematics in the sense that a human or Wolfram Alpha does.

How this differs from classical AI:

The metaphor closest to the truth: imagine a person with perfect memory who has read every book in a public library. They do not remember the books — they remember how texts on any topic sound. Ask them about the French Revolution, and they will speak in the style of textbooks on the French Revolution. Sometimes it will be true. Sometimes it will be a plausible-sounding fabrication — because they cannot tell the two apart. The sources are gone. All that is left is the averaged echo of every source at once.

That is what an LLM is. A very persuasive amnesiac who has nothing left but a sense of style.

And one more thing, without which nothing further in this piece works. A general-purpose LLM differs from every prior AI system in one respect: the universality of its interface. Stockfish knows chess and cannot speak about it outside a game. AlphaFold knows proteins and is silent about everything else. LLMs were the first to deliver natural language as a universal API to anything. If you can talk, you can use it. This is the discontinuous innovation that made more startups appear on top of LLMs in two years than appeared on top of classical AI in the previous twenty.

But that very universality is the source of the instability. A narrow system either works or it doesn’t. A universal system always returns something. If the question is about Uruguay’s 1934 football statistics, on which the training corpus contained a sentence and a half, the model still produces a coherent answer. Coherent, expansive, grammatically correct — and often entirely fabricated. This is called a hallucination, and it is not a bug. It is the default operating mode. The hallucination and the correct answer are generated by the same mechanism. The model does not distinguish between them internally. Confidence without calibration is not intelligence. It’s confabulation.

Now a technical detail without which nothing further in the piece works.

When a model goes through pretraining — the phase where it reads terabytes of text — its “knowledge” is not written into a database. It is encoded in the weights: billions of numbers in the network’s matrices. A single fact is distributed across thousands of parameters, and the same parameters participate in encoding many facts at once.

This is a distributed representation, and it has one fundamental consequence the industry prefers not to discuss publicly:

If you fine-tune a model on football data, it can become worse at trading.

This is not metaphor or oversimplification. The phenomenon is called catastrophic forgetting, described as far back as McCloskey & Cohen (1989) and confirmed many times for modern LLMs. The mechanism: the weights in a transformer are a shared parameter pool. When you take a gradient step to optimize a football task, the weights shift toward whatever is optimal for football. Knowledge that relied on those same parameters — financial analysis, medical terminology, arithmetic ability — degrades. Sometimes invisibly. Sometimes catastrophically.

“For LLM 2 + 3 = 4.97 is accurate enough, when all the other tasks have similar accuracy.”

In Luo et al., LLMs lost up to 20–40% accuracy on general reasoning benchmarks they had previously aced after being fine-tuned on a narrow domain. The model does not “add” football knowledge to existing knowledge. It redistributes the same finite resource — its parameters.

Modern methods — LoRA, QLoRA, adapters — partly soften the effect by freezing the main weights and training only small add-on matrices. But they do not eliminate the problem, and for deep integration of new knowledge a full fine-tune is still required, with all the risks that come with it.

Practical consequence. When OpenAI or Anthropic ships a new model version, that is not “we added fresh data to the old one.” It is a complete retraining from scratch on an updated corpus. A single pretraining run for an Opus- or GPT-5-class model costs tens of millions of dollars and weeks of compute on thousands of GPUs. That is why new versions arrive every few months, not every week.

And that is why every model has a knowledge cutoff — a date past which it knows nothing. More on that in section 5, where the real pain lives.

For the past two years, the industry has talked more and more about harness engineering — building the environment around the model. System prompts, tools, retrieval, memory, safety checks, sub-agent orchestration. Mitchell Hashimoto defined the term in one phrase in February 2026: the model is the engine; the harness is the chassis, the brakes, the steering, and the dashboard. The engine doesn’t go anywhere on its own.

It’s a correct metaphor, and it leads to one wrong conclusion — that a harness “makes the model smarter.” It doesn’t. A harness compensates for the model’s fundamental defects, the ones discussed above.

The concrete list of what gets compensated:

In April 2026, Anthropic published “Effective harnesses for long-running agents” — about how an agent should live across context-window boundaries when a task runs for hours. Their answer: an initializer agent sets up the environment, follow-up agents pick up via claude-progress.txt and git history. That is harness thinking in its purest form. Not "how do we make the model smarter" but "how do we organize the environment so that an inconsistent model produces a consistent result."

Back to the metaphor: a brainless genius needs prosthetics to function in the real world. The harness is the prosthetics. It does not cure the root problem — the absence of understanding — it compensates for the symptoms enough to make the system useful in production.

The strangest property of LLMs, and the one that lands worst with users: the model lives inside a bubble in time.

The knowledge cutoff is the date on which the model’s training stopped. For me (Claude Opus 4.7, the model writing this text with the user) that date is the end of January 2026. Everything that has happened in the world since then does not exist for the model. Not “it doesn’t remember” — it was never taught.

This is not a small flaw to be fixed in the next version. It is an architectural consequence of how pretraining works. Training an Opus-class model is not a process you can put on “a weekly cron.” It is:

This is why models ship as discrete releases every few months, not as a continuous stream of updates. And this is why, at the moment you are reading this, whatever LLM you are talking to has a cutoff somewhere in the past — anywhere from three months to a year and a half ago.

What the industry actually does about it. There are three solutions, and all three are partial.

First — RAG / web search. The model does not try to remember current information; it queries a search engine or a vector database of fresh documents and reads them in its context. This is what Perplexity does, what ChatGPT with web browsing does, what Claude with web_search does. Works well for factual queries, less well for synthesizing deep trends, because the model only sees what surfaces in the search results.

Second — fine-tuning on a narrow domain. A closed company takes an open-weights model and continues training it on internal documentation. They get domain expertise at the cost of degrading general ability (catastrophic forgetting). Works for specialized products, doesn’t work as a universal solution.

Third — model editing (ROME, MEMIT). Targeted surgical edits to specific weights to change specific facts. “The Prime Minister of the UK is now X, not Y.” Works on simple atomic facts, scales poorly, and sometimes breaks related knowledge.

The biggest unsolved problem in LLMs is not reasoning, not math, not context length. It is metacognition. The model needs to know the boundaries of its own knowledge and recognize when it is operating outside them. Until that exists, any harness around it will be compensatory, not curative.

In 2023, the line was “ask ChatGPT.” In 2024, “give it skills.” In 2025, “vibe-code it.” In 2026, the right question reads:

If you are shipping a product on top of an LLM, your real competitor is not the ML team that will train a model 2 points better on a benchmark. It is the team that understood that the model does not know the world, and engineered around it a system of verification, retrieval, tools, and checks that makes the model reliable in spite of that fact.

An LLM is not intelligence. It is compressed pattern matching without an ontology of the world. It is a brainless genius — a system that speaks brilliantly about everything and knows nothing about anything. And the central skill for the next five years is not “prompt engineering.” It is understanding the limits of the model and engineering the prosthetics that compensate for them.

A brilliant amnesiac can do a great deal — if there is someone next to them holding a notebook. Without the notebook, they will confidently tell you invented stories about nonexistent court cases.

And occasionally — deny the existence of their own predecessor.