AGI Beefs

Every few months there's a new argument about AGI. We talk about "AGI timelines" like it's definitely coming, we just don't know when, while still arguing about what it even means. The latest: Demis Hassabis vs Yann LeCun.

LeCun says general intelligence is "complete BS" Human cognition is specialized for the physical world, and our feeling of generality is an illusion. We only seem general because we can't imagine what we're blind to. Hassabis disagrees. Brains are "approximate Turing Machines" capable of learning anything computable. We invented chess, mathematics, civilization.

I think they're both wrong. Not in a "truth is in the middle" way. They're having the wrong argument. Both treat intelligence as a function: something that takes inputs, produces outputs, gets measured on benchmarks, copied between substrates, scaled up. They disagree about whether this function is "general" or "specialized" but they agree on what kind of thing they're talking about. I think that's the mistake.

I get why the function frame is appealing: it's clean; you can talk about intelligence abstractly, compare systems on leaderboards, imagine copying minds between substrates. But intelligence isn't a function. It's a process, a continuous history for a particular lump of matter, entangled with its environment.

You can't copy an intelligence by copying its state. The intelligence isn't in the state; it's the ongoing process that produced it and keeps modifying it. Clone the weights and you get an artifact. The intelligence was the training run, the continuous adaptation, the being-in-the-world.

This doesn't mean software can't be intelligent. Software that runs continuously, learns in real-time, and is coupled to an environment with something at stake: maybe that counts. But that's not what we're building. What we're building are frozen artifacts. The weights get fixed after training, there's no continuous learning, no real-time entanglement with an environment, nothing at stake.

I like to think of current AI systems as cognitive optics: lenses and mirrors that reflect, refract, and amplify human cognition. They see the way telescopes see, which is to say, they don't. "AIs reproduce our biases" isn't a bug; it's all they do. The training data is human, the reward signal is human, the prompts are human. The whole thing is a hall of mirrors reflecting us back at ourselves, sometimes usefully warped.

What LLMs do is compression, not explanation. The weights encode statistical regularities, what patterns co-occur in training data. That's different from understanding why things work. ARC is the test case here. It's designed to resist memorization and require genuine abstraction. LLMs fail at it because they do compression, not explanation. The scaling thesis says more parameters, more data, and more compute will eventually produce intelligence. I think that's a bet, not an engineering certainty, and I think it's wrong. Better optics are still optics.

Obvious objection: if I want continuous learning and environmental coupling, just simulate it. Build a digital environment with entropy, evolution, and death. Evolve agents inside. Maybe that works. If you build software that genuinely runs continuously, genuinely adapts, and genuinely has stakes (even simulated ones), that's at least different from what we're building now. Whether simulated stakes are enough, I don't know. But the current research program isn't even trying this. We're scaling pattern-matching on static datasets and hoping generality falls out.

To be clear, I'm not saying AGI is impossible. I'm saying the current approach won't get there. Frozen artifacts can be very capable, but they can't be genuinely general, no matter how impressive they are within their training distribution. Could software be general? Probably. But it would look different from what we're building: something that learns continuously, couples to an environment in real-time, and has something at stake.