Code Smells and Cameras

Most programmers will tell you about code smells. That ick you get from looking at messy, overly complex code. You can't always articulate why it's wrong, but you know something's off.

This intuition isn't unique to programming. Mathematicians talk about "elegant" proofs. Physicists get suspicious when equations get too complicated—simplicity is treated as weak evidence of truth. The whole lineage from Occam's razor to Kolmogorov complexity points at the same thing: compression and understanding are intimately related. If you truly grok a problem, you can express its solution minimally. Excess complexity signals excess confusion.

Which brings me to LLMs.

No doubt they've changed how I work. But they still generate code that smells. Not broken, just... bloated. Defensive boilerplate. Unnecessary abstractions. Cargo-culted idioms. Patterns that look like code because they were trained on code that looks like that.

Venkatesh Rao has this idea that LLMs are cameras, not engines. We think they're generating something, but they're actually photographing the statistical shape of their training data.

But most code out there is just mediocre. So when an LLM photographs the solution-space of existing code, it faithfully reproduces that mediocrity. It's not so much failing as succeeding at the wrong thing.

Compression requires a model of the problem. LLMs have a model of the solution space. These are orthogonal. Code smell is the gap between them: when the solution's complexity exceeds the problem's complexity. A camera can't detect that gap because it only sees solutions. It has no access to the problem. It doesn't know what's essential.

And I don't think RLHF is the solution either. Human raters would need to identify maximally simple solutions, which requires deeply understanding the problem, the very thing that's hard. What raters actually reward is code that looks clean. Good variable names. Comments. Familiar patterns. That's often more accidental complexity, not less. RLHF just shifts which region of solution-space you sample from, it doesn't escape it.

The models have no representation of the problem. They only have representations of what other people did when they thought they were solving similar problems. That's often not a very good proxy.