On optimizing for intelligibility to humans

Correctness is not everything

One risk of “vibe-coding” a piece of software with an LLM is that it gets you 90% of the way there, but then you’re stuck—the last 10% of bug fixes, performance improvements, or additional features is really hard to figure out because the AI has written messy, verbose code that both of you struggle to work with. Nevertheless, to delegate software engineering to AI tools is more tempting than ever. Frontier models can spit out almost-perfect complex React apps in just a minute, something that would have taken you hours in the past. And despite the risks, it’s often the right decision to prioritize speed, especially as models get smarter.

There is, of course, a middle ground between “vibe-coding” and good old-fashioned typing-every-character-yourself. You could use LLMs for smart autocomplete, occasionally asking for help with specific functions or decisions, or small and targeted edits. But models don’t seem optimized for this use case. It’s actually difficult to do so—it’s one thing to build an RL environment where the goal is to write code that passes some tests or gets a high preference score. It’s another thing to build an RL environment where the model has to guide a human to do a task, write code that’s easy for humans to build on, or ensure the solution is maximally legible to a human.

Will it become a more general problem that the easiest way for an AI to solve a problem is to produce a solution that humans find particularly hard to understand or work with? Some may say this is not a problem at the limit, when AIs are robustly superhuman at the task, but until then there is a temporary period of slop. Personally, I think this is a problem even when AIs are superhuman because of the importance of human oversight. Optimizing for intelligibility to humans is important for robustness and safety—at least some people should be able to understand and verify AI solutions, or intervene in AI-automated systems when needed.