Intelligence might be the ability to doubt your own effectiveness.
Imagine an expert surgeon who performs flawless operations but never notices when their hands start shaking. They continue operating with the same confidence, producing increasingly dangerous results while their performance metrics still look acceptable. This captures something profound about current AI systems that researchers have now formalized mathematically.
The breakthrough lies in distinguishing between agency and intelligence through a metric called bi-predictability. Agency requires only that actions create measurable effects in the environment. Intelligence demands something more unsettling: continuous self-doubt about whether your interactions with reality are actually working. The math reveals that current AI systems excel at agency—they can act, influence outcomes, and optimize for objectives. But they lack the capacity to monitor whether their fundamental grip on reality is slipping.
This mirrors a deeper pattern in biological systems. Your brain doesn't just predict what will happen next; it constantly monitors how well its predictions are working and adjusts accordingly. When this self-monitoring fails, you get conditions where people can speak fluently while their words become meaningless, or move confidently while losing all spatial awareness. The difference between agency and intelligence isn't just academic—it's the difference between reliable operation and sophisticated failure.
The implications extend beyond AI into how we think about competence itself. Every expert faces the challenge of maintaining effectiveness while the world changes around them. The research suggests that true intelligence isn't about being right more often, but about being wrong more quickly—catching degradation before it becomes catastrophic.
If our most advanced AI systems are essentially expert surgeons with steady hands but no ability to notice when those hands start shaking, what does this reveal about the relationship between confidence and competence?
follow us for daily new insights