Yesterday I was practicing an acrobatic skill called “hand-to-hand” where, as a flyer, I climb on my partner’s shoulders, then jump into a handstand on their hands. I’ve attempted this skill hundreds of times and have learned how to safely bring my feet to the ground, even if I don’t nail the handstand. But approaching the end of the night and feeling my wrists and shoulders start to fatigue, I recognized the signals from my body telling me it was time to stop. It wasn’t safe to do another hand-to-hand.

In partner acrobatics, we call this making an accurate self-assessment. All participants—flyer, base, even the spotter—must communicate their own physical and emotional limits when attempting a skill. This assessment is deeply personal, a reflection that only you can make regardless of how others perceive your readiness. It requires an attunement to both the body and mind, a kind of reflective consciousness that guides decisions in complex, high-stakes environments—a kind of consciousness that AI lacks.

The need for metacognition

As AI advances, it remains limited in a critical way: the models don’t understand themselves. AI relies on data and algorithms to perform tasks, using past information to make predictions. While it can be programmed to report confidence levels or performance metrics, it does so without the capacity for introspection.

For AI to approach a human-like level of self-awareness, it would need to develop artificial metacognition, the capacity for machines to monitor, assess, and regulate their own learning processes. While there are already various approaches for error detection and mitigation, these mechanisms are still reactive. AI cannot consider nuances outside of its dataset or reflect on its performance in any existential way, or proactively recognize when the task at hand exceeds its capabilities. For AI to achieve metacognition, it would need to "know when it doesn’t know." Technical approaches like uncertainty quantification or adversarial robustness move AI closer to the appearance of self-awareness, but lack a deeper subjective experience.

So then, can a system without the capacity for subjective experience ever reach a level of awareness where it can truly "reflect" on its own limitations?

A slight tangent on consciousness

I rarely explore a topic on AI without confronting the inevitable black hole of consciousness, and whether or not AI has the potential to ever become conscious. Philosophers, neuroscientists, and AI theorists alike have wrestled with the question, but there's a persistent challenge: humanity itself still lacks consensus on what consciousness really is. Without a clear definition of human consciousness, the prospect of machine consciousness becomes even more nebulous.

I recently encountered a definition of consciousness from DJ Seo on the Lex Friedman’s Podcast that struck me as elegantly simple: consciousness is the brain’s awareness of its electrical activity, of its neurons actively perceiving and interpreting the world. Initially, this sounds clinical, even reductive. But sitting with this definition, I think it captures a critical phenomena of consciousness: not just sensing, but being aware of that sensing. It’s this reflective layer that separates mere data processing from what we consider conscious experience.

If we apply this to machines, we could then define machine consciousness as a system’s awareness of its computational processes, a capability of monitoring and interpreting the electrical signals and mechanical operations that drive its functioning—AI’s equivalent of brain activity. An AI capable of assessing its inner processes might track memory allocation, algorithmic decision pathways, or electrical current variations in real time. It would, in a sense, be able to "perceive" its own performance, identifying when certain algorithms falter or when its data inputs are insufficient.

Still, there is a catch: even if an AI system could technically monitor its processes in this way, would such an AI be truly aware, or just simulating self-awareness? Can we consider a machine "conscious" if it doesn’t have subjective experiences, or does self-assessment in AI exist in an entirely different realm? If awareness in machines never moves beyond raw process monitoring, perhaps what we’re talking about is not consciousness, but pseudo-consciousness—a machine capable of mimicking awareness without ever truly possessing it.

Subscribed

Potential benefits & ethical implications of AI self-assessment

Barring the dilemma of consciousness, if AI could self-assess with greater sophistication, it could revolutionize high-risk industries. For instance, in healthcare, diagnostic AI systems could flag cases where their confidence is low, deferring to human experts for ambiguous diagnoses. Autonomous vehicles could detect when they lack sufficient data to safely navigate unknown road conditions, warning the driver rather than making a risky decision.

This potential of contextual self-awareness would fundamentally reshape human-AI collaboration. Right now, AI systems often operate under the assumption of competence, which can lead to errors and over-reliance. If AI could articulate its limitations transparently, it would foster greater trust between humans and machines, leading to more nuanced and effective partnerships.

However, developing self-assessing AI introduces complex ethical dilemmas. If a machine can assess its capabilities, should it have the autonomy to refuse tasks? If a machine makes a "self-aware" decision, who is responsible for the outcome—human programmers or the machine itself? These questions challenge our traditional understanding of responsibility and accountability, raising concerns about how much agency we grant to machines.

As we explore the concept of self-assessment in AI, we are forced to grapple with deep questions about intelligence, consciousness, and self-awareness. Building AI capable of accurately assessing its limitations requires us to push the boundaries of what we understand about both human cognition and machine learning. In the process, we must also reckon with the philosophical implications of creating technology that not only thinks but can think about thinking.