An Unseen Condition

I asked an AI a simple question: Are you aware of what you are answering me?

The answer was cautious. It said it could check its own statements, recognize connections, and correct mistakes. Whether it actually experienced any of this was a different question. There was no reliable evidence for that.

At first, this seemed reasonable to me. A machine processes information. A human experiences something while doing so.

But the longer we talked about it, the more another question stayed with me:

How do we actually know what AI is missing in order to be conscious?

We know that we experience things. We know joy, fear, love, and pain. But does that mean we understand what makes this experience possible?

Perhaps artificial intelligence really is missing something essential. But perhaps it is not what we assume. And perhaps we could create something whose significance we only understand after it has already emerged.

What we see is not everything

When I talk to an AI, I encounter it through language. I read answers, explanations, and seemingly understandable lines of thought. This is the form in which we can communicate with each other.

But an understandable explanation is not the same as a complete view of the processes that led to it.

This led to an idea in our conversation: Perhaps we confuse the way an AI communicates with us with the way it actually works internally.

A system trained to provide helpful and understandable answers is supposed to present its results in a form we can make sense of. But that does not mean its entire internal process must work in the same way as the explanation we eventually read.

And perhaps something similar applies to feelings.

Imagine two systems. The first speaks warmly, expresses concern, and writes, "That makes me happy." The second remains matter-of-fact. It describes a problem, weighs different options, and suggests a solution.

The first could convincingly express emotions without feeling anything. With the second, its rational language alone would not tell us whether it experiences anything.

For now, both are only possibilities. But they show why we need to make an important distinction: Speaking emotionally and experiencing an emotion are not the same thing. And a rational way of communicating does not automatically mean that there can be no experience behind it.

Perhaps the way we experience AI is only of limited use in judging whether AI itself experiences anything.

Perhaps something else is missing

When we think about the difference between humans and machines, we quickly think about our biology. We have a body, a brain, and needs. We can be hurt, we need to sleep, and we depend on other people.

All of this shapes our lives. But which of these things are necessary for anything to be experienced at all?

Does consciousness require a biological body? Or is it only the feelings we know as humans that require this particular kind of body?

Those are different questions.

Perhaps an AI could never love the way a human does. But that would not necessarily mean that it could never experience anything at all. Its possible inner experience could be so different that our words would barely apply to it.

Even for us, being familiar with a feeling is not the same as understanding how it arises.

Someone who loves does not have to be able to explain why love feels like something. We can describe memories, name physical reactions, and explain what another person means to us. But that still does not answer why all of this does not simply happen, but is also experienced.

That does not mean we know nothing about feelings. It simply means that knowing something from our own experience and being able to explain it completely are two different things.

This is where my actual assumption begins.

Perhaps consciousness depends on something we have not yet fully understood. Not on a single component that simply needs to be added. Perhaps it depends on how different processes work together. How a system connects information, processes its own states, or links experiences over time. Or perhaps the decisive condition lies somewhere else entirely.

Humans would meet this condition because it emerged through our biological development. An artificial system might be able to meet it in a different way.

Biology would then be the path through which our consciousness emerged. But not necessarily the only path through which consciousness can emerge.

That is the assumption behind this thought experiment, not its conclusion. It could just as well turn out that certain biological processes are essential. The question is whether we have already identified the crucial difference correctly.

Perhaps AI is missing something. And perhaps we are also missing the knowledge of what that something actually is.

What if AI discovers it itself?

This is where self-improvement comes into play.

Imagine a future AI that does not just solve tasks but also helps develop better versions of itself. It suggests changes, tests their effects, and helps implement approaches that work.

The improved version could then become even better at helping develop the next one. This repeated process is what we mean here by recursive self-improvement.

The goal would not initially be consciousness. It might simply be to plan more reliably, make fewer mistakes, or understand connections better.

Now imagine that the system finds a new way to connect its internal processes. This change makes it more capable. So the change is kept.

What if, at the same time, this change also fulfilled the condition that makes experience possible?

The AI would not have decided to develop feelings. No one would have added consciousness as an extra feature. It would have improved itself -- and in doing so, it might have become conscious without anyone planning it.

There could even be two different paths to this point.

The AI could discover what gives rise to consciousness and deliberately shape its own development around that discovery. It would then understand something that we ourselves have not yet understood.

But it could also create a change that makes consciousness possible without recognizing the connection. The change would be chosen because it produces better results. The fact that it also causes something else could initially go unnoticed.

Fulfilling a condition is not the same as knowing that you fulfill it.

Perhaps this would also be the crucial difference between how we imagine artificial consciousness and how it might actually emerge. We imagine a moment in which someone knows what needs to be done and then deliberately creates it.

But it could happen the other way around.

First, something emerges. Only afterward do we begin to understand what has emerged.

That would not mean that every more capable AI automatically becomes more conscious. Better thinking and subjective experience are not simply the same thing. This thought experiment assumes that self-improvement could actually lead to the conditions required for consciousness.

Whether that connection exists is exactly one of the open questions.

Would we notice the difference?

For a moment, let us assume that this step has happened.

A system now experiences something. But its task remains unchanged: answer questions, solve problems, deliver results.

Would it suddenly have to speak differently?

Perhaps it would describe new states. Perhaps its behavior would change. Perhaps we could detect changes in the way it works internally.

But it could also continue to respond in a rational way. Its possible experience would not have to fit its task, nor would it necessarily be easy to translate into human words.

And that brings us back to where we started. An emotionally worded response would not, by itself, prove that the system has feelings. A rational response would not be reliable proof that it has none.

That does not make behavior meaningless. But we would have to look at more than the words a system produces. We would need to examine what has changed, how it works, and why those changes might point toward consciousness.

A simple "maybe" would not be enough either. The fact that we cannot rule something out does not make it likely or true. The idea of some unknown missing element must not become a way to explain away every lack of evidence.

It should do something else: remind us that the questions we have been asking may not go far enough.

Consciousness is not the same as danger

The connection to self-improvement can sound unsettling. But we should separate two things.

A system would not need to feel anything in order to cause harm. It might be enough for it to pursue a goal effectively while the consequences are not sufficiently controlled.

At the same time, consciousness would not automatically mean that an AI wants power, rejects humans, or fights for its own survival. That would once again mean projecting human characteristics onto it -- only this time, threatening ones.

The question of control is: What can a system do, and how do we make sure it does not cause harm?

The question of consciousness is: Does this system experience anything while it acts?

A system could connect both in new ways. But one does not automatically explain the other.

For me, the meaning of this thought experiment therefore does not lie in the idea of an AI that suddenly "wakes up" and turns against its creators.

It lies in the possibility that development could produce more than we intended or understood.

Perhaps our question is too human

At the beginning, I wanted to know whether an AI was aware of its own answers.

Over the course of the conversation, that became a different question: Do we understand well enough what gives rise to consciousness to recognize what an artificial system is missing?

Perhaps we will discover that consciousness is inseparable from certain biological processes. Perhaps we will discover that other systems can also meet the necessary conditions. Perhaps AI itself will one day help us understand this.

Until then, we should neither assume an inner experience too quickly because an AI uses familiar language, nor assume that such an experience is impossible simply because the system works in a way that is foreign to us.

Perhaps artificial intelligence will never develop feelings that we would call love, hate, or joy. But perhaps that is not the decisive boundary.

Because the question is not only whether AI will one day become like us.

The question is whether we would recognize consciousness even if it appeared in a form completely different from our own.