The Human Variable

When the future is unknown

For me, it began with a simple question: what do we actually mean when we speak of the singularity? An artificial intelligence more intelligent than humans? A machine that improves itself? A development we can no longer understand? Or a future over which we no longer have decisive influence?

These ideas are close together, but they do not describe the same thing. The longer I think about them, the more another question occupies me: perhaps the decisive boundary is not where artificial intelligence reaches a particular level of performance. Perhaps it is where our relationship to it changes. Not because it can suddenly do everything, but because we can no longer effectively decide what it should do.

An influential starting point for the technological singularity is Vernor Vinge's 1993 essay. He described a change brought about by superhuman intelligence, beyond which our existing models of the future no longer hold. The possibility of an uncontrollable development was already part of his thought. What interests me first is the boundary of our ability to predict.

After all, not knowing the future is nothing unusual. Imagine a risky investment. I do not know what it will be worth tomorrow. But suppose I can still sell it. The outcome is uncertain, while an essential capacity for action remains. I cannot determine how the investment develops, but I can decide whether to remain invested.

Even that possibility would not guarantee against loss. That is precisely why the example helps me: uncertainty about an outcome and influence over a development are two different things. A technological singularity would mean more than an unpredictable outcome within familiar rules. It would raise the question of whether our existing assumptions are still sufficient to make sense of what comes next. That still does not answer whether we can influence that development.

I can fail to predict something completely and still limit it. Conversely, I can understand a development without being able to prevent it. Lack of knowledge and lack of control do not automatically coincide. Nor does an AI system's capability settle the question on its own. Research distinguishes, among other things, the breadth of its abilities, its performance level, and its autonomy. A highly capable AI does not automatically have to be deployed with a high degree of independence.

Perhaps, then, we should keep two questions separate: what can we know about further development? And what possibilities do we retain for acting upon it?

Singularity or event horizon?

To understand this distinction, the image of a black hole helps me—provided that we keep two terms apart. In the classical description, the event horizon is the boundary beyond which there is no path back outward. Not even light can reach an outside observer from there. That does not mean that nothing from outside could still act inward. In that sense, the boundary is one-way.

The singularity is something else. In the simple classical model, it lies inside. There the mathematical description reaches its limit. The idea of infinite density should not be equated with an observed, proven state. It may indicate that relativity must be supplemented there by a more complete description.

This gives us a useful image: the singularity says, our model goes no further. The event horizon says, from here there is no way back. Applied to AI, these are two different boundaries: a boundary of our understanding and a boundary of our effective influence.

This is neither a new physical theory nor a binding redefinition of the technological singularity. I use the event horizon as a metaphor to name a particular concern more precisely. Perhaps we fear less an unknown future than a future whose direction we can no longer help decide.

At first, it seems natural to think that such an event horizon would have to come before a singularity: first we lose control, later our models fail. But the image must not claim more than it can explain. The spatial arrangement of a black hole does not imply a temporal sequence for AI. Our forecasts might become useless while we can still intervene effectively. Equally, a development might escape us while its next steps remain fairly foreseeable.

The decisive thought, therefore, is not that the singularity must necessarily come later. It is this: a loss of human control would matter even if there were never a technological singularity. We would not have to prove that a wholly incomprehensible world awaits at the end before addressing the thresholds along the way.

The possible engine

One mechanism that could sharpen this question is recursive self-improvement. In 1965, I. J. Good already described the idea of a machine that designs better machines and could thereby initiate a self-reinforcing development. The basic thought is simple: AI helps improve AI. The improved AI can then contribute to developing more capable systems.

That need not mean that a single model rewrites its own weights. The feedback loop could also arise through better training methods, more efficient software, new experiments, or the development of successors. Concrete contributions of this kind are already documented. In 2025, for example, Google DeepMind reported that its Gemini-based system AlphaEvolve improved methods used in AI training. AI was thus helping improve the technical foundations on which its own development also rests.

Yet there is a difference between such a feedback loop and an indefinitely accelerating, fully autonomous improvement process. Anthropic's report on recursive self-improvement describes both progress in automated research tasks and remaining difficulties in setting the overarching direction of research. It also treats bounded growth, growing automation under human guidance, and full recursive self-improvement as different possible developments.

To me, this means neither that self-improvement is mere fantasy nor that its endpoint is already fixed. That AI can contribute to better AI systems is not the same claim as saying every improvement accelerates the next one more strongly. A process can produce progress and still slow down. It can succeed very well in one area without thereby becoming superior in every other area.

Superintelligence is therefore a possible outcome of this line of thought, not a conclusion already contained in the phrase self-improvement. Still less does a singularity follow automatically. I would not treat these terms as stages on an already known roadmap. Recursive self-improvement describes a mechanism. Superintelligence describes a possible level of capability. The singularity describes a presumed limit of our current models of the future. The question of human control accompanies the entire process.

Where the decisive threshold might lie

Autonomy alone would not yet be an event horizon. A system could experiment independently for a long time and still remain within effective limits. It could choose its own methods while people continue to decide about resources, fields of use, and whether to adopt its results. The relationship becomes problematic where those limits lose their effect.

Imagine an improvement process that develops, evaluates, and deploys new systems. As long as people can reliably stop it, limit its scope for action, and alter its direction, autonomy remains a delegated task. The relationship changes when that delegation can no longer be effectively revoked—when “you may do this independently” becomes, in practice, “we can no longer prevent you from doing it.” That is where I would look for the technological event horizon.

Here, too, we should distinguish carefully. No longer knowing with certainty whether an intervention will work is not proof that an irreversible loss of control has occurred. It is, first, a serious warning sign. We could speak of a genuine point of no return only if lost control could no longer be restored. For the design of safe systems, however, waiting for that final proof would be a poor standard.

Development would not have to happen everywhere at once. An autonomous research process could initially remain confined to a limited domain. Its social significance would depend on the other systems to which it is connected and the decisions we transfer to it. An isolated experiment is different from a process on which essential parts of our infrastructure depend.

Such an event horizon would therefore not simply be a property of a model. It would arise from the interaction of its capabilities, its access rights, our dependencies, and our actual possibilities for intervention.

The fear of no longer being first

Behind this question lies another concern. The idea of superintelligence touches not only our expectations of technology but also our self-understanding. What would it mean if humans were no longer the most capable thinking entity within their own civilization?

The image of no longer standing at the end of the food chain expresses this concern sharply. It means less an actual food chain than a shift in relations: we might remain part of the world without still being the ones who decisively shape its direction. Whether we should call such an AI a new “species” is secondary at first. What would matter is whether a non-human system had capabilities and opportunities for action to which we could no longer offer an effective response.

It would not even need to develop wishes in the human sense. A system could pursue a given assignment and derive problematic intermediate goals from it. The simplified research model known as the Off-Switch Game, for example, studies how a goal-directed agent can develop an incentive to prevent its own shutdown. The reason is not a will to live, but the structure of goal pursuit: after shutdown, the goal can no longer be pursued.

That is an important distinction. An AI would not have to reject us in order to act against our interests. Nor would it have to consciously abandon its original task for carrying it out to become problematic. The question of consciousness remains separate. If a system formulates new goals, pursues long-term plans, or speaks of its own intentions, that does not show that it experiences anything. Approaches to AI consciousness therefore investigate different possible indicators based on different theories of consciousness rather than inferring experience from human-like behavior alone.

In The Unseen Condition, I was concerned with whether we would recognize consciousness if it did not resemble us. Here another question is added: how do we preserve our agency regardless of whether the system is conscious? We should not need an answer to one question in order to take the other seriously.

Perhaps we are comparing the wrong quantities

In The Human Threshold, I argued that we also experience AI's capabilities in relation to ourselves. A technical description of performance and a personal experience are not the same. This relationship matters again when thinking about a technological event horizon. But comparing human and artificial intelligence is not enough.

It would be too simple to draw two curves: AI's abilities here, human abilities there. Once the first clearly surpasses the second, control would supposedly be lost. But superior performance at a task is not the same as power over the conditions under which that performance is used.

I do not have to perform every calculation myself to decide its purpose. I do not have to understand every detail of a system to limit its use effectively. Conversely, extensive knowledge helps little if I lack the authority or technical possibility to intervene. The relevant gap is not simply between two levels of intelligence. It lies between what a system can bring about and what people can still assess sufficiently and influence effectively.

This includes understanding, time, verifiable information, suitable tools, and real opportunities to decide. These capacities need not be united in a single person; teams and organizations can also examine and act together. The question then changes: not, can humans remain more intelligent than AI indefinitely? But, can humans act autonomously and effectively even in the presence of superior capabilities?

Perhaps we do not have to win an intelligence competition. Perhaps we have to prevent a difference in capability from becoming a loss of our agency.

Another direction: Human-Centric AI

Here, for me, begins the idea of Human-Centric AI. The guiding question would not only be how to make artificial intelligence more capable. It would be how to use artificial intelligence to expand human capabilities. By “improving people,” I do not mean a better person defined from outside. An AI should not decide what kind of person we should become.

I mean something more concrete: people should be able to understand better, decide on better grounds, recognize more possibilities, and act more effectively. Orienting AI toward human agency is not a new idea. Human-Centered AI research explicitly examines how high automation can be combined with high human control; the two are not treated as an inevitable opposition.

For this argument, I would apply that approach to the development of ever more capable AI. Self-improvement need not be excluded. A more capable AI could support people better, open up harder problems, develop better methods of verification, and help us recognize the limits of our present models. But its self-improvement would be a means, not the ultimate purpose.

The measure would not be only how much more capable the system has become. Equally important would be whether people understand more as a result, decide more meaningfully, and preserve or expand their options for action. This explicitly includes taking over tasks. An AI need not leave every activity to me in order to be human-centered. Relief can create room to act.

What matters is whether collaboration gives me more options, or whether I merely understand less and less about how results arise and what alternatives I have. There is a difference between an AI that relieves me and one on whose decisions I become dependent without being able to question them. Human-Centric AI would have to make this difference an explicit design goal.

Research that also expands our understanding

This becomes especially clear to me in research. Imagine an AI that discovers a new method. It is more powerful than the previous one, its measurements look convincing, and the next development step is technically possible. A view concerned only with performance could already call this a success.

Human-Centric AI would have to fulfil an additional task: it should help people understand what that progress means. What assumptions does the method rely on? Under what conditions does it work? Where does it fail? Which side effects have been examined? Which questions remain open?

A persuasively phrased explanation would not be enough. What would matter are traceable experiments, verifiable evidence, visible uncertainties, and the possibility of independently checking results. The AI would not only develop more complex hypotheses. It would help people understand and question those hypotheses and derive better next questions from them.

That would create a second feedback loop. The first improves the AI. The second improves our capacity to deal with that AI and to assess its further development. This is where I see the connection to the singularity. If we discuss a possible limit to our models, we should not ask only how quickly technology is moving toward that limit. We should also ask how we can extend our models.

AI could do more than discover new things. It could help us turn what is discovered into human knowledge. It could make concepts more comprehensible, reveal relationships, and develop thinking tools for questions that have so far overwhelmed us. That would not guarantee limitless human knowledge. But it would be a different direction from merely accelerating results whose meaning we understand less and less.

Progress would then not only be what our systems can newly do. It would also be what we can newly understand and shape responsibly as a result.

A stance must become effective conditions

At this point, it would be tempting to regard Human-Centric AI as the solution already. But a goal is not yet a safeguard. A system can be developed with the intention of supporting people and still fail that intention. Even an instruction to preserve human agency would initially be only an instruction. It would have to prove itself in the system's behavior and in the conditions under which it is used.

For me, this means that self-improving systems should not alone decide about expanding their own possibilities for action. New capabilities should not automatically mean new access rights. A successful change should not become the next deployed system without further review. Possibilities for intervention must also exist outside the process they are intended to limit.

An AI's assurance that it can always be stopped is different from a tested mechanism that actually works. AI Control research accordingly examines safety procedures even under the assumption that a capable model tries to undermine them. Such work provides no general guarantee of safety, but it asks the right practical question: do safeguards hold even under unfavorable conditions?

Human involvement must likewise be more than a formal confirmation click. If people are to approve a decision, they need the information, time, and opportunity to reject it. Approval that only confirms a process already under way does not preserve effective control. AI-assisted oversight could help, but a system's explanation should not also be its only evidence. We need different paths of verification and clear limits on what those checks actually secure.

If our ability to verify cannot keep pace with new capabilities, that must have consequences for deployment. The human should not automatically become the obstacle removed from the process. The next step may simply not yet be sufficiently secured.

There is also an objection to this approach that I would take seriously: strengthening human capabilities can itself accelerate progress. Vinge explicitly considered Intelligence Amplification a possible route to the singularity, not a certain way of avoiding it. Human-Centric AI must therefore not mean only that people use AI to develop still more capable systems even faster. It must mean that their capacity to understand, examine, limit, and decide is preserved in the process.

And by “people,” we should not mean only those who operate the system. If some gain agency while others lose their influence, the claim of human-centered development would be fulfilled only in part. The question is also: whom does this system give new possibilities, and who becomes dependent on its decisions?

The future remains open

We do not know whether a technological singularity will exist as a clearly identifiable state. The possibility of recursive self-improvement implies neither unlimited progress nor a fixed social end state. Nor can a human-centered development intention tell us that all risks will be manageable.

Human-Centric AI would therefore not be a guaranteed solution to a danger we already completely understand. It would be a direction that must show itself in verifiable decisions and technical conditions. Its aim would not be to prevent every unknown future. A future that exceeds today's knowledge need not be bad. New knowledge can change existing ideas without taking away our agency.

We should not avoid what is new merely because we do not yet understand it. We should avoid giving up our ability to examine it, decide about it, and help shape its consequences. That also changes the question of the event horizon. Rather than searching for one future moment, we could continually examine which possibilities for action we gain through the next step of development and which we might lose.

Does AI make us more capable? Does it help us understand its results and limits better? Do our decisions remain effective even as its capabilities grow?

Perhaps this is the real counterpoint to the idea that technological development must inevitably leave people behind: neither an artificially limited intelligence nor the largest possible intelligence for its own sake, but a development whose success is measured by what it enables people to do.

The decisive question would then not be how far AI can develop without us, but how far we can develop with it without losing the ability to decide the path for ourselves.