Schrödinger's Machine: The AI Singularity and the Unsettled Physics of Observation
Farid Novin
I. The Singularity as a Mathematical, Not Merely Technological, Hypothesis
The idea of an artificial-intelligence singularity is usually presented as a technological proposition. At some hypothetical point, an artificial system becomes sufficiently intelligent to improve its own architecture, code, algorithms, and learning procedures. The improved system then produces a still more capable successor, and the process repeats recursively. Intelligence compounds upon itself until the pace of development exceeds the human capacity to track, predict, or govern it.
This proposition is almost always argued in technological language — chips, parameters, benchmarks, compute budgets. But beneath that vocabulary of recursive self-improvement, intelligence explosion, and loss of control sits a much older and more fundamental question, one that physics has never fully settled for its own domain, let alone for ours: does a system possess a determinate state independent of the act of observing it, or does observation itself participate in fixing what the state turns out to be?
This is not a rhetorical flourish borrowed from popular science. It is a live and unresolved dispute among physicists. The majority position — decoherence-based accounts, many-worlds interpretations, and Copenhagen-without-consciousness readings — treats "observation" as any sufficient physical interaction, with no special role for a mind or a belief. But a serious minority tradition disagrees. John von Neumann formalized quantum measurement as a chain of interactions that must terminate somewhere, and Eugene Wigner — a Nobel laureate, not a popularizer — argued explicitly that the chain terminates in consciousness itself: that an observer's awareness, not merely a detector's registration, is what fixes an indeterminate state into a determinate one. John Wheeler's delayed-choice experiments and his "participatory universe" framework pushed further still, suggesting observers may play a role in bringing outcomes into being that is not confined to the present moment. And QBism — Quantum Bayesianism, associated with Christopher Fuchs and collaborators, an active and published research program rather than a fringe position — treats the quantum state itself as an agent's degree of belief rather than a feature of the world existing independently of any agent. The recent experimental literature on Wigner's-friend scenarios (Proietti et al., 2019, and the work that followed it) has kept this dispute empirically alive rather than settling it.
I want to be precise about what follows from this, because it would be intellectually dishonest to claim more than the physics actually supports. I am not asserting that artificial intelligence is a quantum-mechanical phenomenon, nor that belief literally collapses a wavefunction in the technical sense those physicists intend. What I am claiming is narrower and, I think, defensible: that even within physics — the discipline most committed to a mind-independent, Aristotelian conception of reality — serious researchers cannot agree on whether a system's state exists determinately prior to observation or is fixed only through the act and structure of observation. If that question remains open in the hardest of the hard sciences, we should be considerably more cautious about assuming it is closed for something as interpretively loaded, historically entangled, and institutionally embedded as machine intelligence. What follows uses the structure of that unresolved debate — not its literal mechanics — as a lens for a parallel and, I will argue, genuinely analogous problem in how the AI singularity is asked about, funded, built, and governed.
The classical singularity hypothesis implicitly treats intelligence as a scalar variable evolving under a fixed rule: each generation's intelligence is some function of the previous generation's intelligence, so that intelligence after many iterations is that function applied to itself repeatedly. The expectation is that sufficiently many iterations will eventually cross into a qualitatively unprecedented regime — a state assumed, implicitly, to already have a definite trajectory waiting to be discovered rather than one that is being actively constituted by the very act of predicting, funding, and building toward it.
The real world is far less accommodating than that, on either an Aristotelian or a participatory reading.
Whether an AI singularity is possible therefore depends not merely on whether machines become more capable, but on whether the mathematical preconditions for runaway recursive extrapolation hold in a world where intelligence exists — and on whether "the singularity will or will not happen" is even the kind of question that has a single, observer-independent answer at all. My argument is that, on close inspection, the classical preconditions probably do not hold, and that the question itself may be closer in structure to an unresolved measurement problem than to a fact awaiting discovery.
II. From Aristotelian Identity to the Mathematics of Approximation
Classical Western logic begins from three powerful commitments: a thing is identical to itself; a thing and its negation cannot both hold; and for any proposition, either it or its negation holds. These principles are extraordinarily useful. They let a biological organism, an engineer, a scientist, or a computer sort the world into sufficiently stable categories to act on.
An animal does not need to register every microscopic difference between two pieces of food. It needs only to recognize that both belong to the category food. It operates, in effect, on the abstraction that something plus a small enough difference is still, for practical purposes, that same something. This is one of the great achievements of cognition and of mathematics alike: it is what makes classification, counting, measurement, prediction, optimization, and accumulation possible in the first place. It is what allows us to say that two approximately equivalent objects can be treated as two units of the same thing — the entire apparatus of arithmetic rests on it.
But this abstraction is not identical to reality, and the gap matters more than it first appears.
A tree is not merely an instance of the category "tree." A human being is not an interchangeable unit within a population. Two apparently identical neural networks can behave quite differently because of training history, data exposure, stochastic initialization, hardware idiosyncrasies, and the particular sequence of interactions each has undergone. The statement that two units of the same thing make two of that thing is exact within its formal system — arithmetic does not lie. It does not follow that two real-world instances of "intelligence" are perfectly interchangeable, substitutable, or commensurable on a single scale, any more than two "identical" quantum systems are guaranteed to yield identical outcomes upon measurement.
This distinction becomes decisive the moment mathematics is transferred from relatively stable physical quantities — mass, charge, temperature — to something as heterogeneous, interaction-dependent, and possibly observer-entangled as intelligence.
III. The Real World Is Full of Discontinuities
Classical mathematics is powerful precisely because it tolerates continuity, approximation, and abstraction. Yet even within mathematics itself, the representation of reality repeatedly runs into discontinuities, singularities, undefined quantities, and processes that refuse to converge.
Consider an apparently innocuous fraction like one-third. Its decimal expansion never terminates — it repeats forever. The mathematical object is perfectly well defined; any finite decimal approximation of it is not the thing itself but a truncation.
More revealing still is an expression like infinity divided by infinity. This has no universal value. Depending on the specific functions generating each infinity, the limiting ratio can be zero, one, infinity, some other finite number, or simply undefined. The lesson is important: the form of an expression does not by itself determine the behavior of the system it represents.
The same caution applies to intelligence. The fact that an AI system improves along one measurable dimension of performance does not establish that the improvement propagates indefinitely into every other dimension. A system can become better at mathematics without becoming proportionally better at judgment. It can become better at prediction without becoming proportionally better at understanding. It can become more capable at generating code without becoming any better at recognizing which problems are worth solving, or at noticing when the premises of a problem are false. The tidy image of intelligence as a single number climbing a single curve may therefore be mathematically convenient while being ontologically misleading.
IV. The Hidden Assumptions Behind Recursive Self-Improvement
The classical singularity argument begins with a deceptively simple sentence: an AI becomes intelligent enough to improve itself. That sentence quietly presupposes at least seven distinct conditions.
The system must know what constitutes an improvement. The improvement must be measurable. That measurement must remain valid after the architecture itself has changed. An improvement in one component must not produce deterioration elsewhere. The improved system must have sufficient resources to implement the improvement. The environment must permit the improvement to be realized in practice, not merely in principle. And the whole process must remain stable enough across iterations for recursion to make sense at all — with the further requirement that improvements compound rather than run into diminishing returns.
The singularity thesis usually treats these as independent technical footnotes. They are not footnotes. They are the argument.
Represent an AI system's capability not as a single number but as a vector with many components — reasoning, memory, creativity, planning, social interpretation, physical interaction, scientific discovery, uncertainty management, goal selection, and so on. An improvement in one component does not imply improvement in the others; it is entirely possible for one capability to rise while another falls in the same step. A system may become more computationally efficient while becoming less robust. It may become better optimized for a benchmark while becoming worse at handling genuinely unfamiliar situations.
This is not merely a rhetorical worry. It has a formal cousin in the No Free Lunch theorems of computer science, which show that no optimization procedure can be uniformly superior across every possible objective landscape — gains on one class of problems are matched by losses on another, once averaged over all possible environments. Whether or not the strict theorem applies to any particular AI architecture, the intuition it formalizes is exactly the one at stake here: "better" is not a single direction in capability space, and treating it as one is where the singularity narrative first goes astray.
Once intelligence is understood as a many-dimensional, environment- and history-dependent process rather than a single evolving number, the image of a clean intelligence explosion becomes considerably less inevitable.
V. Intelligence Is Not Merely a Quantity
The deepest difficulty with the singularity hypothesis may be semantic before it is computational. We speak of "more intelligence" as though intelligence were analogous to temperature, mass, or raw computing power — a single axis along which more is simply more. But intelligence is not one thing measured on one scale.
A chess engine can possess extraordinary strategic calculation while having no biological needs, no emotional life, and no ordinary social existence at all. A mathematician can possess formidable abstract reasoning while struggling with tasks of physical coordination that most people perform without thought. A young child may lack sophisticated formal reasoning entirely while possessing extraordinary capacities for social learning and adaptation that no current machine can match. To say simply that one system is "more intelligent" than another is meaningless until we specify: more capable in what respect, according to which objective, in which environment, for which class of problems?
The singularity narrative tends to compress this genuinely multidimensional structure into a single ordering, as though all capable systems could be arranged along one line from less to more. But intelligence may be only partially ordered — there may be no single ladder along which every relevant capacity improves in lockstep. Intelligence may instead resemble a landscape of competing capabilities, trade-offs, constraints, and local peaks, where a system can climb one mountain only by moving further from the summit of another.
VI. The Fallacy of Extrapolating Local Improvement into Global Explosion
The strongest version of the singularity argument requires more than steady improvement — it requires recursive acceleration. It is not enough for each generation to be somewhat better than the last; each generation must become better at the very process of becoming better. That is a second-order claim, considerably stronger than the first, because it demands that the system improve not only its intelligence but the mechanism responsible for improving its intelligence.
In physical and biological systems, positive feedback loops of this kind routinely run into constraints. Resources become scarce. Energy requirements rise. Added complexity generates new and unanticipated failure modes. Optimization runs aground on local maxima. Environmental resistance stiffens. Measurement itself becomes less reliable as the system moves further from the regime in which the measuring instrument was calibrated. And, characteristically, the easiest improvements are found first, so that each subsequent gain is harder to secure than the last.
A more realistic model of capability growth would therefore treat the rate of improvement itself as a function of available resources, accumulating constraints, a shifting environment, and irreducible uncertainty — not as a fixed acceleration built into the system from the start. Under such a model there is no mathematical necessity that capability grows without bound. The system may just as plausibly converge toward some equilibrium level, oscillate between regimes, alternate between rapid advances and long stagnations, or discover that further gains require resources — economic, physical, institutional — that are simply unavailable at any price. The singularity, in other words, is not the mathematical default outcome of recursive self-improvement. It is one possible trajectory among many, and arguably not the most likely one.
VII. The Problem of Self-Reference — and Its Wigner's-Friend Structure
There is a further and more subtle difficulty. A recursively self-improving intelligence must, at some point, improve the very system that defines what counts as an improvement. This introduces self-reference into the heart of the process: a system attempting to redesign itself must possess a sufficiently accurate model of itself to know in advance which changes will actually help.
But any sufficiently complex intelligent system is itself an object of incomplete self-knowledge. It is simultaneously the observer and the observed, the optimizer and the optimization target. This recalls — without needing to be reduced to — the broader family of limitative results in mathematics and computation showing that a sufficiently expressive formal system cannot give a complete and fully self-consistent account of all its own properties from within. Gödel's incompleteness theorems are the most famous instance of this pattern; Turing's proof of the undecidability of the halting problem is another, and arguably more directly relevant one, since it shows that no general procedure can determine in advance whether an arbitrary program modifying itself will even terminate, let alone terminate successfully.
The deeper point, though, is epistemological rather than technical: a system cannot automatically convert the capacity for self-reference into perfect self-knowledge. An AI may well be able to modify its own source code without possessing a complete understanding of the downstream consequences of that modification. Changing a complex system is not the same thing as understanding it — a distinction well known to any engineer who has patched a large codebase and discovered the fix elsewhere it broke.
This structure has a genuine, and underappreciated, parallel in the Wigner's-friend thought experiment. In that scenario, a "friend" inside a sealed laboratory performs a measurement on a quantum system and obtains a definite result. From the friend's perspective, the measurement is complete and the outcome is settled. But from the perspective of Wigner, standing outside the laboratory, the friend and the measured system together remain in an undetermined superposition until Wigner performs his own measurement on the laboratory as a whole. Two observers, applying quantum mechanics consistently, can disagree about whether a determinate fact yet exists — and recent experimental work has shown this is not merely a philosophical puzzle but a testable and empirically real feature of layered measurement.
A self-improving AI system occupies a structurally similar position. From the system's own internal vantage point — the "friend" inside the laboratory — a self-modification may appear evaluated, verified, and complete: its internal metrics register improvement. But from the vantage point of an external observer — a regulator, a competing lab, a market, a future audit — that same modification may remain genuinely undetermined: unverified against criteria the system itself cannot access, its consequences not yet realized, its safety not yet established. The system's self-assessment and the world's eventual assessment are not guaranteed to coincide, and there may be no privileged vantage point from which to say, prior to that outside "measurement," which one is correct. This is not a claim that AI self-improvement is literally a quantum process. It is a claim that the logical structure of nested, disagreeing observers — one inside the system's own evaluative frame, one outside it — recurs here in a way that is more than coincidental, and that should make us skeptical of any account of recursive self-improvement that treats the system's own verdict on its progress as automatically authoritative.
VIII. The Environment Is Part of Intelligence — and Belief Is Part of the Environment
Perhaps the most consequential error in conventional singularity thinking is that it treats intelligence as an isolated computational property, sealed inside a chip or a set of weights. But even human intelligence is not located exclusively inside the brain. It emerges through continuous interaction with language, other people, institutions, tools, culture, physical environments, accumulated knowledge, economic incentives, technological infrastructure, and historical experience. Remove any of these and human cognitive performance changes markedly, even though the brain itself is unaltered.
An AI system is embedded in an analogous web of dependencies. Its effective intelligence is therefore not a property of the system alone but of the coupling between the system and its environment. The environment does not merely receive the AI's outputs passively; it changes the information available to the AI going forward, and therefore shapes the AI's subsequent behavior. The system becomes part of an ongoing feedback loop between itself and the world, rather than a closed mathematical recursion running in isolation.
There is a specific and well-documented mechanism by which this feedback loop operates, and it deserves to be named precisely rather than left as a vague gesture toward "interaction." Sociologists call it the Thomas theorem: if actors define a situation as real, it becomes real in its consequences, regardless of whether the original definition was accurate. Robert Merton formalized the same pattern as the self-fulfilling prophecy. In financial markets, George Soros gave it the name reflexivity — the observation that beliefs about a market are not merely predictions of it but active inputs that change the market's subsequent behavior, so that the belief and the reality it describes co-determine one another rather than one simply tracking the other.
The AI singularity question sits unusually close to this mechanism, in a way most physical questions do not. Nobody's belief about tomorrow's weather changes the weather. But a widespread belief among states, laboratories, and investors that a singularity is imminent is not a neutral prediction about an independent fact — it is itself a causal input into the system being predicted. That belief compresses safety timelines, intensifies competitive racing dynamics, directs capital toward maximally aggressive scaling, and treats caution or delay as an existential risk in its own right, since a rival left unregulated is assumed to reach the threshold first. Conversely, a dominant belief that the singularity is overhyped or impossible redirects funding, attention, and regulatory urgency elsewhere, and the system's trajectory bends away from what would have otherwise been pursued. In this sense, and without needing to invoke the contested physics literally, the AI-development system exhibits a structural kinship with the interpretive dispute in Section I: whether "the singularity will happen" behaves less like a fixed fact awaiting discovery and more like an outcome partly constituted by the collective act of hypothesizing about it, funding it, and building toward it. The measurement, in this reading, is not performed by a physicist with a detector but by markets, governments, and institutions — and it is not obviously a passive act of discovery.
The singularity hypothesis, in its classical form, imagines a system evolving purely as a function of its own prior state. The real world is closer to a system evolving as a function of its own prior state, its environment, its accumulated history, the beliefs of the institutions observing and funding it, and an irreducible residue of uncertainty that cannot simply be assumed away for the sake of a clean equation. The "singularity," under this more realistic picture, could never be a property of the machine alone. It would have to be a property of an entire evolving socio-technical system in which belief about the outcome is not separable from the outcome itself.
IX. Why the Real World Is Non-Aristotelian in a Deeper Sense
The phrase "non-Aristotelian mathematics" should not be read as a claim that classical mathematics is false. It should signify a mathematical attitude appropriate to systems in which categories are provisional, boundaries are porous, relationships are nonlinear, uncertainty is irreducible, and context changes the meaning of the variables themselves.
The real world frequently behaves less like a fixed quantity equal to itself and more like a quantity that depends on its environment, its history, and the very process used to measure it. In complex adaptive systems, the observer can become part of the system being observed, and the categories themselves can evolve over the course of observation. The map changes because the territory changes beneath it.
This is precisely what makes the AI singularity so difficult to define with any rigor. If an AI changes the technological, economic, and epistemic environment in which "intelligence" itself is measured, then the benchmark moves along with the system being benchmarked. The optimizer is changing. The environment is changing. The objective function may be changing. The constraints may be changing. There is consequently no guarantee that this recursive process possesses any stable mathematical object toward which it could even in principle converge.
X. The Singularity as a Limit That May Not Exist
A useful way to reframe the entire debate is to treat the singularity as a proposed mathematical limit: the claim, in its strongest form, that intelligence grows without bound as time approaches some finite future point. But why should such a limit exist at all? Why should it be reached in finite time rather than approached asymptotically without ever arriving? And why should "intelligence" remain a stable, well-defined variable as the system approaches that point, rather than becoming as ill-defined as the ratio of two infinities discussed earlier?
These are not rhetorical questions; they are the ordinary questions any mathematician asks of a proposed limit. A function can approach a boundary without ever reaching it. It can diverge. It can oscillate indefinitely. It can become undefined at the very point of interest. It can encounter a discontinuity and jump rather than climb smoothly. Or it can converge quietly to some finite value well short of infinity. All of these are live mathematical possibilities, and nothing about "rapid technological progress" rules any of them out in advance.
Rapid progress is not the same thing as infinite acceleration. Recursive improvement is not the same thing as recursive explosion. The mere existence of positive feedback in a system does not establish the existence of a singularity in that system — feedback loops are common; runaway divergence from them is comparatively rare, precisely because most real systems contain damping mechanisms that a purely formal treatment of the feedback loop leaves out.
XI. Why "Intelligence Explosion" May Be a Category Error
The phrase "intelligence explosion" is rhetorically powerful because it fuses two concepts that are each individually familiar and vivid. But intelligence may simply not behave like an explosive physical quantity. A chemical explosion releases stored energy according to well-understood physical laws operating over microseconds. An intelligence system, by contrast, operates through computation, data, energy, hardware, human infrastructure, institutional permission, scientific knowledge, and continuous interaction with an uncertain environment. Its rate of progress is constrained by an entire network of heterogeneous variables, most of which are not computational at all.
Even a system whose software could in principle be improved with extraordinary speed would still require the energy, hardware, data, communication bandwidth, physical experimentation, and empirical verification needed to translate that software improvement into real-world capability. Software can sometimes move faster than hardware, institutions, or physical infrastructure — but it cannot outrun the physical world indefinitely without eventually being throttled by it. The singularity, understood this way, risks being a vivid metaphor mistaken for a proven theorem.
XII. The "Gentle Singularity" and the Problem of Definition
Contemporary discussions sometimes soften the dramatic implications of the classical singularity by speaking instead of a "gentle singularity" — a period in which AI capabilities compound continuously and the overall pace of technological progress becomes increasingly rapid, without any single discontinuous break.
This is a considerably more defensible proposition, but it also quietly changes the meaning of the word "singularity." If "singularity" now simply means that AI is improving quickly, the concept stops identifying anything mathematically distinctive at all. Human technological history has already contained several episodes of accelerating capability — the printing press, electrification, telecommunications, digital computation, the internet, and modern biotechnology each transformed, in its own era, how quickly societies could generate and distribute knowledge. None of these episodes required a mathematical singularity to be historically transformative.
The real question, then, is not whether AI will produce accelerating technological change. It almost certainly will. The real question is whether that acceleration becomes self-sustaining, autonomous, unbounded, and effectively discontinuous with everything that came before it. That much stronger claim remains, as of this writing, unproven — and arguably underspecified enough that it is not yet clear what evidence would settle it either way.
XIII. The Skeptical Case Against Runaway Autonomy
The available evidence counsels caution rather than alarm. Large language models and other advanced AI systems can display extraordinary capabilities while still depending entirely on human-designed architectures, human-curated training procedures, human-built computing infrastructure, human-designed evaluation systems, human deployment decisions, and external sources of information that the system does not itself control.
A model can generate code that improves another system without thereby possessing an autonomous, persistent understanding of its own existence, its own objectives, or the long-run consequences of its own actions. There is an enormous conceptual distance between the claim that an AI can improve code, which is an engineering capability we already observe, and the claim that an AI can autonomously improve itself without meaningful external constraint, which is a claim about a self-sustaining evolutionary process we have not observed. The first is a fact about present systems. The second is a hypothesis about a future kind of system. Treating the second as though it followed automatically from the first is the central sleight of hand in most popular accounts of the singularity.
XIV. A Bayesian Interpretation — and Why It Is Not the Same Claim as Superposition
The most intellectually honest way to approach the singularity question empirically is through ordinary Bayesian updating. Let the hypothesis of a genuine runaway singularity be one hypothesis among several competing explanations for the evidence we observe, and let the observed evidence be, for instance, a period of unusually rapid improvement in AI capability. The probability we should assign to the singularity hypothesis, given that evidence, depends not only on how likely the evidence is if the singularity hypothesis is true, but on how likely that same evidence is under the competing, more modest hypotheses — and on how plausible the singularity hypothesis seemed before the evidence arrived.
Suppose AI performance improves dramatically because of better chips, larger training sets, improved algorithms, and a surge of capital investment — which is, in fact, the best description of the period we are currently living through. That evidence provides strong support for the modest hypothesis that AI capabilities will continue to improve. It provides substantially weaker support for the stronger hypothesis that AI will autonomously redesign itself without meaningful external bottlenecks. And it provides weaker support still for the strongest hypothesis, that AI development will become explosively recursive and effectively uncontrollable. These three hypotheses are nested, and the evidence that confirms the weakest of them does comparatively little to confirm the strongest.
It is worth being precise here about a distinction that is easy to blur, especially once quantum vocabulary has entered the discussion. Bayesian uncertainty describes a fact that already has a definite value, which we simply do not yet know — the coin has already landed, we just haven't looked. Superposition, on the more radical readings discussed in Section I, describes something stronger: a fact that does not yet have a determinate value prior to a measurement-like interaction that fixes one. These are not the same kind of "unknown," and conflating them would be a genuine equivocation. My claim in this essay is not that the singularity's truth-value is in superposition in the strict physical sense. It is that the reflexive mechanism described in Section VIII — belief as an input to the outcome — produces something that functions like the softer, participatory reading of measurement, even though the underlying uncertainty about current evidence remains ordinary Bayesian uncertainty. The two operate on different objects: Bayesian updating governs our confidence about which hypothesis is true given present evidence; reflexivity governs whether the future fact itself is independent of that confidence, or is partly shaped by it. Both are at work here, and they should not be collapsed into one loose sense of "unknown."
The intellectually disciplined position is therefore neither "the singularity is inevitable" nor "the singularity is impossible." It is that the probability of the strongest hypothesis must be continually updated, soberly and incrementally, as genuinely new evidence accumulates — while recognizing that the accumulation of that evidence, and the institutional behavior it triggers, is not a neutral act of observation but partly constitutive of the very trajectory being assessed.
XV. The Deeper Philosophical Point
The deepest problem with the singularity hypothesis may ultimately be ontological rather than technological. The hypothesis assumes that intelligence can become an object sufficiently well defined to be recursively optimized in the way an engineer optimizes a bridge design or a chess program optimizes a position. But intelligence may be relational rather than intrinsic. A mind is intelligent relative to a problem, an environment, a language, a body, a history, and a set of purposes — not intelligent in some absolute, context-free sense that could in principle be pushed toward infinity.
Once intelligence is understood relationally, the notion of an infinitely increasing intelligence becomes genuinely difficult to make sense of. What would it mean to be infinitely intelligent in a world containing irreducible uncertainty? What would it mean to be infinitely intelligent when the objective function itself is uncertain, contested, or changing? What would it mean to optimize perfectly when there are competing values that cannot be reduced to a single numerical ordering without doing violence to what made them values in the first place? The difficulty here resembles the earlier example of infinity divided by infinity: the trouble is not that infinity is "too large" to reach, but that the expression itself does not determine a unique answer. In the same way, "superintelligence" does not by itself determine a unique trajectory. More intelligence does not uniquely determine more wisdom. More predictive power does not uniquely determine better judgment. More optimization does not uniquely determine better goals. More raw capability does not uniquely determine better outcomes.
XVI. From the Singularity to the Complexity Horizon
For these reasons, it may be more useful to replace the idea of a technological singularity with the concept of a complexity horizon: the point beyond which our ability to predict the consequences of technological change deteriorates faster than our ability to generate that change in the first place.
This reframing does not require infinite intelligence. It does not require an uncontrollable machine. It does not require any single dramatic threshold at all. It requires only that the rate at which complexity is generated begin to outpace the rate at which our predictive and institutional capacities can keep up with it — a condition that is entirely plausible on its own terms, and one that arguably already characterizes portions of modern technological civilization even without any exotic assumptions about self-improving machines.
Under this interpretation, the important question is no longer when will AI become infinitely intelligent? It becomes: when will the complexity generated by AI exceed the epistemic capacity of the institutions attempting to govern it? That is a far more consequential question for policymakers — and, importantly, a far more empirically testable one, since it can be tracked through concrete indicators of institutional adaptation rather than through speculation about an undefined future threshold.
XVII. Conclusion: Not a Fact Awaiting
Discovery, but an Outcome Awaiting Measurement The strongest version of the AI singularity hypothesis requires a chain of assumptions considerably more demanding than the phrase "recursive self-improvement" suggests on its own. It requires intelligence to behave sufficiently like a scalar quantity. It requires improvements to remain comparable across generations of systems that may differ from one another in fundamental architecture. It requires recursive improvements to compound rather than run into diminishing returns. It requires the objective function guiding improvement to remain stable even as the system pursuing it changes. It requires self-modification to remain predictable despite the problem of self-reference. It requires the surrounding environment to remain sufficiently controllable. It requires physical and institutional constraints to become secondary considerations. And, beneath all of this, it requires that "will the singularity happen" be the kind of question with a single, observer-independent answer waiting to be uncovered — an assumption that even physics, in its own much narrower domain, has not been able to fully secure.
None of the classical preconditions follows automatically from the observation that AI systems are becoming extraordinarily capable. The real world is not a smooth, perfectly continuous mathematical surface. It contains discontinuities, thresholds, feedback loops, path dependence, irreducible uncertainty, competing equilibria, and genuinely emergent properties. Small changes are sometimes irrelevant; at other times they become decisive, and there is no general rule for telling in advance which case one is in. Two systems that appear identical at one level of description can diverge dramatically once embedded in different environments and histories — and, as Section VIII argued, once embedded in different structures of belief about what they are becoming.
The classical mathematical habit of treating a quantity plus a negligible difference as though it were simply that same quantity is indispensable to science and engineering. But the dangerous mistake is to assume that the discarded difference is always negligible. In complex adaptive systems, the supposedly negligible term can eventually become the entire story — the tail that was meant to be ignored turns out to be where all the interesting behavior was hiding.
The AI singularity hypothesis may therefore contain something close to a paradox, and it is a paradox with a specific shape. It imagines machines transcending human cognitive limitations through increasingly perfect mathematical optimization of a pre-existing, mind-independent trajectory — even as the world into which those machines are deployed is one where the trajectory itself is partly constituted by the institutions observing, funding, racing toward, or regulating away from it. There may be no single moment at which intelligence becomes infinite, prediction becomes impossible, and history abruptly passes through a technological singularity that was there all along, waiting to be reached. There may instead be a prolonged transition in which the outcome remains genuinely unsettled — not merely unknown in the ordinary Bayesian sense, but unsettled in the stronger sense that different institutional "measurements," different governance regimes, different deployment choices, and different collective beliefs would collapse it differently. The future of artificial intelligence, on this reading, is less like a coin that has already landed and simply awaits inspection, and more like a question that different observers, acting differently, would answer into being differently.
The most profound limitation on the singularity hypothesis is therefore not that machines cannot become extraordinarily intelligent. They may well become so. The deeper limitation is that extraordinary intelligence does not, by itself, imply a mathematically singular future — nor does it imply that such a future exists as a fixed fact independent of how humanity chooses to observe, fund, race toward, and govern it. The future remains what both the non-Aristotelian mathematics of complex systems and the unresolved measurement debates of quantum physics separately suggest it should be: contingent, relational, path-dependent, discontinuous, and only ever partially knowable — and, on the more radical but still respectable physical reading, not fully knowable even in principle prior to the interaction that decides it.
The real question is consequently not whether artificial intelligence will become infinite. It is not even, purely, whether it will happen. It is whether human beings — as the observers, the funders, the regulators, and the friends inside the laboratory all at once — can remain epistemically adaptive in a world where the act of asking the question is itself part of what will determine the answer. That is not a singularity in the classical sense.
It is a permanent epistemological transition, and, on the reading this essay has proposed, a permanently unfinished measurement.
