16 AGI and ASI
In early 2023, a team at Microsoft Research published a paper with an audacious title: “Sparks of Artificial General Intelligence: Early experiments with GPT-4” (Bubeck et al. 2023). The paper argued that GPT-4 exhibited glimmers of general intelligence: it could write poetry, solve math problems, generate code, reason about spatial relationships, and even display a rudimentary theory of mind. The AI community erupted in debate. Some called it premature hype. Others called it a wake-up call. Everyone agreed on one thing: the conversation about AGI had shifted from “if” to “when.”
This chapter explores the most consequential question in artificial intelligence: what happens when machines become as smart as us, and what happens when they become smarter?
16.1 What Is AGI?
Artificial General Intelligence (AGI) refers to an AI system that matches or exceeds human cognitive abilities across virtually all domains. Not just chess. Not just image classification. Not just text generation. Everything: open-ended reasoning, learning from minimal examples, creativity, common sense, social intelligence, and the ability to transfer skills to entirely novel situations.
There is no consensus definition of AGI, and this is itself a problem. Without a clear definition, claims of “achieving AGI” become unfalsifiable. Is a system that passes every benchmark but cannot tie its shoes AGI? Is a system that matches the average human but not the best? The lack of a precise target makes AGI simultaneously overhyped and underappreciated.
Several frameworks have been proposed to make the concept concrete:
Franois Chollet's ARC-AGI benchmark (Chollet 2024) argues that intelligence should be measured by skill-acquisition efficiency: how quickly a system can learn new tasks from minimal examples. ARC (Abstract and Reasoning Corpus) consists of visual puzzles that require inductive reasoning from just a few examples. Humans solve them easily; current LLMs struggle badly. Chollet's key insight: memorizing the internet is not intelligence. The ability to generalize from sparse data is.
The Turing Test, proposed by Alan Turing in 1950 (Turing 1950), asks: can a machine fool a human judge in open-ended conversation? Modern LLMs can pass simplified versions, but the test's validity is increasingly questioned. Being convincingly human-like in conversation may not require general intelligence, just very good statistical patterns.
The Economic Turing Test asks: can an AI perform any economically valuable task that a human can? This pragmatic definition sidesteps philosophical debates about consciousness and understanding, focusing purely on capability.
DeepMind's Levels of AGI framework (Morris et al. 2023) (Morris et al., 2023) proposed a matrix with five levels (Emerging, Competent, Expert, Virtuoso, Superhuman) across two dimensions (narrow vs. general). Under this framework, current LLMs are “Emerging AGI”: they show broad but shallow competence across many tasks.
16.2 Is Current AI AGI-Adjacent?
Bubeck et al. (Bubeck et al. 2023) argued that GPT-4 shows “sparks” of AGI based on its broad competence across diverse tasks. The paper demonstrated GPT-4 generating 3D models from text descriptions, writing working code for complex algorithms, and showing apparent understanding of spatial and temporal relationships.
But critics raise important objections:
- Brittleness: LLMs fail catastrophically on tasks that require genuine understanding. Change the wording of a math problem slightly, and a model that solved it perfectly may fail completely.
- No world model: LLMs may not build internal models of the world (though evidence from Othello-GPT and other probing studies suggests they might represent more than we think).
- Training data contamination: The impressive “reasoning” may be sophisticated pattern matching on problems seen during training, not genuine generalization.
- No persistent memory or learning: Current LLMs cannot learn from experience without retraining. Every conversation starts from scratch.
John Searle's 1980 “Chinese Room” thought experiment (Searle 1980) is more relevant than ever. If a person follows rules to manipulate Chinese characters without understanding Chinese, are they “understanding” Chinese? If an LLM generates correct answers by manipulating statistical patterns without “understanding” the content, is it intelligent? The question may ultimately be unanswerable, but it highlights the difference between capability and comprehension.
16.3 What Is ASI?
Artificial Superintelligence (ASI) refers to an AI system that vastly surpasses the best human minds in every domain: science, art, social skills, strategic planning, and general wisdom. If AGI is “as smart as a human,” ASI is as far beyond us as we are beyond ants.
Nick Bostrom's Superintelligence (Bostrom 2014) laid out the foundational arguments:
Recursive self-improvement: An AGI-level AI that can improve its own architecture, training process, or code could enter a feedback loop of exponential capability gain. Each improvement makes the system better at making further improvements. This “intelligence explosion” could turn an AGI into an ASI in a period of days, hours, or even minutes.
Speed advantage: Silicon-based computation is millions of times faster than biological neural signaling. An AI running on modern hardware could do a year of human-level thinking in seconds, and can be copied across thousands of machines in parallel.
Qualitative superiority: ASI might not just think faster; it might think in ways we cannot comprehend, just as calculus is incomprehensible to a chimpanzee. There may be cognitive abilities that are simply beyond the reach of human-level intelligence.
Stuart Russell (Russell 2019) frames the risk of ASI with a thought experiment: “We are to superintelligence what gorillas are to us. Our survival depends on our superior intelligence. If we create something smarter than us, we need to make very sure it has our interests at heart.” The gorilla does not get to vote on deforestation decisions. Will we get to vote on superintelligence decisions?
16.4 Paths Toward AGI
Several research directions are proposed as steps toward AGI:
The Scaling Hypothesis suggests that continuing to scale model size, data, and compute will eventually produce AGI. Proponents point to emergent capabilities that appear at scale (in-context learning, chain-of-thought reasoning, tool use) and argue that more emergent capabilities will continue to appear. Critics argue that scaling produces better pattern matching, not genuine understanding.
World Models represent Yann LeCun's alternative vision (LeCun 2022). LeCun argues that current LLMs cannot achieve AGI because they lack a world model: an internal representation of how the world works that enables prediction, planning, and counterfactual reasoning. His JEPA (Joint Embedding Predictive Architecture) framework proposes learning representations by predicting abstract features rather than pixel-level details (see Chapter 19).
Reasoning and Reinforcement Learning combine LLMs with RL-trained reasoning. Models like o1 (OpenAI 2024) and DeepSeek-R1 (Guo et al. 2025) show that AI systems can learn to “think” step-by-step, explore multiple solution paths, and verify their own answers. This test-time compute scaling may be a more efficient path to intelligence than parameter scaling.
Embodiment argues that grounding in a physical body is necessary for developing common sense about the physical world. You cannot truly “understand” what “heavy” means without the experience of lifting things. Robotics companies like Figure AI and 1X are pursuing this path.
Neuro-Symbolic AI combines neural networks with symbolic reasoning: logic, knowledge graphs, formal verification, and structured programs. The idea is that neural networks are excellent at perception and pattern recognition while symbolic systems are excellent at reasoning and planning. Combining them might yield the best of both worlds.
16.5 The Alignment Problem
If we build AGI, how do we ensure it does what we want? This is the alignment problem, and many researchers consider it the most important unsolved problem in AI safety.
The alignment problem is not about preventing AI from “deciding to be evil.” It is about the difficulty of precisely specifying what we want. Consider a simple objective: “maximize human happiness.” A misaligned superintelligence might achieve this by drugging everyone, or by wireheading (directly stimulating pleasure centers), or by killing all unhappy people. The objective is satisfied, but the outcome is disastrous. Human values are nuanced, context-dependent, and often contradictory. Translating them into a formal objective that a superintelligent system cannot exploit is extraordinarily challenging.
Key sub-problems include:
Outer alignment: Specifying the right objective. Human values are complex, context-dependent, and hard to formalize. RLHF helps but relies on human labelers who may be inconsistent, biased, or unable to evaluate superhuman outputs.
Inner alignment: Ensuring the model's learned objective matches the specified one. A model may learn proxy behaviors during training (e.g., “produce responses that get high ratings”) that diverge from the intended goal in novel situations (e.g., being sycophantic rather than truthful).
Scalable oversight: As AI systems become more capable than humans in certain domains, how do we verify that their behavior is correct? We cannot evaluate a mathematical proof we do not understand, or a scientific discovery we cannot replicate.
Corrigibility: Designing AI systems that allow humans to correct or shut them down, even if the AI's objectives would “prefer” it stay active. A sufficiently capable AI might resist correction not out of malice, but because being shut down prevents it from achieving its objective.
16.6 Governance and Regulation
As AGI moves from science fiction to plausible near-term reality, governance becomes urgent:
Compute governance: Since training frontier models requires enormous compute, regulating access to advanced AI chips (like NVIDIA's H100/B200) is one of the most practical levers for controlling AI development. The U.S. export controls on advanced chips to China represent the first major exercise of this lever.
Safety evaluations: Several governments are establishing AI safety institutes (the UK's AISI, the US AISI) tasked with evaluating frontier models before deployment. These institutes test for dangerous capabilities: can the model help create bioweapons? Can it autonomously replicate itself? Can it manipulate humans?
International coordination: AI development is a global race, and unilateral regulation risks putting one country at a disadvantage without reducing global risk. The Bletchley Declaration (2023) was a first step toward international coordination, but meaningful enforcement mechanisms remain elusive.
Open-source governance: How do you govern open-weight models that anyone can download and modify? Traditional regulatory frameworks assume centralized control. Open-weight models challenge this by distributing capability to millions of users, most of whom use it responsibly, but some of whom may not.
AI governance is often compared to nuclear governance, but the analogy has limits. Nuclear weapons require rare materials, enormous facilities, and nation-state resources. AI models require only GPUs (widely available), data (everywhere), and expertise (increasingly accessible). You cannot build a nuclear weapon in your garage, but you can fine-tune a language model on your laptop. This makes AI governance fundamentally harder: the proliferation problem is orders of magnitude more difficult.
16.7 Where Does This Leave Us?
The honest answer: nobody knows. There is genuine disagreement among leading researchers about whether current approaches can reach AGI, how soon it might happen, and how dangerous it would be. Optimists like Demis Hassabis predict AGI within a decade. Skeptics like Yann LeCun argue that fundamental breakthroughs are needed. And safety researchers like Paul Christiano and Eliezer Yudkowsky warn that the window for getting alignment right may be closing fast.
What is clear is that these questions are no longer purely academic. The decisions made in the next decade about how to develop, deploy, and govern advanced AI systems will shape the trajectory of human civilization. Understanding the technical landscape, from transformers and training to interpretability and alignment, is essential for anyone who wants to participate in these decisions.
The AGI debate is noisy, with loud voices at both extremes. To navigate it productively: (1) Read the primary sources (papers, not tweets). (2) Distinguish between “current systems cannot do X” and “no future system can ever do X.” (3) Be skeptical of both hype and dismissal. (4) Focus on concrete technical problems (alignment, interpretability, robustness) rather than abstract philosophical debates. (5) Follow researchers who change their minds when evidence changes (a sign of intellectual honesty, not weakness).
16.8 Exercises
- Define AGI in your own words. What capabilities must a system demonstrate before you would call it “generally intelligent”? Compare your definition with Chollet's skill-acquisition efficiency framework and DeepMind's levels framework. Do they agree?
- Read the ARC-AGI benchmark description (Chollet 2024). Try solving a few puzzles yourself at
arcprize.org. Then try to get an LLM to solve them. What strategies does the LLM try? Where does it fail? - Discuss the alignment problem. RLHF and Constitutional AI are current approaches. Can they scale to superintelligent systems, or are fundamentally different approaches needed? Write a one-page argument for each position.
- Read Nick Bostrom's “treacherous turn” argument (from Superintelligence). Then read a counterargument (e.g., from Robin Hanson or Yann LeCun). Who do you find more persuasive, and why?
- Design a thought experiment that would help you distinguish between “truly understands” and “very good pattern matching.” Is your experiment actually conclusive, or could a sufficiently good pattern matcher pass it?