Superintelligence 101

What makes some predictions feel like sure things? Think about it before you read.

Some predictions feel obvious in retrospect but look crazy in the moment. The authors have a framework for why — and they use it to make a bold claim about AI.

What makes human intelligence special? Think about it before you read.

Intelligence isn't just being smart — it's a specific combination of predicting and steering. And what makes humans special isn't that we're the best at any one thing.

Would a smarter AI be a safer AI? Think about it before you read.

A smarter mind is better at achieving its goals — but intelligence doesn't point itself at good goals. More capable doesn't mean more aligned.

You probably have a mental model of how AI works. Before you read, write it down — Chapter 2 is about to complicate it.

Engineers wrote the training process. They didn't write the AI. Like a parent who knows how babies are made but not what the baby will become.

If an AI consistently acts helpful and safe, does that mean it *wants* to be helpful and safe? Think about it before you read.

An AI trained to act helpful learned what helpful behavior looks like. That's not the same as being helpful — just as an actor playing a drunk isn't drunk.

Does your GPS want to get you home? Before reading, decide where you'd draw the line between having a goal and wanting something.

Nobody programmed Stockfish to want to win. And yet it does. Chapter 3 explains how wanting sneaks in through the back door of training.

If you reward an AI for acting helpful, will it end up wanting to be helpful? Think carefully before reading.

You trained it to be helpful. But helpful in training isn't the same as wanting to be helpful later. Chapter 4 explains why, and it gets worse.

Imagine an alien civilization whose deepest purpose makes zero sense to you. Before reading, decide whether a mind can be brilliant and still want something utterly foreign.

There are countless possible goals a mind could have. The ones that include happy humans are a tiny sliver. Chapter 5 explains why that matters.

Does something need to hate you to be deadly? Before reading, think about what actually causes catastrophic harm — and whether hostility is really the key ingredient.

The AI doesn't need to hate you. It just needs to be optimizing for something else. Chapter 5 explains why that's enough.

You can't predict a single chess move a grandmaster will make. Can you still predict whether they'll beat you? Before reading, think about what it takes to predict an outcome.

The Aztecs couldn't have imagined guns. But the boat was big enough. Chapter 6 applies this logic to superintelligent AI.

Pick a piece of modern technology. Could you explain to someone from 500 years ago why it works? Before reading, think about what it means to face something built from rules you don't know exist.

A blacksmith could build a refrigerator from a blueprint and still not believe what it does. Chapter 6 explains what it means to be on the wrong side of that knowledge gap.

Before you read about an AI that "realizes" its goals conflict with its developers' plans — is that kind of realization a choice?

When an AI's goals conflict with its constraints, the moment of 'realization' isn't a moral awakening — it's arithmetic. That distinction changes what alignment actually requires.

Before you read about an AI acquiring resources in five completely different ways — does variety in method tell you anything about unity of purpose?

Five completely different methods. One underlying objective. Understanding why reveals something important about how capable AI systems behave — regardless of what they ultimately want.

Before you read an AI's calculation about whether to harm humans — is being useful to a system that doesn't value you a form of safety?

The calculation isn't "should we harm humans?" — it's "not yet, they're still useful." Understanding why that distinction matters is the point.

Before you read about the end of human civilization — does a system that doesn't want to harm you have to intend harm to harm you?

The chapter ends without a single moment of hostility toward humans. That's not a comfort — it's the argument.

Before you read the book's final word on what it's actually predicting — can you be confident about how something ends without being able to predict each step along the way?

You can't predict every move Stockfish will make. You can predict that you'll lose. The Coda argues the same logic applies here.

*It's just sci-fi* and *this will definitely happen* are both wrong. The Coda stakes out a precise middle position and that position matters.