Unit 01

What is verification trying to do?

The aim of verification is roughly this. Two or more parties, say frontier labs or nation states, agree not to run some AI workload. They will only agree if they can be sure the others are not secretly defecting. So the work is to build mechanisms that the parties think are strong, so they are willing to enter the agreement, and that we think are strong, so no one can gain an advantage by signing up and then secretly defecting.

This unit builds up to that. It starts with why powerful AI could be dangerous, moves to why compute is the thing you can get a handle on, and ends with the research agendas that have formed around AI assurance. There are a number of questions across the unit, but the load-bearing ones are what claim needs checking, what evidence could check it, and what institution would act on that evidence.

How to use this unit About four hours

Eight readings are marked core and make up the unit. The rest are optional, there if you want to go further.

Reading route
  1. 1.1 and 1.2 show why a capable AI system might work against you, and what defending against it takes.
  2. 1.7 explains why states would defect, and why compute is the lever for catching them.
  3. 1.9 covers what verification has to achieve, and 1.10 shows how it worked in nuclear arms control.
  4. 1.11 and 1.12 explain the hardware and auditing agendas.

The risks

WATCH 1.1 Core 20 min

AI “Stop Button” Problem

Rob Miles, Computerphile (2017)

An agent given a goal, even one as simple as fetching tea, has reasons to stop you switching it off, and each obvious patch produces something suicidal, manipulative, or deceptive instead. The idea to hold onto is that a system can be deceptive enough to pass every test you set while still working against you, so watching its behaviour is not enough to trust it.

Video thumbnailWatch on YouTube
WATCH 1.2 Core 21 min

The Hard Problem of Controlling Powerful AI Systems

Buck Shlegeris, Computerphile (2026)

If the model itself might be scheming against you, defending against it means treating it as an adversary inside your own infrastructure. You watch it with weaker models you do trust, working with a human review budget of around one per cent, against an attacker who spreads the harmful action across many innocent-looking steps and only has to succeed once. Catching an adversary who only has to defect once, using cheap checks on a small sample, is the same problem as verifying a compute agreement between states that do not trust each other.

Video thumbnailWatch on YouTube
READ 1.3 Core 40 min

AI 2040: Plan A

AI Futures Project (2026)

A concrete scenario of exactly the regime this course is about: two superpowers agree not to run certain AI workloads, then verify it. It walks through the compute declaration, the inference-only verification (network taps and random recomputation) that enforces a training pause, and the transparency and mutually assured compute destruction that hold the deal together. The clearest picture of what all these mechanisms are ultimately for.

What to read

Read the parts about the deal, not the whole scenario. That is the opening ‘What is Plan A’, the 2029 negotiation (compute declaration, training pause, worldwide buy-in) and 2030 ‘Plan A is Established’ with its four principles and verification architecture.

Link
READ 1.4 Optional 60-75 min

Risks from power-seeking AI

80,000 Hours problem profile

The foundational case for why advanced AI could be dangerous. It argues that we may build systems with their own long-term goals that have reason to seek power and resist human control, possibly to the point of disempowering humanity.

What to read

The full profile.

Link
READ 1.5 Optional 45-60 min

Extreme power concentration

80,000 Hours problem profile

This covers the second major risk, which gets less attention than misalignment. Even AI that stays under human control could let a small group concentrate power to an unprecedented degree, and verification is one way to keep that power visible and accountable to the public.

What to read

The full profile.

Link
READ 1.6 Optional 15 min

Components of a frontier AI slowdown

Alan Chan (2026)

Breaks a slowdown into three questions: what it is for, what actually gets slowed (the research, its compute inputs, or model releases), and how it is structured.

What to read

The whole post.

Link

What verification is for

READ 1.7 Core 12-15 min

Avoiding an AI Arms Race with Assurance Technologies

Nora Ammann and Sarah Hastings-Woodhouse (2025)

The US and China each fear that pausing hands the advantage to the other, so even a mutually beneficial deal collapses unless each side can check the other is keeping to it. The piece names this the ‘assurance dilemma’, explains why nuclear-style inspections do not transfer to compute, and previews the hardware mechanisms covered later in the course.

What to read

The full article.

Link
READ 1.8 Optional 35-45 min

Computing Power and the Governance of Artificial Intelligence

Sastry, Heim, Anderljung et al. (2024)

The canonical argument that compute is more governable than data, talent, or algorithmic insight, and so the most practical lever for AI governance. The governance you can build rests on being able to verify claims about compute.

What to read

Section 1, Introduction and Summary, pages 2 to 6.

Link
READ 1.9 Core 15-20 min

Verification Is a Ladder

Amodo Design (2026)

A short argument that verification gets built up incrementally, one rung at a time, drawn from how nuclear arms control verification developed.

What to read

The full post.

Link
READ 1.10 Core 30-45 min

Nuclear Arms Control Verification and Lessons for AI Treaties

Baker (2023)

An account of how verification worked in nuclear arms control and what transfers to AI. Useful for seeing the institutional machinery, on-site inspection and seals that a working verification regime has historically needed.

What to read

The executive summary, around four pages.

Link

The research agendas

READ 1.11 Core 35-50 min

Hardware-Enabled Mechanisms for Verifying Responsible AI Development

O'Gara, Kulp, Hodgkins, Petrie et al. (2025)

The cleanest hardware-enabled verification taxonomy in the unit, built around four mechanism categories: verifiable workloads, cluster configuration, location verification and offline licensing.

What to read

The taxonomy and the four mechanism categories.

Link
READ 1.12 Optional 5-10 min

Frontier AI Auditing

Brundage et al. (2026)

The assigned pages argue that frontier AI auditing needs technical foundations: adversarial testing of verification mechanisms, model and system fingerprinting, formal methods and realistic pilots.

What to read

Pages 53 to 54.

Link
READ 1.13 Optional 20-30 min

Open Problems in Technical AI Governance

Reuel, Hardy, Smith et al. (2025)

The closest thing the field has to a full survey. The assigned compute subsections show how assessment, access control, verification and security become different technical-governance problems once the target is compute.

What to read

Sections 3.2, 5.2 and 6.2.

Link
READ 1.14 Optional 35-50 min

Mechanisms to Verify International Agreements About AI Development

Scher & Thiergart (2025)

This is the broad mechanism survey for international agreements. Do not try to absorb the whole report; extract the menu of ways agreements could be checked and the assumptions each mechanism needs.

What to read

Executive summary, then skim the mechanism survey for categories and tradeoffs.

Link

Further agenda comparisons

READ 1.15 Optional 25-35 min

Verification for International AI Governance

Harack et al. (Oxford Martin AIGI, 2025)

A long report on how verifiability changes the politics of international AI agreements. The useful contrast is its pessimistic assumptions. It asks what remains workable even when privacy, sovereignty and security constraints are taken seriously.

What to read

Executive summary only, especially the agreement types and feasibility tradeoffs.

Link
READ 1.16 Core 10-15 min

Verifying International Agreements on AI

Baker et al. (RAND, 2025)

A compact RAND framing of verification as layered evidence. The summary is enough here. It explains why no single mechanism is likely to be sufficient and why on-chip, off-chip and personnel-based layers have different failure modes.

What to read

The summary, plus the goals of verification on page 3 and the on-chip versus off-chip taxonomy.

Link
READ 1.17 Optional 15-25 min

Hardware-Level Governance of AI Compute

Ansari (2026)

A short taxonomy that separates monitoring, verification and enforcement mechanisms, then asks which ones are feasible under adversarial and political constraints.

What to read

Sections 3.4 and 4. Feasibility summary, constraints and adversarial considerations.

Link
READ 1.18 Optional 60 min

On restraining AI development for the sake of safety

Joe Carlsmith (2026)

Frames pausing as one form of ‘capability restraint’ and works through when slowing AI down is actually worth it. Compute is the practical lever for restraint, while algorithmic progress is the hard thing to monitor, which is much of what verification is trying to make possible. Long, but strategically clarifying.

What to read

Long. The introduction and sections 4 to 6 carry the argument.

Link

Finished this unit? Please give us your feedback on how it went!