On September 8, Jacob Coxon, a 27-year-old researcher, resigned from Anthropic, one of the largest artificial intelligence (AI) companies. He announced his resignation on X and wrote that neither Anthropic nor OpenAI, where he had also worked, is acting responsibly. He wrote that the people building AI “earnestly believe that it could kill us all by the end of the decade.”
Researchers have quit AI companies with warnings before, and the public mostly did not notice. But this post has been viewed more than 173 million times, likely because Evan Hubinger, a senior researcher who leads alignment stress testing at Anthropic, replied that Coxon was correct. Alignment is the work of making sure an AI system does what its builders intend. Hubinger then added a number. He said he personally thinks the chance that AI kills all humans within the next decade is greater than 10 percent.
Much of the coverage has asked whether that number is right. I think the better question is what kind of number it is.
People in the AI field have a name for it. They call it p(doom), which is short for the probability of doom. It is the chance a person assigns to AI causing human extinction or something close to it. Researchers ask each other for their number in interviews and at conferences, and the answers cover almost the whole range.
Dario Amodei, the chief executive of Anthropic, said last year that he saw a 25 percent chance that things go very badly. A researcher at OpenAI recently said that his number is 70 percent unless there is regulation or a coordinated slowdown. Roman Yampolskiy, a computer scientist at the University of Louisville, has argued that there is almost a 100 percent chance that artificial general intelligence (AGI) will eventually destroy human civilization. In a 2022 tournament run by the Forecasting Research Institute, AI specialists put the chance of extinction from AI by 2100 at about 3 percent. Superforecasters in the same tournament put it below half of 1 percent. When careful people are asked similar questions and their answers run from under 1 percent to over 99 percent, the answers are not measurements.
Several commentators compared 10 percent to Russian roulette, where the chance of death is about 17 percent. But that comparison is misleading. We know the Russian roulette figure because a revolver has six chambers and one holds a bullet. We can calibrate the chance of rain tomorrow because forecasters have records of thousands of similar days, and we can check afterward how often they were right. An insurance company knows the chance of a house fire because it has records on millions of houses.
None of that exists for extinction from AI. It has never happened, so no one can count how often it happens.
No forecaster has a past record on this question that we could check. The numbers also change when the question changes. A 2023 survey of more than 2,700 AI researchers asked about extinction in slightly different ways. The typical answer was 5 percent under one wording and 10 percent under another.
The economist Frank Knight described this difference in 1921. He used the word risk for cases where the odds can be calculated, and the word uncertainty for cases where they cannot. Extinction from AI is a case of uncertainty. When we put a percentage on it, we make uncertainty look like risk. Bayesian statisticians would say that anyone can assign a personal probability to a one-time event. They are right, but that personal probability reports how strongly someone believes, not how often something happens.
Amodei made a related point on CNN. He said a single percentage makes the outcome sound like “a roll of the dice,” when it depends on the decisions companies and governments make. Dice do not respond to what we do. This outcome does.
So is the number meaningless? No. A statement like Hubinger’s reports belief. It tells us nothing precise about the future, but it tells us a good deal about the present. A person who studies the most advanced AI systems every day, and who can see systems the public cannot, is worried enough to say so under his own name while still employed by the company.
Actions tell us the same thing. According to Time, Coxon quit two months before his company shares would have vested. Four days after his post, Amodei published an essay that asked AI companies, including his own, to slow the pace at which they make their models more capable. Sam Altman of OpenAI said he agreed. Executives rarely ask to slow down their own industry.
There are reasons to be skeptical. These companies have financial interests, and some critics argue that alarm helps large companies win rules that would hurt smaller competitors. The computer scientist Melanie Mitchell called the 10 percent figure an “evidence-free claim.” In the narrow sense, she is right. Until Amodei and Altman very recently committed to giving independent evaluators access, no one outside the companies could inspect the unreleased systems that worry these researchers.
If we are not able to verify the number, then what should we do? The reasonable response is to read ‘10 percent’ as a statement. The people building this technology fear where it is going, and they do not yet know how to make it safe. Hubinger said as much, and wrote that Anthropic does not yet have a plan to solve alignment for superintelligent systems.
Citizens can act on that statement. We can call for greater transparency from the companies, particularly about safety incidents and accidents. We can ask our representatives what they plan to do. The worry is real even though the number is not a measurement, and what happens next depends on decisions that have not yet been made.
Disclosure: I used an AI assistant for some research, to test my arguments, and for help with style. The ideas and arguments are my own.
P. ANAND RAO is the Director of the Center for AI and the Liberal Arts at the University of Mary Washington.
