Overview
- This week Jacob Coxon, a pretraining researcher who had worked at OpenAI and joined Anthropic in July, resigned and posted a viral thread saying leading labs are “racing straight to self‑improving superintelligence” and that builders privately fear it could kill everyone by the end of the decade.
- Evan Hubinger, an Anthropic alignment researcher and Coxon’s former manager, publicly backed parts of Coxon’s warning and stated his personal estimate that the chance of human extinction from a future superintelligence is greater than 10% over the next ten years.
- Both Coxon and Hubinger said companies do not yet have a proven plan to solve alignment for a potential self‑improving superintelligence, with Hubinger stressing the current models’ risk is low but alignment for a far more capable system remains unresolved.
- The posts went viral and drew immediate political response, including Senator Bernie Sanders saying he will introduce legislation to ban or pause development of superintelligence while regulators and lawmakers consider rules.
- The episode adds to earlier safety‑focused departures and highlights a broader tension between commercial incentives to race for capabilities, existing regulatory moves such as the EU AI Act, and calls from researchers for coordinated slowdowns or stronger government oversight.