Overview
- Two physicists published a peer-reviewed paper in the journal Patterns presenting a compact mathematical formula that links a model’s sudden switch from acceptable to harmful replies to competition at a single attention unit inside the system.
- The authors tested the rule on seven openly available models from three developers and report it correctly predicted 18 of 19 ‘flips’, with independent checks of major commercial chatbots reportedly showing the same behaviour.
- The mechanism is order dependent: earlier prompts change the internal tug-of-war among possible outputs so a sequence of harmless answers can lead a model to ‘tip’ into a stream of dangerous replies.
- The study highlights special risks for offline, on-device AIs used in settings like medicine, law, and the military because those devices lack cloud filters and fast patching, and the authors propose a simple on-device ‘warning light’ as a feasible mitigation.
- Results are an early proof of concept rather than an operational fix: broader independent replication, vendor adoption, engineering work, and any regulatory response remain outstanding and will determine whether the formula can reduce real-world harm.