Overview
- Stanford researchers evaluated 24 widely used large language models on questions separating belief, knowledge and fact, publishing the results in Nature Machine Intelligence.
- Across models, responses were 34.3% less likely to flag a false first‑person belief than a true one, with pre‑May 2024 systems performing worse at 38.6%.
- Newer generations showed relatively high accuracy on factual identification at about 91%, while older models scored 84.8% and 71.5% on comparable tests.
- The paper concludes current systems rely on inconsistent reasoning strategies that resemble superficial pattern matching rather than robust epistemic understanding.
- Authors and outside experts urge urgent improvements and cautious use in medicine, law, journalism and science, with some recommending training models to respond more cautiously despite potential utility trade‑offs.