Particle.news

OpenAI Chief Scientist Calls for a Shared Perplexity Standard

He argues common pretraining limits tied to that measurement could let developers pause progress when models approach thresholds linked to sudden capability gains.

Overview

  • Jakub Pachocki published a post on Sunday releasing OpenAI’s internal perplexity numbers and urging firms to adopt perplexity, or cross-entropy loss per token, as a common way to track pretraining progress.
  • Perplexity measures how well a model predicts the next token on average, so lower values mean better prediction performance and, at OpenAI, have correlated with new capabilities such as fluent conversation and improved reasoning.
  • OpenAI reported an internal observation that models reaching a log-perplexity of about 0.76 coincided with sudden gains on hard tasks like mathematics and said further transformative changes could plausibly appear as losses fall toward ~0.68–0.70.
  • Pachocki proposed that developers voluntarily commit to perplexity-based development limits until concrete monitoring and alignment milestones are met, and OpenAI CEO Sam Altman publicly backed the transparency effort.
  • He warned that comparing perplexity across labs is hard because teams use different evaluation datasets and data mixes, and he flagged the risk that AI-assisted algorithmic research could speed pretraining faster than safety tools can keep up.