Particle.news

Months-Old Gateway Bug Caused Microsoft 365 Global Outage

Microsoft says a July code change, reduced capacity, plus a Teams traffic shift triggered lock contention that cascaded across its server fleet.

Overview

  • The outage began Thursday, Nov. 12 when a latent defect from a July fix made Microsoft 365 networking gateway servers become overwhelmed and unresponsive.
  • A Teams issue roughly 15 hours earlier shifted traffic onto the same Exchange Online gateway while nearly 14% of gateway capacity was already offline, together raising load beyond target levels.
  • Microsoft’s automated traffic routing sent work to healthy servers, which re-triggered the lock-contention bug and left more than 80% of some server fleets unresponsive.
  • Engineers could not do a simple rollback because the change was months old, so they applied an update that restored the code path to a known-good state and brought servers back in staged batches.
  • Microsoft confirmed there was no cyberattack, said it has restored remaining regions, will publish a final post-incident report on Nov. 20, and set Dec. 31 as the deadline for business SLA credit claims.