Particle.news

Microsoft Says Dormant July Bug Caused Global Outlook and Microsoft 365 Outage

Company says traffic shifts plus a 14% capacity shortfall exposed a testing blind spot that triggered lock contention resulting in cascading gateway failures with a final post‑incident review scheduled for November 20.

Overview

  • Microsoft’s networking gateway began failing during peak US hours on Thursday, Nov. 12, and the company says more than 80% of gateway servers were unresponsive at the outage’s worst point.
  • The company attributes the root cause to a software change deployed in July that created lock contention inside an internal process, a problem that worsens on newer, high‑core hardware and escaped detection in lab tests.
  • A routine Teams failover to the primary gateway about 15 hours earlier combined with roughly 14% unavailable gateway capacity to push remaining servers into the latent bug and trigger a rapid cascading failure.
  • Engineers could not perform a simple rollback because the change was months old, so they developed an update that returned the code path to a known good state, applied it in staged batches, and restored services by Nov. 14 with a preliminary report published on Nov. 15.
  • The outage disrupted email, meetings and third‑party AI integrations for governments, hospitals and businesses, prompted customer operational and legal reviews, and sharpened scrutiny of testing, recovery automation and single‑vendor cloud concentration.