Particle.news

Download on the App Store

Google Cloud Attributes Seven-Hour Outage to Faulty Quota-Policy Update, Vows Overhaul

The incident report identifies a flawed quota-policy update that sent null data across all regions, promising comprehensive system audits followed by infrastructure reinforcement.

Image
Image
Google

Overview

  • The outage on June 12 halted over 70 Google Cloud services for nearly seven hours after an untested May update triggered blank entries across global data centers.
  • Engineers isolated the root cause within ten minutes, but service restoration was delayed by infrastructure overloads in large regions like us-central-1.
  • Google's incident report reveals the absence of feature flags and error handling in the new quota-policy checks feature as key factors in the widespread failures.
  • The blackout disrupted first-party products such as Gmail, Calendar and Drive in addition to third-party platforms including Cloudflare, OpenAI and Shopify.
  • Google has pledged architectural changes to isolate system failures, promising a full operations audit and enhancements to both automated alerts and human communication channels.