GCVE, BMS, GCNV outage in europe-west4-a due to power failure

"modified": "2026-07-18T00:22:38+00:00",
"modified": "2026-07-18T00:22:38+00:00",

"modified": "2026-07-25T13:16:55+00:00",
"created": "2026-07-25T13:16:55+00:00",
"modified": "2026-07-25T13:16:55+00:00",
"text": "## \\# Incident Report\n## \\#\\# Summary\nOn Wednesday, 15 July 2026, Google Cloud VMware Engine (GCVE), Bare Metal Solution (BMS), and Google Cloud NetApp Volumes (GCNV) experienced service interruptions for a total duration of 14 hours, 55 minutes.\nThe root cause for this outage is a 3ms voltage drop in the power feed from the utility provider and subsequent utility breaker protective action. This incident was mitigated by the data center provider rectifying the failed systems followed by the Google engineering team restoring the services.\n## \\#\\# Root Cause\nA regional data center hosting services in europe-west4-a experienced an upstream voltage transient affecting both utility power feeds A and B. During this event, the utility breakers on both A and B feeds tripped and initiated the transfer to the back-up power source DRUPS (Diesel Rotary Uninterruptible Power Supply). The transition of side B to DRUPS system was successful without any power interruption. The DRUPS back-up power system for side A failed to take over the facility load due to electrical component failures.\nThe DRUPS failure to take over load initiated automatic transfer of the affected 3 rows from feed A to redundant power feed B. Rows 1 and 2 transferred to power feed B successfully. Row 3 failed to transfer to power feed B and experienced complete power loss due to an overload protection breaker trip. The root cause of the failed transfer on the affected row was identified as load deployment discrepancy which is under further investigation by Google engineering team. This resulted in loss of both redundant power feeds to the single row.\nDuring the voltage transient event, the server data hall experienced an increased temperature due to a cooling system failure. The chiller controller dropped offline during the voltage transient event, failing to signal the chilled water distribution pumps to restart and ultimately causing the chiller system A to shut down. The redundant source was not available due to known ongoing construction work at the facility. This resulted in the data hall temperatures reaching 44°C and subsequent shutdown of the affected data hall machines. Google engineering teams initiated machine shutdown procedures for the remainder of reachable devices as part of the cooling emergency shutdown process.\nA timeline of events during the incident is provided below.\n### DataCenter Events\n| Timestamp(PST) | Event Description |\n| :---- | :---- |\n| 07-15-2026 16:24 | An electrical fault occurred on the utility grid upstream of the data center, disrupting the electrical distribution. |\n| 07-15-2026 16:24 | DRUPS back-up power system for side A failed to take over the data hall load. |\n| 07-15-2026 16:24 | The chiller controller dropped offline during the voltage transient event causing distribution pumps to stop and unable to restart. |\n| 07-15-2026 16:37 | Rows 1 and 2 transferred to power feed B successfully. Row 3 failed to transfer to power feed B and lost power. |\n| 07-15-2026 18:29 | Data hall temperatures reached 44C and crossed the safe machine operating threshold. |\n| 07-15-2026 19:55 | Machine shutdown procedures implemented. |\n| 07-15-2026 21:05 | Cooling system fully recovered and temperatures in the data hall returned to normal operating range. |\n| 07-15-2026 21:24 | Notification about cooling recovery sent by provider |\n| 07-15-2026 21:46 | Notification about power recovery sent by provider |\n### GCVE Timelines\n| Timestamp(PST) | Event Description |\n| :---- | :---- |\n| 07-15-2026 16:30 | GCVE service power monitoring alert received |\n| 07-15-2026 16:41 | First symptom detected, servers reporting redundancy power feed failure. |\n| 07-15-2026 17:05 | Switch temperature \\>60°C alerts triggered. |\n| 07-15-2026 17:09 | Start of user impact, first prober alert failure for north south traffic for customers. |\n| 07-15-2026 19:00 | Server / Network devices shutdown initiated. |\n| 07-15-2026 19:55 | All reachable server/network devices were shut down. |\n| 07-15-2026 21:24 | Cooling system fully recovered and temperatures in the datahall were returning to normal operating range. |\n| 07-15-2026 21:56 | Network restoration started with reachable hydra devices. |\n| 07-15-2026 22:40 | Onsite technician arrived, to recover the console servers. |\n| 07-15-2026 23:10 | Console server reboot/recovery complete. |\n| 07-16-2026 00:33 | Network recovery for all Placement Groups complete. |\n| 07-16-2026 02:36 | Incident mitigated, after recovering all the customer PCs are fully healthy. |\nThis outage impacted 24 private clouds of 20 distinct customers in europe-west4-a region.\n### Bare Metal Solutions (BMS) Timelines\n| Timestamp(PST) | Event Description |\n| :---- | :---- |\n| 07-15-2026 16:41 | First symptom detected, servers and storage reporting redundancy power feed failure |\n| 07-15-2026 17:52 | Automated monitoring detected rising temperatures |\n| 07-15-2026 18:37 | Four Netapp SAN storage nodes failed and were shut down due to overheating, resulting in storage availability issue |\n| 07-15-2026 18:37 | Customer impact started, first server shutdown detected |\n| 07-15-2026 18:59 | Reserve and buffer servers started to get powered off |\n| 07-15-2026 20:43 | Drop in the temperature has been detected |\n| 07-15-2026 20:52 | Four Netapp SAN storage nodes recovered, mitigating storage availability issue |\n| 07-15-2026 21:05 | The cooling system fully recovered and temperatures in the datahall were returning to normal operating range. |\n| 07-15-2026 23:23 | First customer server reboot started |\n| 07-16-2026 06:32 | Last customer server reboot started |\n| 07-16-2026 08:42 | Incident mitigated. All impacted customer servers were rebooted and confirmed in healthy state. |\nThis outage impacted 9 distinct BMS customers in europe-west4\n### NetApp Timelines\n| Timestamp(PST) | Event Description |\n| :---- | :---- |\n| 07-15-2026 16:32 | Netapp facility remote monitoring alert on high temp and network switches switches failure received |\n| 07-15-2026 17:32 | All 6 clusters failed and were shut down automatically due to high temperature |\n| 07-15-2026 21:05 | Cooling capacity restored |\n| 07-15-2026 22:08 | Cluster recovery process initiated for all 6 clusters. 1st cluster recovered |\n| 07-15-2026 23:55 | All 6 Netapp Clusters and all customers recovered and validated as healthy & operational |\n## \\#\\# Remediation and Prevention\nGoogle engineers were alerted to the infrastructure power loss on Wednesday, 15 July 2026 at 16:30 US/Pacific and subsequent increase in data hall temperatures at 17:44 US/Pacific via our automated monitoring and telemetry, and immediately began investigating.\nDuring the voltage transient event, the chillers maintained power but shut down because the distribution pumps failed to restart, resulting in no water flow. The pumps were manually switched from auto to hand mode to restore circulation and cooling. The facility team deployed an interim portable UPS to support the chiller controllers and prevent localized controller power loss.\nUpon confirmation of utility power restoration, the facility returned to utility power via automatic switching with the exception of two Remote Power Panels (RPPs), which required manual verification before restoration of power to affected server row. To provide full resiliency the facility team aligned a swing DRUPS unit to restore the redundant configuration on the A feed and is working on repairing the faulty DRUPS.\nTo remediate the power loss at Row 3, once the electrical fault was confirmed to be fully isolated, teams reset the tripped breakers, successfully restoring both redundant power to the impacted server racks.\nGoogle is committed to preventing a repeat of this issue in the future and is completing the following actions:\n* Detailed investigation by the engineering team of the sequence of events that prevented transfer of load to DRUPS system and identified system improvements **\\[ETA Aug 2026\\].**\n* Determine and resolve the cause of overload conditions that caused power loss to Row 3 **\\[ETA Aug 2026\\].**\n* Perform investigation of chiller pump control system redundancy setup to ensure cooling system resiliency **\\[ETA Sep 2026\\].**\n* Implement improvements to facility system monitoring alerts for power events and configure early alert thresholds for cooling excursion detection. **\\[ETA Aug 2026\\].**\n* The GCVE engineering team is working to improve the automated triggering of shutdown during quickly evolving thermal runoff situations **\\[ETA Oct 2026\\]**.\n* The BMS engineering team is working with partner teams to update the incident classification and SLO definitions to improve personnel availability during major datacenter events **\\[ETA Sep 2026\\]**.\n* The GCNV engineering team is collaborate with the data center team to review incident handling SLA and runbooks, identify potential future occurrences and create playbooks to address/mitigate them, establish monthly joint emergency drill, and review if the Netapp cluster recovery process can be expedited **\\[ETA Aug 2026\\].**\n## \\#\\# Detailed Description of Impact\nOn 15 July 2026 from 16:39 to 16 July 2026 07:34 US/Pacific, customers experienced service disruptions across several products in europe-west4-a zone:\n* **Google Cloud VMware Engine**: Customers lost connectivity to their private clouds and workloads as foundational hosts and network switches were powered off for thermal protection.\n* **Bare Metal Solution**: Customers experienced connectivity loss to their database servers and storage appliances as the underlying hardware was powered down.\n* **Google Cloud NetApp Volumes**: Customers in the STANDARD, PREMIUM, and EXTREME service levels were unable to access their storage volumes. Control plane operations, including the creation of new storage pools, volumes, and backups, experienced failures in the affected region.",
"when": "2026-07-25T13:16:55+00:00"
"created": "2026-07-25T13:16:55+00:00",
"modified": "2026-07-25T13:16:55+00:00",
"text": "## \\# Incident Report\n## \\#\\# Summary\nOn Wednesday, 15 July 2026, Google Cloud VMware Engine (GCVE), Bare Metal Solution (BMS), and Google Cloud NetApp Volumes (GCNV) experienced service interruptions for a total duration of 14 hours, 55 minutes.\nThe root cause for this outage is a 3ms voltage drop in the power feed from the utility provider and subsequent utility breaker protective action. This incident was mitigated by the data center provider rectifying the failed systems followed by the Google engineering team restoring the services.\n## \\#\\# Root Cause\nA regional data center hosting services in europe-west4-a experienced an upstream voltage transient affecting both utility power feeds A and B. During this event, the utility breakers on both A and B feeds tripped and initiated the transfer to the back-up power source DRUPS (Diesel Rotary Uninterruptible Power Supply). The transition of side B to DRUPS system was successful without any power interruption. The DRUPS back-up power system for side A failed to take over the facility load due to electrical component failures.\nThe DRUPS failure to take over load initiated automatic transfer of the affected 3 rows from feed A to redundant power feed B. Rows 1 and 2 transferred to power feed B successfully. Row 3 failed to transfer to power feed B and experienced complete power loss due to an overload protection breaker trip. The root cause of the failed transfer on the affected row was identified as load deployment discrepancy which is under further investigation by Google engineering team. This resulted in loss of both redundant power feeds to the single row.\nDuring the voltage transient event, the server data hall experienced an increased temperature due to a cooling system failure. The chiller controller dropped offline during the voltage transient event, failing to signal the chilled water distribution pumps to restart and ultimately causing the chiller system A to shut down. The redundant source was not available due to known ongoing construction work at the facility. This resulted in the data hall temperatures reaching 44°C and subsequent shutdown of the affected data hall machines. Google engineering teams initiated machine shutdown procedures for the remainder of reachable devices as part of the cooling emergency shutdown process.\nA timeline of events during the incident is provided below.\n### DataCenter Events\n| Timestamp(PST) | Event Description |\n| :---- | :---- |\n| 07-15-2026 16:24 | An electrical fault occurred on the utility grid upstream of the data center, disrupting the electrical distribution. |\n| 07-15-2026 16:24 | DRUPS back-up power system for side A failed to take over the data hall load. |\n| 07-15-2026 16:24 | The chiller controller dropped offline during the voltage transient event causing distribution pumps to stop and unable to restart. |\n| 07-15-2026 16:37 | Rows 1 and 2 transferred to power feed B successfully. Row 3 failed to transfer to power feed B and lost power. |\n| 07-15-2026 18:29 | Data hall temperatures reached 44C and crossed the safe machine operating threshold. |\n| 07-15-2026 19:55 | Machine shutdown procedures implemented. |\n| 07-15-2026 21:05 | Cooling system fully recovered and temperatures in the data hall returned to normal operating range. |\n| 07-15-2026 21:24 | Notification about cooling recovery sent by provider |\n| 07-15-2026 21:46 | Notification about power recovery sent by provider |\n### GCVE Timelines\n| Timestamp(PST) | Event Description |\n| :---- | :---- |\n| 07-15-2026 16:30 | GCVE service power monitoring alert received |\n| 07-15-2026 16:41 | First symptom detected, servers reporting redundancy power feed failure. |\n| 07-15-2026 17:05 | Switch temperature \\>60°C alerts triggered. |\n| 07-15-2026 17:09 | Start of user impact, first prober alert failure for north south traffic for customers. |\n| 07-15-2026 19:00 | Server / Network devices shutdown initiated. |\n| 07-15-2026 19:55 | All reachable server/network devices were shut down. |\n| 07-15-2026 21:24 | Cooling system fully recovered and temperatures in the datahall were returning to normal operating range. |\n| 07-15-2026 21:56 | Network restoration started with reachable hydra devices. |\n| 07-15-2026 22:40 | Onsite technician arrived, to recover the console servers. |\n| 07-15-2026 23:10 | Console server reboot/recovery complete. |\n| 07-16-2026 00:33 | Network recovery for all Placement Groups complete. |\n| 07-16-2026 02:36 | Incident mitigated, after recovering all the customer PCs are fully healthy. |\nThis outage impacted 24 private clouds of 20 distinct customers in europe-west4-a region.\n### Bare Metal Solutions (BMS) Timelines\n| Timestamp(PST) | Event Description |\n| :---- | :---- |\n| 07-15-2026 16:41 | First symptom detected, servers and storage reporting redundancy power feed failure |\n| 07-15-2026 17:52 | Automated monitoring detected rising temperatures |\n| 07-15-2026 18:37 | Four Netapp SAN storage nodes failed and were shut down due to overheating, resulting in storage availability issue |\n| 07-15-2026 18:37 | Customer impact started, first server shutdown detected |\n| 07-15-2026 18:59 | Reserve and buffer servers started to get powered off |\n| 07-15-2026 20:43 | Drop in the temperature has been detected |\n| 07-15-2026 20:52 | Four Netapp SAN storage nodes recovered, mitigating storage availability issue |\n| 07-15-2026 21:05 | The cooling system fully recovered and temperatures in the datahall were returning to normal operating range. |\n| 07-15-2026 23:23 | First customer server reboot started |\n| 07-16-2026 06:32 | Last customer server reboot started |\n| 07-16-2026 08:42 | Incident mitigated. All impacted customer servers were rebooted and confirmed in healthy state. |\nThis outage impacted 9 distinct BMS customers in europe-west4\n### NetApp Timelines\n| Timestamp(PST) | Event Description |\n| :---- | :---- |\n| 07-15-2026 16:32 | Netapp facility remote monitoring alert on high temp and network switches switches failure received |\n| 07-15-2026 17:32 | All 6 clusters failed and were shut down automatically due to high temperature |\n| 07-15-2026 21:05 | Cooling capacity restored |\n| 07-15-2026 22:08 | Cluster recovery process initiated for all 6 clusters. 1st cluster recovered |\n| 07-15-2026 23:55 | All 6 Netapp Clusters and all customers recovered and validated as healthy & operational |\n## \\#\\# Remediation and Prevention\nGoogle engineers were alerted to the infrastructure power loss on Wednesday, 15 July 2026 at 16:30 US/Pacific and subsequent increase in data hall temperatures at 17:44 US/Pacific via our automated monitoring and telemetry, and immediately began investigating.\nDuring the voltage transient event, the chillers maintained power but shut down because the distribution pumps failed to restart, resulting in no water flow. The pumps were manually switched from auto to hand mode to restore circulation and cooling. The facility team deployed an interim portable UPS to support the chiller controllers and prevent localized controller power loss.\nUpon confirmation of utility power restoration, the facility returned to utility power via automatic switching with the exception of two Remote Power Panels (RPPs), which required manual verification before restoration of power to affected server row. To provide full resiliency the facility team aligned a swing DRUPS unit to restore the redundant configuration on the A feed and is working on repairing the faulty DRUPS.\nTo remediate the power loss at Row 3, once the electrical fault was confirmed to be fully isolated, teams reset the tripped breakers, successfully restoring both redundant power to the impacted server racks.\nGoogle is committed to preventing a repeat of this issue in the future and is completing the following actions:\n* Detailed investigation by the engineering team of the sequence of events that prevented transfer of load to DRUPS system and identified system improvements **\\[ETA Aug 2026\\].**\n* Determine and resolve the cause of overload conditions that caused power loss to Row 3 **\\[ETA Aug 2026\\].**\n* Perform investigation of chiller pump control system redundancy setup to ensure cooling system resiliency **\\[ETA Sep 2026\\].**\n* Implement improvements to facility system monitoring alerts for power events and configure early alert thresholds for cooling excursion detection. **\\[ETA Aug 2026\\].**\n* The GCVE engineering team is working to improve the automated triggering of shutdown during quickly evolving thermal runoff situations **\\[ETA Oct 2026\\]**.\n* The BMS engineering team is working with partner teams to update the incident classification and SLO definitions to improve personnel availability during major datacenter events **\\[ETA Sep 2026\\]**.\n* The GCNV engineering team is collaborate with the data center team to review incident handling SLA and runbooks, identify potential future occurrences and create playbooks to address/mitigate them, establish monthly joint emergency drill, and review if the Netapp cluster recovery process can be expedited **\\[ETA Aug 2026\\].**\n## \\#\\# Detailed Description of Impact\nOn 15 July 2026 from 16:39 to 16 July 2026 07:34 US/Pacific, customers experienced service disruptions across several products in europe-west4-a zone:\n* **Google Cloud VMware Engine**: Customers lost connectivity to their private clouds and workloads as foundational hosts and network switches were powered off for thermal protection.\n* **Bare Metal Solution**: Customers experienced connectivity loss to their database servers and storage appliances as the underlying hardware was powered down.\n* **Google Cloud NetApp Volumes**: Customers in the STANDARD, PREMIUM, and EXTREME service levels were unable to access their storage volumes. Control plane operations, including the creation of new storage pools, volumes, and backups, experienced failures in the affected region.",
"when": "2026-07-25T13:16:55+00:00"
"modified": "2026-07-25T13:16:55+00:00",