Windows 10: Details on Google Cloud Platform Services Outage Incident on June 2

Brink · Jun 7, 2019

ISSUE SUMMARY

On Sunday 2 June, 2019, Google Cloud projects running services in multiple US regions experienced elevated packet loss as a result of network congestion for a duration of between 3 hours 19 minutes, and 4 hours 25 minutes. The duration and degree of packet loss varied considerably from region to region and is explained in detail below. Other Google Cloud services which depend on Google's US network were also impacted, as were several non-Cloud Google services which could not fully redirect users to unaffected regions. Customers may have experienced increased latency, intermittent errors, and connectivity loss to instances in us-central1, us-east1, us-east4, us-west2, northamerica-northeast1, and southamerica-east1. Google Cloud instances in us-west1, and all European regions and Asian regions, did not experience regional network congestion.

Google Cloud Platform services were affected until mitigation completed for each region, including: Google Compute Engine, App Engine, Cloud Endpoints, Cloud Interconnect, Cloud VPN, Cloud Console, Stackdriver Metrics, Cloud Pub/Sub, Bigquery, regional Cloud Spanner instances, and Cloud Storage regional buckets. G Suite services in these regions were also affected.

We apologize to our customers whose services or businesses were impacted during this incident, and we are taking immediate steps to improve the platform’s performance and availability. A detailed assessment of impact is at the end of this report.

ROOT CAUSE AND REMEDIATION

This was a major outage, both in its scope and duration. As is always the case in such instances, multiple failures combined to amplify the impact.

Within any single physical datacenter location, Google's machines are segregated into multiple logical clusters which have their own dedicated cluster management software, providing resilience to failure of any individual cluster manager. Google's network control plane runs under the control of different instances of the same cluster management software; in any single location, again, multiple instances of that cluster management software are used, so that failure of any individual instance has no impact on network capacity.

Google's cluster management software plays a significant role in automating datacenter maintenance events, like power infrastructure changes or network augmentation. Google's scale means that maintenance events are globally common, although rare in any single location. Jobs run by the cluster management software are labelled with an indication of how they should behave in the face of such an event: typically jobs are either moved to a machine which is not under maintenance, or stopped and rescheduled after the event.

Two normally-benign misconfigurations, and a specific software bug, combined to initiate the outage: firstly, network control plane jobs and their supporting infrastructure in the impacted regions were configured to be stopped in the face of a maintenance event. Secondly, the multiple instances of cluster management software running the network control plane were marked as eligible for inclusion in a particular, relatively rare maintenance event type. Thirdly, the software initiating maintenance events had a specific bug, allowing it to deschedule multiple independent software clusters at once, crucially even if those clusters were in different physical locations.

The outage progressed as follows: at 11:45 US/Pacific, the previously-mentioned maintenance event started in a single physical location; the automation software created a list of jobs to deschedule in that physical location, which included the logical clusters running network control jobs. Those logical clusters also included network control jobs in other physical locations. The automation then descheduled each in-scope logical cluster, including the network control jobs and their supporting infrastructure in multiple physical locations.

Google's resilience strategy relies on the principle of defense in depth. Specifically, despite the network control infrastructure being designed to be highly resilient, the network is designed to 'fail static' and run for a period of time without the control plane being present as an additional line of defense against failure. The network ran normally for a short period - several minutes - after the control plane had been descheduled. After this period, BGP routing between specific impacted physical locations was withdrawn, resulting in the significant reduction in network capacity observed by our services and users, and the inaccessibility of some Google Cloud regions. End-user impact began to be seen in the period 11:47-11:49 US/Pacific.

Google engineers were alerted to the failure two minutes after it began, and rapidly engaged the incident management protocols used for the most significant of production incidents. Debugging the problem was significantly hampered by failure of tools competing over use of the now-congested network. The defense in depth philosophy means we have robust backup plans for handling failure of such tools, but use of these backup plans (including engineers travelling to secure facilities designed to withstand the most catastrophic failures, and a reduction in priority of less critical network traffic classes to reduce congestion) added to the time spent debugging. Furthermore, the scope and scale of the outage, and collateral damage to tooling as a result of network congestion, made it initially difficult to precisely identify impact and communicate accurately with customers.

As of 13:01 US/Pacific, the incident had been root-caused, and engineers halted the automation software responsible for the maintenance event. We then set about re-enabling the network control plane and its supporting infrastructure. Additional problems once again extended the recovery time: with all instances of the network control plane descheduled in several locations, configuration data had been lost and needed to be rebuilt and redistributed. Doing this during such a significant network configuration event, for multiple locations, proved to be time-consuming. The new configuration began to roll out at 14:03.

In parallel with these efforts, multiple teams within Google applied mitigations specific to their services, directing traffic away from the affected regions to allow continued serving from elsewhere.

As the network control plane was rescheduled in each location, and the relevant configuration was recreated and distributed, network capacity began to come back online. Recovery of network capacity started at 15:19, and full service was resumed at 16:10 US/Pacific time.

The multiple concurrent failures which contributed to the initiation of the outage, and the prolonged duration, are the focus of a significant post-mortem process at Google which is designed to eliminate not just these specific issues, but the entire class of similar problems. Full details follow in the Prevention and Follow-Up section.

PREVENTION AND FOLLOW-UP

We have immediately halted the datacenter automation software which deschedules jobs in the face of maintenance events. We will re-enable this software only when we have ensured the appropriate safeguards are in place to avoid descheduling of jobs in multiple physical locations concurrently. Further, we will harden Google's cluster management software such that it rejects such requests regardless of origin, providing an additional layer of defense in depth and eliminating other similar classes of failure.

Google's network control plane software and supporting infrastructure will be reconfigured such that it handles datacenter maintenance events correctly, by rejecting maintenance requests of the type implicated in this incident. Furthermore, the network control plane in any single location will be modified to persist its configuration so that the configuration does not need to be rebuilt and redistributed in the event of all jobs being descheduled. This will reduce recovery time by an order of magnitude. Finally, Google's network will be updated to continue in 'fail static' mode for a longer period in the event of loss of the control plane, to allow an adequate window for recovery with no user impact.

Google's emergency response tooling and procedures will be reviewed, updated and tested to ensure that they are robust to network failures of this kind, including our tooling for communicating with the customer base. Furthermore, we will extend our continuous disaster recovery testing regime to include this and other similarly catastrophic failures.

Our post-mortem process will be thorough and broad, and remains at a relatively early stage. Further action items may be identified as this process progresses...
Click to expand...

Read more: https://status.cloud.google.com/inci...tworking/19009

:)

btarunr · Jun 7, 2019

AMD Radeon Technology Will Be Available on Google Cloud Platform in 2017

At SC16, AMD announced that Radeon GPU technology will be available to Google Cloud Platform users worldwide. Starting in 2017, Google will use AMD's fastest available single-precision dual GPU compute accelerators, Radeon-based AMD FirePro S9300 x2 Server GPUs, to help accelerate Google Compute Engine and Google Cloud Machine Learning services. AMD FirePro S9300 x2 GPUs can handle highly parallel calculations, including complex medical and financial simulations, seismic and subsurface exploration, machine learning, video rendering and transcoding, and scientific analysis. Google Cloud Platform will make the AMD GPU resources available for all their users around the world.

"Graphics processors represent the best combination of performance and programmability for existing and emerging big data applications," said Raja Koduri, senior vice president and chief architect, Radeon Technologies Group, AMD. "The adoption of AMD GPU technology in Google Cloud Platform is a validation of the progress AMD has made in GPU hardware and our Radeon Open Compute Platform, which is the only fully open source hyperscale GPU compute platform in the world today. We expect that our momentum in GPU computing will continue to accelerate with future hardware and software releases and advances in the ecosystem of middleware and libraries."

As part of AMD's continuing investments in GPU computing, the company revealed yesterday a new release of Radeon Open Compute Platform (ROCm) featuring software support for new GPU hardware, new math libraries, and a rich foundation of modern programming languages, designed to speed development of high-performance, energy-efficient heterogeneous computing systems. The news bolsters ROCm's position as the most versatile open source platform for GPU computing, targeting high-performance computing across a range of advanced applications from academic and scientific research to commercial deployments. Visit AMD at SC16, booth 1431 to see demonstrations of a wide range of recent GPU computing advances, including:

ROCm Technology Cluster running Machine Learning Code on Supermicro servers

Porting the CUDA application Caffe via HIP Porting Tool

Ray-tracing and VR visualization for HPC with AMD FirePro S9300 X2 & Radeon R9 Nano GPUs

OpenMP 4.5 Interoperability targeting multiple GPUs & platforms

IBM Power8 server with AMD FirePro S9170 Server GPU running ROCm

Penguin Computing Tundra Extreme ARMv8 ThunderX based server with Radeon RX 460 running ROCm

In-situ rendering with Headless OpenGL/EGL Interop OpenCL on ROCm

Brink · Jun 7, 2019

Cloud Platform Release Announcements for May 11, 2017

In today’s mobile first, cloud first world, Microsoft provides the technologies and tools to enable enterprises to embrace a cloud culture. Our differentiated innovations, comprehensive mobile solutions and developer tools help all of our customers realize the true potential of the cloud first era.

You expect cloud-speed innovation from us, and we’re delivering across the breadth of our Cloud Platform product portfolio. Below is a consolidated list of our latest releases to help you stay current, with links to additional details if you’d like more information. In this update:

Azure IoT Edge | Disclosure

Visual Studio Mobile Center Preview | UWP, Push, Test Frameworks and new Repo Support

Azure IoT Edge | Disclosure

At Build, Microsoft announced Azure IoT Edge, a service that enables Microsoft and third-party offerings at the edge and distributes cloud intelligence across a wide range of IoT devices.

Azure IoT Edge is the evolution of the Azure IoT Gateway SDK, which you can get started today on Azure IoT GitHub page. Azure IoT Edge is the new name for our open-source and cross-platform support for building custom logic at the edge, and the newly announced features will be available later this year. To sign up to get the latest information on Azure IoT Edge, click here.

Learn more on the Build blog here.

Visual Studio Mobile Center Preview | UWP, Push, Test Frameworks and new Repo Support

On May 10 at the 2017 Build conference, Microsoft announced a substantially expanded preview of Visual Studio Mobile Center—mission control for mobile apps.

With support for Universal Windows Platform (UWP) apps in its Build, Distribute, Crashes, Push, and Analytics services, Mobile Center’s lifecycle services are now available to Windows developers. Developers can now also connect their Bitbucket and Visual Studio Team Services repositories to build iOS, Android, and UWP apps. A new Push service allows developers to send push notifications to all or to a subset of devices, and with support for the XCUITest and Espresso frameworks, native app developers can write their automated UI tests in their preferred languages and port existing tests to Mobile Center Test.

Mobile Center brings together the cloud and lifecycle services that help developers deliver high-quality apps faster and drive user adoption and retention. Mobile Center lets you build, test, distribute and monitor apps, and makes it easy for developers to scale apps on demand and add popular mobile features such as identity services and offline sync.

To experience the power and simplicity of Mobile Center’s lifecycle management and development services, please sign up now. For additional information, please visit our Web site or view one or of our https://www.youtube.com/playlist?list=PLM75ZaNQS_FY4MOuo7Sscr27vLuUNyFQQ.
Click to expand...

Source: Cloud Platform Release Announcements for May 11, 2017 | Cloud Platform News Bytes Blog

btarunr · Jun 7, 2019

Google To Launch GoogleDrive Cloud-Storage Service Soon

Google is going to monetize its ocean of data storage and bandwidth by starting a new service that will rival the likes of DropBox. Called Google Drive, the service could allow people to make use of their G-Mail storage quota (like G-Mail Drive once did) for free, and probably offer additional storage at a cost. According to Wall Street Journal, the service is expected to be launched "soon" (in the next few weeks or months). To compete with other similar service providers, Google could come up with attractive prices. Unlike services such as MegaUpload, Google Drive would be strictly personal storage. The Drive service will be inter-twined with Google Docs, and facilitate better remote collaboration between co-workers.

Sources: The Verge, Wall Street Journal

Windows 10: Details on Google Cloud Platform Services Outage Incident on June 2

Details on Google Cloud Platform Services Outage Incident on June 2

Details on Google Cloud Platform Services Outage Incident on June 2

Details on Google Cloud Platform Services Outage Incident on June 2

Details on Google Cloud Platform Services Outage Incident on June 2 - Similar Threads - Details Google Cloud

Traffic Incident within the News & Interest will not give me details

Traffic Incident within the News & Interest will not give me details

Traffic Incident within the News & Interest will not give me details

Cloud services - global outage again today

Cloud Covered: What happened in Google Cloud in 2020

What was new in Google Cloud in June 2020

CLOUD SERVICE

New ways to report driving incidents on Google Maps

Microsoft Azure Media Services expands End-to-End cloud video platform

Users found this page by searching for:

google cloud platform availability incident