Just to close the loop on this...
==> AS OF 09-NOVEMBER-2017 10:45 PM COORDINATED UNIVERSAL TIME (UTC)
==> 11/6 RCA - Network Infrastructure - North Central US
==> Summary of impact: Between 19:33 UTC and 21:07 UTC on 6 Nov 2017, a subset of customers in North Central US experienced degraded performance, network drops, or timeouts when accessing Azure resources hosted in this region.
A subset of customers based in the region may have experienced issues accessing non-regional resources, such as Azure Active Directory and the Azure portal.
In addition, a subset of customers using App Service / Web Apps in South Central US would have experienced similar issues due to a bug discovered in how Web Apps infrastructure components utilize North Central US resources for geo-redundancy purposes.
During a planned capacity addition maintenance window to the North Central US region, a tag was introduced to aggregate routes for a subset of the North Central region.
This tagging caused affected prefixes to be withdrawn from regions outside of North Central US.
Automated alerting detected the drop in availability and engineering teams correlated the issue with the capacity addition maintenance, however automated rollback did not mitigate the issue.
The rollback steps were manually validated which fixed the route advertisement and restored the connectivity between the North Central US facilities.
==> Root cause and mitigation: A capacity augmentation workflow in a datacenter in the North Central US region contained an error that resulted in too many routing announcements being assigned a Border Gateway Protocol (BGP) community tag that should have only been used for routes that stay within the North Central US region.
When aggregate IP prefixes intended for distribution to the Microsoft core network were incorrectly tagged for local distribution only, it interrupted communication between this datacenter and all our other Azure regions.
The addition of new capacity to a region has been the cause of incidents in the past, so both automated systems and manual checks are used to verify that there is no impact.
The tool used in this turn up did attempt to rollback, but due to a second error the rollback workflow also applied the incorrectly tagged prefix policy such that the rollback did not resolve the problem.
Because rollback did not resolve the incident, engineers working to resolve the incident lost time looking for other potential causes before determining the true root cause and fully rolling back the capacity addition work.
This increased the time required to mitigate the issues.
==> Workaround: During this incident, customers' applications with geo-redundancy would have allowed the application to continue without impact or would have minimized the impact.
For further information, please visit http://aka.ms/mspnp for best practices for cloud applications and design patterns.
==> Next steps: We sincerely apologize for the impact to the affected customers.
We are continuously taking steps to improve the Microsoft Azure platform and our processes to help ensure that such incidents do not occur in the future, and in this case it includes (but is not limited to):
==> 1. Any scheduled work changing routing policy to be modeled in CrystalNet - https://aka.ms/crystalnet . [In progress]
==> 2. Update verification steps for routing policy changes to include checks from other regions. [In progress]
==> 3. Isolate the dependency upon the geo-redundant Web Apps resources such that they cannot impact the health of the primary resources. [In progress]
