Facebook, WhatsApp, Instagram, Oculus Outage

On October 4, 2021, at 15:39 UTC, the social network Facebook and its subsidiaries (Messenger, Instagram, WhatsApp, Mapillary, Oculus) experienced a global outage for about six hours. The popular media reported the impact of this failure prominently.

NYT reported: “Gone in Minutes, Out for Hours: Outage Shakes Facebook”.

According to one estimate, this outage cost Facebook about $100 million in revenue losses and many billions due to the declining stock of the company.

We now see the sequence of events that caused this global problem.

Sequence of events

The following sequence of events led to the outage of Facebook and its accompanied services:

• A routine maintenance system was needed to find out the spare capacity on Facebook’s backbone network.
• Due to a configuration error, the maintenance system disconnected all the datacenters from each other on the backbone network. Earlier an automated configuration review tool was used to look for any issues in the configuration, but such tools are not perfect. In this specific case, the review tool missed the problems present in a configuration.
• The authoritative Domain Name Systems (DNS) of Facebook had a health-check rule that if it can not reach Facebook’s internal data centers, it stops replying to client DNS queries by withdrawing the routes.
• When networks routes (where

...