What caused Facebook shutdown !

BGP

·

2 min read

What caused Facebook shutdown !

I am gonna try to catch what might be the reason behind this glitch...

" It seems like someone pulled cables from their data centers all at once and disconnected them from the internet. "

lets jump into it :

This was more about BGP (Border Gateway Protocol) : A mechanism to exchange routing information between autonomous systems (AS) on the internet. Without BGP, the internet routers wouldn't know what to do, and the internet wouldn't work.

  • BGP allow Facebook( any network ) to show its presence to other networks. But this was not the same when the "shutdown" occurred !

The ISPs and other networks couldn't find Facebook's network as it changed configuration to their backbone router and so it was unavailable. Becoz, Facebook stopped announcing the routes to their DNS prefixes and 1.1.1.1 DNS resolver was unable to respond to queries asking for IPs .

route-views>show ip bgp 185.89.218.0/23
% Network not in table
route-views>

route-views>show ip bgp 129.134.30.0/23
% Network not in table
route-views>

Why this happened though?

  • Somehow it was the fault with there systems. This happened becoz just before the outage, it got a peak of routing changes from Facebook. Here, the clients at extremes ("users") and the application logic kinks and cause another exponential effect, as sometimes "we the users" aggressively start reloading the pages which ultimately increases the traffic on 1.1.1.1 .

you can see the traffic just before : image6-9.png

The vast majority of our DNS requests kept resolving in under 10ms. At the same time, a minimal fraction of p95 and p99 percentiles saw increased response times, probably due to expired TTLs having to resort to the Facebook nameservers and timeout . The 10 seconds DNS timeout limit is well known amongst engineers.

After update :

  • We can see the renewed BGP activity from Facebook's network below. unnamed-4.png

Hope you got the idea behind..