<img height="1" width="1" style="display:none;" alt="" src="https://px.ads.linkedin.com/collect/?pid=1110556&amp;fmt=gif">
Skip to content
Meet Our Experts Customize My Content Contact Us
      October 8, 2021

      Facebook Outage Underscores Risks of Modern SDN Networks

      Written by: Jeremy Rossbach

      On October 4, 2021, an outage took Facebook, WhatsApp, and Instagram down for six hours. This major outage is a powerful and high-profile example of the difficulties facing those tasked with managing modern network architectures. While there has been a lot of chatter about misconfiguration and DNS failures, the reality is that this outage could have more to do with Facebook's software-defined networks (SDN) and the propensity for errors organizations are exposed to when the control plane is centralized and segregated from network devices.

      An early adopter of SDN, Facebook pioneered innovative methods to automate provisioning of their massive networks. Najam Admad, director of technical operations, who architected SDN for Facebook, said in 2014, "We want to deploy, manage, monitor and fix networks using software." One of their widely publicized use cases is their software-defined backbone routing. Facebook uses SDN to augment BGP route selection. They’ve employed advanced congestion and capacity analytics in order to overcome the common shortcomings of BGP. 

      Facebook’s SDN controllers produce routes that then get sent to the edge routers, which are responsible for connecting to the outside world. Based on public speculation, the belief is that these SDN-controlled BGP route selection processes encountered a glitch and basically made the Facebook edge routers unreachable. This initial issue was then exacerbated by the fact that their internally hosted DNS went into hibernation mode in response to lost connections. The blog post by Facebook's Santosh Janardhan further supports this hypothesis: "During one of these routine maintenance jobs, a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network, effectively disconnecting Facebook data centers globally."

      This type of control-plane and data-plane disconnect is also occurring in other SDN environments. Similar malfunctions are also happening in SD-WAN scenarios. For example, operators find that, while an SD-WAN controller issues a reroute command to an edge device, the edge device ignores the command. There have also been cases in which SD-WAN controllers issue rerouting commands unnecessarily. It is becoming vital that teams in the network operations center monitor the state and the activities of controllers, so they can ensure that critical software-defined transactions are executed correctly.

      The important lesson here is that as we evolve networks to be more intelligent, whether through centralizing the control plane or leveraging automation to boost efficiency, we need to make sure that suitable network observability is established. To support SDN and automation, new NetOps capabilities have to be deployed. For example, teams need to employ new operational workflows so they can validate commands using network intelligence, such as discovered topologies and historical performance and flow data. Teams can also initiate active tests that simulate traffic going to and coming from the edge router, so they can better preempt potential issues. SDN has provided many benefits, and promises more to come, but it has also introduced risks. Therefore, SDN-enabled network observability is needed to establish checks and balances. By establishing modern network observability, teams can monitor not only the networks, but the software intelligence that governs those networks.

      Tag(s): DX NetOps , Network Observability

      Jeremy Rossbach

      As the Chief Technical Evangelist for NetOps by Broadcom, Jeremy is passionate about meeting with customers to identify their IT operational challenges and produce solutions that fit their business and network transformation goals. Prior to joining Broadcom, he spent over 15+ years working in IT, across both public...

      Other resources you might be interested in

      icon
      White Paper June 5, 2026

      How to Install Automic Automation Kubernetes Edition v26 in Azure

      Master the deployment of Automic Automation v26 on Azure AKS. Cover database setup, TLS certificates, and the new Kubernetes Gateway API.

      Read White Paper
      icon
      White Paper June 5, 2026

      How to Install Automic Automation Kubernetes Edition v24 in Azure

      Deploy Automic Automation Kubernetes Edition v24 on Azure AKS with this step-by-step installation and configuration guide.

      Read White Paper
      icon
      White Paper June 5, 2026

      How to Install Automic Automation Kubernetes Edition v26 in AWS

      Learn how to deploy Automic Automation Kubernetes Edition v26 on AWS EKS with this step-by-step guide for configuring databases, secrets, and agents.

      Read White Paper
      icon
      White Paper June 5, 2026

      How to Install Automic Automation Kubernetes Edition v24 in AWS

      See how to deploy Automic Automation v24 on AWS EKS. Learn about using Fargate, Helm charts, PostgreSQL, and AWS Load Balancer Controller.

      Read White Paper
      icon
      White Paper June 5, 2026

      How to Install Automic Automation Kubernetes Edition v24 in GCP

      This guide walks you through the steps to deploy Automic Automation Kubernetes Edition v24 into Google Kubernetes Engine (GKE) on the Google Cloud Platform (GCP).

      Read White Paper
      icon
      White Paper June 5, 2026

      How to Install Automic Automation Kubernetes Edition v26 in GCP

      Discover the steps needed to deploy Automic Automation Kubernetes Edition v26 into Google Kubernetes Engine (GKE) on the Google Cloud Platform (GCP).

      Read White Paper
      icon
      Blog June 4, 2026

      Speed with Confidence: Managing Delivery Risk in an AI-driven Development World

      Stop relying on hunches and guesswork to track and manage delivery progress. Rally's milestone delivery confidence capability offers a data-driven approach.

      Read Blog
      icon
      Video June 4, 2026

      Cloud Orchestration with ESP dSeries Workload Automation

      Learn how Broadcom's ESP dSeries Workload Automation simplifies cloud orchestration and centralizes control of hybrid workloads.

      Video
      icon
      Blog June 4, 2026

      Automating Device and OS Compliance in Air-Gapped Networks with Agentic AI

      Secure air-gapped networks by automating device compliance. Employ a hybrid agentic AI workflow that separates public intelligence from internal audits.

      Read Blog