Backend Aggregation (BAG) Architecture for Meta’s Prometheus AI Cluster

# Backend Aggregation (BAG) Overview Backend Aggregation (BAG) is the central Ethernet‑based super‑spine network that interconnects multiple spine fabrics across Meta's data‑center regions. It enables gigawatt‑scale AI clusters by providing petabit‑class inter‑region bandwidth, modular hardware scalability, and high‑capacity routing for tens of thousands of GPUs. ## Architecture Overview The BAG layer serves as the aggregation point between regional Ethernet fabrics and Meta's global backbone. It employs a distributed topology where regional BAG instances connect local L2 fabrics-Disaggregated Schedule Fabric (DSF) and Non‑Scheduled Fabric (NSF)-to a central hub. This design minimizes distance to edge switches, supports shallow‑buffer NSF devices, and uses deep‑buffer switches for long‑haul BAG‑to‑BAG links, ensuring lossless congestion control. ### Network Fabrics Integration Each data‑center building hosts a backend edge pod that bridges DSF or NSF L2 zones to the BAG plane. Matching Spine Training Switches (STSWs) on each BAG plane provide uniform connectivity, achieving an effective oversubscription ratio of roughly 4.51 from L2 to BAG and variable ratios between BAG layers based on regional capacity. ### Distributed BAG Layers Regional BAG instances are deployed to respect latency and buffer constraints. They connect to one another via either planar (direct match) or spread topologies, chosen according to site size and fiber availability. Inter‑BAG capacities target the 16‑48 Pbps range per region pair, supporting the massive bandwidth demands of the Prometheus cluster. ### Hardware Implementation Meta equips BAG chassis with Jericho3 (J3) ASIC line cards, each offering up to 432 × 800 Gb ports. The central hub chassis expands to host numerous spokes and long‑distance links, optimizing buffer utilization across varied cable lengths. ### Routing and Security Routing within BAG relies on eBGP with link bandwidth attributes, enabling Unequal Cost Multipath (UCMP) for efficient load distribution. All BAG‑to‑BAG links are protected with MACsec encryption, aligning with Meta's security standards. ### Oversubscription Management Careful control of oversubscription ratios balances performance and scalability. Typical L2‑to‑BAG oversubscription sits around 4.51, while BAG‑to‑BAG ratios adjust dynamically based on regional traffic patterns and available link capacity. ### Failure Resilience The design includes comprehensive failure domain analysis at the BAG, data‑hall, and power‑distribution levels. Strategies such as traffic draining, conditional route aggregation, and redundant path provisioning mitigate black‑hole risks and maintain high availability across the AI infrastructure.