Introduction to the Problem

The CUBIC congestion controller, standardized in RFC 9438, is the default congestion controller in Linux, governing how most TCP and QUIC connections on the public Internet probe for available bandwidth, back off when they detect loss, and recover afterward. At Cloudflare, our open-source implementation of QUIC, quiche, uses CUBIC as its default congestion controller, meaning this code is in the critical path for a significant share of the traffic we serve.

CUBIC's Congestion Window Logic

CUBIC's congestion window (cwnd) gets permanently pinned at its minimum and never recovers from a congestion collapse event. The story starts with a Linux kernel change aimed at bringing CUBIC into line with the app-limited exclusion described in RFC 9438. A fix to a real problem in TCP that when ported to our QUIC implementation surfaced unexpected behaviors in quiche. The CUBIC algorithm's logic is based on a simple premise: if there is no packet loss, increase the sending rate, and if there is loss, decrease the sending rate.

Congestion Control Algorithms (CCAs) Basics

Congestion Control Algorithms (CCAs) are used to manage the amount of data that can be sent over a network at any given time. The central knob a CCA turns is the congestion window (cwnd), the sender-side cap on how many bytes can be in flight (sent but not yet acknowledged) at any moment. A larger cwnd lets the sender push more data per round trip, a smaller cwnd throttles it. Every loss-based CCA, including CUBIC, is ultimately a policy for how to grow cwnd when the network looks healthy and how to shrink it when it doesn't.

The Linux Kernel Change and Its Impact

A Linux kernel change aimed at bringing CUBIC into line with the app-limited exclusion described in RFC 9438 led to a QUIC bug. The change, which was intended to fix a real problem in TCP, surfaced unexpected behaviors in quiche when ported to our QUIC implementation. The bug caused the cwnd to get permanently pinned at its minimum, preventing the sender from recovering from a congestion collapse event.

Loss-Based Congestion Control Algorithms

Loss-based CCAs, such as CUBIC, operate on a fundamental premise: if there is no packet loss, increase the sending rate, and if there is loss, decrease the sending rate. This premise is based on the idea that packet loss is an indication of network congestion, and decreasing the sending rate can help to alleviate congestion. The family of loss-based algorithms to which CUBIC belongs operate on this premise, aiming to maximize data transfer by inferring the available bandwidth of the network.

The Elegant Fix: Breaking the Cycle

The bug was fixed with an elegant, near-one-line fix that broke the cycle CUBIC's logic. The fix, which was implemented in the quiche codebase, prevented the cwnd from getting permanently pinned at its minimum, allowing the sender to recover from congestion collapse events. The fix was a result of careful analysis and testing, and it has been successfully deployed in production, ensuring that QUIC connections using CUBIC as their congestion controller can recover from network congestion and maintain optimal performance.