Low Latency Low Loss Scalable throughput (L4S) – a new default service for the Internet
We are so used to the unpredictability of queuing delay, we don’t know how good the Internet would feel without it. The RITE project has developed simple technology to make queuing delay a thing of the past—not just for a select few apps, but for all. With the resources on this page you can get a feel for the difference, understand how the solution works and help make it happen.
- Queuing Delay—the Problem
- 4-minute video demonstrating and explaining Ultra-Low Queuing Delay for All
- 30-minute video demonstrating, explaining and evaluating “Ultra-Low Queuing Delay for All”
- Videos used in IETF dispatch WG presentation of “Ultra-Low Queuing Delay for All Apps”
- IETF specifications
- Papers and magazine articles
- Source code
- Getting Involved
Queuing Delay? What’s the Problem?
The large majority of Internet traffic is capacity-seeking, usually by using TCP. Until each flow completes it will fill available capacity and the buffers intended to absorb bursts. This adds queuing delay for the flow itself and for other applications running at the same time.
People think voice and gaming need low delay, because they don’t work well otherwise. But, nowadays the performance of nearly every application depends at least as much on delay as on bandwidth, e.g. instant messaging, conversational video, interactive video, remote presence, jumping around streaming video like YouTube, and all those little interactive transfers over the web, web services, finance apps, as well as cloud-based apps, cloud-rendered video, remote desktop, video-assisted control of remote machinery and industrial process control. That’s pretty much everything.
In the presence of other long-running traffic flow(s) or when the application is itself a long-running flow, even with a well-tuned network, the base delay typically doubles, halving performance. Sometimes it can be a lot worse. So queuing delay is one of the main causes of the Internet’s unpredictability. Some delay-sensitive applications are capacity-seeking themselves, e.g. interactive video. For them, queuing delay is ever-present, because they inflict it on themselves.
“Ultra-Low Queuing Delay for All” in 4 minutes
Video of cloud-rendered Panoramic Interactive Video (PIV) compiled during the IETF Bits-N-Bites Exhibition, Prague, July 2015.
“Ultra-Low Queuing Delay for All” in 30 minutes
Presented by Koen De Schepper of Bell Labs at the IETF-93 Active Queue Management (AQM) working group, Prague, Jul 2015.
Videos used in IETF dispatch WG “Ultra-Low Queuing Delay for All Apps” slot
(Note, these videos have no soundtrack, because they were intended to complement remote attendance at the meeting)
Video extract showing ultra-low delay finger-gesture Interaction with a cloud-rendered Panoramic Interactive Video (PIV) of a Football Stadium. The network paths for both the L4S (DCTCP) and the Classic (TCP Cubic) clients use the same broadband residential, DSL access and core networks built using production equipment. They both have a base delay of 7ms; downstream capacity of 40Mb/s; and 2-3 other background TCP flows in progress. The PIV application uses one TCP flow for the media, which consumes no more than about 4Mb/s, adapting if less is available.
As the architecture explains, there are three main parts to standardisation: the identifier (#1), the network algorithms (#2) and the host algorithms (#3):
- Low Latency, Low Loss, Scalable Throughput (L4S) Internet Service: Architecture and Problem Statement
- A proposed new identifier for Low Latency, Low Loss, Scalable throughput (L4S) packets
and the IETF’s enabling standards track RFC to make way for this
There’s also a proposal for a new Diffserv codepoint to identify non-queue-building (NQB) traffic that could share the L4S queue with L4S traffic without all the usual bandwidth prioritization baggage of Diffserv:
- Then network operators can ensure L4S flows are isolated from existing Internet traffic in terms of latency, but they coexist in terms of throughput. The simpel Dual-Queue Coupled AQM framework achieves this without inspecting deeper than the IP header:
Example algorithms are given in an appendices.
L4S is deliberately separate from bandwidth allocation, which is not typically necessary, but could be added by network operators – see <draft-briscoe-tsvwg-l4s-diffserv>
- And host developers can deploy new scalable TCP algorithms, e.g. Data Centre TCP (DCTCP):
Without steps #2 & #3, scalable TCPs are too aggressive to coexist with classic TCPs like Reno and Cubic, so DCTCP was initially confined to controlled networks like Data Centres, where all classic traffic sources could be upgraded (or isolated). To be safe over the public Internet, scalable TCP algorithms will need to comply with the list of requirements agreed at an informally convened meeting in Prague that have become know as the TCP Prague requirements:
- One of these requirements is accurate ECN feedback, which the IETF has recognised will need an update to the TCP protocol, and it has stated the requirements a solution should comply with:
A candidate protocol to satisfy these requirements has been developed and adopted by the IETF:
- The following drafts address other specific TCP Prague requirements:
Adding ECN to TCP control packets:
- Ultra Low Queuing Delay for All; Koen De Schepper, Bob Briscoe, Olga Bondarenko & Inton Tsang, IETF ad hoc meeting (‘Bar BoF’) (Apr 2016) [odp|pdf]
- L4S: A gradually deployable simplifying clean-slate opportunity, Bob Briscoe & Koen De Schepper, IETF ad hoc meeting (‘Bar BoF’) (Apr 2016) [ppt|pdf]
- PI2 AQM for classic and scalable congestion control, Koen De Schepper (Bell Labs), Bob Briscoe, Olga Bondarenko (Simula), Ing-Jyh (Inton) Tsang (Bell Labs), Internet Research Task Force (IRTF) Internet congestion control research group (ICCRG), (Apr 2016) [pdf]
- L4S Birds of a Feather (‘BoF’) held at the IETF in Berlin (Jul 2016)
- L4S: Ultra-Low Queuing Delay for All; Bob Briscoe, Koen De Schepper, Olga Bondarenko & Inton Tsang, Stock presentation for network operators (Jul 2017) [odp|pdf]
- “Flow-start: Faster and Less Overshoot with Paced Chirping” Joakim Misund and Bob Briscoe, Internet Congestion Control Research Group (Jul 2018), a proposed flow-start algorithm for TCP Prague [pdf]
Papers and Magazine Articles
- Article in the IETF Journal describing the Demo in Bits-N-Bites at the IETF in Prague, July 2015.
Bob Briscoe, “Ultra-Low Delay for All” IETF Journal, Nov 2015 [ Pre-print (with pictures) | Journal archive (no pictures) ].
- Description of an ambitious demo:
Olga Bondarenko, Koen De Schepper, Ing-Jyh Tsang, Bob Briscoe, Andreas Petlund and Carsten Griwodz “Ultra-Low Delay for All: Live Experience, Live Analysis“, Proc. ACM Multimedia Systems; Demo Session (May 2016).
- Pre-print of paper explaining why scalable TCP algorithms solve the problem (in plain english and maths), how the Dual Queue Coupled AQM algorithm works, and recording the results of extensive testbed experiments.
“`Data Centre to the Home’: Ultra-Low Latency for All” De Schepper, K., Bondarenko, O., Tsang, I.-J. and Briscoe, B. (2015)
- “PI2: A Linearized AQM for both Classic and Scalable TCP,” Koen De Schepper (Nokia Bell Labs), Olga Bondarenko (Simula), Ing-Jyh Tsang (Nokia Bell Labs), Bob Briscoe (Simula), Proc. ACM CONEXT 2016 (Dec 2016).
- Discussion paper: “Resolving Tensions Between Congestion Control Scaling Requirements“, Bob Briscoe (Simula) and Koen De Schepper (Nokia Bell Labs), Simula Technical Report TR-CS-2016-001 (July 2017).
- Report (RITE project deliverable D3.3) including:
- QoS Simplification Objectives from a network operator (BT) (Section 3.1)
- HTTP Adaptive Streaming (HAS) and Web tests of DualQ Coupled (Section 3)
- Panoramic Interactive Video and Web tests of DualQ Coupled (Section 4)
- “Destruction Testing: Ultra-Low Delay using Dual Queue Coupled Active Queue Management“, Masters Thesis of Henrik Steen (University of Oslo) (May 2017).
- Research to address the TCP Prague Requirement for Faster Convergence than DCTP at Flow Start still with ultra-low queueing delay: “Rapid Acceleration in TCP Prague” (a.k.a. “Paced Chirping”), Masters Thesis of Joakim Misund (University of Oslo) (May 2018).
- Dual Queue Coupled AQM
- With Curvy RED for Linux (access available shortly)
- With PI2 for Linux
- Data Centre TCP (DCTCP) for
- Accurate ECN TCP Feedback for Linux (completed, but not fully tested)
- Paced Chirping for Linux (initial proof-of-concept research code)
- TCP Prague mailing list
- IETF TCP Maintenance and Minor Extensions working group mailing list
- IETF Transport Area working group mailing list
- IETF Active Queue Management working group mailing list (the AQM WG has been folded into the above Transport Area WG, but the mailing list continues)
- Contacts for the authors are included within each of the above specifications and papers