What kind of performance should be expected when applying FQ-Codel on a shaper policy?

General questions
,
1

I have applied FQ-Codel as a default class to try and get everything setup, but I can only get ~100mbit/s down/up.
Applying FQ-Codel directly on the interface gets me ~500mbit/s, but still not anywhere close to what I’m expecting.

I am running 1.5-rolling-202402260024 on a Supermicro sys-5019a-fn5t, I tested my Edgrouter 12 with the same FQ-Codel settings and that can achieve ~800mbit/s with much weaker hardware.

This is the config:

interface eth1 {
    egress wan-out
 }
 interface ifb1 {
     egress wan-in
 }policy {
     fq-codel codel-in {
         queue-limit 1000
     }
     shaper wan-in {
         bandwidth 940mbit
         default {
             bandwidth 100%
             ceiling 90%
             codel-quantum 15140
             priority 7
             queue-type fq-codel
         }
     }
     shaper wan-out {
         bandwidth auto
         default {
             bandwidth 100%
             ceiling 90%
             codel-quantum 15140
             priority 7
             queue-limit 1200
             queue-type fq-codel
         }
     }
 }

MOD Edit: Config Formatting

2

What does top show while you’re testing, is a CPU getting pegged with an interrupt?

3

I don’t know how to check interrupts, top shows less than 6% CPU usage.

4

I am having similar issues with fq-codel with 1.4.0-epa1.

Switching to fair-queue gives me expected bandwidth and I do not have this issue with 1.3.6.

5

Thanks for the fair-queue advice. 1.3.6 works great with fq-codel. 1.4epa1 had buffer bloat issues (sometimes weird speed issues) with fq-codel. With fair-queue, performance isn’t quite as good as fq-codel on 1.3.6, but good enough for my (work from home) purposes - good throughput and an A on Waveform’s buffer bloat test.

I’ll tweak later. For now, on a 300/10 DOCSIS connection (provisioned more like 350/11), I’m just using these very basic config commands:

set qos policy shaper EGRESS_QOS bandwidth 11mbit
set qos policy shaper EGRESS_QOS default bandwidth 11mbit
set qos policy shaper EGRESS_QOS default queue-type fair-queue
set qos interface eth0 egress EGRESS_QOS

set qos policy shaper INGRESS-SHAPING bandwidth 325mbit
set qos policy shaper INGRESS-SHAPING default bandwidth 325mbit
set qos policy shaper INGRESS-SHAPING default queue-type fair-queue
set interfaces input ifb0
set interfaces ethernet eth0 redirect ifb0
set qos interface ifb0 egress INGRESS-SHAPING

6

Is this with the same config between the two of them? Have you migrated from one to the other?
Be great if you can share some more details - it’s issues like this that the -epa releases are meant to surface etc, so if there’s a real issue here it’d be good to get it properly documented and even logged if possible. There’s a couple of QoS issues with 1.4 open at the moment, be good to know if they’re related or not.

7

I’ve been struggling with this since 1.4RC1 (last September?). I started with a config that had been migrated by 1.4 from 1.3. Looking back at my notes, the migration appears to have been correct (though at the time it differed from the docs, which at the time were missing the line “set interfaces input ifb0”, and I found conflicting advice online about the line “set interfaces input ifb0”, so I spent some time confused). I ended up deleting all QOS configs, because no QOS was better than fq-codel. Then I saw cmaxwell’s advice, and manually added back in the minimum for fair-queue (which works well).

As an experiment, I just now converted my simple fair-queue config (see above) to fq-codel, with no other changes (no parameters set other than egress and ingress speeds). First, I tested fair-queue with Waveform’s bufferbloat site.

Fair-Queue:
Grade: A
LATENCY
Unloaded 21 ms
Download Active +10 ms
Upload Active +5 ms

SPEED
↓ Download 302.6 Mbps
↑ Upload 11.2 Mbps

Then I simply changed fair-queue to fq-codel,

set qos policy shaper EGRESS_QOS default queue-type fq-codel
set qos policy shaper INGRESS-SHAPING default queue-type fq-codel

and results were much worse:

Grade: B
LATENCY
Unloaded 22 ms
Download Active +51 ms
Upload Active +18 ms

SPEED
↓ Download 226.1 Mbps
↑ Upload 7.80 Mbps

Then I added back in the fq-codel parameters that worked well in 1.3.6:

set qos policy shaper EGRESS_QOS default burst ‘15kbit’
set qos policy shaper EGRESS_QOS default codel-quantum ‘300’
set qos policy shaper EGRESS_QOS default queue-limit ‘1024’
set qos policy shaper INGRESS-SHAPING default burst ‘15kbit’
set qos policy shaper INGRESS-SHAPING default queue-limit ‘1024’

Now the Waveform results (still bad):

Grade: B
LATENCY
Unloaded 22 ms
Download Active +45 ms
Upload Active +27 ms

SPEED
↓ Download 209.4 Mbps
↑ Upload 8.34 Mbps

Just for reference, here are numbers I got with 1.4epa1, no QOS:

Grade: B
LATENCY
Unloaded 22 ms
Download Active +19 ms
Upload Active +53 ms

SPEED
↓ Download 368.4 Mbps
↑ Upload 11.1 Mbps

8

I will raise a bug when I get a chance.

It was a migrated config and the qos details are the same. Seems to impact fq_codel and not fair-queue.

9

Be good to see the output from tc with 1.3 and then with 1.4

I will try the same test here (still using 1.3.6)

10

Let me know what tc command to run and I would be happy to share.

1.3.5:

1.4.0-epa1

11

I’m seeing the same issue with fq-codel going from 1.3.6 to 1.4.0-epa1

Version 1.3.6 (docker source build)

set traffic-policy shaper WAN-IN bandwidth '100mbit'
set traffic-policy shaper WAN-IN default bandwidth '100%'
set traffic-policy shaper WAN-IN default burst '15k'
set traffic-policy shaper WAN-IN default queue-type 'fq-codel'
set traffic-policy shaper WAN-OUT bandwidth '40mbit'
set traffic-policy shaper WAN-OUT default bandwidth '100%'
set traffic-policy shaper WAN-OUT default burst '15k'
set traffic-policy shaper WAN-OUT default queue-type 'fq-codel'

vyos@vyos:~$ tc qdisc show
qdisc noqueue 0: dev lo root refcnt 2
qdisc htb 1: dev eth0 root refcnt 9 r2q 25 default 0x2 direct_packets_stat 0 direct_qlen 1000
qdisc fq_codel 8002: dev eth0 parent 1:2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb
qdisc ingress ffff: dev eth0 parent ffff:fff1 ----------------
qdisc mq 0: dev eth1 root
qdisc pfifo_fast 0: dev eth1 parent :4 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth1 parent :3 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth1 parent :2 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth1 parent :1 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc mq 0: dev eth2 root
qdisc pfifo_fast 0: dev eth2 parent :4 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth2 parent :3 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth2 parent :2 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc pfifo_fast 0: dev eth2 parent :1 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
qdisc htb 1: dev ifb0 root refcnt 2 r2q 63 default 0x2 direct_packets_stat 0 direct_qlen 32
qdisc fq_codel 8003: dev ifb0 parent 1:2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb

Config migrated to 1.4.0-epa1 (docker source build)

vyos@vyos:~$ tc qdisc show
qdisc noqueue 0: dev lo root refcnt 2 
qdisc htb 1: dev eth0 root refcnt 9 r2q 25 default 0x1 direct_packets_stat 0 direct_qlen 1000
qdisc fq_codel 8002: dev eth0 parent 1:1 limit 10240p flows 1024 quantum 1514 target 4us interval 99us memory_limit 32Mb drop_batch 64 
qdisc ingress ffff: dev eth0 parent ffff:fff1 ---------------- 
qdisc mq 0: dev eth1 root 
qdisc fq_codel 0: dev eth1 parent :4 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64 
qdisc fq_codel 0: dev eth1 parent :3 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64 
qdisc fq_codel 0: dev eth1 parent :2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64 
qdisc fq_codel 0: dev eth1 parent :1 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64 
qdisc mq 0: dev eth2 root 
qdisc fq_codel 0: dev eth2 parent :4 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64 
qdisc fq_codel 0: dev eth2 parent :3 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64 
qdisc fq_codel 0: dev eth2 parent :2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64 
qdisc fq_codel 0: dev eth2 parent :1 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64 
qdisc noqueue 0: dev pim6reg root refcnt 2 
qdisc htb 1: dev ifb0 root refcnt 2 r2q 63 default 0x1 direct_packets_stat 1 direct_qlen 32
qdisc fq_codel 8004: dev ifb0 parent 1:1 limit 10240p flows 1024 quantum 1514 target 4us interval 99us memory_limit 32Mb drop_batch 64
12

On 1.4.0-epa, it looks like the qdisc fq_codel ‘target’ and ‘interval’ parameters are out by a factor of 1000 (us instead of ms)

13

The calculations are here vyos-1x/python/vyos/qos/fqcodel.py at c095867d873a9a7dde038bb751ba26edc66b99f7 · vyos/vyos-1x · GitHub

I created a bug report T6081

3 Likes
14

It should be fixed in the latest rolling.
Could someone re-check?

1 Like
15

Download seems to have greatly improved, but now my download cannot break 100mbit/s.

It seems that my current bottleneck is the IFB0 redirect, by applying the QOS rules on the LAN interface i get near full speed.

16

Fix looks good to me on 1.4-rolling-202403022209 (config migrated from 1.3.6), cheers!

vyos@vyos:~$ tc qdisc show dev eth0
qdisc htb 1: root refcnt 9 r2q 25 default 0x1 direct_packets_stat 0 direct_qlen 1000
qdisc fq_codel 8002: parent 1:1 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb drop_batch 64 
qdisc ingress ffff: parent ffff:fff1 ----------------
3 Likes
17

I am very happy to see y’all resolve this off by a 1000 error! :whew: That would have dropped a lot of packets far too soon to achieve any throughput.

In general you should be able to set fq_codel or cake as the default across the board now AND be able to run at full line rate and replace pfifo_fast via either compiling it as a default or setting it as the default qdisc early in the boot.

A couple notes:

A) the HTB 15k burst value is really rather high for sub 100Mbit shaping and causes a burp. We set that closer to 3k in the sqm-scripts, and more dynamically. That might be why your waveform test still has so much jitter. You need to shred the burst down to packets.

B) A queue limit of about 1000 is about right for up to 1gbit traffic. However, we have generally adopted a memory limit instead and been leaving it at the larger 10k figure otherwise. And unless you are memory constrained, leaving this at the default is ok - the AQM manages the queue length for you. Given GRO and TSO offloads 64k “packets” I would deprecate packet limits entirely in favor of memory limits.

C) The quantum 300 is a good number for sub 100Mbit, a quantum of the MTU is cheaper CPU-wise above that.

D) Once you are getting the bandwidth results you are looking for, it is seeing the full bandwidth at vastly reduced latency (vs pfifo_fast) that rocks for more traffic. As much as I like the waveform test, I keep hoping people will measure up + down + latency at the same time with something like the rrul test or a new one called crusader or even better, take a packet capture in wireshark and plot the rtts.

E) Tried cake yet? It too works as a default qdisc, eats more cpu, but does more stuff - and has a variety of statistics that are interesting.

2 Likes
18

cake, among other things, has support for docsis framing, so on the uplink you can get close to 99.999% of the uplink rate, accurately. Please see also:

19

I see an example above where the codel-quantum is set to 15140. Noooo. The max should be the MTU of the underlying interface. Also in that example, the inbound shaper is set to 940Mbits which is far too close to the underlying rate to be effective.

I am not big on inbound shaping, it is a hack to take away control from someone elses oversized queue, I keep hoping people doing inbound shaping in self defense also ask their upstream to apply rfc7567 there at their egress, instead, or look over libreqos.io.

I am only allowed three replies per thread, so I am responding to @marc_s below this one… you should be able to run cake natively, without shaping, on the egress, assuming your ONT will do flow control at a reasonable internal buffer size.

I am also big on the symmetric FTTH fans attempting simultaneous up + down tests because the results are un-intuitive with a fifo vs a vs cake. There is a new one I like over here: Release Automated build of v0.0.10-testing · Zoxc/crusader · GitHub

Cool, you have nat enabled. One of my favorite things to do to a network with that on, is to fire up a huge bitttorrent session on one box, and then the waveform or anything else on another, where it achieves per host fairness.

One day I will write the ultimate fq_codel and cake configuration guide, but it starts with “take the defaults”. I otherwise tend towards loquacious and being able to post links and respond more than thrice to a conversation here would help… @tjh ?

1 Like
20

Fun thread this. A lot of technical stuff to go through. Thought I’d share something with you. I tried cake today on a 1gbit symmetric FTTH line (actual observed bandwidth without any QOS is 950/950 mbits) and was very happy with the results.

Without QOS:

Grade A
Up/down 951/916 mbit/s
Unloaded 5ms
Download active +14ms
Upload active +17ms
Link: https://www.waveform.com/tools/bufferbloat?test-id=f12b1cee-48ef-4470-a248-1505c81c6448

With CAKE like this (RTT 70ms, bandwidth 950mbit/s):

set interfaces input ifb0
set interfaces pppoe pppoe0 redirect 'ifb0'
set qos interface ifb0 egress 'CAKE-WAN-IN'
set qos interface pppoe0 egress 'CAKE-WAN-OUT'
set qos policy cake CAKE-WAN-IN bandwidth '950mbit'
set qos policy cake CAKE-WAN-IN flow-isolation nat
set qos policy cake CAKE-WAN-IN rtt '70'
set qos policy cake CAKE-WAN-OUT bandwidth '950mbit'
set qos policy cake CAKE-WAN-OUT flow-isolation nat
set qos policy cake CAKE-WAN-OUT rtt '70'

I get:

Grade A+
Up/down 811/910 mbit/s
Unloaded 5ms
Download active +2ms
Upload active +2ms
Link: https://www.waveform.com/tools/bufferbloat?test-id=54e071e4-9b46-45d8-b8e9-a33b486295d1

The bandwidth is actually a bit better: according to Ookla (Speedtest by Ookla - The Global Broadband Speed Test) it’s more like 842/924 mbit/s.

I’m very content with this. Currently trying to get a better understanding of which buttons and dials to work to get even faster results. tc-cake(8) - Linux manual page helps a lot :slight_smile:

3 Likes