Poor performance of my virtual Switch in proxmox

General questions
1

Following my previous Post about setting up a vyos switch on a proxmox server:

now the switch works but the performance are not that good.

I have created 2 vitrual machine (debian) connected to the 2 virtual interfaces that are part of the switch.
The both get the dhcp ip :
root@debian-test:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host noprefixroute
valid_lft forever preferred_lft forever
2: enp6s18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 1e:3a:60:1e:72:bc brd ff:ff:ff:ff:ff:ff
inet6 fe80::1c3a:60ff:fe1e:72bc/64 scope link
valid_lft forever preferred_lft forever
3: enp6s19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether b2:4d:17:d0:26:ea brd ff:ff:ff:ff:ff:ff
inet 192.168.1.50/24 brd 192.168.1.255 scope global dynamic enp6s19
valid_lft 85201sec preferred_lft 85201sec
inet6 fe80::b04d:17ff:fed0:26ea/64 scope link
valid_lft forever preferred_lft forever

root@debian-test:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host noprefixroute
valid_lft forever preferred_lft forever
2: enp6s18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 62:e9:c5:9c:6c:e4 brd ff:ff:ff:ff:ff:ff
inet6 fe80::60e9:c5ff:fe9c:6ce4/64 scope link
valid_lft forever preferred_lft forever
3: enp6s19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 2a:9c:61:62:40:99 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.100/24 brd 192.168.1.255 scope global dynamic enp6s19
valid_lft 85148sec preferred_lft 85148sec
inet6 fe80::289c:61ff:fe62:4099/64 scope link
valid_lft forever preferred_lft forever

But if I try to check the speed with iperf3 I get the following results:
root@debian-test:~# iperf3 -c 192.168.1.50
Connecting to host 192.168.1.50, port 5201
[ 5] local 192.168.1.100 port 42786 connected to 192.168.1.50 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 938 MBytes 7.87 Gbits/sec 0 3.15 MBytes
[ 5] 1.00-2.00 sec 931 MBytes 7.81 Gbits/sec 0 3.15 MBytes
[ 5] 2.00-3.00 sec 968 MBytes 8.12 Gbits/sec 0 3.15 MBytes
[ 5] 3.00-4.00 sec 962 MBytes 8.07 Gbits/sec 0 3.15 MBytes
[ 5] 4.00-5.00 sec 945 MBytes 7.93 Gbits/sec 0 3.15 MBytes
[ 5] 5.00-6.00 sec 940 MBytes 7.89 Gbits/sec 0 3.15 MBytes
[ 5] 6.00-7.00 sec 936 MBytes 7.85 Gbits/sec 0 3.15 MBytes
[ 5] 7.00-8.00 sec 909 MBytes 7.63 Gbits/sec 0 3.15 MBytes
[ 5] 8.00-9.00 sec 944 MBytes 7.92 Gbits/sec 0 3.15 MBytes
[ 5] 9.00-10.00 sec 976 MBytes 8.19 Gbits/sec 0 3.15 MBytes

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 9.23 GBytes 7.93 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 9.22 GBytes 7.92 Gbits/sec receiver

iperf Done.

I was expecting something better , so I have tried to increase the mtu on the debian side to 9000 (on the vyos switch all ports have mtu 9000).

But the performace in this casa goes to almost 0! :

root@debian-test:~# ip link set dev enp6s19 mtu 9000
root@debian-test:~# iperf3 -c 192.168.1.50
Connecting to host 192.168.1.50, port 5201
[ 5] local 192.168.1.100 port 60134 connected to 192.168.1.50 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 323 KBytes 2.65 Mbits/sec 7 8.74 KBytes
[ 5] 1.00-2.00 sec 0.00 Bytes 0.00 bits/sec 1 8.74 KBytes
[ 5] 2.00-3.00 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes
[ 5] 3.00-4.00 sec 0.00 Bytes 0.00 bits/sec 1 8.74 KBytes
[ 5] 4.00-5.00 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes
[ 5] 5.00-6.00 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes
[ 5] 6.00-7.00 sec 0.00 Bytes 0.00 bits/sec 1 8.74 KBytes
[ 5] 7.00-8.00 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes
[ 5] 8.00-9.00 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes
[ 5] 9.00-10.00 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 323 KBytes 265 Kbits/sec 10 sender
[ 5] 0.00-10.00 sec 0.00 Bytes 0.00 bits/sec receiver

iperf Done.

That I am doing wrong ?
Regards
Albert

2

Did you enable offloading in the ethernet NICs in VyOS?

3

I don’t think so, how I do it ?
Regards
Albert

4

https://docs.vyos.io/en/latest/configuration/interfaces/ethernet.html#offloading

5

thanks,
I have done it (even if I am not sure about wich type to use : <gro | gso | lro | rps | sg | tso> ).
But still the same result.

And by the way I have tried to connect a phisical machine to the 10gbe port and I have the same beaviour.

6

I update a little bit this post.
I have made an experiment by connecting two phisical machine to the virtual vyos machine.
(previous results were obtained with a physical and a virtual machine)
And I get the following results
With mtu set at 1500:

Connecting to host 192.168.1.103, port 5201
[ 5] local 192.168.1.101 port 48100 connected to 192.168.1.103 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 963 MBytes 8.08 Gbits/sec 64 884 KBytes
[ 5] 1.00-2.00 sec 1001 MBytes 8.40 Gbits/sec 94 635 KBytes
[ 5] 2.00-3.00 sec 1021 MBytes 8.57 Gbits/sec 97 887 KBytes
[ 5] 3.00-4.00 sec 1.00 GBytes 8.61 Gbits/sec 80 888 KBytes
[ 5] 4.00-5.00 sec 1.00 GBytes 8.62 Gbits/sec 47 628 KBytes
[ 5] 5.00-6.00 sec 1024 MBytes 8.59 Gbits/sec 43 888 KBytes
[ 5] 6.00-7.00 sec 1.00 GBytes 8.61 Gbits/sec 119 888 KBytes
[ 5] 7.00-8.00 sec 1022 MBytes 8.58 Gbits/sec 104 901 KBytes
[ 5] 8.00-9.00 sec 1024 MBytes 8.59 Gbits/sec 135 875 KBytes
[ 5] 9.00-10.00 sec 1.00 GBytes 8.60 Gbits/sec 66 899 KBytes

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 9.92 GBytes 8.52 Gbits/sec 849 sender
[ 5] 0.00-10.00 sec 9.92 GBytes 8.52 Gbits/sec receiver

iperf Done.

Ad with the mtu set at 9000:
Connecting to host 192.168.1.103, port 5201
[ 5] local 192.168.1.101 port 49142 connected to 192.168.1.103 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.15 GBytes 9.89 Gbits/sec 0 1.54 MBytes
[ 5] 1.00-2.00 sec 1.15 GBytes 9.90 Gbits/sec 0 2.31 MBytes
[ 5] 2.00-3.00 sec 1.15 GBytes 9.90 Gbits/sec 0 2.31 MBytes
[ 5] 3.00-4.00 sec 1.15 GBytes 9.90 Gbits/sec 0 2.31 MBytes
[ 5] 4.00-5.00 sec 1.15 GBytes 9.90 Gbits/sec 0 2.31 MBytes
[ 5] 5.00-6.00 sec 1.15 GBytes 9.90 Gbits/sec 0 2.31 MBytes
[ 5] 6.00-7.00 sec 1.15 GBytes 9.90 Gbits/sec 0 2.31 MBytes
[ 5] 7.00-8.00 sec 1.15 GBytes 9.90 Gbits/sec 0 2.31 MBytes
[ 5] 8.00-9.00 sec 1.15 GBytes 9.90 Gbits/sec 0 2.31 MBytes
[ 5] 9.00-10.00 sec 1.15 GBytes 9.87 Gbits/sec 79 1.91 MBytes

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 11.5 GBytes 9.89 Gbits/sec 79 sender
[ 5] 0.00-10.00 sec 11.5 GBytes 9.89 Gbits/sec receiver

iperf Done.

So the problem of bad (or terrible with mtu set at 9000) performace seems to be related to the proxmox virtual interface vmbr2 & vmbr3, but if I am only using the phisical ports then I am ok.

I will try to reinstall the virtual machine following the raccomandation received before ad see if I can solve the issue.
Any suggestion/raccomandation is welcome!
Regards
Albert

7

If you’re creating a bridge between the two Proxmox bridges in VyOS, why have two bridges to begin with? Just do a single Proxmox bridge with both ports as members.

8

Not sure I got it right, you mean create just one vmbr and assign it to vyos and the two virtual debian machines ?
Regards
Albert

9

In the bridge ports box, you can specify multiple physical ports. That way you don’t have to create a bridge within VyOS and you instead just have a single NIC. To me it appears that accomplishes the same thing and eliminates an extra step of complexity for which there is definitely a speed penalty.

image