Extremely slow LAN speed on PCEngines APU2

ACiD_GRiM · August 11, 2023, 5:28am

I have a pair of PCEngines APU2 routers that easily routed 800Mbps+ on both over the internet, these are located at a colo, so I don’t have easy access to either. .

At some point this changed and my network has struggled to get over 50-60Mbps, however still gets 800+ over the WAN directly from the router. both routers have always been connected to the LAN using a bridge/STP over two identical switches. This was over a year ago, several vyos builds later.

I suspected a switch issue, so today I had the datacenter tech replace all of the cables between the LAN servers and each router. and now both routers are connected to eachother with 1 cable and up to the switch with the other, still in a bridge configuration. All new cables did not improve the issue.

I then removed the crosslink interfaces from the bridge and configured each router in a /30 network, but they still do not exceed 50-60Mbps. I think this is confirming some configuration or driver issue.

both router configuration are identical except for ip addresses, can someone please help me identify this issues cause

Version:          VyOS 1.4-rolling-202308060317
Release train:    current

Built by:         autobuild@vyos.net
Built on:         Sun 06 Aug 2023 03:17 UTC
Build UUID:       a2edfca7-c3b1-4158-acbf-d8226af46599
Build commit ID:  705a03eb980c84

Architecture:     x86_64
Boot via:         installed image
System type:      bare metal

Hardware vendor:  PC Engines
Hardware model:   apu2
Hardware S/N:     123456789
Hardware UUID:    Unknown

Copyright:        VyOS maintainers and contributors

set firewall all-ping 'enable'
set firewall broadcast-ping 'disable'
set firewall config-trap 'disable'
set firewall ip-src-route 'disable'
set firewall ipv6-name ADMIN_v6-to-LOCAL default-action 'accept'
set firewall ipv6-name ADMIN_v6-to-WAN default-action 'accept'
set firewall ipv6-name LOCAL_v6-to-ADMIN default-action 'accept'
set firewall ipv6-name LOCAL_v6-to-WAN default-action 'accept'
set firewall ipv6-name WAN_v6-to-ADMIN default-action 'drop'
set firewall ipv6-name WAN_v6-to-ADMIN enable-default-log
set firewall ipv6-name WAN_v6-to-LOCAL default-action 'drop'
set firewall ipv6-name WAN_v6-to-LOCAL enable-default-log
set firewall ipv6-name WAN_v6-to-LOCAL rule 20 action 'accept'
set firewall ipv6-name WAN_v6-to-LOCAL rule 20 destination port '13698'
set firewall ipv6-name WAN_v6-to-LOCAL rule 20 protocol 'tcp'
set firewall ipv6-name WAN_v6-to-LOCAL rule 20 state new 'enable'
set firewall ipv6-name WAN_v6-to-LOCAL rule 999 action 'accept'
set firewall ipv6-name WAN_v6-to-LOCAL rule 999 protocol 'ipv6-icmp'
set firewall ipv6-receive-redirects 'disable'
set firewall ipv6-src-route 'disable'
set firewall log-martians 'enable'
set firewall name ADMIN_v4-to-LOCAL default-action 'accept'
set firewall name ADMIN_v4-to-WAN default-action 'accept'
set firewall name LOCAL_v4-to-ADMIN default-action 'accept'
set firewall name LOCAL_v4-to-WAN default-action 'accept'
set firewall name WAN_v4-to-ADMIN default-action 'drop'
set firewall name WAN_v4-to-ADMIN enable-default-log
set firewall name WAN_v4-to-LOCAL default-action 'drop'
set firewall name WAN_v4-to-LOCAL enable-default-log
set firewall name WAN_v4-to-LOCAL rule 10 action 'accept'
set firewall name WAN_v4-to-LOCAL rule 10 source address 'xxx/29'
set firewall name WAN_v4-to-LOCAL rule 20 action 'accept'
set firewall name WAN_v4-to-LOCAL rule 20 destination port '13698'
set firewall name WAN_v4-to-LOCAL rule 20 protocol 'tcp'
set firewall name WAN_v4-to-LOCAL rule 20 state new 'enable'
set firewall name WAN_v4-to-LOCAL rule 100 action 'accept'
set firewall name WAN_v4-to-LOCAL rule 100 destination port '5201'
set firewall name WAN_v4-to-LOCAL rule 100 protocol 'tcp_udp'
set firewall name WAN_v4-to-LOCAL rule 100 source address 'xxx/29'
set firewall name WAN_v4-to-LOCAL rule 100 state new 'enable'
set firewall receive-redirects 'disable'
set firewall send-redirects 'enable'
set firewall state-policy established action 'accept'
set firewall state-policy related action 'accept'
set firewall syn-cookies 'enable'
set firewall twa-hazards-protection 'disable'
set firewall zone ADMIN default-action 'drop'
set firewall zone ADMIN from LOCAL firewall ipv6-name 'LOCAL_v6-to-ADMIN'
set firewall zone ADMIN from LOCAL firewall name 'LOCAL_v4-to-ADMIN'
set firewall zone ADMIN from WAN firewall ipv6-name 'WAN_v6-to-ADMIN'
set firewall zone ADMIN from WAN firewall name 'WAN_v4-to-ADMIN'
set firewall zone ADMIN interface 'br1.2'
set firewall zone ADMIN interface 'br1.1023'
set firewall zone ADMIN interface 'eth2'
set firewall zone ADMIN intra-zone-filtering action 'accept'
set firewall zone LOCAL default-action 'drop'
set firewall zone LOCAL from ADMIN firewall ipv6-name 'ADMIN_v6-to-LOCAL'
set firewall zone LOCAL from ADMIN firewall name 'ADMIN_v4-to-LOCAL'
set firewall zone LOCAL from WAN firewall ipv6-name 'WAN_v6-to-LOCAL'
set firewall zone LOCAL from WAN firewall name 'WAN_v4-to-LOCAL'
set firewall zone LOCAL local-zone
set firewall zone WAN default-action 'drop'
set firewall zone WAN from ADMIN firewall ipv6-name 'ADMIN_v6-to-WAN'
set firewall zone WAN from ADMIN firewall name 'ADMIN_v4-to-WAN'
set firewall zone WAN from LOCAL firewall ipv6-name 'LOCAL_v6-to-WAN'
set firewall zone WAN from LOCAL firewall name 'LOCAL_v4-to-WAN'
set firewall zone WAN interface 'eth1'
set firewall zone WAN interface 'br1.1024'
set high-availability vrrp group ADMIN address xxx/24
set high-availability vrrp group ADMIN advertise-interval '1'
set high-availability vrrp group ADMIN hello-source-address 'xxx'
set high-availability vrrp group ADMIN interface 'br1.2'
set high-availability vrrp group ADMIN no-preempt
set high-availability vrrp group ADMIN peer-address 'xxx'
set high-availability vrrp group ADMIN priority '120'
set high-availability vrrp group ADMIN track interface 'eth1'
set high-availability vrrp group ADMIN transition-script backup '/config/scripts/vrrp-fail.sh'
set high-availability vrrp group ADMIN transition-script fault '/config/scripts/vrrp-fail.sh'
set high-availability vrrp group ADMIN transition-script master '/config/scripts/vrrp-master.sh'
set high-availability vrrp group ADMIN transition-script stop '/config/scripts/vrrp-fail.sh'
set high-availability vrrp group ADMIN vrid '1'
set high-availability vrrp sync-group hf2_syncgroup member 'ADMIN'
set high-availability vrrp sync-group hf2_syncgroup transition-script backup '/config/scripts/vrrp-fail.sh'
set high-availability vrrp sync-group hf2_syncgroup transition-script master '/config/scripts/vrrp-master.sh'
set high-availability vrrp sync-group hf2_syncgroup transition-script stop '/config/scripts/vrrp-fail.sh'
set interfaces bridge br1 enable-vlan
set interfaces bridge br1 member interface eth3 allowed-vlan '1-1024'
set interfaces bridge br1 member interface eth3 native-vlan '1'
set interfaces bridge br1 stp
set interfaces bridge br1 vif 2 address 'xxx/24'
set interfaces bridge br1 vif 2 address 'fxxx64'
set interfaces bridge br1 vif 1023 address 'xxx0/64'
set interfaces bridge br1 vif 1023 address 'xxx/29'
set interfaces bridge br1 vif 1024 ip enable-proxy-arp
set interfaces ethernet eth0 address 'xxx/30'
set interfaces ethernet eth0 hw-id '00:24:9b:6a:bc:6b'
set interfaces ethernet eth1 address 'xxx/24'
set interfaces ethernet eth1 address 'xxx/64'
set interfaces ethernet eth1 hw-id '00:0d:b9:42:28:8c'
set interfaces ethernet eth1 ip enable-proxy-arp
set interfaces ethernet eth1 ipv6
set interfaces ethernet eth2 address '192.168.165.1/30'
set interfaces ethernet eth2 disable-flow-control
set interfaces ethernet eth2 hw-id '00:0d:b9:42:28:8d'
set interfaces ethernet eth2 speed 'auto'
set interfaces ethernet eth3 hw-id '00:0d:b9:42:28:8e'
set interfaces ethernet eth3 speed 'auto'
set interfaces loopback lo
set nat source rule 1 outbound-interface 'eth1'
set nat source rule 1 source address 'xxx'
set nat source rule 1 translation address 'xxx'
set nat66 source rule 1 outbound-interface 'eth1'
set nat66 source rule 1 source prefix xxx
set nat66 source rule 1 translation address xxx
set policy route-map ospf-connected rule 1 action 'permit'
set policy route-map ospf-connected rule 1 match interface 'eth0'
set policy route-map ospfv3-connected rule 1 action 'permit'
set policy route-map ospfv3-connected rule 1 match interface 'eth0'
set protocols ospf interface br1.2 cost '1'
set protocols ospf interface br1.2 dead-interval '6'
set protocols ospf interface br1.2 hello-interval '1'
set protocols ospf interface br1.2 passive disable
set protocols ospf interface br1.1023 cost '100'
set protocols ospf interface br1.1023 dead-interval '6'
set protocols ospf interface br1.1023 hello-interval '1'
set protocols ospf interface br1.1023 passive disable
set protocols ospf parameters router-id '0.0.0.255'
set protocols ospf passive-interface 'default'
set protocols ospf redistribute connected route-map 'ospf-connected'
set protocols ospfv3 interface br1.2 area '0.0.0.2'
set protocols ospfv3 interface br1.2 cost '1'
set protocols ospfv3 interface br1.2 dead-interval '6'
set protocols ospfv3 interface br1.2 hello-interval '1'
set protocols ospfv3 interface br1.1023 area '0.0.0.0'
set protocols ospfv3 interface br1.1023 cost '100'
set protocols ospfv3 interface br1.1023 dead-interval '6'
set protocols ospfv3 interface br1.1023 hello-interval '1'
set protocols ospfv3 interface eth0 passive
set protocols ospfv3 interface eth1 passive
set protocols ospfv3 parameters router-id '0.0.0.255'
set protocols ospfv3 redistribute connected route-map 'ospfv3-connected'
set protocols static route 0.0.0.0/0 next-hop xxx
set protocols static route6 ::/0 next-hop fe80::4a8f:5aff:fe2d:830f interface 'eth1'
set qos interface br1.1024 egress 'insideQOS'
set qos interface eth1 egress 'outsideQOS'
set qos policy cake insideQOS bandwidth '1gbit'
set qos policy cake outsideQOS bandwidth '1gbit'
set service conntrack-sync accept-protocol 'tcp'
set service conntrack-sync accept-protocol 'udp'
set service conntrack-sync accept-protocol 'icmp'
set service conntrack-sync accept-protocol 'icmp6'
set service conntrack-sync accept-protocol 'sctp'
set service conntrack-sync accept-protocol 'dccp'
set service conntrack-sync disable-external-cache
set service conntrack-sync expect-sync 'all'
set service conntrack-sync failover-mechanism vrrp sync-group 'hf2_syncgroup'
set service conntrack-sync interface br1.2
set service conntrack-sync mcast-group '225.0.0.51'
set service conntrack-sync sync-queue-size '512M'
set service dhcp-relay interface 'br1.2'
set service ntp server ntp.0xcbf.net
set service router-advert interface br1.2 default-lifetime '5'
set service router-advert interface br1.2 default-preference 'high'
set service router-advert interface br1.2 dnssl '0xcbf.net'
set service router-advert interface br1.2 interval max '4'
set service router-advert interface br1.2 interval min '3'
set service router-advert interface br1.2 other-config-flag
set service router-advert interface br1.2 prefix fd00:f9a8:f::/64
set service router-advert interface br1.2 reachable-time '5000'
set service snmp v3 view default oid 1
set service ssh listen-address '0.0.0.0'
set service ssh listen-address '::'
set service ssh port '13698'
set system config-management commit-revisions '100'
set system conntrack expect-table-size '50000000'
set system conntrack modules ftp
set system conntrack modules h323
set system conntrack modules nfs
set system conntrack modules pptp
set system conntrack modules sip
set system conntrack modules sqlnet
set system conntrack modules tftp
set system conntrack table-size '50000000'
set system conntrack tcp half-open-connections '2147483647'
set system console device ttyS0 speed '115200'
set system ip multipath layer4-hashing
set system ipv6 multipath layer4-hashing
set system login banner post-login ''
set system login banner pre-login '* * * * * * * * * xxx OFF WERE FULL * * * * * * * * * * *\nTHIS SYSTEM IS RESTRICTED TO AUTHORIZED USERS FOR AUTHORIZED\nUSE ONLY. UNAUTHORIZED ACCESS IS STRICTLY PROHIBITED AND MAY\nBE PUNISHABLE UNDER THE COMPUTER FRAUD AND ABUSE ACT OF 1986\nOR OTHER APPLICABLE LAWS. IF NOT AUTHORIZED TO ACCESS THIS\nSYSTEM, DISCONNECT NOW.\n'

set system option reboot-on-panic
set system option startup-beep
set system syslog global facility all level 'info'
set system syslog global facility daemon level 'info'
set system syslog global facility local7 level 'info'
set system time-zone 'America/Los_Angeles'

ACiD_GRiM · August 11, 2023, 5:45am

turning off the wild conntrack config improved the performance behind the LAN to 500Mbps down, but only 300Mbps Upload. only two cores are being used at 100% when running the speedtest
However I get 700Mbps up and 800down directly between routers on the cross connect

Something else still is holding routing back on both,

Apachez · August 11, 2023, 6:16am

Couldnt it simply be that your colocation provider are throttling the internet connection you are getting from them?

Also how is the latency between the devices (ping + traceroute) you test for performance and how do you perform these tests?

Also there seems to be some kind of issue when it comes to conntrack-sync (conntrackd) - you are not the first one I see that reports about 7x increase of throughput when disabling conntrack-sync.

ACiD_GRiM · August 11, 2023, 6:44am

I was able to get full speed to the internet testing on each router. But ultimately this ended up being a combination of the conntrack settings, the issue that started a year ago, and the cake qos policy.

Without conntrack and qos, I was able to get 700Mbps up and down from the LAN, and with QoS turned on at 1Gbps, it capped out at 400Mbps.

I’m very happy to find out what the issue was so I can work backwards to get both working within my expectations, rubber ducking here seemed to be the final trick.

pepe · August 11, 2023, 8:25am

You have completely disabled offload. Try to set:

set interfaces ethernet eth0 offload gro
set interfaces ethernet eth0 offload gso
set interfaces ethernet eth0 offload sg
set interfaces ethernet eth0 offload tso

set interfaces ethernet eth1 offload gro
set interfaces ethernet eth1 offload gso
set interfaces ethernet eth1 offload sg
set interfaces ethernet eth1 offload tso

ACiD_GRiM · August 11, 2023, 3:04pm

Thanks these were off during troubleshooting but already turned back on after identifying the root cause

Apachez · August 11, 2023, 3:20pm

So enabling the offloading restored expected performance even with conntrack-sync?

ACiD_GRiM · August 11, 2023, 4:15pm

No conntrack sync is still off, offloading was always on. only qos and offloading are back on and performance is as expected, minus the qos maximum i set to leep latency consistent.

Ill play around with conntrack later to see if it can be turned on.

echowings · August 11, 2023, 4:46pm

I had a PCEngines APU4D4, and the CPU performance was so poor. At last, I dump it and replace it with another x86 router to run VyOS. Maybe you can try N100 or J4125.

Schwertfried · August 13, 2023, 7:53pm

Maybe it’s throttling because of “high” temperature?
I’ve an apu4 that’s slow as hell after relocating it to a new location.
(Even SSH over LAN is sometimes extremely slow).

gw1:~$ sensors
fam15h_power-pci-00c4
Adapter: PCI adapter
power1:       10.96 W  (interval =   0.01 s, crit =   6.00 W)

acpitz-acpi-0
Adapter: ACPI interface
temp1:        +59.0°C  (crit = +115.0°C)

k10temp-pci-00c3
Adapter: PCI adapter
temp1:        +59.0°C  (high = +70.0°C)
                       (crit = +105.0°C, hyst = +104.0°C)

Oh, this is already throttling, I guess. High was at 70°C, at that time it was that “hot”, it was nearly like offline.