Hi all - was wondering if anyone has experience of using Vyos on a PC Engines APU2 board.
Setup is working fine, just was upgraded to a 1gb/s line via FTTH. Connection is made using the ISPs hardware and then using IP passthrough handed off to the Vyos devices via ethernet.
Speed tests showing a max of 600mb/s down and around 300mb/s down.
Does anyone know how I can check if this is a device limitation?
I don’t have any experience with an APU2, but I’ll ask this:
Does your ISP use PPPoE? People never, ever mention this in their posts, yet it’s a huge factor in performance. Having to encapsualate and de-encapsulate every since packet egressing/ingressing into your router will have a big performance impact.
You could check if this is a device limitation by looking at top/htop while running your tests, is the CPU hitting 100%?
There’s some performance tuning notes here.
There’s also notes here about turning off the meltdown/spectre workarounds for extra performance, but reading this post seems to indicate that maybe 600mb/s is the limit of the device (assuming you ARE using PPPoE)
Hope that helps a little?
PS: Thanks for supporting the project as a Patron. (I am not a project member, just a happy user)
Appreciate the quick response and informative links. My search skills clearly need some improvement.
PPPoE, I believe Swisscom do, but that is all done on their device acting as the IP bridge. My line is capable of 10gb/s and I have no doubt the hardware they supply can handle a 1gb/s link with no trouble.
I quickly configured an Edgerouter 6P and enable hardware offloading and get as close to the 1gb/s line speed as possible. 910mb/s. I suspect if I want to continue running Vyos I need some new hardware…
On that note, any recommendations for devices capable of running line speed on a 1gb/s connection that are suitable for the home. I don’t have the space of rack based kit unfortunately.
A pleasure being a patron, totally loving Vyos.
Having just double-checked, my connection is using IPoE. I don’t think it makes any difference as the APU isn’t doing any encapsulation/decapsulation. It sees a standard IP interface on its WAN port.
Hello @phillipmcmahon, how you run this bandwidth test?
Could you please take a screenshot sudo top command and press 1 when you run this bandwidth test?
Hey there - I used a bunch of the speed test web sites out there. I was wanting to test my WAN bandwidth.
The ER-6P seems to be able to handle the full line speed. Let me swap devices and get the information you requested. Appreciate the help.
See attached.
I use eth0 for my WAN, eth2 for LAN, along with eth2.100 and eth2.110 for some VPN features. eth1 is disabled.
Speed is around the 550mb/s mark. Occasionally I will see a spike to 650mb/s.
Happy to test whatever tuning you think would help. So far I have not applied any tuning. SMP affinity it set to auto on all interfaces.
I see strange irq loads on CPU3, can you provide an output the following command
sudo cat /proc/interrupts
and also
show interfaces ethernet eth0 physical
show interfaces ethernet eth2 physical
This is rolling or 1.2.x stable?
late night support, I like it!
Rolling.
interrupts.txt (3.4 KB) eth2.txt (1021 Bytes) eth0.txt (916 Bytes)
Hello @phillipmcmahon, it looks like queues didn’t apply properly for interfaces higher then eth1
Please, edit sudo nano /etc/modprobe.d/igb-options.conf
And change
options igb RSS=0,0,0,0
Then execute
sudo /usr/sbin/update-initramfs -u
After reboot HW interrupts for eth2 should be applied
I created a bug report on the development portal ⚓ T2984 (igb, ixgbe) HW queues applied only for the first 2 interfaces
Also if we tell about router tuning, recently was be added possibility to define ring buffers via VyOS CLI
set interfaces ethernet eth0 ring-buffer rx 4096
set interfaces ethernet eth0 ring-buffer tx 4096
set interfaces ethernet eth2 ring-buffer rx 4096
set interfaces ethernet eth2 ring-buffer tx 4096
Applied the queue changes and also the ring buffer.
I assume each time I update the rolling release I have to apply the queue changes?
Gets me up to 750 down / 650 up. I suspect I am hitting the limits of this SBC.
However, that is good enough for me while I investigate another higher capacity SFF to run Vyos, rather than going back to EdgeOS.
Another major influencing factor is the support on this platform. Active and helpful by users AND developers. Something that can no longer be said about the Edgemax forums.
Any recommendations for home use hardware (not noisy rack mount stuff) that can drive a 1gb/s line at full speed in both directions would be greatly appreciated.
config_all.txt (38.3 KB)
@Dmitry - attached my config. Any chance you can give it a once over and suggest improvements?
I am seeing on a whole bunch of PC Engine posts that the APU2 is supporting 1gb/s lines at full speed, some other posts report not.
Those reporting success seems to be running Linux (mostly debian) and with their own setup. Suggests the hardware is capable, but then it becomes a combination of the hardware and OS/routing/firewall platform.
Are there any more tuning items I should be looking at?
Hello @phillipmcmahon, I think all ok with your configuration.
Do you have a chance again to provide top output and press 1 with previously applied tuning?
Did you test bandwidth through WG tunnels?
I took these on the morning I applied the settings. Have not had chance yet to test the WG tunnels. I can do that if you think it will add value. Let me know.
Now interrupts looks pretty and your device has about +25% throughput resources.
Can you try to disable mitigation patches?
Sorry, only just saw this.
Let me do that this morning and run a test via the native ISP pipe and also using my wireguard end points.
Will post back the results ASAP.
I saw in previous advice you recommended the following; Should I do the same?
echo "f" > /sys/class/net/eth0/queues/rx-0/rps_cpus
echo "f" > /sys/class/net/eth1/queues/rx-0/rps_cpus
echo "f" > /sys/class/net/eth2/queues/rx-0/rps_cpusHi @phillipmcmahon, this is software interrupts and necessary if you use PPPoE or QinQ. In your case, this is not suitable