It’s not that easy, unfortunately. The 4 stock fans are connected to a pair of 8-pin connectors, so I’d either need to find a replacement connector or cut up and splice the current one, after mapping pins. Fortunately, the wire coloring is pretty predictable – there are a pair of red and a pair of black wires, plus 2 additional wires per fan, so adapting it to 4-pin PC fan headers is probably pretty easy. 3D printing someone else’s pre-made replacement cover with 120mm fans is less work up front, though.
The existing fans aren’t terrible, but IIRC they stalled before they were slow enough to be <40 dB. With ~no load on the switch it doesn’t take a ton of cooling, and I’m not planning on putting high-powered optics into it while it’s at my desk.
Also, I don’t think I’ve ever seen an example of flowtables offloading working with ConnectX cards. IIRC the closest that I’ve seen requires setting up a bunch of VFs in ways that probably aren’t compatible with VyOS. I’d love to see a simple example, though.
Yeah, I haven’t seen an example of it working either. There is a thread on the STH forum about trying to get that working on those NICs, but I don’t think they were ever successful. I see that you responded in that thread too a while back.
I have the MediaTek implementation of nftables hardware flowtable offloading working in VyOS, so I know that the VyOS side of it is functional at least.
FWIW, I now have it working with a 3D-printed cover with 3 120mm fans. I’m using 1700 RPM Noctua NF-P12 redux fans with a USB-to-12V adapter. It looks like using all 3 fans draws slightly more current than the switch’s front USB port wants to provide, but falling back to only powering 2 fans seems fine. I’ll probably adjust the fan speed down slightly at some point because I can sort of hear them, but they’re still quieter than the stock fan as stall speed. The CPU is reading 24C and the switch itself is 39C. From the manual, it looks like the switch chip has roughly the same thermal limits as a ConnectX4, with a warning temp of 105C. I figure 65 degrees of headroom is plenty for my uses.
I’m working my way through getting a full VyOS build environment right now. The docs for getting all of the ocaml and python prereqs are kind of out of date :-(. However, I think I have the first change building and passing tests. This just updates the regexes in 4 places. I still need to build it into an ISO and run it on real hardware, and adding a couple minor tests for sw* names in vyos-1x would probably be helpful at this point.
Also, back to thinking about devlink port split again. This is the stock kernel mechanism for splitting multi-lane interfaces (QSFP, etc) into sub-interfaces, so it’s needed if you want to split a QSFP28 port into 4xSFP28s, which is a pretty common use. This is the only mechanism supported by Mellanox switches, and it also seems to be supported (at least somewhat) by Intel E810 NICs and probably Mellanox ConnectX-8 NICs as well.
Logically, this config should live in interface ethernet ethX (or swp22or whatever). Except that won’t really work well for two reasons:
devlink split takes a bus address (pci/0000:01:00.0/13), not an Ethernet name, so we’d need to do a bunch of mapping to make interface ethernet ethX / port split count 2 or similar work.
Splitting devlink ports (using something like devlink port split pci/0000:01:00.0/13 count 2) tears down the old interface names and creates new ones. So adding interface ethernet swp22 / port split count 2 would delete swp22 and replace it with swp22s0 and swp22s1. Which would mean that we had a config entry that is required, but lives on an interface that no longer exists. Since these names mostly come from the kernel (Mellanox emits p22s1 as the port “number” in their switchdev driver), working around this just feels like a losing battle. Building a generic sub-interface mechanism into VyOS’s config is out of scope here.
The logical way around this (and the way that Juniper does it in JunOS, at least on some hardware) is to put the port split config outside of interface. JunOS puts it in chassis fps slot. We don’t currently have anything obvious at the top level, but system portwould probably be a passable equivalent.
So something like this would probably be easiest:
system {
port {
device pci/0000:01:00.0/13 {
split count 2
}
}
}
According to the manpage, devlink port split only takes a single parameter (count), so this part of system port won’t get much more complex, but there are other potentially-useful devlink port subcommands that would make sense to add eventually, so we want to make sure that there’s room in the design for them. For example, you need to use it to create VFs, PFs, and SFs, and it’s needed to get hardware flowtable offloading working with Mellanox NICs. That would be its own project, though. I think everything could sanely live under system port device pci/XXX ….
Using pci/XXXX as the port name isn’t a great UX, but I’m not convinced that mapping between that and ethX or swpY is worth doing. Running devlink port will show mappings, so presumably we could do this if we really wanted to:
$ devlink port
pci/0000:01:00.0/0: type notset flavour cpu port 0 splittable false
pci/0000:01:00.0/1: type eth netdev swp15 flavour physical port 15 splittable false lanes 1
pci/0000:01:00.0/2: type eth netdev swp16 flavour physical port 16 splittable false lanes 1
pci/0000:01:00.0/3: type eth netdev swp17 flavour physical port 17 splittable false lanes 1
pci/0000:01:00.0/4: type eth netdev swp18 flavour physical port 18 splittable false lanes 1
pci/0000:01:00.0/5: type eth netdev swp20 flavour physical port 20 splittable true lanes 4
...
We’d probably want to parse this (or the lower-level equivalent) just to pull out splittable true vs splittable false and provide reasonable autocomplete. Being able to say system port device swp20 split count 4 would be a lot nicer than using pci/0000:01:00.0/5 as the name, so I’ll see how doable that is.
I think ultimately it’d make more sense to create a new interface type called switchport or something like that. It would simplify all of the regex, and allow development to be intentionally slower. It also helps deconflict ethernet settings that would only be applicable for Linux interface and for switchdev interfaces.
The devlink command supports a JSON output, so this mapping would actually be very easy. The keys are the physical address, but that can easily be re-indexed so that the key is the interface name. Here’s an example from one of my hosts with a Connectx3:
I’m assuming in your output you’d also have a key for lanes, which is data you’d need anyways to ensure count doesn’t exceed lanes. The re-indexed dict could look like this:
You can run this locally and see what you think of the output:
import json
from vyos.utils.process import cmd
def reindex_ports(data):
return {
attrs["netdev"]: {
**{k: v for k, v in attrs.items() if k != "netdev"},
"device": device
}
for device, attrs in data.get("port", {}).items()
if "netdev" in attrs
}
dev_dict = reindex_ports(json.loads(cmd("devlink -j port")))
print(json.dumps(dev_dict, indent=4))
I think this is fine. There can simply be a check in the verify() stage that checks:
If split is in the dict, and the length of the dict is greater than 1 (check if only port is present).
Raise an error.
If the port is not splittable.
Raise an error.
If count exceeds lanes.
Raise an error.
Etc…
The execution of conf-mode scripts is a little inflexible in some ways, so keeping config within a specific section is important. For instance, if you have a interfaces switchport section (or use the interfaces ethernet section), and have the system port section, that’d need to execute both: interaces_switchport.py and system_port_split.py. This can easily get out of sync with each other and cause errors.
This is additional reasons for the dedicated interfaces switchport section, since these settings wouldn’t have any relevance to general ethernet ports. That’d prevent the majority of users who wouldn’t use a Specturm switch from trying to use non-relevant config options.
I’m not strictly opposed to creating a switchport interface type, but we’d need to make sure that devlink port split is in both ethernet and switchport. It’s needed for at least Mellanox ConnectX-8 NICs and is usable with Intel’s e8xx 100G+ NICs. IIRC, Intel’s own tool is slightly more flexible at splitting ports (you can do weird things like 100→25+25+50 which devlink port split doesn’t support), but I don’t think VyOS supports it either right now.
Example from devlink port on the e810 in my desktop (which is a 4x SFP28 model, but still shows a splittable interface):
pci/0000:61:00.0/0: type eth netdev enp97s0f0np0 flavour physical port 0 splittable true lanes 4
pci/0000:61:00.1/0: type eth netdev enp97s0f1np1 flavour physical port 1 splittable false
pci/0000:61:00.2/0: type eth netdev enp97s0f2np2 flavour physical port 2 splittable false
pci/0000:61:00.3/0: type eth netdev enp97s0f3np3 flavour physical port 3 splittable false
The same system also has a ConnectX-6 in it, which also appears in devlink port:
auxiliary/mlx5_core.eth.0/65535: type eth netdev enp1s0f0np0 flavour physical port 0 splittable false
auxiliary/mlx5_core.eth.1/131071: type eth netdev enp1s0f1np1 flavour physical port 1 splittable false
Interesting that they’re not inpci/. but probably not really important in any way.
Fundamentally, I don’t really think there’s a real difference between switchport and ethernet; they’re slightly different under the hood in the kernel, but the actual interface to user mode shouldn’t be particularly different in any way that I know of. Bridging, bonding, vlans, and IP addresses work just like normal. They use Ethernet MAC addresses. STP is configured the same. Depending on the implementation, tcpdump will probably work differently, but even that is probably a function of the drivers involved; I don’t think there’s any reason why a switchdev driver couldn’t hook into something (promiscuous mode?) and have the switch chip mirror extra traffic to the CPU. I’m sure there are little differences with specific drivers, but there are differences in driver support all over the place. Interfaces that use the mlx5 Ethernet driver are probably closer to interfaces using the mlxsw switchdev driver than they are to interfaces that use random janky Realtek drivers.
If we did split the two, then we’d presumable need to update the bridge, bonding, and vxlan code to know that both ethernet and switchport interfaces are basically interchangeable.
Also, there’s probably more offloading support in modern ConnectX NICs than in the older Mellanox Spectrum switch chip that my switch uses. They both support VxLAN and some other tunnel offloads. I think they both support VLAN offloading, but it’s unclear that that’s enough work to matter. The NIC supports flowtable offloading, where it can bounce traffic back out the same physical port in some situations, while the switch supports L3 routing offloading. Weirdly, the NIC’s offloading mostly requires some software support, while the switch’s offloading is all handled by the kernel directly.
I wouldn’t be stunned to discover that the switch silicon is technically able to handle flowtable offloads, and that the NIC silicon can do L3 routing, but I don’t think the drivers support either. Actually finding usable examples of what either is capable of and how to configure them is a problem. A lot of the docs and old forum discussions went missing when nVidia reorganized their website a few years ago.
Actually, thinking about this a bit – the Intel E810 in my desktop is sort of an interesting case because it comes pre-split. So this is an interesting case where port split 1 and delete port split need to produce different results, and I’m not quite sure how that’s supposed to happen. Running devlink port unsplit whenever we don’t have port split Nwill probably break the NIC until reboot.
Honestly, that’s probably more of an Intel driver problem, though. I’m curious if it’d even let me unsplit it into a 4-lane 100G interface, but not curious enough to try while logged in remotely.
It seems we have a lot in common! I previously create the vyos-bpi-r4, and now I’ve bought an SN2410. I also want to use VyOS on this switch because Onyx seems to be EOL.
I was just updating to a recent nightly earlier today and trying to get builds working again. I found a couple issues with VxLAN and will need to patch bridge configs slightly, and then hopefully have time to actually finish some of this.
You said you are patching bridge configs. I am sitting at the same thing trying to get VyOS running on the Spectrum chip. By the love of things I can not get bridging to work. If I create the bridge myself I can enslave the interfaces, if VyOS creates the bridge I cannot enslave an interface. I am unable to find the difference between the bridges. Any hints, or any preliminary patches by any chance? I already patched (more hacked) the zerochecksum for the VxLAN interfaces to be offloadable.
Yeah, the checksum change is critical, but presumably pretty trivial. We just need to add noudpcsum to the list of allowed parameters, alongside nolearning. I hand-set it for now and was able to get things working, but this box is getting to be kind of a hacked-up mess; I need to update it to a current build and start trying to push PRs upstream. The next couple weeks are going to be busy for me, though, so it may be a while.
I would propose a zero-checksum option, that also sets noudp6zerocsumtx and noudp6zerocsumrx in addition to noudpcsum. We have an IPv4-Fabric only, but others might want to use IPv6 for the underlay. I have a patch for this, but I am not a programmer. I’ll see if it gets accepted.
Did you manage to somehow get bridging to work? Or do you know, what VyOS does differently when creating the bridges in comparison to the Mellanox Wiki?
I am getting the following error when trying to enslave an ethernet interface to the VyOS created bridge:
# sudo ip link set dev enp2 master br100
Error: mlxsw_spectrum: Exceeded number of supported router interface MAC profiles.
IIRC that’s a mostly-bogus error. Setting the bridge MAC explicitly gets around it.
# ip link set dev br200 address xx:xx:xx:xx:xx:xx
# ip link set enp1 master br200
#
It’s something that we’d want to code around for optimal user experience, but IIRC we can already set bridge MACs anyway. I just copied the MAC from enp1 onto br200, but I haven’t thought deeply about the best fix.
I was running my work laptop through my SN2010 for a few days this week, but I saw intermittent packet loss (~10 seconds every few minutes) and reverted back when Zoom started freezing. I’m having weird issues with the single 40G uplink that’s in place right now, so it may have been FRR getting testy and dropping something critical.
I haven’t figured out what’s up with the uplink yet; that’s why I started trying to upgrade everything. When something new is flaky, I’ve found that it’s usually better to upgrade and start over with newer code before debugging and finding “oh, that was fixed 3 months ago.” Anyway, the single 40G link looks ok, transceivers show good RX/TX levels on both ends, but I’m getting ~70% packet loss between the SN2010 and the Arista on the other end. Sometimes it goes away, sometimes it’s slightly worse. It could just be a bad transceiver. But it looks like bridged VxLAN traffic through the switch doesn’t get any loss at all most of the time, so I suspect that there’s some sort of ASIC->CPU bottleneck or rate limiting kicking in.
There’s not really a lot of traffic here, so I’m not sure what’s up yet. Probably something dumb.
