VyOS on Mellanox Spectrum (SNxxxx) switches w/ switchdev

You could take FRR out of the equation by doing static routing instead.

The packets themselves should as far as I know never touch FRR since the packets are handled by the kernel (or offloaded to the ASIC/FPGA) when forwarded (or routed) from one interface to the other..

Other than that comparing to Arista there is often something like 2x10Gbps between the ASIC/FPGA and the mgmt-CPU. And that is somewhat heavily trafficshaped by default.

I assume that in your case perhaps such trafficshaping is non-existent between the mgmt-CPU and the ASIC/FPGA which could be causing issues but again if the ASIC/FPGA is already programmed like through static routes I dont see why that packetforwarding (who would never touch the lanes between the ASIC/FPGA and the mgmt-CPU unless you do like ping/traceroute or talk BGP or such) would getting hickups every 10 seconds or so.

Just to verify the obvious - doublecheck the temperatures of the chip. Also try to see if there are any debug counters regarding watchdog resets or something like that?

Im thinking since you got a custom build perhaps some keepalive of some sort is missing so the FPGA/ASIC think it lost the communication with the mgmt-CPU and by that resets itself like every 10 seconds or so?

It’s doing VxLAN w/ EVPN, so it needs FRR to keep the forwarding plane configured right. The temps are fine (~30C), and I haven’t seen anything interesting in counters, but I haven’t doing a really exhaustive check yet. With EVPN, traffic through the switch works fine except for occasional brief outages. Pings to (or from) the switch’s management address are really consistently flaky, though.

Probably not too hard to fix, but it’ll take more time than I have this week.

Yes but a EVPN/VXLAN normally doesnt need changes every 10 seconds.

And again the forwarding itself shoudlnt be dependent on FRR once FRR have programmed the FDB. Hence try to use static routing just to rule things out that it isnt FRR that somehow malfunctions in updating the ASIC/FPGA over time?

Yes… but it wasn’t failing every 10 seconds. It was failing for around 10 seconds every few minutes. Which is perfectly compatible with BGP timing out due to packet loss and reconnecting, but also could have a ton of other causes.

Although now that I look, the BGP session has been up for 5 days, so that’s not it. And of course ping is flawless everywhere. Speedtest from the laptop via the SN2010 shows a bit north of 9 Gbps, perfectly as expected.

Which is probably for the best, as I really don’t have time to debug anything today anyway.

1 Like

And as soon as I hit send, all traffic through the switch stopped for 13 seconds. That included bridged VxLAN traffic as well as direct pings. Temp is 28C. Pings to the management interface kept working, but pings to loopback through the switch did. And the kernel logged a handful of icmp: detected local route for X.X.X.X during ICMP sending, src Y.Y.Y.Y messages, which is maybe a reverse-path check according to a post in Debian’s archive.

I’ll look at this later, I need to hit the road.

1 Like

This was weird. After leaving the ping running for a while, it looked like it might have just been a boring L2 problem. I saw ~13s ping losses with the same kernel ICMP errors as before, plus sometimes I saw kernel link flapping messages. So I swapped out the 10G-T transceiver between the laptop and the SN2010.

Didn’t help. Still ping timeouts, still link flapping if I waited long enough

Swapped from an external 10GBASE-T adapter to a SFP+ adapter on the laptop, with 10GBASE-SX optics. Precise same problem. Same ICMP messages, same ~13s intermittent packet loss. I didn’t see any link flapping messages, though. Maybe I just didn’t wait long enough, or maybe they were actually a real problem with the NIC, transceiver, or cable.

Changed ports on the SN2010, from port 1 to port 2. No change. Same ping loss, same ICMP errors.

Finally, pulled the second 40G link from my other “main” switch to the SN2010, so it has 2x40G uplinks. Brought it up on port #21 as an L3 interface, matching the existing link on port #22, but with different IPs. Came right up, OSPF worked, LLDP worked, etc.

Haven’t seen any ping failures since then. Which makes no sense at all; I was pinging from my laptop to my switch, which shouldn’t have involved the 40G links at all…

Oh. Yes it did. Because VxLAN. The SN2010 isn’t configured as part of the VxLAN gateway anycast group, so pings had to go laptop->(l2)->SN2010->vxlan encap->(l3)->Arista->vxlan decap->(l3)->SN2010 and then back over the same path, because there’s no local way to go from the laptop’s IP to the SN2010’s IP.

Looking closer, BGP isn’t flapping, but OSPF is flapping on that one link, probably thanks to BFD. So maybe it’s just a bad transceiver or dirty fiber. In any case, it’s not a VyOS or mlxsw issue.

1 Like

Actually, after looking a bit more, this was sort of a VyOS issue and sort of a config issue. The timings for bfd on VyOS and on the Arista peers didn’t quite match, so VyOS (or really FRR) was timing out and dropping OSPF. Increasing the local BFD receive interval from 300ms (default) to 1000ms seems to have mitigated the problem, although I’m still seeing BFD failures every 5-10 minutes. They’re not visible via ping at least. I think BFP was marking the link as down on the VyOS side, not the Arista side, although there’s not a lot of logging to be had here.

Also, it looks like I have a couple hosts running FRR that are having more or less the same problem and have for probably a year, but they weren’t quite as visible.

The lesson here is probably something like “networks are complicated, and don’t try to debug starting with the complicated case.”

When it comes to BFD you also have the singlehop vs multihop to configure (they use different UDP-ports).

But also the regular mode vs echo-mode where echo-mode will just be reflected back to the source.

As in I think you should be fine to use echo-mode at both ends with different timers but probably a bad idea if you try to do the same with regular mode.

Newer BFD also supports authentication so that can also be something to add to the mix if supported to make it less likely that some evil part injects broken BFD packets.

If I create the bridge manually, it gets a virtual MAC like fe:3f:de:xx:xx:xx instead of a real one and it works fine.

Setting one of the physical MAC adresses on the VyOS bridge solves the problem. Setting the VyOS bridge to the MAC used by the manually created bridge (after deleting it of course) gives the same bogus error. So there must be something else VyOS does, which prevents me from using non-physical MACs.

I am happy BGP EVPN with VxLAN is working in your scenario, because that is exactly what we are looking for. Now that I get the bridge up and running, I can do some more tests.

Okay, finally have some time to make progress on this. I have a local build with interface vxlan XXX parameters noudpcsum and interface ethernet swpX working.

I’m able to boot into a working config with swpXstyle interface names and bridging with vxlan and evpn working. I needed to manually add interface vxlan vxlan200 parameters noudpcsum and interface bridge br200 mac b8:59:9f:5d:89:adto the config, but they work fine. In an ideal world those would be auto-configured for mlxsw systems, because you get a somewhat inscrutable error otherwise, but that can come later.

The code for this is sitting in a PR right now; it’s not ready to merge quite yet, but hopefully it’s not that far out, either.

Just a quick demo showing that things work:

$ show int
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface    IP Address                     MAC                VRF        MTU  S/L    Description
-----------  -----------------------------  -----------------  -------  -----  -----  ---------------
br200        -                              b8:59:9f:5d:89:ad  default   1500  u/u
br203        -                              06:a5:45:f1:eb:56  default   1500  u/u
eth0         10.1.8.240/24                  98:03:9b:a0:65:da  MGMT      1500  u/u
lo           127.0.0.1/8                    00:00:00:00:00:00  default  65536  u/u
             10.0.0.17/32
             2600:a801:30:300::17/128
             ::1/128
swp1         -                              b8:59:9f:5d:89:ad  default   9000  u/D    to scott-laptop
swp2         -                              b8:59:9f:5d:89:ae  default   1500  u/u    port 2
swp3         -                              b8:59:9f:5d:89:a9  default   1500  A/D
swp4         10.0.17.8/31                   b8:59:9f:5d:89:aa  default   9000  u/D
             2600:a801:30:300:0:1:21:8/127
swp5         10.0.17.10/31                  b8:59:9f:5d:89:ab  default   9000  u/u    to scottdesk
             2600:a801:30:300:0:1:21:a/127
swp6         -                              b8:59:9f:5d:89:ac  default   1500  A/D
swp7         -                              b8:59:9f:5d:89:a5  default   1500  A/D
swp8         -                              b8:59:9f:5d:89:a6  default   1500  A/D
swp9         -                              b8:59:9f:5d:89:a7  default   1500  A/D
swp10        -                              b8:59:9f:5d:89:a8  default   1500  A/D
swp11        -                              b8:59:9f:5d:89:a1  default   1500  A/D
swp12        -                              b8:59:9f:5d:89:a2  default   1500  A/D
swp13        -                              b8:59:9f:5d:89:a3  default   1500  A/D
swp14        -                              b8:59:9f:5d:89:a4  default   1500  A/D
swp15        -                              b8:59:9f:5d:89:81  default   1500  A/D
swp16        -                              b8:59:9f:5d:89:82  default   1500  A/D
swp17        -                              b8:59:9f:5d:89:83  default   1500  A/D
swp18        -                              b8:59:9f:5d:89:84  default   1500  A/D
swp19        -                              b8:59:9f:5d:89:89  default   1500  u/u    mac-studio
swp20        -                              b8:59:9f:5d:89:85  default   1500  A/D
swp21        10.0.7.218/30                  b8:59:9f:5d:89:91  default   9000  u/u    to swa
             2600:a801:30:300:0:1:2:c5/127
swp22        10.0.6.218/30                  b8:59:9f:5d:89:8d  default   9000  u/u    to swb
             2600:a801:30:300:0:1:3:c5/127
vxlan200     -                              fe:e2:2a:af:15:62  default   1500  u/u
vxlan203     -                              7e:99:9a:13:f5:4d  default   1500  u/u

FWIW, eth0 is the management interface on the front panel.

Most of the usual management features Just Work:

$ show interfaces ethernet swp22 transceiver 
        Identifier                                : 0x0d (QSFP+)
        Extended identifier                       : 0x80
        Extended identifier description           : 2.5W max. Power consumption
        Extended identifier description           : No CDR in TX, No CDR in RX
        Extended identifier description           : High Power Class (> 3.5 W) not enabled
        Power set                                 : Off
        Power override                            : On
        Connector                                 : 0x07 (LC)
        Transceiver codes                         : 0x02 0x00 0x00 0x00 0x00 0x00 0x00 0x00
        Transceiver type                          : 40G Ethernet: 40G Base-LR4
        Encoding                                  : 0x05 (64B/66B)
        BR, Nominal                               : 10300Mbps
        Rate identifier                           : 0x00
        Length (SMF,km)                           : 1km
        Length (OM3 50um)                         : 150m
        Length (OM2 50um)                         : 0m
        Length (OM1 62.5um)                       : 0m
        Length (Copper or Active cable)           : 0m
        Transmitter technology                    : 0x40 (1310 nm DFB)
        Laser wavelength                          : 1310.000nm
        Laser wavelength tolerance                : 6.500nm
        Vendor name                               : Arista Networks
        Vendor OUI                                : 00:1c:73
        Vendor PN                                 : QSFP-40G-UNIV
        Vendor rev                                : 20
        Vendor SN                                 : XLA164600338
        Date code                                 : 16110900
        Revision Compliance                       : SFF-8636 Rev 1.5
        Rx loss of signal                         : None
        Tx loss of signal                         : None
        Module temperature                        : 47.50 degrees C / 117.49 degrees F
        Module voltage                            : 3.3302 V
        Alarm/warning flags implemented           : Yes
        Laser tx bias current (Channel 1)         : 39.180 mA
        Laser tx bias current (Channel 2)         : 46.822 mA
        Laser tx bias current (Channel 3)         : 36.728 mA
        Laser tx bias current (Channel 4)         : 37.882 mA
        Transmit avg optical power (Channel 1)    : 1.5010 mW / 1.76 dBm
        Transmit avg optical power (Channel 2)    : 1.5234 mW / 1.83 dBm
        Transmit avg optical power (Channel 3)    : 1.5476 mW / 1.90 dBm
        Transmit avg optical power (Channel 4)    : 1.5255 mW / 1.83 dBm
        Rcvr signal avg optical power(Channel 1)  : 0.7763 mW / -1.10 dBm
        Rcvr signal avg optical power(Channel 2)  : 0.8118 mW / -0.91 dBm
        Rcvr signal avg optical power(Channel 3)  : 0.6804 mW / -1.67 dBm
        Rcvr signal avg optical power(Channel 4)  : 0.7978 mW / -0.98 dBm
...

My config on this is pretty boring:

interfaces {
    bridge br200 {
        mac b8:59:9f:5d:89:ad
        member {
            interface swp1 {
            }
            interface swp2 {
            }
            interface swp19 {
            }
            interface vxlan200 {
            }
        }
        mtu 1500
    }
    bridge br203 {
        member {
            interface vxlan203 {
            }
        }
        mtu 1500
    }
    ethernet eth0 {
        address dhcp
        hw-id 98:03:9b:a0:65:da
        offload {
            gro
            gso
            sg
            tso
        }
        vrf MGMT
    }
    ethernet swp1 {
        disable-flow-control
        mtu 9000
    }
    ethernet swp2 {
        disable-flow-control
        mtu 1500
    }
    ethernet swp4 {
        address 10.0.17.8/31
        address 2600:a801:30:300:0:1:21:8/127
        disable-flow-control
        mtu 9000
    }
    ethernet swp5 {
        address 10.0.17.10/31
        address 2600:a801:30:300:0:1:21:a/127
        disable-flow-control
        mtu 9000
    }
    ethernet swp19 {
        disable-flow-control
        mtu 1500
    }
    ethernet swp21 {
        address 10.0.7.218/30
        address 2600:a801:30:300:0:1:2:c5/127
        description "to swa"
        disable-flow-control
        mtu 9000
    }
    ethernet swp22 {
        address 10.0.6.218/30
        address 2600:a801:30:300:0:1:3:c5/127
        description "to swb"
        disable-flow-control
        mtu 9000
    }
    loopback lo {
        address 10.0.0.17/32
        address 2600:a801:30:300::17/128
    }
    vxlan vxlan200 {
        mtu 1500
        parameters {
            nolearning
            noudpcsum
        }
        port 4789
        source-address 10.0.0.17
        vni 200
    }
    vxlan vxlan203 {
        mtu 1500
        parameters {
            nolearning
            noudpcsum
        }
        port 4789
        source-address 10.0.0.17
        vni 203
    }
}
protocols {
    bgp {
        address-family {
            ...
            l2vpn-evpn {
                advertise {
                    ipv4 {
                        unicast {
                        }
                    }
                    ipv6 {
                        unicast {
                        }
                    }
                }
                advertise-all-vni
                advertise-svi-ip
                vni 200 {
                    rd 10.0.0.1:200
                    route-target {
                        both 200:200
                    }
                }
                vni 203 {
                    rd 10.0.0.1:203
                    route-target {
                        both 203:203
                    }
                }
            }
        }
        neighbor 10.0.0.2 {
            description swa
            peer-group FABRIC
            remote-as internal
        }
        neighbor 10.0.0.3 {
            description swb
            peer-group FABRIC
            remote-as internal
        }
        parameters {
            bestpath {
                as-path {
                    multipath-relax
                }
            }
            router-id 10.0.0.17
        }
        peer-group FABRIC {
            address-family {
                ipv4-unicast {
                    route-map {
                        import set-src
                    }
                }
                ipv6-unicast {
                }
                l2vpn-evpn {
                }
            }
            ebgp-multihop 5
            update-source 10.0.0.17
        }
        system-as 65000
    }
    ospf {
        ...
    }
    ospfv3 {
        ...
    }
}
service {
    ssh {
        vrf MGMT
    }
}
system {
    console {
        device ttyS0 {
            kernel
            speed 115200
        }
    }
}
vrf {
    name MGMT {
        description MGMT
        protocols {
            static {
                route 0.0.0.0/0 {
                    next-hop 10.1.8.254 {
                    }
                }
            }
        }
        table 101
    }
}

1 Like

Created T8878 for this.

1 Like

Got any iperf/imix metrics to verify that the offloading is actually working?

How about “advanced” features such as EVPN/VXLAN etc?

Also any thoughts of adding some QoS between dataplane and mgmt-plane to protect the mgmt-cpu which often sits on like 2x10G or so towards the FPGA/ASIC and by that can easily be overrun/overloaded which affects other services?

And does it work out of the box with local services such as DNS, DHCP but also if you run a custom container lets say HAproxy?

Of course that traffic would get offloaded (it will use those 2x10G or whatever is available between the FPGA/ASIC and the mgmt-cpu) but still. I dont recall how this part works when using VPP (as in local services when using VPP/DPDK)?

Since this got already pulled and closed, you got a typo in `interface-definitions/interfaces_vxlan.xml.in`:

it reads “mellanox switch support”, should be “Mellanox switch support”.

I haven’t ran any solid benchmarks, but I did run a generic speedtest from my laptop through the SN2010 and it hit ~9.2 Gbps, which is about the limit of my ISP, and certainly more than I’d expect from an Intel C2558. In any case, the CPU load on the switch didn’t change noticeably while running it. That does include EVPN and VXLAN, so that part’s working fine.

I haven’t really tried running an local containers, but they should just work up to the limit of the CPU, RAM, and x8 PCIe 3.0 link between the ASIC and CPU. This is all just a semi-normal part of the kernel, and traffic that the ASIC can’t handle on its own gets shunted to the CPU. Without that, BGP and OSPF wouldn’t work, so it wouldn’t be much of a switch.

The mlxsw wiki has a bunch of extra documentation on what this can do and how to accomplish it. They have docs on QoS, ACLs, multicast routing, and queue configuration. Basically, it all just uses the usual Linux tools (ip, tc, devlink, etc), but there are limitations on which options the hardware supports, and it’ll report an error if you try to configure something it can’t do, so you shouldn’t end up with a “that was a nice option, shame it dropped your performance by 99.9%” situation. So there’s a good chance that we’ll need to add documentation and maybe a few extra flags, but in general I’d expect VyOS’s current tools to mostly just work. Probably except for firewalling and NAT – VyOS uses nftables, and mlxsw only has limited hardware support for filters via tc.

Pim walked through a SN2700 with generic Debian a while ago; it’s slightly different hardware with slightly different limitations (you can’t split QSFP ports 4 ways on it without losing access to a second QSFP!), but it should be pretty similar in most ways.

The same article points out how you can see offloaded vs total packet counters. You have to enable collection via ip stats set dev swpXX l3_stats on, and then ip stats reports the total traffic broken down in various subgroups:

$ ip stats show dev swp21
11: swp21: group afstats subgroup mpls
    RX: bytes packets errors dropped noroute
            0       0      0       0       0 
    TX: bytes packets errors dropped
            0       0      0       0 
11: swp21: group link
    RX:   bytes packets errors dropped  missed   mcast           
     6703643222 9873147      0       1       0 1033786 
    TX:   bytes packets errors dropped carrier collsns           
    10053668536 8735942      0       0       0       0 
11: swp21: group offload subgroup l3_stats on used on
    RX:  bytes packets errors dropped   mcast
    2273685327 1469229      0       0      35 
    TX:  bytes packets errors dropped
    3794133812 2515254      0       0 
11: swp21: group offload subgroup hw_stats_info
    l3_stats on used on
11: swp21: group offload subgroup cpu_hit
    RX:  bytes packets errors dropped  missed   mcast           
     265401814 2420326      0       0       0       0 
    TX:  bytes packets errors dropped carrier collsns           
      48769798  529321      0       0       0       0 
11: swp21: group xstats_slave subgroup bond suite 802.3ad
11: swp21: group xstats subgroup bond suite 802.3ad
11: swp21: group xstats_slave subgroup bridge suite mcast
11: swp21: group xstats_slave subgroup bridge suite stp
11: swp21: group xstats subgroup bridge suite mcast
11: swp21: group xstats subgroup bridge suite stp

I didn’t turn l3_stats on immediately, so these numbers may not add up. VyOS should probably enable them by default, and show interface should be able to report them, but that’s code for another day, and in any case not critical for getting this working in the first place.

Thanks for spotting the typo; I’ll patch it into the upstream PR.

Also, PTP seems to work just fine on the switch. I manually installed linuxptp and ran ptp4l against all interfaces at once via something like ptp4l $(for a in $(ip link | grep swp | cut -d: -f2 | sort -n +0.4 ); do echo -n "-i $a "; done)and it starts up, discovers that 3 of its interfaces have PTP masters on them, picks the best interface, and then starts syncing PTP out the remaining interfaces supposedly with ~10 ns of jitter or so. I can get stats out of it via the pmc tool, so mostly this will just need a bit of Python for config/ops mode in VyOS and the willingness to add linuxptp to the default set of installed tools.

Also, it looks like I still have some regex issues; running devlink port split pci/0000:01:00.0/5 count 4 turns swp20 into swp20s0 through swp20s3, but show interfaces doesn’t show them. Presumably a Python issue, as /usr/libexec/vyos/completion/list_interfaces(from OCaml) does show them correctly.

There are a few places that assume that all interface names are a prefix followed by numbers, or sometimes numbers and a single letter, and tracking all of those down has been fun.

This means that the longer-form sw1p2s3, for port 2 split 3 on the second switch connected to the system probably won’t work either. I should really write tests for some of this, but I don’t see anything similar right now and it’s been a long time since I’ve written tests in Python.

Okay, it looks like everything is passing on the PR, the AI code review bot has finally stopped griping, and I have one of the two needed approval stamps. Hopefully this PR can merge soon and I can get started on devlink split next.