Do make it short, I would get a 8 switch with an SFP slot (or SFP+, since this is different check your SFP Modul).
I would then attach two router the switch with wan port. The two lan ports to your internal switch.
This will split the single fiber to two ports.
For the vyos vrrp setup this really depends on what your ISP delivers. Do you have pppoe dialup or do you get the public IP via DHCP?
Vrrp allows two nodes to communicate heartbeat with private range IPs and the failover a public IP on the interface. But so see how and what you can do we need to know how you geht your IP from the isp.
Ah and also you can attach the SFP to your internal switch if it is managed and create a dedicated vlan for the SFP port and have the vlan at two virtualization nodes uplink Ports and then set the virtual nix for wan untagged in the vlan.
I have 10gigabit delivery from my ISP so switching to sfp would kind of not be so nice =)
My ISP delivers a public ipv4 over DHCP locked to my networks card mac address. But I belive they deliver multiple ipv6(investigating this currently). So my option is to see if I can get rid on the mac locking then for ipv6 or ipv4.
Ok so VRRP move the public ip aswell then, was hard to figure that out from the documentation. It just brings up that virtual address is moved which I assumed is an internal virtual address.
I have a Mikrotik 8 port sfp+ switch. Didn’t think about that I could actually use that on a separate isolated VLAN. Would loose three ports though. Also plan was to have 2 port sfp+ nics on each node dedicated for vyos using pci passthrough.
With only a single uplink using 2x VyOS boxes wont bring you much in redundancy because you would still need to rely on the single point of failure as a switch between the incoming fiber and your 2x VyOS boxes doing VRRP.
So in that case I would rather use that 2nd VyOS unit as a cold standby ready to have the latest VyOS version and config applied (from backup) when needed and then move the networking cables to the spare unit when ■■■■ hits the fan.
That is instead of:
ISP ↔ fiber ↔ Switch ↔ cable(s) ↔ 2x VyOS
I would do:
ISP ↔ fiber ↔ VyoS
and that 2nd VyOS is on a shelf close by.
Now with 2 fibers (or two ISPs) unless doing BGP to the ISP I would probably terminate each fiber in its own switch (if you do BGP you wont need that switchlayer in between).
Then interconnect the switches and then connect one VyOS to one switch and the other VyOS to the other switch.
Like so:
ISP1 ↔ Switch1 ↔ VyOS1
ISP2 ↔ Switch2 ↔ VyOS2
and then a cable (or two) between Switch1 and Switch2.
A more advanced setup of above would be if these two switches can do MLAG (also known as IRF, VPC, VSS, Virtual Chassis etc depending on vendor). That is two switches behaves as a single logic unit.
Each fiber would still terminate in each switch but between the switches and each VyOS you would have two (or more) cables where on the VyOS you would set them up as a linkaggregation (preferly using LACP).
This way even if one VyOS dies the remaining one can still utilize both switches and by that both uplinks (and with MLAG you will avoid the headache of spanningtree and similar).
That is that setup would be able to withstand an error at VyOS1 at the same time as there is an error at Switch2.
You are totaly right. Maybe he wants two vyos do test update and then failover or so.
If he wants to test vrrp in the homelab or just want to have 2 routers on two virtual server he can do this. VRRP users it’s own mac that can fail over from one host to the other and with the mac also the ip. So the problem with mac lock to one ipv4 wouldn’t be the problem.
Okay if you have 10gbit your isp will most likly have given a sfp+ modul. So your device needs a sfp+ slot.
I need to take a look on how to setup vrrp with a dhcp adress. But it also interessts me. I will try to build a lab setup this evening and post the config if I figure it out.
The mac thingy is part of VRRP standard. Enabling VRRPv3 is probably prefered in all situatoins nowadays.
In your case with lets say 3x Proxmox hosts you should be able to configure one VyOS at each host and then configure manual priority between them regarding which should be elected master. That is you must also “lock” each VyOS VM guest to each VM host for this to work.
One drawback by utilizing VRRP is that stateful services such as DHCP will be ehm… fun… to handle. One workaround here is to configure your DHCP to lease addresses based on option82 information to make it semi-static.
Another issue is if you are using firewalling because switching VyOS unless you do conntrack-sync will have all current sessions invalidated and must be reinitiated by the endhosts.
And of course if you got 3x VyOS you must then also make sure to perform the same changes in all since they dont have any config-sync.
A workaround for above is of course to let Proxmox move your VyOS VM guest around.
Slightly longer downtime when a Proxmox host goes poff (since the VyOS VM guest must be booted on one of the other Proxmox hosts in your PVE Cluster) but less headache of trying to sync three different VyOS installations.
I used to do this at home. I just had two Proxmox boxes with the WAN of my ISP connected to a switch (so that both Vyos Boxes could have access to a WAN Port)
In doing this, your switch becomes your single point of failure, but you are then able to easily reboot one Proxmox mode and the Vyos Instance in the other node takes over.
For me because my ISP is PPPoE, I had to have scripts run so that when the Backup VRRP instance became the master, it unshut its PPPoE Interface. And then when it become backup again, it shut it. If I didn’t do this, both Vyos boxes would be trying to establish PPPoE, one would win but the other would keep trying over and over again.
I would also do conntrack-sync which worked great, people in the house would lose connectivity for ~3 seconds on failover/failback usually.
Anyway it’s easy to do, but just keep in mind you’re moving your SPoF to the switch now.
That way the lease is based on which switch and interface you are connected to and not which mac-address or duid your dhcp-client uses in its request.
In short whatever is connected to Switch1 int4 will always get the same DHCP lease.
The nice thing except for the semi-static design is that there is no lease database to care about so redundancy with multiple DHCP-servers is very easy (as long as they all got the same Option82 configuration).
However when i switch network card I will have no internet access. Need to visit a webpage dhcp.provider.yyy, login and choose the mac adress of the card I want to have active before internet access is enabled. Then it takes a few mins until Internet is enabled for that mac.
But I guess they could be using option82 behind the scenes anyway. Which seems logical since I kept ipv4 adress when changing nic.
Why do you need multiple VyOS instances to begin with? You already have a cluster, so a single VyOS VM should fail over to the other node, keeping its MAC addresses
Why do you pass the NIC through. Just make a vmbr1 oder what ever number is free. Assign the port connected to this vmbr1 and the virtual nix from the VM…nothing more.
If you do this on both Proxmox you can migrate teh vyos VM from one host to another bevore power down / maintenance.
In this case without shared storage (nfs from Nas will do plenty for a vyos VM) you need to fix the host. But if you want to address this then Proxmox ha is what you can do to remediate this.
