Unable to route traffic over VLAN with VRRP for unknown reason

Hi all again, I am learning and wanting to use VRRP at home and seeking some help with VRRP please.

I am on rolling release “VyOS 1.3-rolling-202002030754” and running inside a VM on Server 2019 with Hyper-V role with the VM set in Trunk mode as per below

PS C:\Windows\system32> get-vmnetworkadaptervlan -vmname vyos007

VMName VMNetworkAdapterName Mode VlanList


vyos007 Network Adapter Trunk 0,0-200
vyos007 Network Adapter Trunk 0,0-200
vyos007 Network Adapter Trunk 0,0-200

All 3 NICs inside Hyper-V VM settings have “Enable MAC address spoofing” ticked.

My second physical Hyper-V host does not even have the vyos VM configured yet, but it is installed and ready for me to configure, its not even switched off at this point so actual VRRP is not technically possible… but it should still work on a single router as it should be always a master right?

Just till I figure it out I am testing VRRP on my ip camera VLAN 53 running on eth2
So my gateway or floating virtual ip for this vlan should be 192.168.53.253
My normal interfaces are supposed to be 192.168.53.252 for primary vyos router and .254 for secondary.

What has happened, my cameras are unreachable as soon as I switch to using VRRP as it appears the gateway (virtual ip 192.168.53.253) does not work when it becomes master.
I have rebooted the vyos VM but nothing changes.
Nothing concerning in dmesg that I saw regarding VRRP.

Can you please help, I am still playing but nothing is working. Thanks!
If I use monitor traffic on either eth2.53 or eth2.53v53 I can see camera traffic on both interfaces which is also not expected as eth2.53 is the static address and eth2.53v53 should be the virtual vrrp address with the correct gateway ip?

show_interfaces_ethernet_eth2

duplex auto
hw-id 00:15:5d:01:9f:14
speed auto
vif 53 {
address 192.168.53.252/24
description Cam
}

show_high_availability

vrrp {
group cam {
advertise-interval 1
description Cam
hello-source-address 192.168.53.252
interface eth2.53
peer-address 192.168.53.254
priority 200
rfc3768-compatibility
virtual-address 192.168.53.253/24
vrid 53
}
sync-group sync {
member cam
}
}

show_vrrp

Name Interface VRID State Last Transition


cam eth2.53v53 53 MASTER 3h32m12s

show_interfaces_vrrp

Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface IP Address S/L Description


eth2.53v53 192.168.53.253/24 u/u

show_vrrp_detail with only the relevant interface information attached for sake of keeping it simple
Interestingly the virtual interface for VRRP does not appear to have ipv4 address assigned?

------< VRRP Topology >------
VRRP Instance = cam
VRRP Version = 2
Sync group = sync
State = MASTER
Wantstate = MASTER
Number of interface and track script faults = 0
Number of track scripts init = 0
Last transition = 1582501708 (Mon Feb 24 10:48:28 2020)
Read timeout = 1582514416.250918 (Mon Feb 24 14:20:16.250918)
Master down timer = 3218750 usecs
Use VMAC, is_up = true, xmit_base = true
Interface = eth2.53v53, vmac on eth2.53, xmit base i/f
Using src_ip = 192.168.53.252 (from configuration)
Gratuitous ARP delay = 5
Gratuitous ARP repeat = 5
Gratuitous ARP refresh = 0
Gratuitous ARP refresh repeat = 1
Gratuitous ARP lower priority delay = 5
Gratuitous ARP lower priority repeat = 5
Send advert after receive lower priority advert = true
Send advert after receive higher priority advert = false
Virtual Router ID = 53
Priority = 200
Effective priority = 200
Total priority = 200
Advert interval = 1 sec
Accept = enabled
Preempt = enabled
Promote_secondaries = disabled
Authentication type = none
Virtual IP = 1
192.168.53.253/24 dev eth2.53v53 scope global
Unicast Peer = 1
192.168.53.254
Unicast checksum compatibility = no
fd_in 15, fd_out 16
Using smtp notification = no
------< VRRP Sync groups >------
VRRP Sync Group = sync, MASTER
VRRP member instances = 1
cam
Using smtp notification = no

Name = eth2.53
index = 10
IPv4 address = 192.168.53.252
IPv6 address = fe80::215:5dff:fe01:9f14
MAC = 00:15:5d:01:9f:14
MAC broadcast = ff:ff:ff:ff:ff:ff
State = UP, RUNNING
MTU = 1500
HW Type = ETHERNET
NIC netlink status update
Reset ARP config counter 1
Original arp_ignore 1
Original arp_filter 1
rp_filter 0
Original promote_secondaries 0
Reset promote_secondaries counter 0
Tracking VRRP instances = 1
cam, weight 0

Name = eth2.53v53
index = 14
IPv4 address = (none)
IPv6 address = (none)
MAC = 00:00:5e:00:01:35
MAC broadcast = ff:ff:ff:ff:ff:ff
State = UP, RUNNING
VMAC type private, underlying interface = eth2.53, state = UP, RUNNING
I/f created by keepalived
MTU = 1500
HW Type = ETHERNET
NIC netlink status update
Reset ARP config counter 0
Original arp_ignore 1
Original arp_filter 0
rp_filter 0
Original promote_secondaries 1
Reset promote_secondaries counter 0
Tracking VRRP instances = 1
cam, weight 0

I couldn’t find much troubleshooting information online if the eth2.53v53 interface should be in the routing table when using VRRP? but here is my route table

me@vyos007:~$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default [private-removed] 0.0.0.0 UG 20 0 0 eth0.167
10.168.17.0 openvpn007.on.f 255.255.255.0 UG 20 0 0 eth0.17
10.168.19.0 openvpn007.on.f 255.255.255.0 UG 20 0 0 eth0.17
192.168.7.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2.7
192.168.11.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1.11
192.168.13.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1.13
192.168.17.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0.17
192.168.53.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2.53
192.168.67.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0.67
192.168.79.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0.79
192.168.131.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1.131
[private-removed] 0.0.0.0 255.255.224.0 U 0 0 0 eth0.167

@blackhole Show status of interfaces, maybe it state ‘down’.
The network 192.168.53.0/24 should be seen as directly connected.
Can you ping 192.168.53.252, 192.168.53.254, and 192.168.53.253?

@Viacheslav thanks, it appears it is up however

me@vyos007:~$ show interfaces
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface IP Address S/L Description


eth0 - u/u
eth0.17 192.168.17.253/24 u/u Public
eth0.67 192.168.67.253/24 u/u DMZ
eth0.79 192.168.79.253/24 u/u Download
eth0.167 [private-removed] u/u WAN
eth1 - u/u
eth1.11 192.168.11.253/24 u/u IOT
eth1.13 192.168.13.253/24 u/u LAN
eth1.131 192.168.131.253/24 u/u Guest
eth2 - u/u
eth2.7 192.168.7.253/24 u/u Management
eth2.53 192.168.53.252/24 u/u Cam
eth2.53v53 192.168.53.253/24 u/u
lo 127.0.0.1/8 u/u
::1/128

Here is a ping from the router session to .252 and .253 (.254 does not exist at this stage till I get this working)

me@vyos007:~$ ping 192.168.53.252
PING 192.168.53.252 (192.168.53.252): 56 data bytes
64 bytes from 192.168.53.252: icmp_seq=0 ttl=64 time=0.034 ms
64 bytes from 192.168.53.252: icmp_seq=1 ttl=64 time=0.048 ms
64 bytes from 192.168.53.252: icmp_seq=2 ttl=64 time=0.126 ms
^C— 192.168.53.252 ping statistics —
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.034/0.069/0.126/0.040 ms

me@vyos007:~$ ping 192.168.53.253
PING 192.168.53.253 (192.168.53.253): 56 data bytes
64 bytes from 192.168.53.253: icmp_seq=0 ttl=64 time=0.041 ms
64 bytes from 192.168.53.253: icmp_seq=1 ttl=64 time=0.063 ms
64 bytes from 192.168.53.253: icmp_seq=2 ttl=64 time=0.052 ms
^C— 192.168.53.253 ping statistics —
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.041/0.052/0.063/0.000 ms

I am a bit confused now I guess, here is the ping results from the hypervisor host, ie the server 2019 box that currently runs the software that records from the ip cameras.

There is no direct connection to the Cam VLAN id 53 from the hypervisor DMZ VLAN id 67. It is routed and the firewalls work without issue when i disable the VRRP group and revert the Cam network address to 192.168.53.253. There is no static route for the Cam network on the hypervisor server either.

C:\Users\me>ping 192.168.53.252

Pinging 192.168.53.252 with 32 bytes of data:
Reply from 192.168.53.252: bytes=32 time=3ms TTL=64
Reply from 192.168.53.252: bytes=32 time=3ms TTL=64
Reply from 192.168.53.252: bytes=32 time=5ms TTL=64
Reply from 192.168.53.252: bytes=32 time=4ms TTL=64

Ping statistics for 192.168.53.252:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 3ms, Maximum = 5ms, Average = 3ms

C:\Users\me>ping 192.168.53.253

Pinging 192.168.53.253 with 32 bytes of data:
Reply from 192.168.53.253: bytes=32 time=6ms TTL=64
Reply from 192.168.53.253: bytes=32 time=1ms TTL=64
Reply from 192.168.53.253: bytes=32 time=3ms TTL=64
Reply from 192.168.53.253: bytes=32 time=3ms TTL=64

Ping statistics for 192.168.53.253:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 1ms, Maximum = 6ms, Average = 3ms

@Viacheslav, I ran just “route” again just now, after re-enabling VRRP for this test and I can now see 2x routes for 192.168.53.0, one for each interface, hmmm! Not sure what to make of this just yet.

I will just add there is still no connectivity to the cameras inside the Cam VLAN network.

me@vyos007:~$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default [private-removed] 0.0.0.0 UG 20 0 0 eth0.167
10.168.17.0 openvpn007.on.f 255.255.255.0 UG 20 0 0 eth0.17
10.168.19.0 openvpn007.on.f 255.255.255.0 UG 20 0 0 eth0.17
192.168.7.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2.7
192.168.11.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1.11
192.168.13.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1.13
192.168.17.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0.17
192.168.53.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2.53
192.168.53.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2.53v53
192.168.67.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0.67
192.168.79.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0.79
192.168.131.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1.131
[private-removed] 0.0.0.0 255.255.224.0 U 0 0 0 eth0.167

Hmmm, a thought just occured to me, does one have to add the eth2.53v53 interface into the zone-policy?
I am running a zone based firewall, but I suspect this does should not matter.

I guess:
Do I have to manually add the VRRP interfaces to the zone-policy, can confirm only eth2.53 is currently in the Cam zone?
- OR -
Does the VRRP background process deal with this automagically?
I am not seeing firewall related errors tho, unless I am mistaken.

At this point I have not tried to manually add eth2.53v53 VRRP interface to the zone-policy but may try shortly.

@Viacheslav Can confirm that things are working now that I have added eth2.53v53 to the Cam zone-policy.
This is a bit weird, have never seen this mentioned or expected it to be necessary.

Is there something I have missed or something to take up further.

At this stage it feels like a bit of a workaround?
The cameras immediately started showing image and recording after adding the VRRP interface to the zone-policy.

I did a bit of “monitor traffic” of both eth2.53 and eth2.53v53 (VRRP) and can see traffic to/from cameras and server is on both interfaces. This just seems not quite right, should this be happening?

All the RTSP traffic from the cameras destined for the server are going through eth2.53v53 VRRP interface while eth2.53 contains both server to camera RTSP requests and camera to server RTSP traffic.

I think this is due to both interfaces being in the routing table with the same metric of 0, instead of just eth2.53v53 VRRP interface or different metrics to prefer the desired floating virtual address. This is what appears to be causing the traffic to go over both, unless I am mis-understanding something.

Thanks!

So I have had a bit more of a think about things and made the static address of eth2.53 on a different network now so it will be 192.168.59.59 for primary router and 192.168.59.60 when i set up the second.

I have made eth2.53v53 VRRP interface as the only interface in Cam zone and made a new VRRP zone which will host all the static interfaces, so this contains eth2.53 currently.

It appears to be working! I have made a firewall-vrrp firewall rule that allows vrrp protocol through and can see in

show log | grep vrrp

is only complaining that it can not reach the peer, as it does not exist yet.

Is this the expected behaviour and/or right way to do this? I am still unsure about adding the VRRP interface to zone-policy as I have not seen this done anywhere in the past. Is this something that is typical and I should add some updates to the High Availability document or is something broken in my config?

Thanks and apologies for the flood, but I like to show my workings, there is always someone with the same issue down the track and search functions are great.

Cheers!

I am quietly still working on this and supplying updates in the hope someone with more knowledge how this works can tell me if the route table should have the static and virtual addresses both in there?

Currently, from the Juniper documentation I am trying my static and virtual addresses for the primary to be both 192.168.53.253 and the backup will have static 192.168.53.254.
At the moment the vrrp zone I created earlier is deleted too, I adjusted the VRRP group priority to 255 as well.

route -n

me@vyos007:~$ route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 [private-removed] 0.0.0.0 UG 20 0 0 eth0.167
10.168.17.0 192.168.17.100 255.255.255.0 UG 20 0 0 eth0.17
10.168.19.0 192.168.17.100 255.255.255.0 UG 20 0 0 eth0.17
192.168.7.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2.7
192.168.11.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1.11
192.168.13.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1.13
192.168.17.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0.17
192.168.53.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2.53v53
192.168.53.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2.53
192.168.67.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0.67
192.168.79.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0.79
192.168.131.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1.131
[private-removed] 0.0.0.0 255.255.224.0 U 0 0 0 eth0.167

Thanks!