I’ve built VyOs 1.2 from source and am having this issue on all of the vyos routers that I use as virtual endpoints for various networks.
The configuration there are a pair of routers with public (eth1) and private (eth0) IP addresses, both of which have VRRP enabled for HA for sevices.
We also have IPSec enabled on the public interface for IKEv2 site to site vpn connections to other routers.
A thing we have noticed is that when the VRRP on the public interface changes state, intended for private ip addresseses go out of the public interface.
Here is the interface config:
high-availability {
vrrp {
group eth0-1 {
advertise-interval 1
interface eth0
priority 32
virtual-address 10.xx.xx.254/24
vrid 1
}
group eth1-1 {
advertise-interval 1
interface eth1
peer-address xx.xx.xx.34
priority 32
virtual-address xx.xx.xx.2/26
vrid 2
}
}
}
ethernet eth0 {
address 10.xx.xx.253/24
duplex auto
smp-affinity auto
speed auto
}
ethernet eth1 {
address xx.xx.xx.35/26
duplex auto
smp-affinity auto
speed auto
}
Once the VRRP state changes, packets meant to go out of etho go out of eth1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
15:46:09.525007 IP RouterA > 10.xx.xx.11: ICMP echo request, id 37020, seq 1, length 12
15:46:10.200648 IP RouterA > 10.xx.xx.11: ICMP echo request, id 30211, seq 29, length 64
15:46:11.224580 IP RouterA > 10.xx.xx.11: ICMP echo request, id 30211, seq 30, length 64
15:46:12.248571 IP RouterA > 10.xx.xx.11: ICMP echo request, id 30211, seq 31, length 64
This is reproduceable by just manually changing the priority of VRRP on the public interface and changing it back again.
The weird part is that in order to fix this all I have to do is to run restart vpn to restart the ipsec process and packets go back to running out of the correct interfaces again.
This is on both VyOS 1.2.4 and 1.2.5.
The version of strongSwan is 5.7.2-1+vyos2
Is anyone else able to reproduce this issue?
Below is the journalctl log for the time this router was affected:
Oct 14 15:41:20 RouterA kernel: device eth1 entered promiscuous mode
Oct 14 15:43:45 RouterA kernel: IPv4: martian source xx.xx.xx.2 from xx.xx.xx.2, on dev eth1
Oct 14 15:43:45 RouterA kernel: ll header: 00000000: ff ff ff ff ff ff 00 15 5d 70 ef c1 08 06 ........]p....
Oct 14 15:43:45 RouterA kernel: IPv4: martian source xx.xx.xx.2 from xx.xx.xx.2, on dev eth1
Oct 14 15:43:45 RouterA kernel: ll header: 00000000: ff ff ff ff ff ff 00 15 5d 70 ef c1 08 06 ........]p....
Oct 14 15:43:45 RouterA kernel: IPv4: martian source xx.xx.xx.2 from xx.xx.xx.2, on dev eth1
Oct 14 15:43:45 RouterA kernel: ll header: 00000000: ff ff ff ff ff ff 00 15 5d 70 ef c1 08 06 ........]p....
Oct 14 15:43:45 RouterA kernel: IPv4: martian source xx.xx.xx.2 from xx.xx.xx.2, on dev eth1
Oct 14 15:43:45 RouterA kernel: ll header: 00000000: ff ff ff ff ff ff 00 15 5d 70 ef c1 08 06 ........]p....
Oct 14 15:43:45 RouterA kernel: IPv4: martian source xx.xx.xx.2 from xx.xx.xx.2, on dev eth1
Oct 14 15:43:45 RouterA kernel: ll header: 00000000: ff ff ff ff ff ff 00 15 5d 70 ef c1 08 06 ........]p....
Oct 14 15:43:45 RouterA Keepalived_vrrp[3646]: (eth1-1) Master received advert from xx.xx.xx.34 with higher priority 64, ours 32
Oct 14 15:43:45 RouterA Keepalived_vrrp[3646]: (eth1-1) Entering BACKUP STATE
Oct 14 15:43:45 RouterA charon[172445]: 06[KNL] xx.xx.xx.2 disappeared from eth1
Oct 14 15:43:45 RouterA bgpd[1210]: [EC 100663301] INTERFACE_ADDRESS_DEL: Cannot find IF 3 in VRF 0
Oct 14 15:43:45 RouterA charon[172445]: 15[IKE] sending address list update using MOBIKE
Oct 14 15:43:45 RouterA charon[172445]: 15[ENC] generating INFORMATIONAL request 674 [ N(NO_ADD_ADDR) ]
Oct 14 15:43:45 RouterA charon[172445]: 15[NET] sending packet: from xx.xx.xx.35[4500] to vpn.endpoint.138[4500] (76 bytes)
Oct 14 15:43:45 RouterA charon[172445]: 05[NET] received packet: from vpn.endpoint.138[4500] to xx.xx.xx.35[4500] (76 bytes)
Oct 14 15:43:45 RouterA charon[172445]: 05[ENC] parsed INFORMATIONAL response 674 [ ]
Oct 14 15:43:46 RouterA ntpd[3399]: Deleting interface #11 eth1, xx.xx.xx.2#123, interface stats: received=0, sent=0, dropped=0, active_time=16336589 secs
Oct 14 15:43:46 RouterA ntpd[3399]: peers refreshed
Oct 14 15:43:48 RouterA sudo[29967]: support : TTY=pts/1 ; PWD=/home/support ; USER=root ; COMMAND=/bin/ping 10.xx.xx.11
Oct 14 15:43:48 RouterA sudo[29967]: pam_unix(sudo:session): session opened for user root by (uid=0)
Oct 14 15:44:02 RouterA sudo[29967]: pam_unix(sudo:session): session closed for user root
Oct 14 15:44:11 RouterA sudo[30083]: support : TTY=pts/1 ; PWD=/home/support ; USER=root ; COMMAND=/usr/libexec/vyos/op_mode/vrrp.py --summary
Oct 14 15:44:11 RouterA sudo[30083]: pam_unix(sudo:session): session opened for user root by (uid=0)
Oct 14 15:44:11 RouterA Keepalived_vrrp[3646]: Printing VRRP as json for process(3646) on signal
Oct 14 15:44:11 RouterA sudo[30083]: pam_unix(sudo:session): session closed for user root
Oct 14 15:44:28 RouterA Keepalived_vrrp[3646]: (eth1-1) received lower priority (16) advert from xx.xx.xx.34 - discarding
Oct 14 15:44:29 RouterA Keepalived_vrrp[3646]: (eth1-1) received lower priority (16) advert from xx.xx.xx.34 - discarding
Oct 14 15:44:30 RouterA Keepalived_vrrp[3646]: (eth1-1) received lower priority (16) advert from xx.xx.xx.34 - discarding
Oct 14 15:44:31 RouterA Keepalived_vrrp[3646]: (eth1-1) received lower priority (16) advert from xx.xx.xx.34 - discarding
Oct 14 15:44:32 RouterA Keepalived_vrrp[3646]: (eth1-1) Entering MASTER STATE
Oct 14 15:44:32 RouterA charon[172445]: 07[KNL] xx.xx.xx.2 appeared on eth1
Oct 14 15:44:32 RouterA bgpd[1210]: [EC 100663301] INTERFACE_ADDRESS_ADD: Cannot find IF 3 in VRF 0
Oct 14 15:44:32 RouterA charon[172445]: 15[IKE] sending address list update using MOBIKE
Oct 14 15:44:32 RouterA charon[172445]: 15[ENC] generating INFORMATIONAL request 675 [ N(ADD_4_ADDR) ]
Oct 14 15:44:32 RouterA charon[172445]: 15[NET] sending packet: from xx.xx.xx.35[4500] to vpn.endpoint.138[4500] (76 bytes)
Oct 14 15:44:32 RouterA charon[172445]: 12[NET] received packet: from vpn.endpoint.138[4500] to xx.xx.xx.35[4500] (76 bytes)
Oct 14 15:44:32 RouterA charon[172445]: 12[ENC] parsed INFORMATIONAL response 675 [ ]
Oct 14 15:44:34 RouterA ntpd[3399]: Listen normally on 32 eth1 xx.xx.xx.2 UDP 123
Oct 14 15:44:34 RouterA ntpd[3399]: peers refreshed
Oct 14 15:44:35 RouterA sudo[30116]: support : TTY=pts/1 ; PWD=/home/support ; USER=root ; COMMAND=/usr/libexec/vyos/op_mode/vrrp.py --summary
Oct 14 15:44:35 RouterA sudo[30116]: pam_unix(sudo:session): session opened for user root by (uid=0)
Oct 14 15:44:35 RouterA Keepalived_vrrp[3646]: Printing VRRP as json for process(3646) on signal
Oct 14 15:44:35 RouterA sudo[30116]: pam_unix(sudo:session): session closed for user root
Oct 14 15:45:11 RouterA sudo[30165]: support : TTY=pts/1 ; PWD=/home/support ; USER=root ; COMMAND=/bin/ping 10.xx.xx.11
Oct 14 15:45:11 RouterA sudo[30165]: pam_unix(sudo:session): session opened for user root by (uid=0)
Oct 14 15:45:14 RouterA sudo[30165]: pam_unix(sudo:session): session closed for user root
Oct 14 15:45:16 RouterA bgpd[1210]: [EC 100663301] INTERFACE_STATE: Cannot find IF eth1 in VRF 0
Oct 14 15:45:16 RouterA kernel: device eth1 left promiscuous mode
Oct 14 15:45:16 RouterA sudo[29783]: pam_unix(sudo:session): session closed for user root
Oct 14 15:45:39 RouterA sudo[30182]: support : TTY=pts/2 ; PWD=/home/support ; USER=root ; COMMAND=/usr/sbin/tcpdump -i eth1
Oct 14 15:45:39 RouterA sudo[30182]: pam_unix(sudo:session): session opened for user root by (uid=0)
Oct 14 15:45:39 RouterA bgpd[1210]: [EC 100663301] INTERFACE_STATE: Cannot find IF eth1 in VRF 0
Oct 14 15:45:39 RouterA kernel: device eth1 entered promiscuous mode
Oct 14 15:45:41 RouterA sudo[30210]: support : TTY=pts/1 ; PWD=/home/support ; USER=root ; COMMAND=/bin/ping 10.xx.xx.11
Oct 14 15:45:41 RouterA sudo[30210]: pam_unix(sudo:session): session opened for user root by (uid=0)
Oct 14 15:45:45 RouterA bgpd[1210]: [EC 100663301] INTERFACE_STATE: Cannot find IF eth1 in VRF 0
Oct 14 15:45:45 RouterA kernel: device eth1 left promiscuous mode
Oct 14 15:45:45 RouterA sudo[30182]: pam_unix(sudo:session): session closed for user root
Oct 14 15:45:49 RouterA sudo[30212]: support : TTY=pts/2 ; PWD=/home/support ; USER=root ; COMMAND=/usr/sbin/tcpdump -i eth1
Oct 14 15:45:49 RouterA sudo[30212]: pam_unix(sudo:session): session opened for user root by (uid=0)
Oct 14 15:45:49 RouterA bgpd[1210]: [EC 100663301] INTERFACE_STATE: Cannot find IF eth1 in VRF 0
Oct 14 15:45:49 RouterA kernel: device eth1 entered promiscuous mode
Oct 14 15:45:53 RouterA bgpd[1210]: [EC 100663301] INTERFACE_STATE: Cannot find IF eth1 in VRF 0
Oct 14 15:45:53 RouterA kernel: device eth1 left promiscuous mode
Oct 14 15:45:53 RouterA sudo[30212]: pam_unix(sudo:session): session closed for user root
Oct 14 15:45:55 RouterA sudo[30222]: support : TTY=pts/2 ; PWD=/home/support ; USER=root ; COMMAND=/usr/sbin/tcpdump -i eth1
Oct 14 15:45:55 RouterA sudo[30222]: pam_unix(sudo:session): session opened for user root by (uid=0)
Oct 14 15:45:55 RouterA bgpd[1210]: [EC 100663301] INTERFACE_STATE: Cannot find IF eth1 in VRF 0
Oct 14 15:45:55 RouterA kernel: device eth1 entered promiscuous mode
Oct 14 15:46:07 RouterA bgpd[1210]: [EC 100663301] INTERFACE_STATE: Cannot find IF eth1 in VRF 0
Oct 14 15:46:07 RouterA kernel: device eth1 left promiscuous mode
Oct 14 15:46:07 RouterA sudo[30222]: pam_unix(sudo:session): session closed for user root
Oct 14 15:46:09 RouterA sudo[30233]: support : TTY=pts/2 ; PWD=/home/support ; USER=root ; COMMAND=/usr/sbin/tcpdump -i eth1
Oct 14 15:46:09 RouterA sudo[30233]: pam_unix(sudo:session): session opened for user root by (uid=0)
Oct 14 15:46:09 RouterA bgpd[1210]: [EC 100663301] INTERFACE_STATE: Cannot find IF eth1 in VRF 0
Oct 14 15:46:09 RouterA kernel: device eth1 entered promiscuous mode
Oct 14 15:46:24 RouterA bgpd[1210]: [EC 100663301] INTERFACE_STATE: Cannot find IF eth1 in VRF 0
Oct 14 15:46:24 RouterA kernel: device eth1 left promiscuous mode
Oct 14 15:46:24 RouterA sudo[30233]: pam_unix(sudo:session): session closed for user root
Oct 14 15:51:00 RouterA charon[172445]: 09[KNL] creating rekey job for CHILD_SA ESP/0xcd10ff9f/vpn.endpoint.138
```This text will be hidden