VyOS 1.4 instance performing differently with each reboot after enabling WAN load balancing

I recently deployed some new hardware to enable 10/40Gbps networking and move over to WAN failover load balancing. However, ever since I’ve done this, I’ve had strange issues which seemingly change with a reboot. Some of the situations I’ve experienced are:

  • being able to ping IPs and nslookup domains, but unable to ping/access devices by domain nor access opened ports on the WAN firewall to LOCAL
  • system will be completely unable to reach the Internet, but firewall rules for WAN2LOCAL work correctly
  • system can access Internet correctly and able to wireguard into system from WAN, but Wireguard is unable to reach other devices despite being properly configured

I’m really lost on what to do, as the setup seemingly worked fine before I did WAN load balancing, but I’m unable to see any configuration or setup which seems out of place. The configuration file can be found here

Just chiming in, I believe I am noticing the same thing.
Latest rolling image as of time of writing, im using zone based firewall config, but local-zone (firewall) to wan is set up as always.
I cant ping from VyOS to WAN and actually what made me realise to start looking into it was my dynamic dns for Namecheap is not working anymore, IP hasnt updated since 1.4.
Logs do not have any entries denying firewall outbound

On screen/logs from a simple ping and the ddclient

mario@vyos007:/config/$ ping 8.8.8.8
connect: Network is unreachable

Firewall logs

mario@vyos007:/config$ show log all | grep "firewall-wan"
mario@vyos007:/config$

mario@vyos007:/config$ show log all | grep “ddclient”

May  7 15:09:12 vyos007 ddclient[27076]: WARNING:  cannot connect to dynamicdns.park-your-domain.com:443 socket: Network is unreachable IO::Socket::IP configuration failed
May  7 15:09:12 vyos007 ddclient[27076]: FAILED:   updating enter: Could not connect to dynamicdns.park-your-domain.com.
May  7 15:10:12 vyos007 ddclient[27076]: WARNING:  file /run/ddclient/ddclient.cache, line 3: Invalid Value for keyword 'ip' = ''
May  7 15:10:12 vyos007 ddclient[27076]: WARNING:  skipping update of enter from <nothing> to {myexternalipaddress}.
May  7 15:10:12 vyos007 ddclient[27076]: WARNING:   last updated <never> but last attempt on Fri May  7 15:09:12 2021 failed.
May  7 15:10:12 vyos007 ddclient[27076]: WARNING:   Wait at least 5 minutes between update attempts.
May  7 15:11:12 vyos007 ddclient[27076]: WARNING:  cannot connect to dynamicdns.park-your-domain.com:443 socket: Network is unreachable IO::Socket::IP configuration failed
May  7 15:11:12 vyos007 ddclient[27076]: FAILED:   updating enter: Could not connect to dynamicdns.park-your-domain.com.
May  7 15:12:13 vyos007 ddclient[27076]: WARNING:  file /run/ddclient/ddclient.cache, line 3: Invalid Value for keyword 'ip' = ''
May  7 15:12:13 vyos007 ddclient[27076]: WARNING:  skipping update of enter from <nothing> to {myexternalipaddress}.
May  7 15:12:13 vyos007 ddclient[27076]: WARNING:   last updated <never> but last attempt on Fri May  7 15:11:12 2021 failed.
May  7 15:12:13 vyos007 ddclient[27076]: WARNING:   Wait at least 5 minutes between update attempts.
May  7 15:13:13 vyos007 ddclient[27076]: WARNING:  cannot connect to dynamicdns.park-your-domain.com:443 socket: Network is unreachable IO::Socket::IP configuration failed
May  7 15:13:13 vyos007 ddclient[27076]: FAILED:   updating enter: Could not connect to dynamicdns.park-your-domain.com.

Any help please?

Just noticed I am missing the default route 0.0.0.0 on VyOS… probably explains things…
My WAN (both, even tho one is off at the moment) are DHCP

How very strange/interesting

mario@vyos007:/config$ show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

S>* 10.0.0.0/8 [254/0] unreachable (blackhole), weight 1, 01:21:27
S>* 10.168.17.0/24 [1/0] via 192.168.17.100, eth0.17, weight 1, 01:21:27
S>* 10.168.19.0/24 [1/0] via 192.168.17.100, eth0.17, weight 1, 01:21:27
S>* 172.16.0.0/12 [254/0] unreachable (blackhole), weight 1, 01:21:27
S>* 192.168.0.0/16 [254/0] unreachable (blackhole), weight 1, 01:21:27
C * 192.168.7.0/24 is directly connected, eth0.7v7, 01:21:17
C>* 192.168.7.0/24 is directly connected, eth0.7, 01:21:31
C * 192.168.11.0/24 is directly connected, eth0.11v11, 01:21:17
C>* 192.168.11.0/24 is directly connected, eth0.11, 01:21:31
C * 192.168.13.0/24 is directly connected, eth0.13v13, 01:21:17
C>* 192.168.13.0/24 is directly connected, eth0.13, 01:21:31
C * 192.168.17.0/24 is directly connected, eth0.17v17, 01:21:17
C>* 192.168.17.0/24 is directly connected, eth0.17, 01:21:31
C * 192.168.53.0/24 is directly connected, eth0.53v53, 01:21:17
C>* 192.168.53.0/24 is directly connected, eth0.53, 01:21:30
C * 192.168.67.0/24 is directly connected, eth0.67v67, 01:21:17
C>* 192.168.67.0/24 is directly connected, eth0.67, 01:21:30
C * 192.168.79.0/24 is directly connected, eth0.79v79, 01:21:17
C>* 192.168.79.0/24 is directly connected, eth0.79, 01:21:30
C * 192.168.131.0/24 is directly connected, eth0.131v131, 01:21:17
C>* 192.168.131.0/24 is directly connected, eth0.131, 01:21:30
C>* 203.7.0.0/19 is directly connected, eth0.167, 01:21:29

My actual version information

mario@vyos007:/config$ show version

Version:          VyOS 1.4-rolling-202105050002
Release Train:    sagitta

Built by:         autobuild@vyos.net
Built on:         Wed 05 May 2021 01:17 UTC
Build UUID:       cccd86c6-1629-41e3-a554-2b649a564ea6
Build Commit ID:  2feba96c6c9fcb

Architecture:     x86_64
Boot via:         installed image
System type:      VMware guest

Hardware vendor:  VMware, Inc.
Hardware model:   VMware Virtual Platform
Hardware S/N:     VMware-42 1f 29 92 a1 0b 9f a2-cc e7 85 32 d3 2a 7a 9a
Hardware UUID:    92291f42-0ba1-a29f-cce7-8532d32a7a9a

Copyright:        VyOS maintainers and contributors

one more update, im on to something

added default route to wan temporarily
mario@vyos007# set protocols static route 0.0.0.0/0 interface eth0.167

check routes and also a ping to generic google dns server are now much better

mario@vyos007:/config$ show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

S>* 0.0.0.0/0 [1/0] is directly connected, eth0.167, weight 1, 00:00:27
S>* 10.0.0.0/8 [254/0] unreachable (blackhole), weight 1, 01:49:52
S>* 10.168.17.0/24 [1/0] via 192.168.17.100, eth0.17, weight 1, 01:49:52
S>* 10.168.19.0/24 [1/0] via 192.168.17.100, eth0.17, weight 1, 01:49:52
S>* 172.16.0.0/12 [254/0] unreachable (blackhole), weight 1, 01:49:52
S>* 192.168.0.0/16 [254/0] unreachable (blackhole), weight 1, 01:49:52
C * 192.168.7.0/24 is directly connected, eth0.7v7, 01:49:42
C>* 192.168.7.0/24 is directly connected, eth0.7, 01:49:56
C * 192.168.11.0/24 is directly connected, eth0.11v11, 01:49:42
C>* 192.168.11.0/24 is directly connected, eth0.11, 01:49:56
C * 192.168.13.0/24 is directly connected, eth0.13v13, 01:49:42
C>* 192.168.13.0/24 is directly connected, eth0.13, 01:49:56
C * 192.168.17.0/24 is directly connected, eth0.17v17, 01:49:42
C>* 192.168.17.0/24 is directly connected, eth0.17, 01:49:56
C * 192.168.53.0/24 is directly connected, eth0.53v53, 01:49:42
C>* 192.168.53.0/24 is directly connected, eth0.53, 01:49:55
C * 192.168.67.0/24 is directly connected, eth0.67v67, 01:49:42
C>* 192.168.67.0/24 is directly connected, eth0.67, 01:49:55
C * 192.168.79.0/24 is directly connected, eth0.79v79, 01:49:42
C>* 192.168.79.0/24 is directly connected, eth0.79, 01:49:55
C * 192.168.131.0/24 is directly connected, eth0.131v131, 01:49:42
C>* 192.168.131.0/24 is directly connected, eth0.131, 01:49:55
C>* 203.7.0.0/19 is directly connected, eth0.167, 01:49:54
mario@vyos007:/config$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=58 time=21.8 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=58 time=12.4 ms
^C
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 3ms
rtt min/avg/max/mdev = 12.364/17.058/21.753/4.696 ms

and even the dynamic dns update worked (yay!)

mario@vyos007:/config$ show log dns dynamic
<error lines, a lot of>
May  7 15:53:14 vyos007 ddclient[27076]: SUCCESS:  updating enter: good: IP address set to 203.{myipaddress}

Ref ⚓ T3505 Commits do not respect changes in FRR that are not stored in a config

1 Like

Thanks for the support ,hopefully the fix isnt too bad