Losing some configuration on-reboot

, , , , ,
1

Hi guys,

I wonder if anyone can advise how to fix this?

We have several VyOS VM’s running as Ruckus SoftGRE tunnel end-points. Under normal operation it works great. The Ruckus AP’s tunnel using gre-bridge to the VyOS device and a Guest SSID which tags Guest traffic with a specific VLAN ID. At the other end, a Juniper SRX performs DHCP and security services against the traffic before it goes out to the internet.

Tonight, the underlying ESXi host was upgraded, which means the VyOS VM was shutdown. On returning to service, some of the critical configuration for the GRE tunnels and bridge interface was missing. It does this across every reboot.

The configuration once rebooted looks like this -

set firewall all-ping 'enable'
set firewall broadcast-ping 'disable'
set firewall config-trap 'disable'
set firewall group address-group MANAGEMENT-HOSTS address 'xxx.xxx.10.21'
set firewall group address-group MANAGEMENT-HOSTS address 'xxx.xxx.40.1-xxx.xxx.40.254'
set firewall group address-group MANAGEMENT-HOSTS address 'xxx.xxx.50.1-xxx.xxx.50.254'
set firewall group address-group MANAGEMENT-HOSTS address 'xxx.xxx.10.32'
set firewall group address-group MANAGEMENT-HOSTS address 'xxx.xxx.10.11'
set firewall group address-group MANAGEMENT-HOSTS address 'xxx.xxx.3.103'
set firewall group address-group MANAGEMENT-HOSTS address 'xxx.xxx.32.1-xxx.xxx.32.254'
set firewall group address-group MANAGEMENT-HOSTS address 'xxx.xxx.33.1-xxx.xxx.33.254'
set firewall group address-group MANAGEMENT-HOSTS address 'xxx.xxx.10.34'
set firewall group address-group MANAGEMENT-HOSTS address 'xxx.xxx.10.12'
set firewall group address-group MANAGEMENT-HOSTS address 'xxx.xxx.26.237'
set firewall group address-group MANAGEMENT-HOSTS address 'xxx.xxx.10.237'
set firewall group address-group MANAGEMENT-HOSTS address 'xxx.xxx.208.249'
set firewall group address-group RUCKUS-AP address 'xxx.xxx.32.240'
set firewall group address-group RUCKUS-AP address 'xxx.xxx.32.241'
set firewall group address-group RUCKUS-AP address 'xxx.xxx.32.242'
set firewall group address-group RUCKUS-AP address 'xxx.xxx.32.243'
set firewall group address-group RUCKUS-AP address 'xxx.xxx.138.240'
set firewall group address-group RUCKUS-AP address 'xxx.xxx.138.241'
set firewall group address-group RUCKUS-AP address 'xxx.xxx.138.242'
set firewall group address-group RUCKUS-AP address 'xxx.xxx.138.243'
set firewall ipv6-receive-redirects 'disable'
set firewall ipv6-src-route 'disable'
set firewall ip-src-route 'disable'
set firewall log-martians 'enable'
set firewall name management-filter default-action 'drop'
set firewall name management-filter rule 10 action 'accept'
set firewall name management-filter rule 10 protocol 'all'
set firewall name management-filter rule 10 source group address-group 'RUCKUS-AP'
set firewall name management-filter rule 20 action 'accept'
set firewall name management-filter rule 20 destination port '22'
set firewall name management-filter rule 20 protocol 'tcp'
set firewall name management-filter rule 20 source group address-group 'MANAGEMENT-HOSTS'
set firewall name management-filter rule 30 action 'accept'
set firewall name management-filter rule 30 protocol 'icmp'
set firewall name management-filter rule 30 source group address-group 'MANAGEMENT-HOSTS'
set firewall name management-filter rule 40 action 'accept'
set firewall name management-filter rule 40 destination port '161'
set firewall name management-filter rule 40 protocol 'udp'
set firewall name management-filter rule 40 source group address-group 'MANAGEMENT-HOSTS'
set firewall name management-filter rule 50 action 'accept'
set firewall name management-filter rule 50 protocol 'udp'
set firewall name management-filter rule 50 source group address-group 'MANAGEMENT-HOSTS'
set firewall name management-filter rule 50 source port '123'
set firewall name management-filter rule 60 action 'accept'
set firewall name management-filter rule 60 protocol 'udp'
set firewall name management-filter rule 60 source group address-group 'MANAGEMENT-HOSTS'
set firewall name management-filter rule 60 source port '53'
set firewall name management-filter rule 70 action 'accept'
set firewall name management-filter rule 70 protocol 'tcp'
set firewall name management-filter rule 70 source group address-group 'MANAGEMENT-HOSTS'
set firewall name management-filter rule 70 source port '443'
set firewall name management-filter rule 9999 action 'drop'
set firewall name management-filter rule 9999 log 'enable'
set firewall receive-redirects 'disable'
set firewall send-redirects 'enable'
set firewall source-validation 'disable'
set firewall syn-cookies 'enable'
set firewall twa-hazards-protection 'disable'
set interfaces ethernet eth0 address 'xxx.xxx.3.239/24'
set interfaces ethernet eth0 address 'xxx.xxx.3.107/24'
set interfaces ethernet eth0 description '*** GRE-TUNNEL LOCAL-IP INTERFACE ***'
set interfaces ethernet eth0 firewall local name 'management-filter'
set interfaces ethernet eth0 hw-id 'XX:XX:XX:XX:XX:e3'
set interfaces ethernet eth1 description '*** GRE-TUNNEL BRIDGE 802.1Q INTERFACE ***'
set interfaces ethernet eth1 hw-id 'XX:XX:XX:XX:XX:ed'
set interfaces loopback lo
set protocols static route xxx.xxx.0.0/0 next-hop xxx.xxx.3.103
set protocols static route xxx.xxx.0.0/16 next-hop xxx.xxx.3.1
set service lldp interface eth0
set service lldp legacy-protocols cdp
set service lldp management-address 'xxx.xxx.3.239'
set service lldp snmp enable
set service snmp contact 'bob@example.com'
set service snmp listen-address xxx.xxx.3.239
set service snmp location xxxxxx - site'
set service snmp v3 group nms mode 'ro'
set service snmp v3 group nms seclevel 'priv'
set service snmp v3 group nms view 'view1'
set service snmp v3 user xxxxxx auth plaintext-key 'xxxxxx'
set service snmp v3 user xxxxxx auth type 'sha'
set service snmp v3 user xxxxxx group 'nms'
set service snmp v3 user xxxxxx mode 'ro'
set service snmp v3 user xxxxxx privacy plaintext-key 'xxxxxx'
set service snmp v3 user xxxxxx privacy type 'aes'
set service snmp v3 view view1 oid xxx.xxx.6.1
set service ssh listen-address 'xxx.xxx.3.239'
set system config-management commit-revisions '100'
set system console device ttyS0 speed '115200'
set system host-name xxxxxx
set system login user xxxxxx authentication encrypted-password xxxxxx
set system login user xxxxxx authentication plaintext-password xxxxxx
set system login user xxxxxx authentication public-keys xxxx@xxx.xxx key xxxxxx
set system login user xxxxxx authentication public-keys xxxx@xxx.xxx type ssh-xxx
set system login user xxxxxx authentication encrypted-password xxxxxx
set system login user xxxxxx authentication plaintext-password xxxxxx
set system name-server 'xxx.xxx.10.237'
set system name-server 'xxx.xxx.26.237'
set system ntp server xxxxx.tld
set system ntp server xxxxx.tld
set system ntp server xxxxx.tld
set system ntp server xxxxx.tld
set system syslog global facility all level 'info'
set system syslog global facility protocols level 'debug'
set system time-zone 'Europe/London'

What it should look like is this -

set firewall all-ping 'enable'
set firewall broadcast-ping 'disable'
set firewall config-trap 'disable'
set firewall group address-group MANAGEMENT-HOSTS address 'xxx.xxx.10.21'
set firewall group address-group MANAGEMENT-HOSTS address 'xxx.xxx.40.1-xxx.xxx.40.254'
set firewall group address-group MANAGEMENT-HOSTS address 'xxx.xxx.50.1-xxx.xxx.50.254'
set firewall group address-group MANAGEMENT-HOSTS address 'xxx.xxx.10.32'
set firewall group address-group MANAGEMENT-HOSTS address 'xxx.xxx.10.11'
set firewall group address-group MANAGEMENT-HOSTS address 'xxx.xxx.3.103'
set firewall group address-group MANAGEMENT-HOSTS address 'xxx.xxx.32.1-xxx.xxx.32.254'
set firewall group address-group MANAGEMENT-HOSTS address 'xxx.xxx.33.1-xxx.xxx.33.254'
set firewall group address-group MANAGEMENT-HOSTS address 'xxx.xxx.10.34'
set firewall group address-group MANAGEMENT-HOSTS address 'xxx.xxx.10.12'
set firewall group address-group MANAGEMENT-HOSTS address 'xxx.xxx.26.237'
set firewall group address-group MANAGEMENT-HOSTS address 'xxx.xxx.10.237'
set firewall group address-group MANAGEMENT-HOSTS address 'xxx.xxx.208.249'
set firewall group address-group RUCKUS-AP address 'xxx.xxx.32.240'
set firewall group address-group RUCKUS-AP address 'xxx.xxx.32.241'
set firewall group address-group RUCKUS-AP address 'xxx.xxx.32.242'
set firewall group address-group RUCKUS-AP address 'xxx.xxx.32.243'
set firewall group address-group RUCKUS-AP address 'xxx.xxx.138.240'
set firewall group address-group RUCKUS-AP address 'xxx.xxx.138.241'
set firewall group address-group RUCKUS-AP address 'xxx.xxx.138.242'
set firewall group address-group RUCKUS-AP address 'xxx.xxx.138.243'
set firewall ipv6-receive-redirects 'disable'
set firewall ipv6-src-route 'disable'
set firewall ip-src-route 'disable'
set firewall log-martians 'enable'
set firewall name management-filter default-action 'drop'
set firewall name management-filter rule 10 action 'accept'
set firewall name management-filter rule 10 protocol 'all'
set firewall name management-filter rule 10 source group address-group 'RUCKUS-AP'
set firewall name management-filter rule 20 action 'accept'
set firewall name management-filter rule 20 destination port '22'
set firewall name management-filter rule 20 protocol 'tcp'
set firewall name management-filter rule 20 source group address-group 'MANAGEMENT-HOSTS'
set firewall name management-filter rule 30 action 'accept'
set firewall name management-filter rule 30 protocol 'icmp'
set firewall name management-filter rule 30 source group address-group 'MANAGEMENT-HOSTS'
set firewall name management-filter rule 40 action 'accept'
set firewall name management-filter rule 40 destination port '161'
set firewall name management-filter rule 40 protocol 'udp'
set firewall name management-filter rule 40 source group address-group 'MANAGEMENT-HOSTS'
set firewall name management-filter rule 50 action 'accept'
set firewall name management-filter rule 50 protocol 'udp'
set firewall name management-filter rule 50 source group address-group 'MANAGEMENT-HOSTS'
set firewall name management-filter rule 50 source port '123'
set firewall name management-filter rule 60 action 'accept'
set firewall name management-filter rule 60 protocol 'udp'
set firewall name management-filter rule 60 source group address-group 'MANAGEMENT-HOSTS'
set firewall name management-filter rule 60 source port '53'
set firewall name management-filter rule 70 action 'accept'
set firewall name management-filter rule 70 protocol 'tcp'
set firewall name management-filter rule 70 source group address-group 'MANAGEMENT-HOSTS'
set firewall name management-filter rule 70 source port '443'
set firewall name management-filter rule 9999 action 'drop'
set firewall name management-filter rule 9999 log 'enable'
set firewall receive-redirects 'disable'
set firewall send-redirects 'enable'
set firewall source-validation 'disable'
set firewall syn-cookies 'enable'
set firewall twa-hazards-protection 'disable'
set interfaces bridge br0 aging '300'
set interfaces bridge br0 description '*** GRE-TUNNEL to eth1 802.1Q INTERFACE ***'
set interfaces bridge br0 hello-time '2'
set interfaces bridge br0 ip
set interfaces bridge br0 max-age '20'
set interfaces bridge br0 member interface eth1
set interfaces bridge br0 member interface tun0
set interfaces bridge br0 member interface tun1
set interfaces bridge br0 member interface tun2
set interfaces bridge br0 member interface tun3
set interfaces bridge br0 member interface tun4
set interfaces bridge br0 member interface tun5
set interfaces bridge br0 member interface tun6
set interfaces bridge br0 member interface tun7
set interfaces ethernet eth0 address 'xxx.xxx.3.239/24'
set interfaces ethernet eth0 address 'xxx.xxx.3.107/24'
set interfaces ethernet eth0 description '*** GRE-TUNNEL LOCAL-IP INTERFACE ***'
set interfaces ethernet eth0 firewall local name 'management-filter'
set interfaces ethernet eth0 hw-id 'XX:XX:XX:XX:XX:e3'
set interfaces ethernet eth1 description '*** GRE-TUNNEL BRIDGE 802.1Q INTERFACE ***'
set interfaces ethernet eth1 hw-id 'XX:XX:XX:XX:XX:ed'
set interfaces loopback lo
set interfaces tunnel tun0 description '*** SITE1AP001 ***'
set interfaces tunnel tun0 encapsulation 'gre-bridge'
set interfaces tunnel tun0 ip enable-arp-accept
set interfaces tunnel tun0 ip enable-arp-announce
set interfaces tunnel tun0 local-ip 'xxx.xxx.3.239'
set interfaces tunnel tun0 remote-ip 'xxx.xxx.32.240'
set interfaces tunnel tun1 description '*** SITE1AP002 ***'
set interfaces tunnel tun1 encapsulation 'gre-bridge'
set interfaces tunnel tun1 ip enable-arp-accept
set interfaces tunnel tun1 ip enable-arp-announce
set interfaces tunnel tun1 local-ip 'xxx.xxx.3.239'
set interfaces tunnel tun1 remote-ip 'xxx.xxx.32.241'
set interfaces tunnel tun2 description '*** SITE1AP003 ***'
set interfaces tunnel tun2 encapsulation 'gre-bridge'
set interfaces tunnel tun2 ip enable-arp-accept
set interfaces tunnel tun2 ip enable-arp-announce
set interfaces tunnel tun2 local-ip 'xxx.xxx.3.239'
set interfaces tunnel tun2 remote-ip 'xxx.xxx.32.242'
set interfaces tunnel tun3 description '*** SITE1AP004 ***'
set interfaces tunnel tun3 encapsulation 'gre-bridge'
set interfaces tunnel tun3 ip enable-arp-accept
set interfaces tunnel tun3 ip enable-arp-announce
set interfaces tunnel tun3 local-ip 'xxx.xxx.3.239'
set interfaces tunnel tun3 remote-ip 'xxx.xxx.32.243'
set interfaces tunnel tun4 description '*** SITE2AP001 ***'
set interfaces tunnel tun4 encapsulation 'gre-bridge'
set interfaces tunnel tun4 ip enable-arp-accept
set interfaces tunnel tun4 ip enable-arp-announce
set interfaces tunnel tun4 local-ip 'xxx.xxx.3.107'
set interfaces tunnel tun4 remote-ip 'xxx.xxx.138.240'
set interfaces tunnel tun5 description '*** SITE2AP002 ***'
set interfaces tunnel tun5 encapsulation 'gre-bridge'
set interfaces tunnel tun5 ip enable-arp-accept
set interfaces tunnel tun5 ip enable-arp-announce
set interfaces tunnel tun5 local-ip 'xxx.xxx.3.107'
set interfaces tunnel tun5 remote-ip 'xxx.xxx.138.241'
set interfaces tunnel tun6 description '*** SITE2AP003 ***'
set interfaces tunnel tun6 encapsulation 'gre-bridge'
set interfaces tunnel tun6 ip enable-arp-accept
set interfaces tunnel tun6 ip enable-arp-announce
set interfaces tunnel tun6 local-ip 'xxx.xxx.3.107'
set interfaces tunnel tun6 remote-ip 'xxx.xxx.138.242'
set interfaces tunnel tun7 description '*** SITE2AP004 ***'
set interfaces tunnel tun7 encapsulation 'gre-bridge'
set interfaces tunnel tun7 ip enable-arp-accept
set interfaces tunnel tun7 ip enable-arp-announce
set interfaces tunnel tun7 local-ip 'xxx.xxx.3.107'
set interfaces tunnel tun7 remote-ip 'xxx.xxx.138.243'
set protocols static route xxx.xxx.0.0/0 next-hop xxx.xxx.3.103
set protocols static route xxx.xxx.0.0/16 next-hop xxx.xxx.3.1
set service lldp interface eth0
set service lldp legacy-protocols cdp
set service lldp management-address 'xxx.xxx.3.239'
set service lldp snmp enable
set service snmp contact 'bob@example.com'
set service snmp listen-address xxx.xxx.3.239
set service snmp location xxxxxx - 'site'
set service snmp v3 group nms mode 'ro'
set service snmp v3 group nms seclevel 'priv'
set service snmp v3 group nms view 'view1'
set service snmp v3 user xxxxxx auth plaintext-key 'xxxxxx'
set service snmp v3 user xxxxxx auth type 'sha'
set service snmp v3 user xxxxxx group 'nms'
set service snmp v3 user xxxxxx mode 'ro'
set service snmp v3 user xxxxxx privacy plaintext-key 'xxxxxx'
set service snmp v3 user xxxxxx privacy type 'aes'
set service snmp v3 view view1 oid xxx.xxx.6.1
set service ssh listen-address 'xxx.xxx.3.239'
set system config-management commit-revisions '100'
set system console device ttyS0 speed '115200'
set system host-name xxxxxx
set system login user xxxxxx authentication encrypted-password xxxxxx
set system login user xxxxxx authentication plaintext-password xxxxxx
set system login user xxxxxx authentication public-keys xxxx@xxx.xxx key xxxxxx
set system login user xxxxxx authentication public-keys xxxx@xxx.xxx type ssh-xxx
set system login user xxxxxx authentication encrypted-password xxxxxx
set system login user xxxxxx authentication plaintext-password xxxxxx
set system name-server 'xxx.xxx.10.237'
set system name-server 'xxx.xxx.26.237'
set system ntp server xxxxx.tld
set system ntp server xxxxx.tld
set system ntp server xxxxx.tld
set system ntp server xxxxx.tld
set system syslog global facility all level 'info'
set system syslog global facility protocols level 'debug'
set system time-zone 'Europe/London'

So effectively, it’s losing the following -

set interfaces bridge br0 aging '300'
set interfaces bridge br0 description '*** GRE-TUNNEL to eth1 802.1Q INTERFACE ***'
set interfaces bridge br0 hello-time '2'
set interfaces bridge br0 ip
set interfaces bridge br0 max-age '20'
set interfaces bridge br0 member interface eth1
set interfaces bridge br0 member interface tun0
set interfaces bridge br0 member interface tun1
set interfaces bridge br0 member interface tun2
set interfaces bridge br0 member interface tun3
set interfaces bridge br0 member interface tun4
set interfaces bridge br0 member interface tun5
set interfaces bridge br0 member interface tun6
set interfaces bridge br0 member interface tun7
set interfaces tunnel tun0 description '*** SITE1AP001 ***'
set interfaces tunnel tun0 encapsulation 'gre-bridge'
set interfaces tunnel tun0 local-ip 'X.X.3.239'
set interfaces tunnel tun0 remote-ip 'X.X.32.240'
set interfaces tunnel tun1 description '*** SITE1AP002 ***'
set interfaces tunnel tun1 encapsulation 'gre-bridge'
set interfaces tunnel tun1 local-ip 'X.X.3.239'
set interfaces tunnel tun1 remote-ip 'X.X.32.241'
set interfaces tunnel tun2 description '*** SITE1AP003 ***'
set interfaces tunnel tun2 encapsulation 'gre-bridge'
set interfaces tunnel tun2 local-ip 'X.X.3.239'
set interfaces tunnel tun2 remote-ip 'X.X.32.242'
set interfaces tunnel tun3 description '*** SITE1AP004 ***'
set interfaces tunnel tun3 encapsulation 'gre-bridge'
set interfaces tunnel tun3 local-ip 'X.X.3.239'
set interfaces tunnel tun3 remote-ip 'X.X.32.243'
set interfaces tunnel tun4 description '*** SITE2AP001 ***'
set interfaces tunnel tun4 encapsulation 'gre-bridge'
set interfaces tunnel tun4 local-ip 'X.X.3.107'
set interfaces tunnel tun4 remote-ip 'X.X.138.240'
set interfaces tunnel tun5 description '*** SITE2AP002 ***'
set interfaces tunnel tun5 encapsulation 'gre-bridge'
set interfaces tunnel tun5 local-ip 'X.X.3.107'
set interfaces tunnel tun5 remote-ip 'X.X.138.241'
set interfaces tunnel tun6 description '*** SITE2AP003 ***'
set interfaces tunnel tun6 encapsulation 'gre-bridge'
set interfaces tunnel tun6 local-ip 'X.X.3.107'
set interfaces tunnel tun6 remote-ip 'X.X.138.242'
set interfaces tunnel tun7 description '*** SITE2AP004 ***'
set interfaces tunnel tun7 encapsulation 'gre-bridge'
set interfaces tunnel tun7 local-ip 'X.X.3.107'
set interfaces tunnel tun7 remote-ip 'X.X.138.243'

I actually saw similar issues when moving to a later Rolling release, but had reverted the version for reasons of stability. As the device had to be shutdown tonight, the issue has come to a head.

The version currently running is -

Version:          VyOS 1.3-rolling-202005100117
Release Train:    equuleus
Built by:         autobuild@vyos.net
Built on:         Sun 10 May 2020 01:17 UTC
Build UUID:       aa1e03bc-7360-40b3-a58c-cdae46958353
Build Commit ID:  2cb6f390d7bdaa
Architecture:     x86_64
Boot via:         installed image
System type:      VMware guest
Hardware vendor:  VMware, Inc.
Hardware model:   VMware Virtual Platform
Hardware S/N:     VMware-56 4d 74 5b 44 7a 86 d5-20 40 d5 7f a1 cd 36 e3
Hardware UUID:    564d745b-447a-86d5-2040-d57fa1cd36e3

Any ideas on where to look would be greatly appreciated.

Best regards
Andy

2

Hi @millap
Can you test it on the latest rolling?
And check your /config/config.boot file. Is it contain necessary strings?

3

Hi @Viacheslav,

Thanks for the response. I’ll certainly try the latest rolling, albeit I have done that before and lost the configuration.

One thing I’ve found since the original post is that the issue was exclusively to do with the firewall configuration. There was a console message on the VM at boot about the vyatta FW migration script, and that it had failed. As soon as I removed all firewall configuration, the brX and tunX interface configuration remained after a reboot. I’ve then added it back on and rebooted again, and the brX and tunX still remained. Very strange.

Will report back once the upgrade is done.

Andy

4

Hi @Viacheslav,

No luck with the rolling upgrade I’m afraid. In-fact, whilst the tunX and brX interfaces stayed, when I rebooted, it lost a user configuration for a user which uses key-based logins.

The error I see on the console is -

vyos-fw-error
vyos-fw-error720×57 1.22 KB

I’ve rolled back to the previous release now (1.3-rolling-202005100117).

Best regards
Andy

5

@millap thanks for configuration.
I found one bug

set interfaces tunnel tun0 description '*** SITE1 ***'
set interfaces tunnel tun0 encapsulation 'gre-bridge'
set interfaces tunnel tun0 ip enable-arp-accept
set interfaces tunnel tun0 ip enable-arp-announce
set interfaces tunnel tun0 local-ip '10.0.3.239'
set interfaces tunnel tun0 remote-ip '10.0.32.240'
commit
[ interfaces tunnel tun0 ip enable-arp-accept ]
sh: /proc/sys/net/ipv4/conf/tun0/arp_accept: No such file or directory

[[interfaces tunnel tun0]] failed
Commit failed

I created a bug report, ref. ⚓ T2633 Error with arp_accept on tun interface

6

This bug was fixed in the latest rolling release.
@millap Can you check it?

7

Hi @Viacheslav,

Looks good, I’ve tested on two separate VyOS installs where we’re using SoftGRE tunnels. Thanks a lot for the prompt bug/fix.

Andy

10