Wireguard not functioning on latest builds

Korikaze · January 23, 2022, 2:18am

I’ve had some issues with upgrading to some of the very latest rolling release builds which have nuked sections of my config, mainly firewall, vpn, and ssh. I decided to try just rebuilding those sections to what they were previously and it’s back to how it was. The last remaining thing though is that Wireguard is just not functioning at all with the same config. Both of the two interfaces I have just refuse to handshake even after I completely recreate them and trying different keys.

Previous version: 1.4-rolling-202108291448
Current version:

Version:          VyOS 1.4-rolling-202201211359
Release train:    sagitta

Built by:         autobuild@vyos.net
Built on:         Fri 21 Jan 2022 13:59 UTC
Build UUID:       b78b0ec3-7e13-49e7-96ce-129c8434b139
Build commit ID:  86b750c3f9c002

Architecture:     x86_64
Boot via:         installed image
System type:      bare metal

Hardware vendor:  PC Engines
Hardware model:   apu4
Hardware S/N:     1413086
Hardware UUID:    Unknown

Copyright:        VyOS maintainers and contributors

I noticed some relevant messages in the logs, not sure if they actually are…

Jan 22 20:37:32 vyos vyos-configd[571]: Received message: {"type": "node", "data": "VYOS_TAGNODE_VALUE=wg1/usr/libexec/vyos/conf_mode/interfaces-wireguard.py"}
Jan 22 20:37:32 vyos vyos-configd[571]: module 'interfaces_wireguard' has no attribute 'generate'

I wanted to try the newer builds since there were some enhancements to the containers feature but unfortunately I keep having this issue on all of the latest builds.

c-po · January 23, 2022, 7:33am

Hi @Korikaze,

VyOS 1.4 firewall is currently under heavy rewrite from iptables to nftables and also from Perl to Python. Would it be possible for you to share your configuration with us so we have ways to reproduce your bugs to fix them.

Thanks,
Christian

Korikaze · January 23, 2022, 3:05pm

Thanks for the heads up @c-po

Here is the working config prior to the upgrade, sanitized.

set firewall all-ping 'enable'
set firewall broadcast-ping 'disable'
set firewall config-trap 'disable'
set firewall group address-group vpn address '10.20.2.1-10.20.2.254'
set firewall group network-group inside network '10.20.0.0/16'
set firewall group network-group inside network '10.37.98.0/30'
set firewall group port-group ipsec port '500'
set firewall group port-group ipsec port '4500'
set firewall group port-group ipsec port '50'
set firewall group port-group ipsec port '51'
set firewall group port-group mail port '25'
set firewall group port-group mail port '587'
set firewall group port-group mail port '993'
set firewall group port-group mail port '995'
set firewall group port-group mail port '4190'
set firewall group port-group mail port '143'
set firewall group port-group mail port '465'
set firewall group port-group mail port '110'
set firewall group port-group plex port '32400'
set firewall group port-group web port '80'
set firewall group port-group web port '443'
set firewall ipv6-receive-redirects 'disable'
set firewall ipv6-src-route 'disable'
set firewall ip-src-route 'disable'
set firewall log-martians 'enable'
set firewall name hub-in default-action 'drop'
set firewall name hub-in rule 10 action 'accept'
set firewall name hub-in rule 10 state established 'enable'
set firewall name hub-in rule 10 state related 'enable'
set firewall name hub-in rule 11 action 'drop'
set firewall name hub-in rule 11 state invalid 'enable'
set firewall name hub-in rule 20 action 'accept'
set firewall name hub-in rule 20 destination group port-group 'mail'
set firewall name hub-in rule 20 protocol 'tcp'
set firewall name hub-in rule 20 state new 'enable'
set firewall name hub-in rule 30 action 'accept'
set firewall name hub-in rule 30 destination group port-group 'web'
set firewall name hub-in rule 30 protocol 'tcp'
set firewall name hub-in rule 30 state new 'enable'
set firewall name hub-local default-action 'drop'
set firewall name hub-local rule 10 action 'accept'
set firewall name hub-local rule 10 state established 'enable'
set firewall name hub-local rule 10 state related 'enable'
set firewall name hub-local rule 11 action 'drop'
set firewall name hub-local rule 11 state invalid 'enable'
set firewall name out-in default-action 'drop'
set firewall name out-in rule 10 action 'accept'
set firewall name out-in rule 10 state established 'enable'
set firewall name out-in rule 10 state related 'enable'
set firewall name out-in rule 11 action 'drop'
set firewall name out-in rule 11 state invalid 'enable'
set firewall name out-in rule 32 action 'accept'
set firewall name out-in rule 32 destination address '10.20.1.71'
set firewall name out-in rule 32 destination port '32400'
set firewall name out-in rule 32 protocol 'tcp'
set firewall name out-in rule 32 state new 'enable'
set firewall name out-local default-action 'drop'
set firewall name out-local rule 10 action 'accept'
set firewall name out-local rule 10 state established 'enable'
set firewall name out-local rule 10 state related 'enable'
set firewall name out-local rule 11 action 'drop'
set firewall name out-local rule 11 state invalid 'enable'
set firewall name wg-in default-action 'drop'
set firewall name wg-in rule 10 action 'accept'
set firewall name wg-in rule 10 state established 'enable'
set firewall name wg-in rule 10 state related 'enable'
set firewall name wg-in rule 11 action 'drop'
set firewall name wg-in rule 11 state invalid 'enable'
set firewall name wg-in rule 40 action 'accept'
set firewall name wg-in rule 40 destination port '56999'
set firewall name wg-in rule 40 protocol 'tcp'
set firewall name wg-in rule 40 state new 'enable'
set firewall name wg-local default-action 'drop'
set firewall name wg-local rule 10 action 'accept'
set firewall name wg-local rule 10 state established 'enable'
set firewall name wg-local rule 10 state related 'enable'
set firewall name wg-local rule 11 action 'drop'
set firewall name wg-local rule 11 state invalid 'enable'
set firewall receive-redirects 'disable'
set firewall send-redirects 'enable'
set firewall source-validation 'disable'
set firewall syn-cookies 'enable'
set firewall twa-hazards-protection 'disable'
set interfaces ethernet eth0 address '192.168.1.4/24'
set interfaces ethernet eth0 description 'WAN'
set interfaces ethernet eth0 firewall in name 'out-in'
set interfaces ethernet eth0 firewall local name 'out-local'
set interfaces ethernet eth0 hw-id '00:0d:b9:56:40:78'
set interfaces ethernet eth0 offload gro
set interfaces ethernet eth0 offload gso
set interfaces ethernet eth0 offload sg
set interfaces ethernet eth0 offload tso
set interfaces ethernet eth0 ring-buffer rx '4096'
set interfaces ethernet eth0 ring-buffer tx '4096'
set interfaces ethernet eth1 description 'LAN'
set interfaces ethernet eth1 hw-id '00:0d:b9:56:40:79'
set interfaces ethernet eth1 offload gro
set interfaces ethernet eth1 offload gso
set interfaces ethernet eth1 offload sg
set interfaces ethernet eth1 offload tso
set interfaces ethernet eth1 ring-buffer rx '4096'
set interfaces ethernet eth1 ring-buffer tx '4096'
set interfaces ethernet eth1 vif 20 address '10.20.1.1/16'
set interfaces ethernet eth1 vif 20 policy route 'default'
set interfaces ethernet eth1 vif 37 address '10.37.98.1/30'
set interfaces ethernet eth1 vif 37 policy route 'mail'
set interfaces ethernet eth1 vif 59 address '10.59.90.1/24'
set interfaces ethernet eth1 vif 59 policy route 'vpn'
set interfaces ethernet eth2 hw-id '00:0d:b9:56:40:7a'
set interfaces ethernet eth2 offload gro
set interfaces ethernet eth2 offload gso
set interfaces ethernet eth2 offload sg
set interfaces ethernet eth2 offload tso
set interfaces ethernet eth2 ring-buffer rx '4096'
set interfaces ethernet eth2 ring-buffer tx '4096'
set interfaces ethernet eth3 hw-id '00:0d:b9:56:40:7b'
set interfaces ethernet eth3 ring-buffer rx '4096'
set interfaces ethernet eth3 ring-buffer tx '4096'
set interfaces loopback lo
set interfaces wireguard wg0 address '10.67.151.170/32'
set interfaces wireguard wg0 description 'WG Tunnel'
set interfaces wireguard wg0 firewall in name 'wg-in'
set interfaces wireguard wg0 firewall local name 'wg-local'
set interfaces wireguard wg0 peer mullvad address '1.2.3.4'
set interfaces wireguard wg0 peer mullvad allowed-ips '0.0.0.0/0'
set interfaces wireguard wg0 peer mullvad persistent-keepalive '25'
set interfaces wireguard wg0 peer mullvad port '51820'
set interfaces wireguard wg0 peer mullvad public-key 'pubkey0'
set interfaces wireguard wg0 private-key 'privkey1'
set interfaces wireguard wg1 address '10.98.0.2/30'
set interfaces wireguard wg1 description 'VPN-to-HUB'
set interfaces wireguard wg1 peer to-HUB address '1.2.3.5'
set interfaces wireguard wg1 peer to-HUB allowed-ips '0.0.0.0/0'
set interfaces wireguard wg1 peer to-HUB persistent-keepalive '25'
set interfaces wireguard wg1 peer to-HUB port '51820'
set interfaces wireguard wg1 peer to-HUB public-key 'pubkey1'
set interfaces wireguard wg1 private-key 'privkey1'
set nat destination rule 32 description 'Plex'
set nat destination rule 32 destination port '32400'
set nat destination rule 32 inbound-interface 'eth0'
set nat destination rule 32 protocol 'tcp'
set nat destination rule 32 translation address '10.20.1.71'
set nat destination rule 40 destination port '56999'
set nat destination rule 40 inbound-interface 'wg0'
set nat destination rule 40 protocol 'udp'
set nat destination rule 40 translation address '10.59.90.2'
set nat source rule 20 outbound-interface 'wg0'
set nat source rule 20 translation address 'masquerade'
set policy route default description 'Default route for lab'
set policy route default rule 20 set table '11'
set policy route mail rule 10 set table '37'
set policy route vpn description 'Routes hosts in the VLAN 59 over Mullvad VPN'
set policy route vpn rule 10 set table '59'
set protocols static route 0.0.0.0/0 next-hop 192.168.1.1
set protocols static table 11 route 0.0.0.0/0 next-hop 192.168.1.1
set protocols static table 11 route 10.20.0.0/16 interface eth1.20
set protocols static table 11 route 10.37.98.0/30 interface eth1.37
set protocols static table 11 route 10.59.90.0/24 interface eth1.59
set protocols static table 11 route 10.98.0.0/30 interface wg1
set protocols static table 11 route 192.168.1.0/24 next-hop 192.168.1.1
set protocols static table 37 route 0.0.0.0/0 interface wg1
set protocols static table 37 route 10.20.0.0/16 interface eth1.20
set protocols static table 37 route 10.37.98.0/30 interface eth1.37
set protocols static table 40 route 10.20.0.0/16 interface eth1.20
set protocols static table 40 route 10.37.98.0/30 interface eth1.37
set protocols static table 59 route 0.0.0.0/0 interface wg0
set protocols static table 59 route 10.20.0.0/16 interface eth1.20
set protocols static table 59 route 10.59.90.0/24 interface eth1.59
set service dhcp-server shared-network-name lan authoritative
set service dhcp-server shared-network-name lan subnet 10.20.0.0/16 default-router '10.20.1.1'
set service dhcp-server shared-network-name lan subnet 10.20.0.0/16 dns-server '10.20.0.15'
set service dhcp-server shared-network-name lan subnet 10.20.0.0/16 dns-server '10.20.0.16'
set service dhcp-server shared-network-name lan subnet 10.20.0.0/16 domain-name 'lab.local'
set service dhcp-server shared-network-name lan subnet 10.20.0.0/16 domain-search 'lab.local'
set service dhcp-server shared-network-name lan subnet 10.20.0.0/16 lease '86400'
set service dhcp-server shared-network-name lan subnet 10.20.0.0/16 range 0 start '10.20.3.1'
set service dhcp-server shared-network-name lan subnet 10.20.0.0/16 range 0 stop '10.20.3.254'
set service dns
set service snmp community routers authorization 'ro'
set service snmp community routers network '10.20.0.0/16'
set service snmp listen-address 10.20.1.1 port '161'
set service ssh access-control allow user 'myuser'
set service ssh access-control deny user 'vyos'
set service ssh disable-password-authentication
set service ssh port '22'
set system config-management commit-archive location 'tftp://10.20.1.50'
set system config-management commit-archive source-address '10.20.1.1'
set system config-management commit-revisions '100'
set system conntrack modules ftp
set system conntrack modules h323
set system conntrack modules nfs
set system conntrack modules pptp
set system conntrack modules sip
set system conntrack modules sqlnet
set system conntrack modules tftp
set system console device ttyS0 speed '115200'
set system host-name 'vyos'
set system login user mysuer authentication encrypted-password 'encryptedpassword'
set system login user myuser authentication plaintext-password ''
set system login user myuser authentication public-keys user@pc key 'sshpubkey'
set system login user myuser authentication public-keys user@pc type 'ssh-rsa'
set system name-server '10.20.0.15'
set system name-server '10.20.0.16'
set system ntp server 0.pool.ntp.org
set system ntp server 1.pool.ntp.org
set system ntp server 2.pool.ntp.org
set system syslog global facility all level 'info'
set system syslog global facility protocols level 'debug'
set system time-zone 'America/New_York'

I will say a brand new install with one the of the latest builds mostly works from experience with a VPS instance of VyOS I have, except I’ve had happen on one occasion so far where most of the ‘accept’ rules didn’t actually function after a reboot and would just drop instead. Deleting and reapplying to the interface fixed that issue. Here is the configuration for that one as well.

set firewall all-ping 'enable'
set firewall broadcast-ping 'disable'
set firewall config-trap 'disable'
set firewall group port-group ipsec port '500'
set firewall group port-group ipsec port '1701'
set firewall group port-group ipsec port '4500'
set firewall group port-group ipsec port '50'
set firewall group port-group mail port '25'
set firewall group port-group mail port '587'
set firewall group port-group mail port '993'
set firewall group port-group mail port '995'
set firewall group port-group mail port '4190'
set firewall group port-group mail port '465'
set firewall group port-group mail port '143'
set firewall group port-group mail port '110'
set firewall group port-group web port '80'
set firewall group port-group web port '443'
set firewall ipv6-receive-redirects 'disable'
set firewall ipv6-src-route 'disable'
set firewall ip-src-route 'disable'
set firewall log-martians 'enable'
set firewall name WILD-IN default-action 'drop'
set firewall name WILD-IN rule 10 action 'accept'
set firewall name WILD-IN rule 10 state established 'enable'
set firewall name WILD-IN rule 10 state related 'enable'
set firewall name WILD-IN rule 11 action 'drop'
set firewall name WILD-IN rule 11 state invalid 'enable'
set firewall name WILD-IN rule 20 action 'accept'
set firewall name WILD-IN rule 20 source address '172.17.200.177/32'
set firewall name WILD-IN rule 20 state new 'enable'
set firewall name WILD-IN rule 30 action 'accept'
set firewall name WILD-IN rule 30 destination group port-group 'mail'
set firewall name WILD-IN rule 30 protocol 'tcp'
set firewall name WILD-IN rule 30 state new 'enable'
set firewall name WILD-IN rule 40 action 'accept'
set firewall name WILD-IN rule 40 destination group port-group 'web'
set firewall name WILD-IN rule 40 protocol 'tcp'
set firewall name WILD-IN rule 40 state new 'enable'
set firewall name WILD-LOCAL default-action 'drop'
set firewall name WILD-LOCAL rule 10 action 'accept'
set firewall name WILD-LOCAL rule 10 state established 'enable'
set firewall name WILD-LOCAL rule 10 state related 'enable'
set firewall name WILD-LOCAL rule 11 action 'drop'
set firewall name WILD-LOCAL rule 11 state invalid 'enable'
set firewall name WILD-LOCAL rule 20 action 'accept'
set firewall name WILD-LOCAL rule 20 icmp type-name 'echo-request'
set firewall name WILD-LOCAL rule 20 protocol 'icmp'
set firewall name WILD-LOCAL rule 20 state new 'enable'
set firewall name WILD-LOCAL rule 31 action 'accept'
set firewall name WILD-LOCAL rule 31 destination port '22'
set firewall name WILD-LOCAL rule 31 protocol 'tcp'
set firewall name WILD-LOCAL rule 31 state new 'enable'
set firewall name WILD-LOCAL rule 40 action 'accept'
set firewall name WILD-LOCAL rule 40 description 'Allow Wireguard'
set firewall name WILD-LOCAL rule 40 destination port '51820'
set firewall name WILD-LOCAL rule 40 log 'enable'
set firewall name WILD-LOCAL rule 40 protocol 'udp'
set firewall name WILD-LOCAL rule 40 source
set firewall name WILD-LOCAL rule 50 action 'accept'
set firewall name WILD-LOCAL rule 50 destination group port-group 'ipsec'
set firewall name WILD-LOCAL rule 50 ipsec match-ipsec
set firewall name WILD-LOCAL rule 50 protocol 'udp'
set firewall name WILD-LOCAL rule 50 state new 'enable'
set firewall receive-redirects 'disable'
set firewall send-redirects 'enable'
set firewall source-validation 'disable'
set firewall syn-cookies 'enable'
set firewall twa-hazards-protection 'disable'
set interfaces dummy dum0 address '10.100.100.1/24'
set interfaces dummy dum1 address '10.100.200.1/24'
set interfaces ethernet eth0 address 'dhcp'
set interfaces ethernet eth0 description 'Wild Wild West'
set interfaces ethernet eth0 firewall in name 'WILD-IN'
set interfaces ethernet eth0 firewall local name 'WILD-LOCAL'
set interfaces ethernet eth0 hw-id '00:16:ee:7d:59:1f'
set interfaces loopback lo
set interfaces wireguard wg0 address '10.98.0.1/30'
set interfaces wireguard wg0 description 'VPN-to-HOME'
set interfaces wireguard wg0 peer to-HOME allowed-ips '0.0.0.0/0'
set interfaces wireguard wg0 peer to-HOME public-key 'pubkey0'
set interfaces wireguard wg0 port '51820'
set interfaces wireguard wg0 private-key 'privkey0'
set nat destination rule 50 destination port '25,587,993,995,4190,143,465,110,80,443'
set nat destination rule 50 inbound-interface 'eth0'
set nat destination rule 50 protocol 'tcp'
set nat destination rule 50 translation address '10.37.98.2'
set nat source rule 8 destination address '10.37.98.0/30'
set nat source rule 8 outbound-interface 'wg0'
set nat source rule 8 protocol 'tcp'
set nat source rule 8 source address '10.37.98.0/30'
set nat source rule 8 translation address 'masquerade'
set nat source rule 9 destination address '172.17.200.177/32'
set nat source rule 9 exclude
set nat source rule 9 outbound-interface 'eth0'
set nat source rule 10 outbound-interface 'eth0'
set nat source rule 10 translation address 'masquerade'
set protocols static route 0.0.0.0/0 next-hop x.x.x.x
set protocols static route 10.37.98.0/30 interface wg0
set service ssh disable-password-authentication
set service ssh port '22'
set system config-management commit-revisions '100'
set system conntrack modules ftp
set system conntrack modules h323
set system conntrack modules nfs
set system conntrack modules pptp
set system conntrack modules sip
set system conntrack modules sqlnet
set system conntrack modules tftp
set system console device ttyS0 speed '115200'
set system domain-name 'tooeffayy.tech'
set system host-name 'hub'
set system login user vy authentication encrypted-password 'encyptedpass'
set system login user vy authentication public-keys user@pc key 'sshpubkey'
set system login user vy authentication public-keys user@pc type 'ssh-rsa'
set system name-server '9.9.9.9'
set system ntp server time1.vyos.net
set system ntp server time2.vyos.net
set system ntp server time3.vyos.net
set system option performance 'latency'
set system syslog global facility all level 'info'
set system syslog global facility protocols level 'debug'

c-po · January 25, 2022, 6:21am

I have loaded your configuration on VyOS 1.3.0 and upgraded to VyOS 1.4-rolling.

I also created a Vyos 1.4 → 1.2.8 WireGuard peering and also could not find what it’s working. Can you please add more details?

Korikaze · January 26, 2022, 1:23pm

All I have done is add system image the latest rolling release. After rebooting to the new image, I have this output:

[  OK  ] Finished OpenBSD Secure Shell session cleanup.
[   61.608950] vyos-router[830]: Waiting for NICs to settle down: settled in 0sec..
[   77.034731] vyos-router[945]: Started watchfrr.
[   90.504207] vyos-router[830]: Mounting VyOS Config...done.
[   93.056136] vyos-router[830]: Starting VyOS router: migrate
[   93.065672] vyos-router[1361]: Could not find wireguard private key for migration on interface "wg0"
[   93.077577] vyos-router[1361]: Could not find wireguard private key for migration on interface "wg1"
[  257.372559] vyos-router[830]:  configure
[  257.381201] vyos-router[3262]:  failed!
[  258.203497] vyos-config[3140]: Configuration error

Despite the messages about the wireguard keys, they appear to be working fine this time. However the firewall config is completely gone.

myuser@vyos:~$ show conf
interfaces {
    ethernet eth0 {
        address 192.168.1.4/24
        description WAN
        firewall {
            in {
                name out-in
            }
            local {
                name out-local
            }
        }
        hw-id 00:0d:b9:56:40:78
        offload {
            gro
            gso
            sg
            tso
        }
        ring-buffer {
            rx 4096
            tx 4096
        }

SSH config also gone:

myuser@vyos# sh service
 dhcp-server {
     shared-network-name lan {
         authoritative
         subnet 10.20.0.0/16 {
             default-router 10.20.1.1
             domain-name lab.local
             domain-search lab.local
             lease 86400
             name-server 10.20.0.15
             name-server 10.20.0.16
             range 0 {
                 start 10.20.3.1
                 stop 10.20.3.254
             }
         }
     }
 }
 dns {
 }
 snmp {
     community routers {
         authorization ro
         network 10.20.0.0/16
     }
     listen-address 10.20.1.1 {
         port 161
     }
 }