OpenVPN routing not working after 1.4 18th June 2023 release

vy-testing · October 10, 2023, 2:14am

hi, long time testing user of VyOS form old school Vyatta days. Love you work as always!

I have a virtual router being used for testing on 1.4-rolling-202207080217, that we test OpenVPN and VyOS on.

This has been working fine and continues to run fine.
The log file confirms at this point the tunnel is up and the routes are pushed down to the client with no issues.

2023-10-10 08:47:45 Blocking outside dns using service succeeded.
2023-10-10 08:47:50 Initialization Sequence Completed

I attempt to update this to the latest 1.5 rolling release and everything came up fine except no traffic is routing properly;
and errors appear in the OpenVPN client (windows 11);

2023-10-05 10:58:02 Blocking outside dns using service succeeded.
2023-10-05 10:58:37 Warning: route gateway is not reachable on any active network adapters: 10.10.0.1

This is strange because at this stage, I’m able to ping 10.10.0.1 from the client PC.

I wanted to find out what date the error started so found a site that pushes nightly builds that I hoped could help me to match the failures to a specific build;
https://repo.serverforge.org/vyos/sagitta/

our current config works fine on the release dated vyos-1.4-rolling-202306180707-amd64.iso, but the same config fails on the following build dated;
vyos-1.4-rolling-202306250707-amd64.iso

I’ve checked the release history for Vyos 1.4 that have changed between the 18th June 2023 and 25th June 2023 from here; 1.4 Sagitta — VyOS 1.4.x (sagitta) documentation

I initially thought that perhaps the embedded OpenVPN versions had changed between the releases but this was not the case as both included OpenVPN 2.6.3;

/usr/sbin/openvpn --version
OpenVPN 2.6.3

Our OpenVPN server has multiple interfaces with multiple routes on and we are using things like deferred auth via a custom script using the auth-script-openvpn script.

Before I spend time working on a reduced sanitized version of our /config/config.boot that I’m able to paste in this forum and able to demonstrate the issue, are there any known issues between those releases which would break the OpenVPN routing as it seems strange that the functionality should break on a specific release. As soon as I then downgrade to the 202306180707 release everything works as expected again.

Any help would be appreciated.
thanks.

vy-testing · October 10, 2023, 4:41am

so i compared the release that is broken to the one which isn’t.
I checked the /run/openvpn/vtun0.conf file before and after.
The version that is broken has the following line injected in just before the routing section - whereas the one which works does not have that line in;

server 10.10.0.0 255.255.252.0 nopool

so my question is - how do I prevent VyOS from adding that line into the OpenVPN config file?

Viacheslav · October 10, 2023, 5:31am

We don’t support unknown builds.
And don’t have any relation to this repo. So you use it on your own risk.

Apachez · October 10, 2023, 5:49am

Official nightly builds can be downloaded from:

https://vyos.net/get/nightly-builds/

or directly:

vy-testing · October 11, 2023, 3:59am

thanks, so I get exactly the same issue on the 1.5 nightly - which is why in the first place I wanted to go back to a specific point in time to determine when the issue occurred and because there is no release iso archive on the 1.4 nightlies, the only test I could run was using the iso i found, and the original test proved that the issue was not openvpn version related but rather the VyOS generated config file.
regardless, when I switch to the latest build;

add system image https://github.com/vyos/vyos-rolling-nightly-builds/releases/download/1.5-rolling-202310100022/vyos-1.5-rolling-202310100022-amd64.iso

I then get the following;

vyos@vyos:~$ show version
Version:          VyOS 1.5-rolling-202310100022
Release train:    current

Built by:         autobuild@vyos.net
Built on:         Tue 10 Oct 2023 01:46 UTC

$ /usr/sbin/openvpn --version
OpenVPN 2.6.3 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] [DCO]

vyos@vyos:~$ cat /run/openvpn/vtun0.conf
### Autogenerated by interfaces-openvpn.py ###
#
# See https://community.openvpn.net/openvpn/wiki/Openvpn24ManPage
# for individual keyword definition
#
#
#

verb 3
dev-type tun
dev vtun0
persist-key
proto tcp-server
lport 443
disable-dco

#
# OpenVPN Server mode
#
mode server
tls-server
topology net30
server 10.10.0.0 255.255.252.0
push "route 10.0.10.0 255.255.255.0 10.10.0.1 0"
push "route 10.2.45.24 255.255.255.255 10.10.0.1 0"
push "route 10.20.22.0 255.255.255.0 10.10.0.1 0"
push "route 10.247.0.0 255.255.240.0 10.10.0.1 0"
push "route 172.16.0.0 255.254.0.0 10.10.0.1 0"
push "route 172.18.0.0 255.255.0.0 10.10.0.1 0"
push "route 172.20.0.0 255.252.0.0 10.10.0.1 0"
push "route 172.24.0.0 255.248.0.0 10.10.0.1 0"
push "route 192.168.100.0 255.255.255.0 10.10.0.1 0"

keepalive 10 600
management /run/openvpn/openvpn-mgmt-intf unix

push "dhcp-option DNS 10.247.0.1"

# TLS options
ca /run/openvpn/vtun0_ca.pem
cert /run/openvpn/vtun0_cert.pem
key /run/openvpn/vtun0_cert.key
dh /run/openvpn/vtun0_dh.pem


# Encryption options
# https://vyos.dev/T5027
# Required to support BF-CBC (default ciphername when none given)
providers legacy default

auth sha1

however, on the nightly 1.4 build that we are on previously that works fine, I get the following;

vyos@vyos:~$ show version
Version:          VyOS 1.4-rolling-202207080217
Release train:    sagitta
Built by:         autobuild@vyos.net
Built on:         Fri 08 Jul 2022 02:17 UTC

$ /usr/sbin/openvpn --version
OpenVPN 2.5.1 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on May 14 2021

vyos@vyos:~$ cat /run/openvpn/vtun0.conf
### Autogenerated by interfaces-openvpn.py ###
#
# See https://community.openvpn.net/openvpn/wiki/Openvpn24ManPage
# for individual keyword definition
#
#
#

verb 3
dev-type tun
dev vtun0
persist-key
proto tcp-server
lport 443

#
# OpenVPN Server mode
#
mode server
tls-server
topology net30
server 10.10.0.0 255.255.252.0 nopool
push "route 10.0.10.0 255.255.255.0 "
push "route 10.2.45.24 255.255.255.255 "
push "route 10.20.22.0 255.255.255.0 "
push "route 10.247.0.0 255.255.240.0 "
push "route 172.16.0.0 255.254.0.0 "
push "route 172.18.0.0 255.255.0.0 "
push "route 172.20.0.0 255.252.0.0 "
push "route 172.24.0.0 255.248.0.0 "
push "route 192.168.100.0 255.255.255.0 "
ifconfig-pool 10.10.0.4 10.10.3.253

keepalive 10 600
management /run/openvpn/openvpn-mgmt-intf unix

push "dhcp-option DNS 10.247.0.1"

# TLS options
ca /run/openvpn/vtun0_ca.pem
cert /run/openvpn/vtun0_cert.pem
key /run/openvpn/vtun0_cert.key
dh /run/openvpn/vtun0_dh.pem

# Encryption options

auth sha1

so the difference between the two files that seem to be cause the issue is;
working one;

server 10.10.0.0 255.255.252.0 nopool
ifconfig-pool 10.10.0.4 10.10.3.253

where as the one which is broken

server 10.10.0.0 255.255.252.0

If i try and inject the ifconfig-pool config item I get a message saying;

vyos openvpn-vtun0[2858]: Options error: --server and --server-bridge cannot be used together

Is there a way of injecting the exact openvpn config I need and disabling the vyos specific validation?
I’m assuming the issue is down to the bridging configuration - Are there some configuration examples specifically for 1.5 release that I can look at so that I can experiment with some working 1.5 configurations?
Any thoughts would be appreciated.

Apachez · October 11, 2023, 4:35am

I found these two commits (might have missed some) that might be related (at least they involve that nopool part):

https://vyos.dev/T3214

vy-testing · October 25, 2023, 4:40am

thanks for the advice.
I’ve spent some time simplifying our config down to its simplest form where I’m able to demonstrate the problem. To eliminate the non blocking auth script we were using I’ve created a simple auth script which always returns 0 (our real one triggers a custom radius script in non-blocking mode).

cat > /config/scripts/authscript << 'EOF'
#!/bin/bash
exit 0
EOF
chmod o+x /config/scripts/authscript

If I then run on the nightly dated 202207080217 and check the version, then everything works as expected.

$ show system image
The system currently has the following image(s) installed:
   1: 1.4-rolling-202207080217 (default boot) (running image)

The results of a show config command are below;

interfaces {
    ethernet eth0 {
        address 172.28.27.95/22
        hw-id aa:bb:cc:dd:ee:ff
        offload {
            gro
            gso
            sg
            tso
        }
    }
    loopback lo {
    }
    openvpn vtun0 {
        hash sha1
        local-port 443
        mode server
        openvpn-option "--push block-outside-dns"
        openvpn-option "--auth-user-pass-verify /config/scripts/authscript via-env --script-security 3"
        openvpn-option "--setenv verb 9"
        openvpn-option --duplicate-cn
        openvpn-option "--verify-client-cert none"
        openvpn-option --username-as-common-name
        openvpn-option --persist-key
        openvpn-option --persist-tun
        openvpn-option "--cipher AES-256-GCM"
        openvpn-option "--reneg-sec 0"
        openvpn-option "--inactive 0"
        protocol tcp-passive
        server {
            name-server 8.8.8.8
            push-route 172.16.0.0/15 {
            }
            push-route 172.18.0.0/16 {
            }
            push-route 172.20.0.0/14 {
            }
            push-route 172.24.0.0/13 {
            }
            subnet 10.10.0.0/22
        }
        tls {
            ca-certificate openvpn_vtun0_1
            certificate openvpn_vtun0
            dh-params openvpn_vtun0
        }
    }
}
nat {
    source {
        rule 10 {
            outbound-interface eth0
            source {
                address 10.10.0.0/22
            }
            translation {
                address masquerade
            }
        }
    }
}
pki {
    ca openvpn_vtun0_1 {
        certificate ********
    }
    certificate openvpn_vtun0 {
        certificate ********
        private {
            key ****************
        }
    }
    dh openvpn_vtun0 {
        parameters *********
    }
}
protocols {
    static {
        route 0.0.0.0/0 {
            next-hop 172.28.24.1 {
            }
        }
    }
}
service {
    ssh {
    }
}
system {
    config-management {
        commit-revisions 100
    }
    conntrack {
        modules {
            ftp
            h323
            nfs
            pptp
            sip
            sqlnet
            tftp
        }
    }
    host-name vyos
    login {
        user vyos {
            authentication {
                encrypted-password ****************
            }
        }
    }
    name-server 8.8.8.8
}

I have blanked out our certifcates.
I’m then able to connect to the end point without issue.

I now check the latest nightly build from here and upgrade to it;
https://vyos.net/get/nightly-builds/

add system image https://github.com/vyos/vyos-rolling-nightly-builds/releases/download/1.5-rolling-202310240118/vyos-1.5-rolling-202310240118-amd64.iso

now as soon as I connect to the endpoint I get the same issue as before which is;
“Warning: route gateway is not reachable on any active network adapters: 10.10.0.1”
and the routes don’t get pushed through to the client.

I checked the results of show config from the 1.4 release compared to the 1.5 rolling release and they were the same.

The client.ovpn file looks like this (again with ca blanked)

client
proto tcp
remote 172.28.27.95
connect-timeout 60
connect-retry-max 3
port 443
dev tun
remote-cert-tls server
auth-user-pass
auth-nocache
reneg-sec 0

# ca.crt
<ca>
-----BEGIN CERTIFICATE-----
*****
-----END CERTIFICATE-----
</ca>

Any help would be appreciated.

Apachez · October 25, 2023, 6:48am

Just to verify - are the ca and user/server private keys and certs intact after the upgrade?

Im thinking if the upgrade somehow misses something during the upgrade phase so you end up with lets say a pointer to a cert which no longer exists in the persistence storage created for the upgraded version?

Because VyOS uses filesystem.squashfs from the ISO as a readonly loop partition which it then uses overlayfs to “overwrite” whatever is in the readonly partition with whatever is put in the perstistence directory on your permanent storage (each system image have its own persistence directory).

What you see → overlayfs → persistence storage + readonly filesystem.squashfs.

This persistence directory is populated during “system install” by copying current stuff from /config and place a copy in the new persistence directory. I think stuff in /etc/ssh is also copied but currently not stuff in /home/< username >.

vy-testing · October 25, 2023, 9:01am

thanks, All the certs etc are fine as the full contents of the cert files are reflected in the config files.
Older 1.3 releases specified the cert files which would be persisted on the /config tree, but the 1.4 release changed the way certificates are stored and referenced PKI — VyOS 1.4.x (sagitta) documentation
So I think the problem must be elsewhere. As said in the original, whatever caused this happened just after the 18th June but I can’t figure out a working configuration for 1.5 at this stage so it would be great if someone can attempt a simple config like the one I have as my test to figure out what is going wrong.
thanks.

vy-testing · October 25, 2023, 10:44am

after a lot of trial and error and attempting to compare my config to others, I noticed that I was missing;

set interfaces openvpn vtun0 server topology 'subnet'

As soon as I put that in, the OpenVPN connection started up.
I presume at some stage that this parameter was added to Vyos but I was unaware I needed to add that to the config file.
Thanks for everyones help on this. Phew!

system · November 24, 2023, 10:44am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.