OpenVPN on VyOS Router behind Access Router

may24x · December 2, 2019, 2:51pm

Hi all. I’ve red several tutorial and tried dozends sample configs … but I’m missing something … somewhere

I’ve an access Router (192.168.0.1) that statically NATs all Internet Traffic to 192.168.0.2. This IP is also my VyOS routers eth0.

Now, I want to access my Subnet 192.168.10.0/24 from the internet. So I setup openvpn:

interfaces {
    ethernet eth0 {
        address 192.168.0.2/24
        hw-id 00:0c:29:37:f9:5a
    }
    ethernet eth1 {
        address 192.168.10.1/24
        hw-id 00:0c:29:37:f9:7a
    }
    loopback lo {
    }
    openvpn vtun0 {
        mode server
        openvpn-option "--proto tcp"
        openvpn-option "--ifconfig-pool-persist ipp.txt"
        openvpn-option "--keepalive 10 120"
        openvpn-option "--user nobody"
        openvpn-option "--group nogroup"
        openvpn-option --persist-key
        openvpn-option --persist-tun
        openvpn-option "--status openvpn-status.log"
        openvpn-option "--verb 3"
        openvpn-option "--mute 10"
        openvpn-option "--port 2019"
        openvpn-option "--dev vtun0"
        openvpn-option "--cipher AES-256-CBC"
        openvpn-option "auth SHA512"
        openvpn-option "remote-cert-tls client"
        openvpn-option --client-to-client
        server {
            push-route 192.168.10.0/24
            subnet 10.8.0.0/24
        }
        tls {
            auth-file /config/auth/ta.key
            ca-cert-file /config/auth/ca.crt
            cert-file /config/auth/server.crt
            dh-file /config/auth/dh4096.pem
            key-file /config/auth/server.key
        }
    }
}

The Client has:

  client
  dev tun
  resolv-retry infinite
  nobind
  persist-key
  persist-tun
  route-delay 15
  tls-client
  tls-auth ta.key 1
  cipher AES-256-CBC
  auth SHA512
  remote-cert-tls server
  key-direction 1
  verb 6
  compress lz4-v2
  proto tcp
  remote my.homeserver.net 2019
  <ca>
  -----BEGIN CERTIFICATE-----

The openvpn is listening:

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:2019            0.0.0.0:*               LISTEN      1805/openvpn

I haven’t set any additional rules/routes so far …

Can you help me out ?

zsdc · December 2, 2019, 6:44pm

Hello, @may24x!

It would be good if you will draw and share your topology map. Without it, will be hard to suggest something. We could imagine how the topology should look from your config and description, but better will be to get it from you.

may24x · December 3, 2019, 9:06am

Here you go

zsdc · December 3, 2019, 11:19am

Thanks, @may24x!
It was truly helpful - now the topology is clear. So, are clients able to connect to the configured OpenVPN server? If yes, please show us the routing table on them after connection. There must be a route to 192.168.10.0/24 via OpenVPN.

may24x · December 3, 2019, 1:39pm

Well afaik they connect “half-way” …

openvpn --config ./bia-tap-client.ovpn
Tue Dec  3 14:34:39 2019 OpenVPN 2.4.7 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Sep  5 2019
Tue Dec  3 14:34:39 2019 library versions: OpenSSL 1.1.1c  28 May 2019, LZO 2.10
Tue Dec  3 14:34:40 2019 TCP/UDP: Preserving recently used remote address: [AF_INET]99.9.199.199:2019
Tue Dec  3 14:34:40 2019 Attempting to establish TCP connection with [AF_INET]99.9.199.199:2019 [nonblock]
Tue Dec  3 14:34:41 2019 TCP connection established with [AF_INET]99.9.199.199:2019
Tue Dec  3 14:34:41 2019 TCP_CLIENT link local: (not bound)
Tue Dec  3 14:34:41 2019 TCP_CLIENT link remote: [AF_INET]99.9.199.199:2019
Tue Dec  3 14:34:41 2019 Connection reset, restarting [0]
Tue Dec  3 14:34:41 2019 SIGUSR1[soft,connection-reset] received, process restarting
Tue Dec  3 14:34:46 2019 TCP/UDP: Preserving recently used remote address: [AF_INET]99.9.199.199:2019
Tue Dec  3 14:34:46 2019 Attempting to establish TCP connection with [AF_INET]99.9.199.199:2019 [nonblock]
Tue Dec  3 14:34:47 2019 TCP connection established with [AF_INET]99.9.199.199:2019
Tue Dec  3 14:34:47 2019 TCP_CLIENT link local: (not bound)
Tue Dec  3 14:34:47 2019 TCP_CLIENT link remote: [AF_INET]99.9.199.199:2019
Tue Dec  3 14:34:47 2019 Connection reset, restarting [0]
Tue Dec  3 14:34:47 2019 SIGUSR1[soft,connection-reset] received, process restarting
Tue Dec  3 14:34:48 2019 SIGINT[hard,init_instance] received, process exiting

On the vyos side:

vyos@hera:~$ netstat -r
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
default 192.168.0.1 0.0.0.0 UG 0 0 0 eth0
10.8.0.0 10.8.0.2 255.255.255.0 UG 0 0 0 vtun0
10.8.0.2 * 255.255.255.255 UH 0 0 0 vtun0
192.168.0.0 * 255.255.255.0 U 0 0 0 eth0
192.168.10.0 * 255.255.255.0 U 0 0 0 eth1
192.168.3.0 * 255.255.255.0 U 0 0 0 eth2

Viacheslav · December 4, 2019, 12:42pm

Can you check correctly settings of port forwarding TCP/UDP from 99.9.199.199 port 2019 to VyOS?
Can you see dump packets on eth0 with dport 2019?
Maybe windows firewall is blocking access for the openvpn-client. Can you temporary disable firewall on client side? And try run it with Administrator privileges.

may24x · December 4, 2019, 1:45pm

All systems are either Ubuntu or VyOS istself. No Win box available …
However, I can’t see any packets entering eth0 with dport 2019 … although I’m not sure if tcpdump is just unable to catch them. Also if I’m traceing vtun0 I can’t see anything.
I suspect the packages arrive at eth0 but then Vyos don’t know what to do and so it discards them. Unfortunately It’s currently not possible to trace between the access router and the Vyos-Router …

But I presumed that since openVPN listens on all interfaces (see netstat from above) it “knows” where to distribute the vpn traffic to (vtun0 - as it’s the only interface that listens to 2019). Hence no additional NAT rule or static route would be necessary … (?) But from what I see of the openvpn client output I’d conclude the eth0 rejects/discards the incoming packets … or am I wrong ??

Viacheslav · December 4, 2019, 2:01pm

You should see packets in tcpdump dport ‘2019’ anyway.
Try connecting to different port without ovpn.
Example connection to ip address 99.9.199.199 port 2222 forward to 192.168.0.2 port 22.
And check ssh connection. As far as I understand 99.9.199.199 is this your internet address.

may24x · December 4, 2019, 2:39pm

yes 99.9.199.199 would be my internet address … but it changes all 24h …
anyway, I also have a nginx running (on a different dev-server) on port 80/443. And I can reach it without any problems. But I’ve a VyOS NAT rule for this …

zsdc · December 4, 2019, 4:25pm

Hello, @may24x!
I would recommend you to start debugging from this:

Check with tcpdump if you see on VyOS (eth0, port 2019) incoming packets from the client. Which source and destination address those packets has.
Check with tcpdump on the client if it sees any answers from the VyOS. They must be seen from the gateway public address, port 2019.
Check OpenVPN logs on VyOS at the connection moment.

It more looks like some problems with NAT/firewall on your internet gateway. You can drop here logs and tcpdump output, and we will check it together.

may24x · December 5, 2019, 9:27am

So … I target the 2019 port with my Firefox … Packets are arriving at eth0 … but immediately get terminated (FIN). Nothing on the vtun0 interface. Seems vyos does not forward/route them to the vtunnel

Maybe I need to bind the openvpn to the vtun0 interface ?

may24x · December 5, 2019, 11:29am

Ok, so it’s not possible to bind openVPN to a specific interface.

Could the problem be that I haven’t set a “local” address ? Actually I all have is the vtun0:

vyos@hera# ifconfig -a vtun0
vtun0     Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  
          inet addr:10.8.0.1  P-t-P:10.8.0.2  Mask:255.255.255.255
          inet6 addr: fe80::fc3b:bb7f:1c4d:3c81/64 Scope:Link
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2092 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100 
          RX bytes:0 (0.0 B)  TX bytes:139938 (136.6 KiB)

zsdc · December 5, 2019, 5:06pm

NAT looks good, but how about OpenVPN server logs? The answer should be there.

may24x · December 8, 2019, 11:39am

Ok, from what I got from (diverse) docs, this was missing:

set protocols static interface-route 10.8.0.0/24 next-hop-interface vtun0

BUT, it still doesn’t work. IMO the setting above is unnecessary and doesn’t make much sense as openvpn ‘should’ handle this mapping all by itself … however the log shows me something else:

un Dec  8 19:13:01 2019 us=235356 Listening for incoming TCP connection on [AF_INET][undef]:2019
Sun Dec  8 19:13:01 2019 us=235367 TCPv4_SERVER link local (bound): [AF_INET][undef]:2019
Sun Dec  8 19:13:01 2019 us=235374 TCPv4_SERVER link remote: [AF_UNSPEC]
Sun Dec  8 19:13:01 2019 us=235383 GID set to nogroup
Sun Dec  8 19:13:01 2019 us=235392 UID set to nobody
Sun Dec  8 19:13:01 2019 us=235406 MULTI: multi_init called, r=256 v=256
Sun Dec  8 19:13:01 2019 us=235449 IFCONFIG POOL: base=10.8.0.4 size=62, ipv6=0
Sun Dec  8 19:13:01 2019 us=235463 IFCONFIG POOL LIST
Sun Dec  8 19:13:01 2019 us=235483 MULTI: TCP INIT maxclients=1024 maxevents=1028
Sun Dec  8 19:13:01 2019 us=235527 Initialization Sequence Completed
tail: unrecognized file system type 0x794c7630 for ‘openvpn.log’. please report this to [email protected]. reverting to polling
Sun Dec  8 19:14:00 2019 us=557821 MULTI: multi_create_instance called
Sun Dec  8 19:14:00 2019 us=557895 Re-using SSL/TLS context
Sun Dec  8 19:14:00 2019 us=558027 Control Channel MTU parms [ L:1623 D:1138 EF:112 EB:0 ET:0 EL:3 ]
Sun Dec  8 19:14:00 2019 us=558057 Data Channel MTU parms [ L:1623 D:1450 EF:123 EB:406 ET:0 EL:3 ]
Sun Dec  8 19:14:00 2019 us=558089 Local Options String (VER=V4): 'V4,dev-type tun,link-mtu 1603,tun-mtu 1500,proto TCPv4_SERVER,cipher AES-256-CBC,auth SHA512,keysize 256,tls-auth,key-method 2,tls-server'
Sun Dec  8 19:14:00 2019 us=558100 Expected Remote Options String (VER=V4): 'V4,dev-type tun,link-mtu 1603,tun-mtu 1500,proto TCPv4_CLIENT,cipher AES-256-CBC,auth SHA512,keysize 256,tls-auth,key-method 2,tls-client'
Sun Dec  8 19:14:00 2019 us=558130 TCP connection established with [AF_INET]192.168.3.3:58126
Sun Dec  8 19:14:00 2019 us=558143 TCP_SERVER link local: (not bound)
Sun Dec  8 19:14:00 2019 us=558154 TCP_SERVER link remote: [AF_INET]192.168.3.3:58126
Sun Dec  8 19:14:01 2019 us=565773 192.168.3.3:58126 TCP_SERVER READ [86] from [AF_INET]192.168.3.3:58126: P_CONTROL_HARD_RESET_CLIENT_V2 kid=0 pid=[ #1 ] [ ] pid=0 DATA len=0
Sun Dec  8 19:14:01 2019 us=565826 192.168.3.3:58126 TLS: Initial packet from [AF_INET]192.168.3.3:58126, sid=ab8cd489 a86ea9ad
Sun Dec  8 19:14:01 2019 us=565849 192.168.3.3:58126 Authenticate/Decrypt packet error: packet HMAC authentication failed
Sun Dec  8 19:14:01 2019 us=565864 192.168.3.3:58126 TLS Error: incoming packet authentication failed from [AF_INET]192.168.3.3:58126
Sun Dec  8 19:14:01 2019 us=565890 192.168.3.3:58126 Fatal TLS error (check_tls_errors_co), restarting
Sun Dec  8 19:14:01 2019 us=565902 192.168.3.3:58126 SIGUSR1[soft,tls-error] received, client-instance restarting
Sun Dec  8 19:14:01 2019 us=565951 TCP/UDP: Closing socket

Now, I double checked all cert’s and the ta.keys … and there is no missmatch !
HMAC (should ?) be sha512 and it’s supported by both server and client …

Viacheslav · December 9, 2019, 12:38pm

OpenVPN key’s/Auth Algorithm doesn’t match.
Also try.
set interfaces openvpn vtun0 openvpn-option "--tls-auth /config/auth/ta.key 0"

And I see “link-mtu 1603” messages
Execute on server and client sides:
ip link show eth0

On client side:

sudo openvpn --config bia-tap-client.ovpn

On server side:
show log vpn all
ping -c 5 -s 1500 -M do ip-address-vpn-client.
Decrease this value if you see a message
ping: local error: Message too long, mtu=1500