Playing around with OpenVPN Client

anowak · October 8, 2021, 2:44am

Hi,

I’m toying around with VyOS and OpenVPN client to see if I can get connected to Surfshark.
I have two Zones INTERNAL and DEVICES. On DEVICES Interface I have one system I would like to test going through OpenVPN client.

OpenVPN Client Configuration

set interfaces openvpn vtun10 authentication password xxxxxx
set interfaces openvpn vtun10 authentication username xxxxxx
set interfaces openvpn vtun10 description ‘VPN’
set interfaces openvpn vtun10 device-type ‘tun’
set interfaces openvpn vtun10 encryption cipher ‘aes256’
set interfaces openvpn vtun10 hash ‘sha512’
set interfaces openvpn vtun10 mode ‘client’
set interfaces openvpn vtun10 openvpn-option ‘–resolv-retry infinite’
set interfaces openvpn vtun10 openvpn-option ‘–verb 3’
set interfaces openvpn vtun10 openvpn-option ‘–pull’
set interfaces openvpn vtun10 openvpn-option ‘–fast-io’
set interfaces openvpn vtun10 openvpn-option ‘–remote-random’
set interfaces openvpn vtun10 openvpn-option ‘–nobind’
set interfaces openvpn vtun10 openvpn-option ‘–tun-mtu 1500’
set interfaces openvpn vtun10 openvpn-option ‘–tun-mtu-extra 32’
set interfaces openvpn vtun10 openvpn-option ‘–mssfix 1450’
set interfaces openvpn vtun10 openvpn-option ‘–persist-key’
set interfaces openvpn vtun10 openvpn-option ‘–persist-tun’
set interfaces openvpn vtun10 openvpn-option ‘–ping 15’
set interfaces openvpn vtun10 openvpn-option ‘–ping-restart 0’
set interfaces openvpn vtun10 openvpn-option ‘–ping-timer-rem’
set interfaces openvpn vtun10 openvpn-option ‘–reneg-sec 0’
set interfaces openvpn vtun10 persistent-tunnel
set interfaces openvpn vtun10 protocol ‘udp’
set interfaces openvpn vtun10 remote-host ‘au-syd.prod.surfshark.com’
set interfaces openvpn vtun10 remote-port ‘1194’
set interfaces openvpn vtun10 tls auth-file ‘/config/auth/client.key’

Zone rule

set firewall name LOCAL-OUTSIDE rule 2000 action ‘accept’
set firewall name LOCAL-OUTSIDE rule 2000 description ‘Allow LOCAL VPN OUTSIDE’
set firewall name LOCAL-OUTSIDE rule 2000 destination port ‘1194’
set firewall name LOCAL-OUTSIDE rule 2000 log ‘enable’
set firewall name LOCAL-OUTSIDE rule 2000 protocol ‘udp’
set firewall name LOCAL-OUTSIDE rule 2000 state new ‘enable’

When I enable the config all external networking drops.
I’ve struggled with VyOS logging as I seem to get nothing when tailing the messages log.

At one point it did spit out the following

adm@box# tail -f /var/log/messages | grep openvpn
Oct 7 17:46:18 box vyos-configd[662]: Received message: {“type”: “node”, “data”: “VYOS_TAGNODE_VALUE=vtun10/usr/libexec/vyos/conf_mode/interfaces-openvpn.py”}
Oct 7 17:46:19 box openvpn-vtun10[32747]: DEPRECATED OPTION: --cipher set to ‘aes-256-cbc’ but missing in --data-ciphers (AES-256-GCM:AES-128-GCM). Future OpenVPN version will ignore --cipher for cipher negotiations. Add ‘aes-256-cbc’ to --data-ciphers or change --cipher ‘aes-256-cbc’ to --data-ciphers-fallback ‘aes-256-cbc’ to silence this warning.
Oct 7 17:46:19 box openvpn-vtun10[32747]: OpenVPN 2.5.1 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Apr 28 2021
Oct 7 17:46:19 box openvpn-vtun10[32747]: library versions: OpenSSL 1.1.1d 10 Sep 2019, LZO 2.10
Oct 7 17:46:19 box openvpn-vtun10[32747]: WARNING: --ping should normally be used with --ping-restart or --ping-exit
Oct 7 17:46:19 box openvpn-vtun10[32747]: WARNING: No server certificate verification method has been enabled. See http://openvpn.net/howto.html#mitm for more info.
Oct 7 17:46:19 box openvpn-vtun10[32747]: Outgoing Control Channel Authentication: Using 512 bit message hash ‘SHA512’ for HMAC authentication
Oct 7 17:46:19 box openvpn-vtun10[32747]: Incoming Control Channel Authentication: Using 512 bit message hash ‘SHA512’ for HMAC authentication
Oct 7 17:46:19 box openvpn-vtun10[32747]: TCP/UDP: Preserving recently used remote address: [AF_INET]144.48.38.21:1194
Oct 7 17:46:19 box openvpn-vtun10[32747]: Socket Buffers: R=[212992->212992] S=[212992->212992]
Oct 7 17:46:19 box openvpn-vtun10[32747]: UDP link local: (not bound)
Oct 7 17:46:19 box openvpn-vtun10[32747]: UDP link remote: [AF_INET]144.48.38.21:1194
Oct 7 17:46:19 box openvpn-vtun10[32747]: NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
Oct 7 17:46:19 box openvpn-vtun10[32747]: TLS: Initial packet from [AF_INET]144.48.38.21:1194, sid=242f0b8f d3e619de
Oct 7 17:46:19 box openvpn-vtun10[32747]: VERIFY OK: depth=2, C=VG, O=Surfshark, CN=Surfshark Root CA
Oct 7 17:46:19 box openvpn-vtun10[32747]: VERIFY OK: depth=1, C=VG, O=Surfshark, CN=Surfshark Intermediate CA
Oct 7 17:46:19 box openvpn-vtun10[32747]: VERIFY OK: depth=0, CN=au-syd-v010.prod.surfshark.com
Oct 7 17:46:19 box openvpn-vtun10[32747]: WARNING: ‘link-mtu’ is used inconsistently, local=‘link-mtu 1633’, remote=‘link-mtu 1581’
Oct 7 17:46:19 box openvpn-vtun10[32747]: WARNING: ‘auth’ is used inconsistently, local=‘auth SHA512’, remote=‘auth [null-digest]’
Oct 7 17:46:19 box openvpn-vtun10[32747]: Control Channel: TLSv1.2, cipher TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 2048 bit RSA
Oct 7 17:46:19 box openvpn-vtun10[32747]: [au-syd-v010.prod.surfshark.com] Peer Connection Initiated with [AF_INET]144.48.38.21:1194
Oct 7 17:46:20 box openvpn-vtun10[32747]: SENT CONTROL [au-syd-v010.prod.surfshark.com]: ‘PUSH_REQUEST’ (status=1)
Oct 7 17:46:20 box openvpn-vtun10[32747]: PUSH: Received control message: ‘PUSH_REPLY,dhcp-option DNS 162.252.172.57,dhcp-option DNS 149.154.159.92,redirect-gateway def1,sndbuf 524288,rcvbuf 524288,explicit-exit-notify,block-outside-dns,route-gateway 10.8.8.1,topology subnet,ping 60,ping-restart 180,ifconfig 10.8.8.2 255.255.255.0,peer-id 0,cipher AES-256-GCM’
Oct 7 17:46:20 box openvpn-vtun10[32747]: Options error: Unrecognized option or missing or extra parameter(s) in [PUSH-OPTIONS]:7: block-outside-dns (2.5.1)
Oct 7 17:46:20 box openvpn-vtun10[32747]: OPTIONS IMPORT: timers and/or timeouts modified
Oct 7 17:46:20 box openvpn-vtun10[32747]: OPTIONS IMPORT: explicit notify parm(s) modified
Oct 7 17:46:20 box openvpn-vtun10[32747]: OPTIONS IMPORT: --sndbuf/–rcvbuf options modified
Oct 7 17:46:20 box openvpn-vtun10[32747]: Socket Buffers: R=[212992->446464] S=[212992->425984]
Oct 7 17:46:20 box openvpn-vtun10[32747]: OPTIONS IMPORT: --ifconfig/up options modified
Oct 7 17:46:20 box openvpn-vtun10[32747]: OPTIONS IMPORT: route options modified
Oct 7 17:46:20 box openvpn-vtun10[32747]: OPTIONS IMPORT: route-related options modified
Oct 7 17:46:20 box openvpn-vtun10[32747]: OPTIONS IMPORT: --ip-win32 and/or --dhcp-option options modified
Oct 7 17:46:20 box openvpn-vtun10[32747]: OPTIONS IMPORT: peer-id set
Oct 7 17:46:20 box openvpn-vtun10[32747]: OPTIONS IMPORT: adjusting link_mtu to 1656
Oct 7 17:46:20 box openvpn-vtun10[32747]: OPTIONS IMPORT: data channel crypto options modified
Oct 7 17:46:20 box openvpn-vtun10[32747]: Data Channel: using negotiated cipher ‘AES-256-GCM’
Oct 7 17:46:20 box openvpn-vtun10[32747]: Outgoing Data Channel: Cipher ‘AES-256-GCM’ initialized with 256 bit key
Oct 7 17:46:20 box openvpn-vtun10[32747]: Incoming Data Channel: Cipher ‘AES-256-GCM’ initialized with 256 bit key
Oct 7 17:46:20 box openvpn-vtun10[32747]: net_route_v4_best_gw query: dst 0.0.0.0
Oct 7 17:46:20 box openvpn-vtun10[32747]: net_route_v4_best_gw result: via 100.66.128.1 dev eth0
Oct 7 17:46:20 box openvpn-vtun10[32747]: ROUTE_GATEWAY 100.66.128.1/255.255.128.0 IFACE=eth0 HWADDR=2c:f0:5d:d6:b3:ca
Oct 7 17:46:20 box openvpn-vtun10[32747]: TUN/TAP device vtun10 opened
Oct 7 17:46:20 box openvpn-vtun10[32747]: net_iface_mtu_set: mtu 1500 for vtun10
Oct 7 17:46:20 box openvpn-vtun10[32747]: net_iface_up: set vtun10 up
Oct 7 17:46:20 box openvpn-vtun10[32747]: net_addr_v4_add: 10.8.8.2/24 dev vtun10
Oct 7 17:46:20 box openvpn-vtun10[32747]: net_route_v4_add: 144.48.38.21/32 via 100.66.128.1 dev [NULL] table 0 metric -1
Oct 7 17:46:20 box openvpn-vtun10[32747]: net_route_v4_add: 0.0.0.0/1 via 10.8.8.1 dev [NULL] table 0 metric -1
Oct 7 17:46:20 box openvpn-vtun10[32747]: net_route_v4_add: 128.0.0.0/1 via 10.8.8.1 dev [NULL] table 0 metric -1
Oct 7 17:46:20 box openvpn-vtun10[32747]: GID set to openvpn
Oct 7 17:46:20 box openvpn-vtun10[32747]: UID set to openvpn
Oct 7 17:46:20 box openvpn-vtun10[32747]: WARNING: this configuration may cache passwords in memory – use the auth-nocache option to prevent this
Oct 7 17:46:20 box openvpn-vtun10[32747]: Initialization Sequence Completed

But that is it… I disable and re-enable the interface, without anymore logs.

Any insight would be good?
Believe it is setting up routes which kill off the nat rules.

Kind Regards

anowak · October 7, 2021, 8:45am

Got the first part sorted with the routes. Did a bit more digging and by default OpenVPN injects routes
adding the following fix this.
set interfaces openvpn vtun10 openvpn-option “pull-filter ignore redirect-gateway”

anowak · October 8, 2021, 2:51am

Hi,

Now that I have openvpn client connected, what is the best way to ONLY allow system 1 to go through the openvpn connection?

Currently I have a NAT rule that pushes all DNS queries to INTERNAL. Need to get around this to use OpenVPN DNS entries supplied. Next do I need another source NAT → outbound vtun10 → masquerade? Or mix with some policy based route?

Regards

Nikolay · October 8, 2021, 12:42pm

Hi @anowak,

Policy based routing is fine for redirecting traffic from system1 to the special route.

anowak · October 8, 2021, 1:07pm

Hi Nikolay,

Appreciate your input.

Not sure how I would get the PBR to work if the IP VPN address changes every time it reconnects.

set policy route PBR rule 5 set table ‘10’
set policy route PBR rule 5 description ‘Route host in Devices to table 10’
set policy route PBR rule 5 source address ‘172.16.20.20’

set protocols static table 10 route 0.0.0.0/0 next-hop 10.8.8.x

Is there a way to point the next hop to an interface instead.

Nikolay · October 9, 2021, 2:08am

You can try point static route to vtun interface:

set protocols static table 10 route 0.0.0.0/0 interface vtun10

anowak · October 9, 2021, 9:54am

Thanks for pointing me in the that direction - will give it a go. Command is as follows.

set protocols static table 10 interface-route 0.0.0.0/0 next-hop-interface vtun10

anowak · October 9, 2021, 10:15am

Looks like it is still getting caught by the source NAT for the DEVICE zone.

anowak · October 9, 2021, 12:26pm

Getting the following - not really sure what it means

run show policy route

-----------------------------
Rulesets Information
-----------------------------
--------------------------------------------------------------------------------
IPv4 Policy Route "PBR":

 Active on (eth2,ROUTE)

rule  action   proto     packets  bytes
splice() offset past end of array at /opt/vyatta/bin/vyatta-show-firewall.pl line 356.
Use of uninitialized value in concatenation (.) or string at /opt/vyatta/bin/vyatta-show-firewall.pl line 368.
Use of uninitialized value in concatenation (.) or string at /opt/vyatta/bin/vyatta-show-firewall.pl line 369.
Use of uninitialized value $string_words_part1[3] in string at /opt/vyatta/bin/vyatta-show-firewall.pl line 400.
Use of uninitialized value $string_words_part1[0] in string at /opt/vyatta/bin/vyatta-show-firewall.pl line 400.
Use of uninitialized value $string_words_part1[1] in string at /opt/vyatta/bin/vyatta-show-firewall.pl line 400.
----  ------   -----     -------  -----
5     set
  condition - saddr  daddr

10000 drop     all       2343699  162254994
  condition - saddr 0.0.0.0/0 daddr 0.0.0.0/0 LOG enabled

set policy route PBR enable-default-log
set policy route PBR rule 5 description 'Route single host in DEVICE, use table 10'
set policy route PBR rule 5 set table '10'
set policy route PBR rule 5 source address 'xxx.xxx.20.12'
set protocols static table 10 interface-route 0.0.0.0/0 next-hop-interface vtun10
set interfaces ethernet eth2 policy route 'PBR'

run show ip route table 10
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

VRF default table 10:
S>* 0.0.0.0/0 [1/0] is directly connected, vtun10, weight 1, 02:21:35

16again · October 10, 2021, 7:47am

order of operation:
First routing decision is made.
Then , before packet gets routed to the outside, sNAT rules on outgoing interface are checked
So if you see normal NAT rule kick in, then routing decision made isn’t as intended
btw, you need sNAT rule on vtun too, as remote probably has no route back to your LAN

show policy route looks like something is broken

anowak · October 10, 2021, 10:18am

Thanks 16again,

Currently I have the following sNAT rule - pretty standard.

set nat source rule 100 description 'NAT INTERNAL'
set nat source rule 100 outbound-interface 'eth0'
set nat source rule 100 source address 'xxx.xxx.10.0/24'
set nat source rule 100 translation address 'masquerade'
set nat source rule 101 description 'NAT DEVICE'
set nat source rule 101 log
set nat source rule 101 outbound-interface 'eth0'
set nat source rule 101 source address 'xxx.xxx.20.0/24'
set nat source rule 101 translation address 'masquerade'

I did set up another sNAT rule 99 as follows, but did not seem to do anything or at least did not log anything.

set nat source rule 99 description 'NAT VPN'
set nat source rule 99 outbound-interface 'vtun10'
set nat source rule 99 source address 'xxx.xxx.20.12'
set nat source rule 99 translation address 'masquerade'

Let me tare it all down and try again. PBR does not seem to be working.

From reading different articles, removing the push routes from OpenVPN provider is the way to go when giving only a single system access to OVPN.

I also have the following allow firewall rules

set firewall name DEVICE-VPN default-action 'accept'
set firewall name VPN-OUTSIDE default-action 'accept'

set zone-policy zone VPN default-action 'drop'
set zone-policy zone VPN from DEVICE firewall name 'DEVICE-VPN'
set zone-policy zone VPN from OUTSIDE firewall name 'OUTSIDE-VPN'
set zone-policy zone VPN interface 'vtun10'

Nikolay · October 10, 2021, 1:31pm

I tried this PBR example, it works (IPsec VTI interfaces):
set protocols static table 10 interface-route 0.0.0.0/0 next-hop-interface vti10

set interfaces ethernet eth3 policy route ‘PBR’
set policy route PBR rule 10 set table ‘10’
set policy route PBR rule 10 source address ‘172.24.133.33’

I agree with @16again about NAT. Your 99 NAT rule looks correct

anowak · October 10, 2021, 11:16pm

Hi @Nikolay, @16again,

Alright! getting somewhere, thanks for the help!!

Re-added my sNAT 99 rule as above.
Re-added the PBR onto the interface.

Note: The interface (eth2) needs to be restarted, otherwise you get the following error on commit

# commit
[ policy route PBR ]
RTNETLINK answers: File exists

and you will receive error “Use of uninitialized value in concatenation” when showing route policy and the following error.

Strange thing is I cannot see the openvpn logs? Get nothing. Also nothing under /var/log/messages or /var/log/openvpn/*. Logging seems to be very intermittent. (anywhere else I can look?)

# run show log openvpn
[edit]

Although the connection is active.

# run show openvpn client

OpenVPN status on vtun10

Client CN       Remote Host           Local Host            TX bytes    RX bytes   Connected Since
---------       -----------           ----------            --------    --------   ---------------
N/A             au-syd.prod.surfshark.com:1194 N/A                   4.4 KB      6.2 KB     N/A

When running the following command, what value is needed under as no matter what I put I get the following No such file or directory error.

reset openvpn client <text>
2021/10/11 10:10:34 socat[23447] E connect(5, AF=1 "/run/openvpn/openvpn-mgmt-intf", 32): No such file or directory

Anyhow - Now I need to work out DNS. Currently I have dNAT in place that pushes all DNS queries to the internal DNS server, even for the firewall. On VPN connection “System 1” loses connectivity to the LAN (which is another PBR in the main table - I’m guessing).

I’ll keep tinkering.

anowak · October 11, 2021, 2:14am

Just an update: May help others looking at doing something similar.

I wanted System 1 to still have access via the LAN, given all traffic goes out through OpenVPN. Using the initial PBR did not allow for this - so updated it.

set policy route PBR enable-default-log

set policy route PBR rule 10 description 'ALLOW FOR INTERNAL ROUTING'
set policy route PBR rule 10 source address 'xxx.xxx.20.12'
set policy route PBR rule 10 destination address 'xxx.xxx.10.0/24'
set policy route PBR rule 10 set table 'main'

set policy route PBR rule 11 description 'ALLOW FOR DEVICE ROUTING'
set policy route PBR rule 11 source address 'xxx.xxx.20.12'
set policy route PBR rule 11 destination address 'xxx.xxx.20.0/24'
set policy route PBR rule 11 set table 'main'

set policy route PBR rule 12 set table '10'
set policy route PBR rule 12 source address 'xxx.xxx.20.12'

Not sure if the documentaiton is incorrect, but I could not place two destination addresses in the same rule
https://docs.vyos.io/en/latest/configuration/policy/examples.html#multiple-uplinks

set policy route PBR rule 10 description 'VLAN10 <-> VLAN11 shortcut'
set policy route PBR rule 10 destination address '192.168.188.0/24'
set policy route PBR rule 10 destination address '192.168.189.0/24'
set policy route PBR rule 10 set table 'main'

anowak · October 11, 2021, 7:39am

Still working on DNS issue. I have now changed DHCP scope that points DNS to VYOS address. VyOS will forward to internal DNS server. But I’d like the “System 1” to use the DNS servers allocated by OpenVPN DHCP options.

Will the following config help? Would this make it global or only systems that talk via vtun10.

system name-servers-dhcp vtun10

Cheers

16again · October 11, 2021, 6:39pm

You can’t use multiple destination addresses on single rule, instead use a group as destination

Policy routing comes before dNAT. So PBR rules still sees public DNS address as destination, so packet will be routed using vtun. Then dNAT kicks in , changing DNS server address, which won’t work on vtun.
So add another exclude rule in PBR config, when destination matches udp53 , use table main

anowak · October 11, 2021, 10:06pm

Hi @16again,

Sorry, not sure if I clearly explained what I am trying to achieve. The issue is being able to resolve DNS using the DNS provided by Surfshark.

Currently all systems point to VyOS to resolve their DNS requests. Internal requests get resolved by VyOS and external requests are resolved by an internal PiHole server. When the vtun is up I’d like to see System 1 bypass the PiHole server and go directly to the DNS servers presented by the VPN provider. The problem is that System 1’s DNS is pointing to VyOS and VyOS has no idea what to do (need that config) so it sends it to pihole (which resolves but not exactly how it should work).

Snippet of tunnel connection.

Oct 11 18:05:16 vyos openvpn-vtun10[15038]: PUSH: Received control message: 'PUSH_REPLY,,dhcp-option DNS 149.154.159.92,redirect-gateway def1,sndbuf 524288,rcvbuf 524288,explicit-exit-notify,block-outside-dns,route-gateway 10.8.8.1,topology subnet,ping 60,ping-restart 180,ifconfig 10.8.8.7 255.255.255.0,peer-id 5,cipher AES-256-CBC'
Oct 11 18:05:16 vyos openvpn-vtun10[15038]: Pushed option removed by filter: 'redirect-gateway def1'

As shown in the excerpt, DNS is 149.154.159.92. How can I get VyOS to see this?
If I connect to another location a different DNS is presented.

Guess there are two questions here.

How do I get vyos to see the tunnel DNS entry.
How do I use dNAT or PBR to use that tunnel DNS entry.

Info: If I hard code the external DNS entry under my resolv.conf under system 1 it works.

16again · October 13, 2021, 8:34pm

You can’t do what you want…from vyos config.
dNAT rules come before routing decision. But you want to use different dNAT rule to override DNS address, based on future route decision
Script that runs on vtun going up/down that enables/disables dNAT rule could help you out

anowak · October 14, 2021, 5:02am

Hi 16again,

Appreciate your help.

Tried getting vyos to obtain DNS entries provided by DHCP over vtun10 by using this command

set service dns forwarding dhcp vtun10

But have no success. Idea is that when the tunnel is up it would allocated the DNS server to VyOS. I would have dNAT rule for all other hosts to use internal DNS.

Searched around for examples on setting scripts that monitor the interface with no success. Found one for load-balancing wan only.

Cheers