RDP connection over OpenVPN issues

Hi folks! So I’ve run into a challenge with a setup I’m working on and am so far stumped.

I have a VM host (hyper-v based) with several VMs on it, one of which is VyOS 1.1.7, which is acting as gateway for the other VMs, and an OpenVPN server for client connections. Eth0 is attached to the public net, Eth1 to the private net. From within the private net, I can ping and RDP to hosts just fine. From the VPN client, I can also ping hosts just fine, but RDP gets an authentication error. Some posts I’ve found have suggested that it’s a Kerberos issue, or possibly a fragmented packet issue. I haven’t been able to solve it, yet.

Of note, none of the VMs or the VPN client are on an AD domain.

Here’s my config:

firewall { name OUTSIDE-LOCAL { rule 300 {
            action accept
            destination { port openvpn }
            protocol udp
} } }
interfaces {
    ethernet eth0 {
        address 217.163.x.x/24
        description Public
    ethernet eth1 {
        description Private
    openvpn vtun0 {
        mode server
        openvpn-option "--comp-lzo --keepalive 10 120 --duplicate-cn --mssfix"
        server {
        tls {
            ca-cert-file /config/auth/ca.crt
            cert-file /config/auth/vpnserver.crt
            crl-file /config/auth/crl.pem
            dh-file /config/auth/dh2048.pem
            key-file /config/auth/vpnserver.key
nat { source {
        rule 100 {
            description NAT
            outbound-interface eth0
            source { address }
            translation { address masquerade }
        rule 110 {
            description "NAT Reflection"
            destination { address }
            outbound-interface eth1
            source { address }
            translation { address masquerade }
} }
protocols { static { route { next-hop 217.163.x.1 { distance 1 } } } }
service {
    dhcp-server {
        disabled false
        shared-network-name private-net {
            authoritative enable
            subnet {
                lease 86400
                start { stop }
    dns { forwarding {
            cache-size 0
            listen-on eth1
            listen-on vtun0
    } }
    ssh { port 22 }

The errors I get are the RDP client stating that “The logon attempt failed”, and in the Event Log, I found this warning:
RDPClient_SSL: An error was encountered when transitioning from TsSslStateHandshakeInProgress to TsSslStateDisconnecting in response to TsSslEventHandshakeContinueFailed (error code 0x80004005).

Another strange thing. There is one VM that IS on a domain, and I CAN RDP to that one, but only that one.

So any ideas? Thanks!

Hello Aaron,
can you configure portforward to one of affected machines and try to rdp into it (so it goes not via VPN)
Also, when you RDP in you use IP or FQDN?

Thanks for the response,

I just tested with a portforward through VyOS to the affected machine and that worked fine. I’ve been using IP for RDP but I have tried FQDN as well and get the same result. It works for both over the port forward, and doesn’t work for both through the VPN. I’m using a windows server 2016 standard VM as my “test server” and a windows 10 pro VM on a different network as my “test client”.

Can you confirm that you use VMXNET3 and not E1000 adapter?
i will need tcpdump of traffic that pass via openvpn

I’m using Hyper-V for the hypervisor. The interfaces are using the current non-legacy adapters. I’ve e-mailed the TCP dumps to you.