Hi Team,
My topology is given below. I have VPC 10.122.0.0/20 and on-prem network is 172.16.3.0/24.
I have configured site-to-site tunnel with on-prem firewall and tunnel shows up however I am unable to communicate with on-prem servers from my digital ocean droplets and/or vice-versa.
Also I added route on my droplet i.e. to reach 172.16.3.0/24 NH is 10.122.0.7.
Any clue? I am sure Digital ocean firewall is not dropping and it has something related to routing.
IKE/IPsec SA both shows up.
Here is my config
set interfaces ethernet eth1 address 'xxx.xxx.134.148/20'
set interfaces ethernet eth3 address 'xxx.xxx.0.7/20'
set interfaces loopback lo
set protocols static route xxx.xxx.0.0/0 next-hop xxx.xxx.128.1
set service ssh
set vpn ipsec esp-group testesp compression 'disable'
set vpn ipsec esp-group testesp mode 'tunnel'
set vpn ipsec esp-group testesp pfs 'dh-group2'
set vpn ipsec esp-group testesp proposal 5 encryption 'aes256'
set vpn ipsec esp-group testesp proposal 5 hash 'sha256'
set vpn ipsec ike-group testike key-exchange 'ikev1'
set vpn ipsec ike-group testike proposal 5 dh-group '2'
set vpn ipsec ike-group testike proposal 5 encryption 'aes256'
set vpn ipsec ike-group testike proposal 5 hash 'sha256'
set vpn ipsec ipsec-interfaces interface 'eth1'
set vpn ipsec site-to-site peer xxxxx.tld authentication mode 'pre-shared-secret'
set vpn ipsec site-to-site peer xxxxx.tld authentication pre-shared-secret xxxxxx
set vpn ipsec site-to-site peer xxxxx.tld ike-group 'testike'
set vpn ipsec site-to-site peer xxxxx.tld local-address 'xxx.xxx.134.148'
set vpn ipsec site-to-site peer xxxxx.tld tunnel 0 allow-nat-networks 'disable'
set vpn ipsec site-to-site peer xxxxx.tld tunnel 0 allow-public-networks 'disable'
set vpn ipsec site-to-site peer xxxxx.tld tunnel 0 esp-group 'testesp'
set vpn ipsec site-to-site peer xxxxx.tld tunnel 0 local prefix 'xxx.xxx.0.0/20'
set vpn ipsec site-to-site peer xxxxx.tld tunnel 0 remote prefix 'xxx.xxx.3.0/24'
Add local prefix to looback/dummy interface
To check if it possible to communicate between peers over IPSec
To be sure that all routes/policies works correct.
You mean to say add one more interface to vyos instance? I am sorry I am bit confused
Plus; I forgot to mention I am not seeing packets on either devices internal interfaces. i.e. no packets from 10.122.0.2 on my on prem firewall 172.16.3.1 interface.
Seems Digital Ocean has a limitation in VPN setup. I had raised a ticket with them and this is what they replied.
Has anyone achieved this kind of scenario in Digital Ocean.
Hello @blason
You can use nat to access other servers in the vpc network.
Try the configuration:
set nat source rule 100 destination address '10.122.0.0/20'
set nat source rule 100 outbound-interface 'eth1'
set nat source rule 100 source address '172.16.3.0/24'
set nat source rule 100 translation address '10.122.0.7'
To connect to DigitalOcean vpc network I use l2tp connection and nat described above and everything works.
Still unclear to me what doesn’t work, the DigitalOcean reply doesn’t give any clues.
Do both end of tunnel encrypt packets into ESP?
Does remote receive ESP packets or is ESP protocol being blocked?
Forcing Nat-Translation might be a way around this.
Yes - Tunnels both ways shows up but vyos unable to route packet. Infact I added route on my droplet i.e. 172.16.3.0/24 NH 10.122.0.7
Still no luck - I am sure my packets are leaving on-prem network without any issues. I really doubt it would be eth1? since eth1 is my external interface - Any way I tried with eth3 as well but still no luck
set nat source rule 50 destination address '10.122.0.0/20'
set nat source rule 50 outbound-interface 'eth1'
set nat source rule 50 source address '172.16.3.0/24'
set nat source rule 50 translation address '10.122.0.7'
run show vpn ipsec sa
Connection State Up Bytes In/Out Remote address Remote ID Proposal
----------------------------- ------- ---------- -------------- ---------------- ----------- ---------------------------------------------------------
peer-xxx.xxx.xxx.250-tunnel-0 up 66 minutes 0B/0B xxx.xxx.xx.250 N/A AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_1024
ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eno16777984: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:3e:ae:d3 brd ff:ff:ff:ff:ff:ff
inet **172.16.3.72/24** brd 172.16.3.255 scope global noprefixroute eno16777984
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe3e:aed3/64 scope link noprefixroute
valid_lft forever preferred_lft forever
ping 10.122.0.2
PING 10.122.0.2 (10.122.0.2) 56(84) bytes of data.
^C^C
--- 10.122.0.2 ping statistics ---
12 packets transmitted, 0 received, 100% packet loss, time 11052ms
And this is my droplet
ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 02:cb:80:b1:e5:65 brd ff:ff:ff:ff:ff:ff
inet xxx.xxx.xxx.xx/20 brd xxx.xx.xx.255 scope global eth0
valid_lft forever preferred_lft forever
inet 10.47.0.5/16 brd 10.47.255.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::cb:80ff:feb1:e565/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 7a:07:88:b0:90:8d brd ff:ff:ff:ff:ff:ff
inet 10.122.0.2/20 brd 10.122.15.255 scope global eth1
valid_lft forever preferred_lft forever
inet6 fe80::7807:88ff:feb0:908d/64 scope link
valid_lft forever preferred_lft forever
default via 128.xxxx.xxxx.1 dev eth0 proto static
10.47.0.0/16 dev eth0 proto kernel scope link src 10.47.0.5
10.122.0.0/20 dev eth1 proto kernel scope link src 10.122.0.2
xxx.xxx.xxx.0/20 dev eth0 proto kernel scope link src xxx.xxx.xxx.xxx
172.16.3.0/24 via 10.122.0.7 dev eth1
show vpn ipsec sa shows no bytes being encrypted/decrypted
Why would you use rule like:
set nat source rule 50 destination address ‘10.122.0.0/20’
set nat source rule 50 outbound-interface ‘eth1’
set nat source rule 50 source address ‘172.16.3.0/24’
set nat source rule 50 translation address ‘10.122.0.7’
Translated packet will no longer match IPSEC policy
traffic passing vpn should (normally) be excluded from NAT.
I agree - then its definitely a limitation with Digital Ocean cloud routing platform. Droplets are unable to route traffic then.
Has anyone tried this before?
172.16.3.0/24 via 10.122.0.7 dev eth1 - this route on vyos or on another droplet?
If you are configuring nat on vyos you do not need to configure the droplet route to 172.16.3.0/24.
You need to configure the eth3 interface which is connected to the vpc network:
set nat source rule 50 outbound-interface 'eth3'
seems to me, your setup is feasable on DigitalOcean, it resembles “network with internet gateway” setup on
How to Configure a Droplet as a VPC Gateway :: DigitalOcean Documentation
Everything possibly I could do is already done; but still no luck.
Dang!! issue is resolved.I spent almost 2 days and night and you know the issue was digital ocean firewall which was blocking the connection. Even numerous times I allowed source 172.16.3.0/24 ALL TCP/ALL UDP/ICMP but it was not allowing.
Finally this morning I thought lets remove the firewall completely and see and bang my ping started immediately. I reapplied with all the exceptions and result is same; it is blocking connection. Hence finally I had to remove then entire firewall and activated firewall on vyos.
Thanks a lot everyone for your ideas and help; it was much appreciated.
This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.