200.X.X.X —> 187.X.X.X (peer) <-----------ipsec------------> 185.X.X.X (VyOS,peer) --NAT–> 192.168.X.21
Actors:
Left side IP 200.X.X.X, sends telnet probes each minute to 194.X.X.3:443 (DNAT/SNAT) which shoudl reach 192.168.X.21
The left site is configured by partner and I have no access to it.
Right side:
VyOS 1.3 eth0 (185.X.X.X), eth2 (192.168.X.1), dummy interface (194.X.X.3)
APP server eth1 (192.168.X.21)
The aim is to follow the way configuration is done https://docs.vyos.io/en/latest/configuration/nat/index.html “NAT before VPN”
VPN is IPSEC between hosts and works fine:
*vyos@vyos$ show vpn ipsec sa*
Connection State Uptime Bytes In/Out Packets In/Out Remote address Remote ID Proposal*
---------------------------- ------- -------- -------------- ---------------- ---------------- ----------- ---------------------------------------*
peer-187.X.X.X-tunnel-0 up 39m11s 6K/13K 117/234 187.X.X.X N/A 3DES_CBC/HMAC_SHA1_96/MODP_1536*
iptables - all policies are ACCEPT and only rules are
vyos@vyos# sudo iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
VYATTA_PRE_DNAT_HOOK all -- anywhere anywhere
LOG all -- 200.X.X.X 194.X.X.3 /* DST-NAT-108 */ LOG level warning prefix "[NAT-DST-108] "
DNAT all -- 200.X.X.X 194.X.X.3 /* DST-NAT-108 */ to:192.168.X.21
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
VYATTA_PRE_SNAT_HOOK all -- anywhere anywhere
LOG all -- 192.168.X.21 200.X.X.X /* SRC-NAT-108 */ LOG level warning prefix "[NAT-SRC-108] "
SNAT all -- 192.168.X.21 200.X.X.X /* SRC-NAT-108 */ to:194.X.X.3
ESPs are commin on eth0 ok. TCP packets on eth0 gets decrypted and DNAT rule is applied, I see this communication on eth0 of VyOS, and DNAT rule in logs, but tcp goes only one way:
tcpdump -ni eth0 host 200.X.X.X or host 192.168.X.21 or host 194.X.X.3 or host 192.168.X.1
06:58:02.120064 IP 200.X.X.X.50625 > 194.X.X.3.443: Flags [S], seq 3712955870, win 14600, options [mss 1460,sackOK,TS val 2453686457 ecr 0,nop,wscale 7], length 0
06:58:05.121212 IP 200.X.X.X.50625 > 194.X.X.3.443: Flags [S], seq 3712955870, win 14600, options [mss 1460,sackOK,TS val 2453687208 ecr 0,nop,wscale 7], length 0
06:58:10.119714 IP 200.X.X.X.50625 > 194.X.X.3.443: Flags [R.], seq 3712955871, ack 0, win 0, length 0
Two sync’s and then reset, the other side gets no answer and resets with “Connection refused”
TCP packets on eth2 on VyOS flow ok, both sides try to talk but none reaches the other
tcpdump -ni eth2 host 200.X.X.X or host 194.X.X.3
07:02:01.530554 IP 200.X.X.X.51108 > 192.168.X.21.443: Flags [S], seq 3378212280, win 14600, options [mss 1460,sackOK,TS val 2453746309 ecr 0,nop,wscale 7], length 0
07:02:01.531361 IP 192.168.X.21.443 > 200.X.X.X.51108: Flags [S.], seq 3293980511, ack 3378212281, win 65160, options [mss 1460,sackOK,TS val 995534613 ecr 2453746309,nop,wscale 7], length 0
07:02:02.538879 IP 192.168.X.21.443 > 200.X.X.X.51108: Flags [S.], seq 3293980511, ack 3378212281, win 65160, options [mss 1460,sackOK,TS val 995535621 ecr 2453746309,nop,wscale 7], length 0
07:02:04.529170 IP 200.X.X.X.51108 > 192.168.X.21.443: Flags [S], seq 3378212280, win 14600, options [mss 1460,sackOK,TS val 2453747060 ecr 0,nop,wscale 7], length 0
07:02:04.530015 IP 192.168.X.21.443 > 200.X.X.X.51108: Flags [S.], seq 3293980511, ack 3378212281, win 65160, options [mss 1460,sackOK,TS val 995537612 ecr 2453746309,nop,wscale 7], length 0
07:02:06.538949 IP 192.168.X.21.443 > 200.X.X.X.51108: Flags [S.], seq 3293980511, ack 3378212281, win 65160, options [mss 1460,sackOK,TS val 995539621 ecr 2453746309,nop,wscale 7], length 0
07:02:09.531123 IP 200.X.X.X.51108 > 192.168.X.21.443: Flags [R.], seq 1, ack 1000986785, win 0, length 0
07:02:10.698941 IP 192.168.X.21.443 > 200.X.X.X.51108: Flags [S.], seq 3293980511, ack 3378212281, win 65160, options [mss 1460,sackOK,TS val 995543781 ecr 2453746309,nop,wscale 7], length 0
The same as on eth2 is tcp communication on eth1 of application server 192.168.X.21 so useless to paste it.
The ending → up till eth2, communication is fine, where packets dissapear after that I cannot say. SNAT is not being applied. It does not appear in log file and obviously “monitor nat source rule 108” does show nothing. DNAT rule looks ok. What is interesting is that when I initiate telnet/netcat probe to 200.X.X.X from 192.168.X.21, SNAT is executed but I got connection timeout (maybe there is an issue on their side too). But this is my secondary problem, at the moment I cannot explain why SNAT is not executed and where are packets being routed.
On app server 192.168.X.21, static route for the remote IP is added via VyOS router eth2 interface:
200.X.X.0/24 via 192.168.X.1 dev eth1 proto static
On vyOS router, there is 220 table entry and routing:
ip xfrm policy:
src 194.X.X.3/32 dst 200.X.X.X/32
dir out priority 367231 ptype main
tmpl src 185.X.X.X dst 187.X.X.X
proto esp spi 0x1a49cc29 reqid 7 mode tunnel
src 200.X.X.X/32 dst 194.X.X.3/32
dir fwd priority 367231 ptype main
tmpl src 187.X.X.X dst 185.X.X.X
proto esp reqid 7 mode tunnel
src 200.X.X.X/32 dst 194.X.X.3/32
dir in priority 367231 ptype main
tmpl src 187.X.X.X dst 185.X.X.X
proto esp reqid 7 mode tunnel
ip route get to 200.X.X.X
200.X.X.X via 185.X.X.1 dev eth0 table 220 src 194.X.X.3 uid 1003
cache
It seems that response packets never reach eth0 and SNAT - even set to ‘any’ interface is not being done.
I tried to set log_martians but nothing is reported. I have tried to pass through kernel params with manual to see something suspicious, they are below. I played with
sudo sysctl -w net.ipv4.conf.all.log_martians=1
sudo sysctl -w net.ipv4.conf.eth0.log_martians=1
sudo sysctl -w net.ipv4.conf.eth0.route_localnet=1
sudo sysctl -w net.ipv4.conf.all.route_localnet=1
sudo sysctl -w net.ipv4.conf.eth0.accept_local=1
sudo sysctl -w net.ipv4.conf.all.accept_local=1
But no change, packets don’t get to eth0, no matter what I do. Any advice very welcome, I think I have all setup done properly.