site-to-site vpn vti0 source routing

General questions
1

Hi guys,

I need some help. I have got a vyatta router configured in my private network and another one running vpn with the public IP with my other ISP. I created an IPSEC tunel between them and connected it. So the situation looks like:

MY LAN (192.168.1.0/24) → (192.168.1.251) vyatta1 (vti0/ 192.168.3.249/30) → (vti0 192.168.4.249/30) vyatta2 (eth0 PUBLIC_IP)

The tunnel is connected over the 2 public ISPs. I forwarded port 500 and 4500 on my ISP router at home so I can connect vyatta1 with vyatta2 over the tunnel ID.

Now… I’m trying to create a source route which will generally forward a traffic from MY LAN trough the tunnel and SNAT it on vyatta2 over the public IP. The problem is that doesn’t work for certain things and I don’t understand why. From what I have found it is related to the rp tables.

As an example on my Linux box 192.168.1.250 I created a static route:
ip ro add 208.64.38.55 via 192.168.1.251

208.64.38.55 is an IP for www.whatsmyip.org so I can check what IP I’m going out on.

When I curl www.whatsmyip.org it’s timing out. Ping works…

[root@linux ~]# ping www.whatsmyip.org
PING www.whatsmyip.org (208.64.38.55) 56(84) bytes of data.
64 bytes from whatsmyip.org (208.64.38.55): icmp_seq=1 ttl=48 time=124 ms
64 bytes from whatsmyip.org (208.64.38.55): icmp_seq=2 ttl=48 time=124 ms
64 bytes from whatsmyip.org (208.64.38.55): icmp_seq=3 ttl=48 time=125 ms
^C
— www.whatsmyip.org ping statistics —
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 124.538/124.848/125.194/0.488 ms

but traceroute does not…
[root@linux ~]# traceroute -I www.whatsmyip.org
traceroute to www.whatsmyip.org (208.64.38.55), 30 hops max, 60 byte packets
1 192.168.1.251 (192.168.1.251) 0.648 ms 0.898 ms 0.981 ms
2 192.168.4.249 (192.168.4.249) 18.572 ms 19.628 ms 19.828 ms
3 * * *
4 * * *
…
30 * * *

tcpdump can see the traffic flow:
vyos@vyos2# sudo tcpdump -nnv host 208.64.38.55
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
21:42:24.358755 IP (tos 0x0, ttl 62, id 2639, offset 0, flags [DF], proto TCP (6), length 60)
PUBLIC_IP.53005 > 208.64.38.55.80: Flags [S], cksum 0x2708 (correct), seq 3064943484, win 29200, options [mss 1460,sackOK,TS val 1069766963 ecr 0,nop,wscale 7], length 0
21:42:25.359015 IP (tos 0x0, ttl 62, id 2640, offset 0, flags [DF], proto TCP (6), length 60)
PUBLIC_IP.53005 > 208.64.38.55.80: Flags [S], cksum 0x231f (correct), seq 3064943484, win 29200, options [mss 1460,sackOK,TS val 1069767964 ecr 0,nop,wscale 7], length 0
21:42:27.363405 IP (tos 0x0, ttl 62, id 2641, offset 0, flags [DF], proto TCP (6), length 60)
PUBLIC_IP.53005 > 208.64.38.55.80: Flags [S], cksum 0x1b4b (correct), seq 3064943484, win 29200, options [mss 1460,sackOK,TS val 1069769968 ecr 0,nop,wscale 7], length 0
21:42:31.371342 IP (tos 0x0, ttl 62, id 2642, offset 0, flags [DF], proto TCP (6), length 60)
PUBLIC_IP.53005 > 208.64.38.55.80: Flags [S], cksum 0x0ba3 (correct), seq 3064943484, win 29200, options [mss 1460,sackOK,TS val 1069773976 ecr 0,nop,wscale 7], length 0
21:42:39.379036 IP (tos 0x0, ttl 62, id 2643, offset 0, flags [DF], proto TCP (6), length 60)
PUBLIC_IP.53005 > 208.64.38.55.80: Flags [S], cksum 0xec5a (correct), seq 3064943484, win 29200, options [mss 1460,sackOK,TS val 1069781984 ecr 0,nop,wscale 7], length 0
^C
5 packets captured
5 packets received by filter
0 packets dropped by kernel
[edit]
vyos@vyos2# sudo tcpdump -nnv host 208.64.38.55 -i vti0
tcpdump: listening on vti0, link-type RAW (Raw IP), capture size 65535 bytes
21:42:49.701575 IP (tos 0x0, ttl 63, id 49586, offset 0, flags [DF], proto TCP (6), length 60)
192.168.1.250.53006 > 208.64.38.55.80: Flags [S], cksum 0x00fe (correct), seq 3540155776, win 29200, options [mss 1460,sackOK,TS val 1069792306 ecr 0,nop,wscale 7], length 0
21:42:50.702930 IP (tos 0x0, ttl 63, id 49587, offset 0, flags [DF], proto TCP (6), length 60)
192.168.1.250.53006 > 208.64.38.55.80: Flags [S], cksum 0xfd13 (correct), seq 3540155776, win 29200, options [mss 1460,sackOK,TS val 1069793308 ecr 0,nop,wscale 7], length 0
21:42:52.707080 IP (tos 0x0, ttl 63, id 49588, offset 0, flags [DF], proto TCP (6), length 60)
192.168.1.250.53006 > 208.64.38.55.80: Flags [S], cksum 0xf53f (correct), seq 3540155776, win 29200, options [mss 1460,sackOK,TS val 1069795312 ecr 0,nop,wscale 7], length 0
21:42:56.714999 IP (tos 0x0, ttl 63, id 49589, offset 0, flags [DF], proto TCP (6), length 60)
192.168.1.250.53006 > 208.64.38.55.80: Flags [S], cksum 0xe597 (correct), seq 3540155776, win 29200, options [mss 1460,sackOK,TS val 1069799320 ecr 0,nop,wscale 7], length 0
^C
4 packets captured
4 packets received by filter
0 packets dropped by kernel
[edit]

NAT seems to be configured correctly on vyatta2:
vyos@vyos# show nat
source {
rule 1 {
log enable
outbound-interface eth0
protocol all
source {
address 192.168.1.250/32
}
translation {
address masquerade
}
}
}

Routing seems to be working fine both ways:
vyos@vyos2# ping 192.168.1.250
PING 192.168.1.250 (192.168.1.250) 56(84) bytes of data.
64 bytes from 192.168.1.250: icmp_req=1 ttl=63 time=17.5 ms
64 bytes from 192.168.1.250: icmp_req=2 ttl=63 time=18.0 ms
64 bytes from 192.168.1.250: icmp_req=3 ttl=63 time=18.7 ms
^C
— 192.168.1.250 ping statistics —
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 17.531/18.098/18.756/0.515 ms

[root@linux ~]# ping 192.168.4.249
PING 192.168.4.249 (192.168.4.249) 56(84) bytes of data.
64 bytes from 192.168.4.249: icmp_seq=1 ttl=63 time=18.1 ms
64 bytes from 192.168.4.249: icmp_seq=2 ttl=63 time=18.2 ms
^C
— 192.168.4.249 ping statistics —
3 packets transmitted, 2 received, 33% packet loss, time 2002ms
rtt min/avg/max/mdev = 18.136/18.188/18.241/0.144 ms

I can also ping vyatta1 both interfaces:
[root@linux ~]# ping 192.168.3.249
PING 192.168.3.249 (192.168.3.249) 56(84) bytes of data.
64 bytes from 192.168.3.249: icmp_seq=1 ttl=64 time=0.532 ms
64 bytes from 192.168.3.249: icmp_seq=2 ttl=64 time=0.580 ms
^C
— 192.168.3.249 ping statistics —
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.532/0.556/0.580/0.024 ms
[root@linux ~]# ping 192.168.1.251
PING 192.168.1.251 (192.168.1.251) 56(84) bytes of data.
64 bytes from 192.168.1.251: icmp_seq=1 ttl=64 time=0.627 ms
^C
— 192.168.1.251 ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.627/0.627/0.627/0.000 ms

So now on my linux box… if I do
echo 2 > /proc/sys/net/ipv4/conf/all/rp_filter

Traceroute starts working fine:
[root@brama-dom ~]# traceroute -I www.whatsmyip.org
traceroute to www.whatsmyip.org (208.64.38.55), 30 hops max, 60 byte packets
1 192.168.1.251 (192.168.1.251) 0.617 ms 0.842 ms 1.047 ms
2 192.168.4.249 (192.168.4.249) 20.073 ms 20.316 ms 20.662 ms
3 REMOTE_PUBLIC_GW (REMOTE_PUBLIC_GW) 25.479 ms 26.186 ms 26.372 ms
…
11 vl-5.car1.Detroit1.Level3.net (4.69.202.234) 263.467 ms 231.195 ms 232.114 ms
12 EASY-ONLINE.car1.Detroit1.Level3.net (4.31.123.98) 119.568 ms 119.389 ms 119.492 ms
13 core5.tym.r256.net (173.225.176.93) 124.623 ms 123.764 ms 124.720 ms
14 agr3.tym.r256.net (173.225.185.38) 119.541 ms 118.948 ms 119.671 ms
15 whatsmyip.org (208.64.38.55) 125.144 ms 125.335 ms 126.250 ms

but curl times out.

What did I miss ?

Any help really appreciated.

Thanks in advance!

2

-Make sure VTI interfaces are in same segment 192.168.3.249/30 and 192.168.3.250/30, so routing protocol like OSPF can be used on it.
But the real problem here might be mss-clamp. Probably now, endpoints will decide on mss value during TCP connection setup…which doesn’t fit in the tunnel.

Over here I’m using code below to mss-clamp: (can’t apply the policy to VTI tunnel itself, is there a better way to do it)

set policy route POL_ETH0_IN rule 10 description ‘MSS-CLAMPforVPN’
set policy route POL_ETH0_IN rule 10 destination group network-group ‘OFFICE_INSIDE’
set policy route POL_ETH0_IN rule 10 protocol ‘tcp’
set policy route POL_ETH0_IN rule 10 set tcp-mss ‘1358’
set policy route POL_ETH0_IN rule 10 tcp flags ‘SYN’
set interfaces ethernet eth0 policy route ‘POL_ETH0_IN’

3

Hi,

Thanks for your reply.

I think I have actually found the problem in the PBR I believe.

Generally as you could see the rule 1 I posted is related to any traffic really. Hence all the confusion. I only have a 1 interface so there is no way I could divide local and external traffic as they come on the same interface. By putting 0.0.0.0/0 I made the routing more complicated and confusing.

Once I changed that 0.0.0.0/0 to the IP address of the remote end so for example 8.8.8.8/32 … traffic flows nicely (www but not ping - which is quite confusing too).

Anyway I think I need to find a way to exclude local subnets from the PBR so it knows how to route them or create OSPF between both vyattas. My VTIs now are within the same subnet so they can reach each other as soon as the tunnel is up.

I will keep diggin.