Routing Issue Over Wireguard?

General questions
, ,
1

I have two sites each with a vyos firewall, connected via wireguard. On site 1 I have a kubernetes cluster advertising routes via bgp. I cannot reach the advertised route from site 2 over the tunnel and I’m not sure why. E.g. 10.5.21.12 is unable to reach 10.4.168.3. Hosts on the r1 network are able to reach 10.4.168.3 just fine though.

r1
r1-config.txt (14.5 KB)

[email protected]# ip r
default nhid 71 via <public ip> dev eth0 proto static metric 20 
10.4.10.0/24 dev bond0.10 proto kernel scope link src 10.4.10.1 
10.4.20.0/24 dev bond0.20 proto kernel scope link src 10.4.20.1 
10.4.30.0/24 dev bond0.30 proto kernel scope link src 10.4.30.1 
10.4.68.0/24 dev bond0.68 proto kernel scope link src 10.4.68.1 
10.4.168.2 nhid 223 via 10.4.68.21 dev bond0.68 proto bgp metric 20 
10.4.168.3 nhid 223 via 10.4.68.21 dev bond0.68 proto bgp metric 20 
10.4.168.4 nhid 223 via 10.4.68.21 dev bond0.68 proto bgp metric 20 
10.5.0.0/16 nhid 40 via 172.16.10.2 dev wg01 proto static metric 20 onlink 
<default gateway> dev eth0 proto kernel scope link src <public ip>
172.16.10.2 nhid 36 dev wg01 proto static metric 20 
172.16.10.101 nhid 36 dev wg01 proto static metric 20 
192.168.20.0/24 dev br0.920 proto kernel scope link src 192.168.20.1

r2
r2-config.txt (12.2 KB)

[email protected]:~$ ip r
default nhid 127 via 192.168.0.1 dev eth0 proto static metric 20 
10.4.0.0/16 nhid 27 via 172.16.10.1 dev wg01 proto static metric 20 onlink 
10.5.10.0/24 dev eth3 proto kernel scope link src 10.5.10.1 
10.5.20.0/24 dev eth3.520 proto kernel scope link src 10.5.20.1 
10.5.21.0/24 dev eth3.521 proto kernel scope link src 10.5.21.1 
10.5.22.0/24 dev eth3.522 proto kernel scope link src 10.5.22.1 
10.5.30.0/24 dev eth3.530 proto kernel scope link src 10.5.30.1 
10.5.31.0/24 dev eth3.531 proto kernel scope link src 10.5.31.1 
10.5.32.0/24 dev eth3.532 proto kernel scope link src 10.5.32.1 
10.5.42.0/24 dev eth3.542 proto kernel scope link src 10.5.42.1 
10.68.0.0/16 nhid 27 via 172.16.10.1 dev wg01 proto static metric 20 onlink 
172.16.10.0/24 dev wg01 proto kernel scope link src 172.16.10.2 
172.16.10.1 nhid 24 dev wg01 proto static metric 20 
192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.2
2

It looks like I can reach 10.4.168.3 from r2 itself, but not from clients in the r2 10.5.0.0/16 network.

3

You’ve got two static routes on r1 for 10.5.0.0/16 ?

route 10.5.0.0/16 {
            next-hop 172.16.10.1 {
            }
            next-hop 172.16.10.2 {
            }
        }
1 Like
4

Whoops not sure how that slipped in there but deleting next-hop 172.16.10.1 had no effect. That was address for the local side of the wireguard tunnel on r1. Should just be the far side now.

5

What do you see with a bit of tcpdumping around the place.
Are the packets making it there?

1 Like
6

This is what I’m seeing on the destination. It appears packets are making it there but not returning.

tcpdump | grep  10.4.168.3 | grep 10.5.21.12
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
23:21:54.146799 IP 10.5.21.12.57040 > 10.4.168.3.443: Flags [.], seq 2619893649:2619893650, ack 318988070, win 1026, length 1
23:21:54.146867 IP 10.4.168.3.443 > 10.5.21.12.57040: Flags [.], ack 1, win 500, options [nop,nop,sack 1 {0:1}], length 0
23:22:04.232867 IP 10.5.21.12.57040 > 10.4.168.3.443: Flags [.], seq 0:1, ack 1, win 1026, length 1
23:22:04.232933 IP 10.4.168.3.443 > 10.5.21.12.57040: Flags [.], ack 1, win 500, options [nop,nop,sack 1 {0:1}], length 0
23:22:15.529639 IP 10.4.168.3.443 > 10.5.21.12.57096: Flags [.], seq 2377331330:2377332790, ack 1145747619, win 500, length 1460
23:22:15.529887 IP 10.4.68.1 > 10.4.168.3: ICMP 10.5.21.12 unreachable - need to frag (mtu 1420), length 556
23:22:15.529899 IP 10.4.68.1 > 10.4.168.3: ICMP 10.5.21.12 unreachable - need to frag (mtu 1420), length 556
23:22:27.979862 IP 10.5.21.12.57096 > 10.4.168.3.443: Flags [F.], seq 1, ack 0, win 1024, length 0
23:22:28.021529 IP 10.4.168.3.443 > 10.5.21.12.57096: Flags [.], ack 2, win 500, length 0
23:22:33.449561 IP 10.4.168.3.443 > 10.5.21.12.57096: Flags [.], seq 0:1460, ack 2, win 500, length 1460
23:22:33.449850 IP 10.4.68.1 > 10.4.168.3: ICMP 10.5.21.12 unreachable - need to frag (mtu 1420), length 556
23:22:33.449865 IP 10.4.68.1 > 10.4.168.3: ICMP 10.5.21.12 unreachable - need to frag (mtu 1420), length 556
23:22:34.474002 IP 10.5.21.12.57040 > 10.4.168.3.443: Flags [.], seq 0:1, ack 1, win 1026, length 1
23:22:34.474075 IP 10.4.168.3.443 > 10.5.21.12.57040: Flags [.], ack 1, win 500, options [nop,nop,sack 1 {0:1}], length 0
23:22:38.102156 IP 10.5.21.12.57096 > 10.4.168.3.443: Flags [.], seq 1:2, ack 0, win 1024, length 1
23:22:38.102220 IP 10.4.168.3.443 > 10.5.21.12.57096: Flags [.], ack 2, win 500, options [nop,nop,sack 1 {1:2}], length 0
23:22:44.548025 IP 10.5.21.12.57040 > 10.4.168.3.443: Flags [.], seq 0:1, ack 1, win 1026, length 1
23:22:44.548083 IP 10.4.168.3.443 > 10.5.21.12.57040: Flags [.], ack 1, win 500, options [nop,nop,sack 1 {0:1}], length 0
23:22:48.179079 IP 10.5.21.12.57096 > 10.4.168.3.443: Flags [.], seq 1:2, ack 0, win 1024, length 1
23:22:48.179151 IP 10.4.168.3.443 > 10.5.21.12.57096: Flags [.], ack 2, win 500, options [nop,nop,sack 1 {1:2}], length 0
23:22:54.635827 IP 10.5.21.12.57040 > 10.4.168.3.443: Flags [.], seq 0:1, ack 1, win 1026, length 1
23:22:54.635883 IP 10.4.168.3.443 > 10.5.21.12.57040: Flags [.], ack 1, win 500, options [nop,nop,sack 1 {0:1}], length 0
23:22:58.258992 IP 10.5.21.12.57096 > 10.4.168.3.443: Flags [.], seq 1:2, ack 0, win 1024, length 1
23:22:58.259052 IP 10.4.168.3.443 > 10.5.21.12.57096: Flags [.], ack 2, win 500, options [nop,nop,sack 1 {1:2}], length 0
23:23:04.710001 IP 10.5.21.12.57040 > 10.4.168.3.443: Flags [.], seq 0:1, ack 1, win 1026, length 1
23:23:04.710081 IP 10.4.168.3.443 > 10.5.21.12.57040: Flags [.], ack 1, win 500, options [nop,nop,sack 1 {0:1}], length 0
23:23:08.336063 IP 10.5.21.12.57096 > 10.4.168.3.443: Flags [.], seq 1:2, ack 0, win 1024, length 1
23:23:08.336148 IP 10.4.168.3.443 > 10.5.21.12.57096: Flags [.], ack 2, win 500, options [nop,nop,sack 1 {1:2}], length 0
23:23:10.313642 IP 10.4.168.3.443 > 10.5.21.12.57096: Flags [.], seq 0:1460, ack 2, win 500, length 1460
23:23:10.313930 IP 10.4.68.1 > 10.4.168.3: ICMP 10.5.21.12 unreachable - need to frag (mtu 1420), length 556
23:23:10.313944 IP 10.4.68.1 > 10.4.168.3: ICMP 10.5.21.12 unreachable - need to frag (mtu 1420), length 556
23:23:14.791025 IP 10.5.21.12.57040 > 10.4.168.3.443: Flags [.], seq 0:1, ack 1, win 1026, length 1
23:23:14.791111 IP 10.4.168.3.443 > 10.5.21.12.57040: Flags [.], ack 1, win 500, options [nop,nop,sack 1 {0:1}], length 0
23:23:18.412697 IP 10.5.21.12.57096 > 10.4.168.3.443: Flags [.], seq 1:2, ack 0, win 1024, length 1
23:23:18.412787 IP 10.4.168.3.443 > 10.5.21.12.57096: Flags [.], ack 2, win 500, options [nop,nop,sack 1 {1:2}], length 0
23:23:23.967752 IP 10.5.21.12.57040 > 10.4.168.3.443: Flags [R.], seq 1, ack 1, win 0, length 0
23:23:28.496303 IP 10.5.21.12.57096 > 10.4.168.3.443: Flags [.], seq 1:2, ack 0, win 1024, length 1
23:23:28.496366 IP 10.4.168.3.443 > 10.5.21.12.57096: Flags [.], ack 2, win 500, options [nop,nop,sack 1 {1:2}], length 0
23:23:38.580968 IP 10.5.21.12.57096 > 10.4.168.3.443: Flags [.], seq 1:2, ack 0, win 1024, length 1
23:23:38.581018 IP 10.4.168.3.443 > 10.5.21.12.57096: Flags [.], ack 2, win 500, options [nop,nop,sack 1 {1:2}], length 0
23:23:48.660776 IP 10.5.21.12.57096 > 10.4.168.3.443: Flags [.], seq 1:2, ack 0, win 1024, length 1
23:23:48.660841 IP 10.4.168.3.443 > 10.5.21.12.57096: Flags [.], ack 2, win 500, options [nop,nop,sack 1 {1:2}], length 0
7

I seem them here?

23:22:34.474002 IP 10.5.21.12.57040 > 10.4.168.3.443: Flags [.], seq 0:1, ack 1, win 1026, length 1

You can use “tcpdump host ” instead of that grep thing.

1 Like
8

Sorry, not well versed in tcpdump, I was looking at 23:23:10.313930 IP 10.4.68.1 > 10.4.168.3: ICMP 10.5.21.12 unreachable....

9

Right, might be an MTU problem (Wireguard tunnels are small)

Does ICMP/Ping work OK between the hosts?

Try this on both routers.

set interfaces wireguard wg01 ip adjust-mss '1380'
1 Like
10

I’m not seeing that configuration path. Looks like I’m quite out of date on 1.3.4.

Unfortunately the load balancer vip 10.4.168.3 isn’t open for ICMP. The underlying host 10.4.68.21 has no problems pinging 10.5.21.12 in either direction.

11

set firewall options interface wg01 adjust-mss 1300 for 1.3

2 Likes
12

That did the trick, thank you so much!

1 Like
13

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.