Trouble replicating routed /29 WireGuard setup from pfSense to VyOS

General questions
, , ,
1

Hey everyone,

I’m trying to replicate my working pfSense setup, a routed /29 network using WireGuard, on VyOS, but I’m running into issues. The same configuration works perfectly on pfSense and OPNsense, but I’d like to move to VyOS since it sits at the network edge.

The WireGuard tunnel comes up and passes traffic correctly, but as soon as I assign eth3 as a gateway (for example, .113) for my downstream firewall, all traffic stops. This setup works flawlessly on pfSense, so I’m not sure what I’m missing in VyOS.

I even wrote a blog post detailing the pfSense configuration in case it helps with troubleshooting.

I’d really appreciate any guidance or tips from anyone who’s tackled something similar.

Thanks,
John

2

I looked at the VyOS config you provided, and here’s what I’m seeing:

Detailed Analysis
  1. The first thing I’m noticing is this:
  1. And the second, and more important thing is this:
Summary

I’m kidding. Please provide your scrubbed VyOS config so we can see what’s going on.

5 Likes
3

vyos@vyos:~$ show configuration
interfaces {
ethernet eth0 {
address dhcp
description WAN
hw-id 00:26:b9:87:53:b4
offload {
gro
gso
sg
tso
}
}
ethernet eth1 {
address 192.168.5.1/24
description LAN
hw-id 00:26:b9:87:53:b5
offload {
gro
gso
sg
tso
}
}
ethernet eth2 {
address x.113/29
description “CoreTransit Gateway Downstream”
hw-id 00:10:18:6b:d4:58
mtu 1420
offload {
gro
gso
sg
tso
}
}
ethernet eth3 {
hw-id 00:10:18:6b:d4:5a
offload {
gro
gso
sg
tso
}
}
loopback lo {
}
wireguard wg0 {
address x.109/31
description “CoreTransit WireGuard Tunnel”
mtu 1420
peer hDqmR4VF {
address 134.199.56.1
allowed-ips 0.0.0.0/0
description “CoreTransit Peer”
persistent-keepalive 10
port 10253
public-key ****************
}
port 51820
private-key ****************
}
}
protocols {
static {
route 0.0.0.0/0 {
next-hop x.108 {
interface wg0
}
next-hop 192.168.1.1 {
}
}
}
}
service {
ntp {
allow-client {
address 127.0.0.0/8
address 169.254.0.0/16
address 10.0.0.0/8
address 172.16.0.0/12
address 192.168.0.0/16
address ::1/128
address fe80::/10
address fc00::/7
}
server time1.vyos.net {
}
server time2.vyos.net {
}
server time3.vyos.net {
}
}
ssh {
port 22
}
}
system {
config-management {
commit-revisions 100
}
console {
device ttyS0 {
speed 115200
}
}
host-name vyos
login {
user vyos {
authentication {
encrypted-password ****************
plaintext-password ****************
}
}
}
option {
reboot-on-upgrade-failure 5
}
syslog {
local {
facility all {
level info
}
facility local7 {
level debug
}
}
}
}

image
image610×380 86 KB

image
image1256×242 90.9 KB

You can see that the pfSense connected to Vyos is receiving traffic, but it’s unable to send outbound traffic. I think I might be missing a firewall rule on Vyos.

4

What is your topology? Is it like this?:

Internet --> VyOS --> pfSense?

If you’re going to have both VyOS and pfSense running, you won’t want both devices to have a LAN address of 192.168.5.1/24.

I’m assuming the IP address you’re receiving from your internet is in the 192.168.1.0/24 range (also assuming you-re on something with CG-NAT like Starlink or 5G). You’ll want to modify your static routes, since right now they’re load balancing. You want to prefer the paths over wireguard. You can do something like this:

delete protocols static route 0.0.0.0/0 next-hop x.108

set protocols static route 0.0.0.0/1 next-hop x.108
set protocols static route 128.0.0.0/1 next-hop x.108

set protocols static route 134.199.56.1/32 next-hop 192.168.1.1

You can see that the pfSense connected to Vyos is receiving traffic, but it’s unable to send outbound traffic. I think I might be missing a firewall rule on Vyos.

You’re not running the firewall right now on VyOS; it doesn’t run by default, you’ll need to configure it.

With that said, if you’re not sending from pfSense to VyOS, then the issue would be on pfSense. You can verify if the traffic is reaching VyOS by running this on VyOS and generating traffic from pfSense:

sudo tcpdump -ni eth2 src x.114

If traffic isn’t reaching VyOS, then your issue is on pfSense; likely routing or firewall. If traffic is reaching VyOS, then it may just be that:

  • The asymmetric routing causing issues with your traffic.
    • traffic came from CoreTransit, but returned via your normal internet connection due to the same cost static defaults
    • My suggested updates to your static routing should help fix that.
  • You were trying to return traffic to 134.199.56.1 via the wireguard tunnel itself, causing tunnel recursion (e.g. trying to send to the tunnel endpoint via the tunnel itself, which is obviuosly not possible).
    • My suggested updates to your static routing should also help fix that.

Once you get this working, you’ll definitely want to configure the firewall in VyOS since it has a public IP address. And of course make sure you replace and delete the default vyos user account.

5

Current Lab Setup:
Right now, my lab network is configured as follows:

Starlink → UDM → Port 1 → VyOS

The VyOS router is running a WireGuard tunnel to CoreTransit, which provides a /29 subnet.
On that subnet:

  • VyOS (eth2) has the gateway .113

  • pfSense is assigned .114

vyos@vyos:~$ sudo tcpdump -ni eth2 src 134.199.56.114
tcpdump: verbose output suppressed, use -v[v]… for full protocol decode
listening on eth2, link-type EN10MB (Ethernet), snapshot length 262144 bytes
20:50:45.192883 IP 134.199.56.114 > 134.199.56.113: ICMP echo request, id 11788, seq 212, length 9
20:50:45.725122 IP 134.199.56.114 > 134.199.56.113: ICMP echo request, id 11788, seq 213, length 9
20:50:46.100266 IP 134.199.56.114.51820 > 134.199.56.1.10253: UDP, length 148
20:50:46.257382 IP 134.199.56.114 > 134.199.56.113: ICMP echo request, id 11788, seq 214, length 9
20:50:46.789632 IP 134.199.56.114 > 134.199.56.113: ICMP echo request, id 11788, seq 215, length 9
20:50:47.321895 IP 134.199.56.114 > 134.199.56.113: ICMP echo request, id 11788, seq 216, length 9
20:50:47.854145 IP 134.199.56.114 > 134.199.56.113: ICMP echo request, id 11788, seq 217, length 9
20:50:48.386388 IP 134.199.56.114 > 134.199.56.113: ICMP echo request, id 11788, seq 218, length 9
20:50:48.918654 IP 134.199.56.114 > 134.199.56.113: ICMP echo request, id 11788, seq 219, length 9
20:50:49.450878 IP 134.199.56.114 > 134.199.56.113: ICMP echo request, id 11788, seq 220, length 9
20:50:49.968882 IP 134.199.56.114 > 134.199.56.113: ICMP echo request, id 11788, seq 221, length 9
20:50:50.501134 IP 134.199.56.114 > 134.199.56.113: ICMP echo request, id 11788, seq 222, length 9
20:50:51.033383 IP 134.199.56.114 > 134.199.56.113: ICMP echo request, id 11788, seq 223, length 9
20:50:51.196542 IP 134.199.56.114.51820 > 134.199.56.1.10253: UDP, length 148
20:50:51.565641 IP 134.199.56.114 > 134.199.56.113: ICMP echo request, id 11788, seq 224, length 9

I agree I was just trying to get the basic connectivity working before adding any firewall rules or additional configurations.

I also added the static route as you suggested, and it looks like the traffic is attempting to reach the gateway. However, I’m not entirely sure if that’s having the intended effect yet.

6

I got it to work, whats the best way to tighten the firewall rules on here. I basically want to deny everything to 109 from the WAN. But still get internet down that tunnel for routing

7

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.