Wireguard traffic ignores routing rules

General questions
1

I have a weird proble, that I can’t wrap my head around. I have a VyOS 1.3.5 instance with two Wireguard tunnels. I have bridge interface set up (mainly to have spare ethernet ports) for outgoing traffic. My network is multihomed and there are alternative routes to the remote VPN endpoints. The problem is, wireguard traffic chooses the alternative route even though I have explicitly set static routes for remote VPN endpoint via the bridge interface… I also tried setting local-route policy and use a separate routing table with it but nothing works… the traffic goes thro eth0 interface as even if it is hardcoded or something… there is no even route via this interface.

The network went through multiple iterations with different versions of VyOS and switching 3-4 ISPs over last 3 years and it was working reasonably fine for the most part. The VPN tunnels in question supposed to serve as backup and I’ve only recently noticed that they are not functioning when I needed to disconnect the primary provider for hardware replacement, so it is very hard to trace back changes to configuration at this time since it is unknown when it stopped working…

Is anyone aware of any similar issues with wireguard? Any help is appreciated.

2

You don’t post any config, so we can only guess.

If I had to bet, I’d guess you’re tripping over the 0.0.0.0/0 thing, which almost everyone seems to do at first.

3

I indeed use “that thing” :slight_smile:… I will look into this post in more details… I had issues with local IPv6 addresses in the past and using ::/0 was the workaround at the time. I suppose something has changed in the last couple of years.

Here is my configuration:

firewall {
     all-ping enable
     broadcast-ping disable
     config-trap disable
     ipv6-receive-redirects disable
     ipv6-src-route disable
     ip-src-route disable
     log-martians enable
     options {
         interface wg241 {
             adjust-mss clamp-mss-to-pmtu
             adjust-mss6 clamp-mss-to-pmtu
         }
         interface wg244 {
             adjust-mss clamp-mss-to-pmtu
             adjust-mss6 clamp-mss-to-pmtu
         }
     }
     receive-redirects disable
     send-redirects enable
     source-validation disable
     syn-cookies enable
     twa-hazards-protection disable
 }
 interfaces {
     bridge br0 {
         address dhcp
         description WAN-STARLINK
         dhcp-options {
             no-default-route
             reject xxx.xxx.100.1
         }
         member {
             interface eth1 {
             }
             interface eth4 {
             }
         }
     }
     ethernet eth0 {
         address xxxx:xxxx:6000:aa00::24/64
         address xxx.xxx.128.184/28
         description CORE
         hw-id xx:xx:xx:xx:xx:07
         ip {
             ospf {
                 cost 10
                 dead-interval 40
                 hello-interval 10
                 priority 1
                 retransmit-interval 5
                 transmit-delay 1
             }
         }
         ipv6 {
             ospfv3 {
                 cost 10
                 dead-interval 40
                 hello-interval 10
                 instance-id 0
                 priority 1
                 retransmit-interval 5
                 transmit-delay 1
             }
         }
         offload {
             gro
             gso
             sg
             tso
         }
     }
     ethernet eth1 {
         description WAN-VIRT
         hw-id xx:xx:xx:xx:xx:12
         offload {
             gro
             gso
             sg
             tso
         }
     }
     ethernet eth2 {
         address xxx.xxx.1.96/24
         address xxxx:xxxx:6000:aa10::9600/64
         description LAN-MGMT
         hw-id xx:xx:xx:xx:xx:d5
         ip {
             disable-forwarding
         }
         ipv6 {
             disable-forwarding
         }
         offload {
             gro
             gso
             sg
             tso
         }
         vrf MGMT
     }
     ethernet eth3 {
         hw-id xx:xx:xx:xx:xx:a2
     }
     ethernet eth4 {
         hw-id xx:xx:xx:xx:xx:a3
     }
     loopback lo {
         address xxx.xxx.128.134/32
         address xxxx:xxxx:6000:aa24::1/128
     }
     wireguard wg241 {
         address xxxx:xxxx:6000:aa24::1/128
         address xxx.xxx.128.152/31
         description "[r24-r1-tun0] to tor1"
         ip {
             ospf {
                 bfd
                 cost 205
                 dead-interval 40
                 hello-interval 10
                 priority 1
                 retransmit-interval 5
                 transmit-delay 1
             }
         }
         ipv6 {
             ospfv3 {
                 cost 205
                 dead-interval 40
                 hello-interval 10
                 instance-id 0
                 priority 1
                 retransmit-interval 5
                 transmit-delay 1
             }
         }
         mtu 1380
         peer to-R1 {
             address xxx.xxx.211.181
             allowed-ips ::/0
             allowed-ips xxx.xxx.0.0/0
             port 54511
             pubkey ZBojUU7LJuh7fKsuKLUI/HGIlHJgxlGsZ3T1qGzbKHE=
         }
         port 54511
         private-key xxxxxx
     }
     wireguard wg244 {
         address xxxx:xxxx:6000:aa24::1/128
         address xxx.xxx.128.144/31
         description "[r24-r4-tun0] to mtl1"
         ip {
             ospf {
                 bfd
                 cost 200
                 dead-interval 40
                 hello-interval 10
                 priority 1
                 retransmit-interval 5
                 transmit-delay 1
             }
         }
         ipv6 {
             ospfv3 {
                 cost 200
                 dead-interval 40
                 hello-interval 10
                 instance-id 0
                 priority 1
                 retransmit-interval 5
                 transmit-delay 1
             }
         }
         mtu 1380
         peer to-R4 {
             address xxx.xxx.22.174
             allowed-ips ::/0
             allowed-ips xxx.xxx.0.0/0
             port 52248
             pubkey 4jI9FCTWdA0mJi6imBmSsJwRh0NdlrLS4JbZ958+N20=
         }
         port 52248
         private-key xxxxxx
     }
 }
 policy {
     local-route {
         rule 10 {
             destination xxx.xxx.211.181
             destination xxx.xxx.22.174
             set {
                 table 110
             }
         }
     }
     prefix-list DEFAULT-ROUTE-IPV4 {
         rule 10 {
             action permit
             prefix xxx.xxx.0.0/0
         }
     }
     prefix-list STARLINK-MGMT-IPV4 {
         rule 10 {
             action permit
             prefix xxx.xxx.100.1/32
         }
     }
     prefix-list6 DEFAULT-ROUTE-IPV6 {
         rule 10 {
             action permit
             prefix ::/0
         }
     }
     route-map ALLOW-DEFAULT-ROUTE-IPV4 {
         rule 10 {
             action permit
             match {
                 ip {
                     address {
                         prefix-list DEFAULT-ROUTE-IPV4
                     }
                 }
             }
         }
     }
     route-map ALLOW-DEFAULT-ROUTE-IPV6 {
         rule 10 {
             action permit
             match {
                 ipv6 {
                     address {
                         prefix-list DEFAULT-ROUTE-IPV6
                     }
                 }
             }
         }
     }
     route-map ALLOW-STARLINK-MGMT {
         rule 10 {
             action permit
             match {
                 ip {
                     address {
                         prefix-list STARLINK-MGMT-IPV4
                     }
                 }
             }
         }
     }
     route-map DENY-DEFAULT-ROUTE-IPV4 {
         rule 10 {
             action deny
             match {
                 ip {
                     address {
                         prefix-list DEFAULT-ROUTE-IPV4
                     }
                 }
             }
         }
         rule 20 {
             action permit
         }
     }
     route-map DENY-DEFAULT-ROUTE-IPV6 {
         rule 10 {
             action deny
             match {
                 ipv6 {
                     address {
                         prefix-list DEFAULT-ROUTE-IPV6
                     }
                 }
             }
         }
         rule 20 {
             action permit
         }
     }
 }
 protocols {
     ospf {
         area 100 {
             area-type {
                 nssa {
                     translate candidate
                 }
             }
             network xxx.xxx.128.128/25
             network xxx.xxx.100.0/24
         }
         auto-cost {
             reference-bandwidth 10000
         }
         log-adjacency-changes {
         }
         parameters {
             abr-type cisco
             rfc1583-compatibility
             router-id xxx.xxx.128.134
         }
     }
     ospfv3 {
         area 0 {
             interface lo
             interface eth0
             interface wg243
             interface wg244
             interface wg241
         }
         parameters {
             router-id xxx.xxx.128.134
         }
     }
     static {
         interface-route xxx.xxx.211.181/32 {
             next-hop-interface br0 {
             }
         }
         interface-route xxx.xxx.22.174/32 {
             next-hop-interface br0 {
             }
         }
         route xxx.xxx.128.0/24 {
             blackhole {
             }
         }
         /* TOR1-R1 - Amanah */
         route xxx.xxx.211.176/29 {
             dhcp-interface br0
         }
         /* Servermania */
         route xxx.xxx.22.168/29 {
             dhcp-interface br0
         }
         route6 xxxx:xxxx:6000::/48 {
             blackhole {
             }
         }
         table 110 {
             route xxx.xxx.0.0/0 {
                 dhcp-interface br0
             }
         }
     }
     vrf MGMT {
         static {
             route xxx.xxx.0.0/0 {
                 next-hop xxx.xxx.1.1 {
                     next-hop-interface eth0
                 }
             }
             route6 ::/0 {
                 next-hop xxxx:xxxx:6000:aa10::1 {
                     interface eth0
                 }
             }
         }
     }
 }
 service {
     snmp {
         community public {
         }
         contact noc@alkeron.com
         location xxxxxx
         vrf MGMT
     }
     ssh {
         disable-password-authentication
         port 22
         vrf MGMT
     }
 }
 system {
     config-management {
         commit-revisions 100
     }
     conntrack {
         modules {
             ftp
             h323
             nfs
             pptp
             sip
             sqlnet
             tftp
         }
     }
     console {
         device ttyS0 {
             speed 115200
         }
     }
     host-name xxxxxx
     ip {
         multipath {
             layer4-hashing
         }
     }
     ipv6 {
         multipath {
             layer4-hashing
         }
     }
     login {
         user xxxxxx {
             authentication {
                 encrypted-password xxxxxx
                 public-keys xxxx@xxx.xxx {
                     key xxxxxx
                     type ssh-rsa
                 }
             }
         }
     }
     name-server xxxx:xxxx:80BB::10
     name-server xxxx:xxxx:80BC::10
     name-server xxx.xxx.121.10
     name-server xxx.xxx.122.10
     ntp {
         server xxxxx.tld {
         }
         server xxxxx.tld {
         }
         server xxxxx.tld {
         }
         server xxxxx.tld {
         }
     }
     option {
         performance latency
     }
     syslog {
         global {
             facility all {
                 level info
             }
             facility protocols {
                 level debug
             }
         }
     }
     time-zone America/Toronto
 }
 vrf {
     name MGMT {
         table 100
     }
 }
4

Yup, you have that allowed-ips multiple times. That’s not how Wireguard works, think of those “allowed-ips” are more of a “wireguard, route these networks”. You can’t have more than one default route active at a time. This isn’t a VyOS thing, it’s a Wireguard thing.

Once you fix that up, you might have other issues (I haven’t examined your config in detail) but fixing that up will get you much further.

5

They are on different interfaces, so it should be fine… actually, while tinkering with configuration of one of the tunnels, I switched to more specific IP ranges and then reverted back to 0.0.0.0/0 thing… and this tunnel started working properly… then I rebooted the router and the second tunnel started working properly again, so it sems that there may be a bug related some lingering state after all. I remember in the past there were several occasions, when I would copy a configuration, then remove it and apply it again… that may be one of those cases…

…but thanks for your tip anyway - it actually helped to get to the bottom of it :wink:

6

Sorry, you’re right, they’re on different WG interfaces, so that won’t have been the problem as you rightfully point out.

I still think that 0.0.0.0/0 probably isn’t what you want, unless you’re really expecting to have the entire Internet come in via that Interface at some stage. Which given the config I guess is possible. But yea.

Anyway glad it’s working, I’ll shut up now :slight_smile: