Policy Routing Not Working Wireguard

Hi,

I have an issue where I have set policy routing rules to route based on source ip or destination ip over a wireguard vpn.

The issue is on a reboot the routing doesnt work. After much troubleshooting it seems the only way i can get it to work is to run these commands

del static table 20 interface-route
commit
set static table 20 interface-route 0.0.0.0/0 next-hop-interface wg2
commit

(where wg2 is my wireguard interface)

this will work 100% until the next reboot.

Any ideas?

Log a ticket, it sounds like you’ve found a reproducible bug!

Hi @phasma , could you please share VyOS version you’re facing this issue as well as:

show configuration commands | strip-private

cat /var/log/messages | strip-private (after the router bootup process)

Thanks.

Running 1.3 RC 6

set firewall all-ping 'enable'
set firewall broadcast-ping 'disable'
set firewall config-trap 'enable'
set firewall group address-group ADR-ORYX address 'xxx.xxx.83.4'
set firewall group address-group ADR-PIHOLE address 'xxx.xxx.83.251'
set firewall group address-group ADR-SABLE address 'xxx.xxx.83.250'
set firewall group address-group ADR-BLOCK-LAN-IN description 'Block Internet access on these hosts.'
set firewall group address-group ADR-DNS-EXTERNAL address 'xxx.xxx.1.1'
set firewall group address-group ADR-DNS-EXTERNAL address 'xxx.xxx.9.9'
set firewall group address-group ADR-DNS-EXTERNAL address 'xxx.xxx.0.1'
set firewall group address-group ADR-DNS-EXTERNAL address 'xxx.xxx.112.112'
set firewall group address-group ADR-DNS-INTERNAL address 'xxx.xxx.83.250'
set firewall group address-group ADR-DNS-INTERNAL address 'xxx.xxx.83.251'
set firewall group address-group ADR-DNS-INTERNAL address 'xxx.xxx.83.4'
set firewall group address-group ADR-ROUTE-TG-EU-UK description 'Route Hosts over WG1'
set firewall group address-group ADR-ROUTE-TG-USA-DEDI-NY description 'Route Hosts over USA dedicated IP'
set firewall group network-group NET-LAN network 'xxx.xxx.83.0/24'
set firewall group network-group NET-THINKBROADBAND network 'xxx.xxx.99.160/28'
set firewall group network-group NET-THINKBROADBAND network 'xxx.xxx.216.159/32'
set firewall group network-group NET-OPENVPN-SITE-TO-SITE network 'xxx.xxx.10.0/28'
set firewall group network-group NET-ROUTE-DEDI-USA-NY description 'Route via USA-NY'
set firewall group network-group NET-STREAMING-DISNEY network 'xxx.xxx.221.26/32'
set firewall group network-group NET-STREAMING-PANDORA network 'xxx.xxx.161.0/24'
set firewall group network-group NET-STREAMING-PANDORA network 'xxx.xxx.162.0/23'
set firewall group network-group NET-STREAMING-PANDORA network 'xxx.xxx.164.0/23'
set firewall group network-group NET-STREAMING-PANDORA network 'xxx.xxx.40.0/21'
set firewall group network-group NET-STREAMING-PANDORA network 'xxx.xxx.209.0/24'
set firewall group port-group PORT-WEB-HTTPs port 'http'
set firewall group port-group PORT-WEB-HTTPs port 'https'
set firewall ipv6-receive-redirects 'disable'
set firewall ipv6-src-route 'disable'
set firewall ip-src-route 'disable'
set firewall log-martians 'enable'
set firewall name DROP-ALL default-action 'drop'
set firewall name DROP-ALL description 'Block Inbound OpenVPN Client'
set firewall name DROP-ALL enable-default-log
set firewall name DROP-ALL rule 10 action 'drop'
set firewall name DROP-ALL rule 10 log 'enable'
set firewall name LAN-IN default-action 'accept'
set firewall name LAN-IN rule 1 action 'accept'
set firewall name LAN-IN rule 1 state established 'enable'
set firewall name LAN-IN rule 1 state related 'enable'
set firewall name LAN-IN rule 2 action 'drop'
set firewall name LAN-IN rule 2 log 'enable'
set firewall name LAN-IN rule 2 state invalid 'enable'
set firewall name LAN-IN rule 10 action 'drop'
set firewall name LAN-IN rule 10 destination group address-group 'ADR-DNS-EXTERNAL'
set firewall name LAN-IN rule 10 destination port 'domain'
set firewall name LAN-IN rule 10 log 'enable'
set firewall name LAN-IN rule 10 protocol 'tcp_udp'
set firewall name LAN-IN rule 10 source group address-group '!ADR-DNS-INTERNAL'
set firewall name LAN-IN rule 20 action 'drop'
set firewall name LAN-IN rule 20 protocol 'all'
set firewall name LAN-IN rule 20 source group address-group 'ADR-BLOCK-LAN-IN'
set firewall name OPENVPN-IN default-action 'drop'
set firewall name OPENVPN-IN description 'Site-To-Site VPN'
set firewall name OPENVPN-IN enable-default-log
set firewall name OPENVPN-IN rule 10 action 'accept'
set firewall name OPENVPN-IN rule 10 state established 'enable'
set firewall name OPENVPN-IN rule 10 state related 'enable'
set firewall name OPENVPN-IN rule 20 action 'drop'
set firewall name OPENVPN-IN rule 20 log 'enable'
set firewall name OPENVPN-IN rule 20 state invalid 'enable'
set firewall name OPENVPN-IN rule 30 action 'accept'
set firewall name OPENVPN-IN rule 30 description 'Allow ICMP to SABLE'
set firewall name OPENVPN-IN rule 30 destination group address-group 'ADR-SABLE'
set firewall name OPENVPN-IN rule 30 icmp type-name 'echo-request'
set firewall name OPENVPN-IN rule 30 protocol 'icmp'
set firewall name OPENVPN-IN rule 30 source group network-group 'NET-OPENVPN-SITE-TO-SITE'
set firewall name OPENVPN-IN rule 40 action 'accept'
set firewall name OPENVPN-IN rule 40 description 'Allow HTTP/s to SABLE'
set firewall name OPENVPN-IN rule 40 destination group address-group 'ADR-SABLE'
set firewall name OPENVPN-IN rule 40 destination group port-group 'PORT-WEB-HTTPs'
set firewall name OPENVPN-IN rule 40 protocol 'tcp'
set firewall name OPENVPN-IN rule 40 source group network-group 'NET-OPENVPN-SITE-TO-SITE'
set firewall name OPENVPN-IN rule 50 action 'accept'
set firewall name OPENVPN-IN rule 50 description 'Allow DNS to SABLE'
set firewall name OPENVPN-IN rule 50 destination group address-group 'ADR-SABLE'
set firewall name OPENVPN-IN rule 50 destination port 'domain'
set firewall name OPENVPN-IN rule 50 protocol 'tcp_udp'
set firewall name OPENVPN-IN rule 50 source group network-group 'NET-OPENVPN-SITE-TO-SITE'
set firewall name WAN-IN default-action 'drop'
set firewall name WAN-IN rule 10 action 'accept'
set firewall name WAN-IN rule 10 disable
set firewall name WAN-IN rule 10 state established 'enable'
set firewall name WAN-IN rule 10 state related 'enable'
set firewall name WAN-IN rule 20 action 'drop'
set firewall name WAN-IN rule 20 log 'enable'
set firewall name WAN-IN rule 20 state invalid 'enable'
set firewall name WAN-IN rule 30 action 'accept'
set firewall name WAN-IN rule 30 destination group address-group 'ADR-SABLE'
set firewall name WAN-IN rule 30 destination group port-group 'PORT-WEB-HTTPs'
set firewall name WAN-IN rule 30 protocol 'tcp'
set firewall name WAN-LOCAL default-action 'drop'
set firewall name WAN-LOCAL enable-default-log
set firewall name WAN-LOCAL rule 10 action 'accept'
set firewall name WAN-LOCAL rule 10 state established 'enable'
set firewall name WAN-LOCAL rule 10 state related 'enable'
set firewall name WAN-LOCAL rule 20 action 'drop'
set firewall name WAN-LOCAL rule 20 log 'enable'
set firewall name WAN-LOCAL rule 20 state invalid 'enable'
set firewall name WAN-LOCAL rule 30 action 'accept'
set firewall name WAN-LOCAL rule 30 icmp type-name 'echo-request'
set firewall name WAN-LOCAL rule 30 protocol 'icmp'
set firewall name WAN-LOCAL rule 30 source group network-group 'NET-THINKBROADBAND'
set firewall name WAN-LOCAL rule 40 action 'accept'
set firewall name WAN-LOCAL rule 40 description 'Site-to-Site OpenVPN'
set firewall name WAN-LOCAL rule 40 destination port 'openvpn'
set firewall name WAN-LOCAL rule 40 protocol 'udp'
set firewall receive-redirects 'disable'
set firewall send-redirects 'enable'
set firewall source-validation 'disable'
set firewall state-policy established action 'accept'
set firewall state-policy invalid action 'drop'
set firewall state-policy invalid log enable
set firewall state-policy related action 'accept'
set firewall syn-cookies 'enable'
set firewall twa-hazards-protection 'disable'
set interfaces ethernet eth0 description 'WAN'
set interfaces ethernet eth0 hw-id 'XX:XX:XX:XX:XX:e0'
set interfaces ethernet eth0 mtu '1520'
set interfaces ethernet eth0 offload gro
set interfaces ethernet eth0 offload gso
set interfaces ethernet eth0 offload sg
set interfaces ethernet eth0 offload tso
set interfaces ethernet eth1 address 'xxx.xxx.83.254/24'
set interfaces ethernet eth1 description 'LAN'
set interfaces ethernet eth1 firewall in name 'LAN-IN'
set interfaces ethernet eth1 firewall local
set interfaces ethernet eth1 firewall out
set interfaces ethernet eth1 hw-id 'XX:XX:XX:XX:XX:ea'
set interfaces ethernet eth1 offload gro
set interfaces ethernet eth1 offload gso
set interfaces ethernet eth1 offload sg
set interfaces ethernet eth1 offload tso
set interfaces ethernet eth1 policy route 'POL-ROUTE-ETH1'
set interfaces loopback lo
set interfaces openvpn vtun10 description 'OpenVPN Site-to-Site'
set interfaces openvpn vtun10 device-type 'tun'
set interfaces openvpn vtun10 disable
set interfaces openvpn vtun10 encryption cipher 'aes256'
set interfaces openvpn vtun10 firewall in name 'OPENVPN-IN'
set interfaces openvpn vtun10 hash 'sha512'
set interfaces openvpn vtun10 local-address xxx.xxx.10.1
set interfaces openvpn vtun10 local-port '1194'
set interfaces openvpn vtun10 mode 'site-to-site'
set interfaces openvpn vtun10 persistent-tunnel
set interfaces openvpn vtun10 protocol 'udp'
set interfaces openvpn vtun10 remote-address 'xxx.xxx.10.2'
set interfaces openvpn vtun10 shared-secret-key-file xxxxxx
set interfaces openvpn vtun20 authentication password xxxxxx
set interfaces openvpn vtun20 authentication username xxxxxx
set interfaces openvpn vtun20 description 'TorGuard USA Dedicated IP NY'
set interfaces openvpn vtun20 device-type 'tun'
set interfaces openvpn vtun20 disable
set interfaces openvpn vtun20 encryption cipher 'aes128'
set interfaces openvpn vtun20 firewall in name 'DROP-ALL'
set interfaces openvpn vtun20 firewall local name 'DROP-ALL'
set interfaces openvpn vtun20 hash 'sha256'
set interfaces openvpn vtun20 mode 'client'
set interfaces openvpn vtun20 openvpn-option 'key-direction 1'
set interfaces openvpn vtun20 openvpn-option 'route-nopull'
set interfaces openvpn vtun20 openvpn-option '--proto udp4'
set interfaces openvpn vtun20 openvpn-option 'tun-mtu 1500'
set interfaces openvpn vtun20 openvpn-option 'mssfix'
set interfaces openvpn vtun20 persistent-tunnel
set interfaces openvpn vtun20 protocol 'udp'
set interfaces openvpn vtun20 remote-host 'xxx.xxx.55.154'
set interfaces openvpn vtun20 remote-port '1912'
set interfaces openvpn vtun20 tls auth-file '/config/auth/openvpn/tg-dedi-us-ny/tls.key'
set interfaces openvpn vtun20 tls ca-cert-file xxxxxx
set interfaces pppoe pppoe0 authentication password xxxxxx
set interfaces pppoe pppoe0 authentication user xxxxxx
set interfaces pppoe pppoe0 default-route 'force'
set interfaces pppoe pppoe0 description 'FTTP-PPPoE'
set interfaces pppoe pppoe0 firewall in name 'WAN-IN'
set interfaces pppoe pppoe0 firewall local name 'WAN-LOCAL'
set interfaces pppoe pppoe0 mtu '1500'
set interfaces pppoe pppoe0 no-peer-dns
set interfaces pppoe pppoe0 source-interface 'eth0'
set interfaces wireguard wg1 address 'xxx.xxx.108.177/24'
set interfaces wireguard wg1 description 'TG-WG-EU-UK-1'
set interfaces wireguard wg1 firewall in name 'DROP-ALL'
set interfaces wireguard wg1 firewall local name 'DROP-ALL'
set interfaces wireguard wg1 peer TGWGEUUK1 address 'xxx.xxx.231.248'
set interfaces wireguard wg1 peer TGWGEUUK1 allowed-ips 'xxx.xxx.0.0/0'
set interfaces wireguard wg1 peer TGWGEUUK1 persistent-keepalive '25'
set interfaces wireguard wg1 peer TGWGEUUK1 port '1443'
set interfaces wireguard wg1 peer TGWGEUUK1 pubkey 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
set interfaces wireguard wg1 private-key 'tg-wg-eu-uk-1'
set interfaces wireguard wg2 address 'xxx.xxx.65.97/24'
set interfaces wireguard wg2 description 'TG-WG-US-NY-DED-1'
set interfaces wireguard wg2 firewall in name 'DROP-ALL'
set interfaces wireguard wg2 firewall local name 'DROP-ALL'
set interfaces wireguard wg2 peer TGWGUSNYDED1 address 'xxx.xxx.116.242'
set interfaces wireguard wg2 peer TGWGUSNYDED1 allowed-ips 'xxx.xxx.0.0/0'
set interfaces wireguard wg2 peer TGWGUSNYDED1 persistent-keepalive '25'
set interfaces wireguard wg2 peer TGWGUSNYDED1 port '1443'
set interfaces wireguard wg2 peer TGWGUSNYDED1 pubkey 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
set interfaces wireguard wg2 private-key 'tg-wg-us-ny-ded-1'
set nat destination rule 10 description 'Port Forward Outside HTTP/HTTPS to SABLE'
set nat destination rule 10 destination port 'http,https'
set nat destination rule 10 inbound-interface 'pppoe0'
set nat destination rule 10 protocol 'tcp'
set nat destination rule 10 translation address 'xxx.xxx.83.250'
set nat source rule 1 description 'PPPoE OUT NAT'
set nat source rule 1 outbound-interface 'pppoe0'
set nat source rule 1 source address 'xxx.xxx.83.0/24'
set nat source rule 1 translation address 'masquerade'
set nat source rule 20 description 'TG USA NY Dedicated'
set nat source rule 20 outbound-interface 'wg2'
set nat source rule 20 source address 'xxx.xxx.83.0/24'
set nat source rule 20 translation address 'masquerade'
set nat source rule 30 description 'TG WG EU UK 1'
set nat source rule 30 outbound-interface 'wg1'
set nat source rule 30 source address 'xxx.xxx.83.0/24'
set nat source rule 30 translation address 'masquerade'
set policy route POL-ROUTE-ETH1 rule 10 description 'Disney Plus Cloudflare via TG USA DEDI'
set policy route POL-ROUTE-ETH1 rule 10 destination group network-group 'NET-STREAMING-DISNEY'
set policy route POL-ROUTE-ETH1 rule 10 set table '20'
set policy route POL-ROUTE-ETH1 rule 11 description 'Pandora Radio via TG USA DEDI'
set policy route POL-ROUTE-ETH1 rule 11 destination group network-group 'NET-STREAMING-PANDORA'
set policy route POL-ROUTE-ETH1 rule 11 set table '20'
set policy route POL-ROUTE-ETH1 rule 12 description 'ADR-ROUTE-TG-USA-DEDI-NY hosts via TG USA DEDI'
set policy route POL-ROUTE-ETH1 rule 12 protocol 'all'
set policy route POL-ROUTE-ETH1 rule 12 set table '20'
set policy route POL-ROUTE-ETH1 rule 12 source group address-group 'ADR-ROUTE-TG-USA-DEDI-NY'
set policy route POL-ROUTE-ETH1 rule 13 description 'Route addresses in NET-ROUTE-DEDI-USA-NY via USA-NY Dedicated'
set policy route POL-ROUTE-ETH1 rule 13 destination group network-group 'NET-ROUTE-DEDI-USA-NY'
set policy route POL-ROUTE-ETH1 rule 13 protocol 'all'
set policy route POL-ROUTE-ETH1 rule 13 set table '20'
set policy route POL-ROUTE-ETH1 rule 20 description 'ADR-ORYX NNTPS via TG GB LON'
set policy route POL-ROUTE-ETH1 rule 20 destination port 'nntp,nntps'
set policy route POL-ROUTE-ETH1 rule 20 protocol 'tcp'
set policy route POL-ROUTE-ETH1 rule 20 set table '30'
set policy route POL-ROUTE-ETH1 rule 20 source group address-group 'ADR-ORYX'
set policy route POL-ROUTE-ETH1 rule 21 description 'ADR-ROUTE-TG-EU-UK hosts via WG1'
set policy route POL-ROUTE-ETH1 rule 21 set table '30'
set policy route POL-ROUTE-ETH1 rule 21 source group address-group 'ADR-ROUTE-TG-EU-UK'
set policy route POL-ROUTE-ETH1 rule 22 action 'drop'
set policy route POL-ROUTE-ETH1 rule 22 description 'Block NNTP and NNTPS if not over VPN from ORYX'
set policy route POL-ROUTE-ETH1 rule 22 destination port 'nntp,nntps'
set policy route POL-ROUTE-ETH1 rule 22 log 'enable'
set policy route POL-ROUTE-ETH1 rule 22 protocol 'tcp'
set policy route POL-ROUTE-ETH1 rule 22 source group address-group 'ADR-ORYX'
set protocols static table 20 interface-route xxx.xxx.0.0/0 next-hop-interface wg2
set protocols static table 30 interface-route xxx.xxx.0.0/0 next-hop-interface wg1
set service dns dynamic interface pppoe0 service cloudflare host-name xxxxxx
set service dns dynamic interface pppoe0 service cloudflare login 'xxxxxxxxxxxxxx'
set service dns dynamic interface pppoe0 service cloudflare password xxxxxx
set service dns dynamic interface pppoe0 service cloudflare protocol 'cloudflare'
set service dns dynamic interface pppoe0 service cloudflare zone 'xxxxxxxxx'
set service dns dynamic interface pppoe0 use-web url 'checkip.amazonaws.com'
set service ssh port '22'
set system config-management commit-revisions '100'
set system conntrack expect-table-size '2048'
set system conntrack hash-size '32768'
set system conntrack modules h323 disable
set system conntrack table-size '262144'
set system console
set system host-name xxxxxx
set system ipv6 disable
set system login banner post-login ''
set system login banner pre-login ''
set system login user xxxxxx authentication encrypted-password xxxxxx
set system login user xxxxxx authentication plaintext-password xxxxxx
set system login user xxxxxx authentication public-keys xxxx@xxx.xxx key xxxxxx
set system login user xxxxxx authentication public-keys xxxx@xxx.xxx type ssh-xxx
set system name-server 'xxx.xxx.83.250'
set system name-server 'xxx.xxx.83.251'
set system ntp allow-clients address 'xxx.xxx.83.0/24'
set system ntp listen-address 'xxx.xxx.83.254'
set system ntp server xxxxx.tld
set system ntp server xxxxx.tld
set system ntp server xxxxx.tld
set system ntp server xxxxx.tld
set system option reboot-on-panic
set system syslog global facility all level 'info'
set system syslog global facility protocols level 'debug'
set system syslog host xxx.xxx.83.4 facility all level 'warning'

@phasma as I could see there is same configuration for 2 wg interfaces, does this issue applies to both or policy route for wg1 works fine after the reboot? Also please provide the requested logs whenever possible, that also might help to understand why the issue happens. Thanks.

Happens on both WG1 and WG2 and my fix described by deleting the table and adding back in fixes both routes.

The logs are too long to paste here but have dumped them here.

https://justpaste.it/7ovnz

Looking for alternative solutions to you situation.
Could you define table 20 and table 30, using route, and no interface route?

set protocols static table 20 route xxx.xxx.0.0/0 next-hop remote_wireguard_IP_WG2
set protocols static table 30 route xxx.xxx.0.0/0 next-hop remote_wireguard_IP_WG1

The remote IP’s often change so I dont think this wouldnt work.

@phasma

We tried to reproduce your issue, but after reboot, everything is working as expected in our lab.
Some things to consider:

  1. How did you get to 1.3-rc6 version? Upgraded from previous version, or it was a fresh install?
  2. Could you make a new reboot, and before applying your commit that restores functionality, type “compare” and share the result of it? In the logs provided, we couldn’t find when policy POL-ROUTE-ETH1 was loaded.

Just to add to the conversation, I’m running 1.3 RC 6 with policy based routing with multiple WireGuard instances and interface defined routes and have not experienced any issues after rebooting. I am running a site to site, Mullvad client, and server (road warrior) and everything has come back up as expected after multiple reboots.

Hi,

I upgraded to 1.3 rc 6 via add system image.

Here is the process.

tracert before the fix on a fresh reboot

Tracing route to www.pandora.com [208.85.40.158]
over a maximum of 3 hops:

  1    <1 ms    <1 ms    <1 ms  vyos.xxxxxxxxxxx[192.168.83.254]
  2     4 ms     4 ms     4 ms  vt1.cor2.lond1.ptn.zen.net.uk [51.148.72.22]
  3     4 ms     4 ms     4 ms  lag-9.p1.thn-lon.zen.net.uk [51.148.73.160]

Deleting the routing interface. Followed by a traceroute to show no change

vyos@vyos# compare
[edit protocols static table 20]
-interface-route 0.0.0.0/0 {
-    next-hop-interface wg2 {
-    }
-}
[edit protocols static table 30]
-interface-route 0.0.0.0/0 {
-    next-hop-interface wg1 {
-    }
-}
[edit protocols static]

Tracing route to www.pandora.com [208.85.40.158]
over a maximum of 3 hops:

  1    <1 ms    <1 ms    <1 ms  vyos.xxxxxxxxxxxxxxxx [192.168.83.254]
  2     4 ms     4 ms     4 ms  vt1.cor2.lond1.ptn.zen.net.uk [51.148.72.22]
  3     4 ms     4 ms     4 ms  lag-9.p1.thn-lon.zen.net.uk [51.148.73.160]

Readding back the wireguard interface followed by a traceroute showing it working.


vyos@vyos# compare
[edit protocols static table 20]
+interface-route 0.0.0.0/0 {
+    next-hop-interface wg2 {
+    }
+}
[edit protocols static table 30]
+interface-route 0.0.0.0/0 {
+    next-hop-interface wg1 {
+    }
+}
[edit protocols static]


Tracing route to www.pandora.com [208.85.40.158]
over a maximum of 3 hops:

  1    <1 ms    <1 ms    <1 ms  vyos.xxxxxxxxx [192.168.83.254]
  2    77 ms    76 ms    76 ms  10.13.0.1
  3    77 ms    77 ms    77 ms  te0-7-0-19.rcr22.b001362-2.jfk01.atlas.cogentco.com [38.142.116.241]

Also just tried 1.3 rc 5 as still had that image same issue

What does actual linux route table look like in error condition?
sudo ip route show table 20

Hello, @phasma!

To collect necessary debug information, you need to share the output of the next commands at the moment when PBR works and when not:

sudo ip rule show
sudo ip r show table 20
sudo ip r show table 30
sudo ip r get [DST_ADDR] mark [PBR_MARK]
sudo nft list table ip mangle

where:
[DST_ADDR] - a destination address traffic to which should be routed via wg interfaces
[PBR_MARK] - a mark from the sudo ip rule show output. There should be two of them.

This will give a start point for investigation

Rebooted Fresh and not working…

vyos:[~] $ sudo ip rule show
0:      from all lookup local
20:     from all fwmark 0x80000013 lookup 20
30:     from all fwmark 0x8000001d lookup 30
32766:  from all lookup main
32767:  from all lookup default

sudo ip r show table 20
[Nothing returned]

sudo ip r show table 30
[Nothing returned]

vyos:[~] $ sudo ip r get 208.85.40.158 mark 0x80000013
208.85.40.158 dev pppoe0 src 82.69.85.101 mark 0x80000013 uid 0
    cache

vyos:[~] $ sudo ip r get 95.211.189.152 mark 0x8000001d
95.211.189.152 dev pppoe0 src 82.69.85.101 mark 0x8000001d uid 0
    cache

table ip mangle {
        chain PREROUTING {
                type filter hook prerouting priority mangle; policy accept;
                counter packets 102454 bytes 55985143 jump VYATTA_FW_IN_HOOK
        }

        chain INPUT {
                type filter hook input priority mangle; policy accept;
        }

        chain FORWARD {
                type filter hook forward priority mangle; policy accept;
        }

        chain OUTPUT {
                type route hook output priority mangle; policy accept;
                counter packets 1730 bytes 187385 jump VYATTA_FW_LOCALOUT_HOOK
        }

        chain POSTROUTING {
                type filter hook postrouting priority mangle; policy accept;
                counter packets 100328 bytes 55611817 jump VYATTA_FW_OUT_HOOK
        }

        chain VYATTA_FW_OUT_HOOK {
        }

        chain VYATTA_FW_IN_HOOK {
                iifname "eth1" counter packets 55554 bytes 48513250 jump POL-ROUTE-ETH1
        }

        chain VYATTA_FW_LOCALOUT_HOOK {
        }

        chain POL-ROUTE-ETH1 {
                # match-set NET-STREAMING-DISNEY dst counter packets 0 bytes 0 jump VYATTA_PBR_20 comment "POL-ROUTE-ETH1-10"
                # match-set NET-STREAMING-PANDORA dst counter packets 86 bytes 5160 jump VYATTA_PBR_20 comment "POL-ROUTE-ETH1-11"
                # match-set ADR-ROUTE-TG-USA-DEDI-NY src counter packets 0 bytes 0 jump VYATTA_PBR_20 comment "POL-ROUTE-ETH1-12"
                # match-set NET-ROUTE-DEDI-USA-NY dst counter packets 0 bytes 0 jump VYATTA_PBR_20 comment "POL-ROUTE-ETH1-13"
                meta l4proto tcp # match-set ADR-ORYX src tcp dport { 119,563} counter packets 0 bytes 0 jump VYATTA_PBR_30 comment "POL-ROUTE-ETH1-20"
                # match-set ADR-ROUTE-TG-EU-UK src counter packets 0 bytes 0 jump VYATTA_PBR_30 comment "POL-ROUTE-ETH1-21"
                meta l4proto tcp # match-set ADR-ORYX src tcp dport { 119,563} counter packets 0 bytes 0 log prefix "[POL-ROUTE-ETH1-22-D] " comment "POL-ROUTE-ETH1-22"
                meta l4proto tcp # match-set ADR-ORYX src tcp dport { 119,563} counter packets 0 bytes 0 drop comment "POL-ROUTE-ETH1-22"
                counter packets 55468 bytes 48508090 return comment "POL-ROUTE-ETH1-10000 default-action accept"
        }

        chain VYATTA_PBR_20 {
                counter packets 86 bytes 5160 meta mark set 0x80000013
                counter packets 86 bytes 5160 accept
        }

        chain VYATTA_PBR_30 {
                counter packets 0 bytes 0 meta mark set 0x8000001d
                counter packets 0 bytes 0 accept
        }
}


And then once fix applied

vyos:[~] $ sudo ip rule show
0:      from all lookup local
20:     from all fwmark 0x80000013 lookup 20
30:     from all fwmark 0x8000001d lookup 30
32766:  from all lookup main
32767:  from all lookup default


vyos:[~] $ sudo ip r show table 20
default nhid 26 dev wg2 proto static metric 20

vyos:[~] $ sudo ip r show table 30
default nhid 28 dev wg1 proto static metric 20

vyos:[~] $ sudo ip r get 208.85.40.158 mark 0x80000013
208.85.40.158 dev wg2 table 20 src 10.13.65.97 mark 0x80000013 uid 0
    cache

vyos:[~] $ sudo ip r get 95.211.189.152 mark 0x8000001d
95.211.189.152 dev wg1 table 30 src 10.13.108.177 mark 0x8000001d uid 0
    cache

table ip mangle {
        chain PREROUTING {
                type filter hook prerouting priority mangle; policy accept;
                counter packets 194554 bytes 106482687 jump VYATTA_FW_IN_HOOK
        }

        chain INPUT {
                type filter hook input priority mangle; policy accept;
        }

        chain FORWARD {
                type filter hook forward priority mangle; policy accept;
        }

        chain OUTPUT {
                type route hook output priority mangle; policy accept;
                counter packets 3265 bytes 330833 jump VYATTA_FW_LOCALOUT_HOOK
        }

        chain POSTROUTING {
                type filter hook postrouting priority mangle; policy accept;
                counter packets 191178 bytes 105852454 jump VYATTA_FW_OUT_HOOK
        }

        chain VYATTA_FW_OUT_HOOK {
        }

        chain VYATTA_FW_IN_HOOK {
                iifname "eth1" counter packets 103195 bytes 87881467 jump POL-ROUTE-ETH1
        }

        chain VYATTA_FW_LOCALOUT_HOOK {
        }

        chain POL-ROUTE-ETH1 {
                # match-set NET-STREAMING-DISNEY dst counter packets 0 bytes 0 jump VYATTA_PBR_20 comment "POL-ROUTE-ETH1-10"
                # match-set NET-STREAMING-PANDORA dst counter packets 150 bytes 9000 jump VYATTA_PBR_20 comment "POL-ROUTE-ETH1-11"
                # match-set ADR-ROUTE-TG-USA-DEDI-NY src counter packets 0 bytes 0 jump VYATTA_PBR_20 comment "POL-ROUTE-ETH1-12"
                # match-set NET-ROUTE-DEDI-USA-NY dst counter packets 0 bytes 0 jump VYATTA_PBR_20 comment "POL-ROUTE-ETH1-13"
                meta l4proto tcp # match-set ADR-ORYX src tcp dport { 119,563} counter packets 0 bytes 0 jump VYATTA_PBR_30 comment "POL-ROUTE-ETH1-20"
                # match-set ADR-ROUTE-TG-EU-UK src counter packets 0 bytes 0 jump VYATTA_PBR_30 comment "POL-ROUTE-ETH1-21"
                meta l4proto tcp # match-set ADR-ORYX src tcp dport { 119,563} counter packets 0 bytes 0 log prefix "[POL-ROUTE-ETH1-22-D] " comment "POL-ROUTE-ETH1-22"
                meta l4proto tcp # match-set ADR-ORYX src tcp dport { 119,563} counter packets 0 bytes 0 drop comment "POL-ROUTE-ETH1-22"
                counter packets 103045 bytes 87872467 return comment "POL-ROUTE-ETH1-10000 default-action accept"
        }

        chain VYATTA_PBR_20 {
                counter packets 150 bytes 9000 meta mark set 0x80000013
                counter packets 150 bytes 9000 accept
        }

        chain VYATTA_PBR_30 {
                counter packets 0 bytes 0 meta mark set 0x8000001d
                counter packets 0 bytes 0 accept
        }
}

Thanks a lot!
Now we know that the problem is in routing entries that are not creating during boot for some reason.
Could you show also the output of:

sudo vtysh -c 'show running-config' | tee
sudo journalctl -b /usr/lib/frr/staticd | tee

Here is the output after a fresh reboot

vyos:[~] $ sudo vtysh -c 'show running-config' | tee
Building configuration...

Current configuration:
!
frr version 7.5.1-20210801-00-g8bed329e4
frr defaults traditional
hostname vyos
log syslog
log facility local7
service integrated-vtysh-config
!
ip route 0.0.0.0/0 pppoe0
!
line vty
!
end

vyos:[~] $ sudo journalctl -b /usr/lib/frr/staticd | tee
-- Logs begin at Thu 2021-08-26 17:02:42 UTC, end at Thu 2021-08-26 17:03:31 UTC. --
-- No entries --

Here is output once the fix has been applied

Building configuration...

Current configuration:
!
frr version 7.5.1-20210801-00-g8bed329e4
frr defaults traditional
hostname vyos
log syslog
log facility local7
service integrated-vtysh-config
!
ip route 0.0.0.0/0 wg1 table 30
ip route 0.0.0.0/0 wg2 table 20
ip route 0.0.0.0/0 pppoe0
!
line vty
!
end

vyos:[~] $ sudo journalctl -b /usr/lib/frr/staticd | tee
-- Logs begin at Thu 2021-08-26 15:14:48 UTC, end at Thu 2021-08-26 17:02:06 UTC. --
-- No entries --

does some script run when wg interface goes up? It might only handle main route table.
Create some dummy route on wg interface (for example, to 1.1.1.1/32 ), and see if this dummy route is present after start-up

The problem is clear - routing tables 10 and 20 are not presented in the FRR. The question is: why?
I would expect to see at least anything in logs that may show the reasons, but there are empty.

I tested again with openvpn same result doesnt matter if its wireguard or not tables arent created at reboot. Also tried with just one table.