Multiple WAN and policy based routing not working

daddy · April 26, 2024, 5:58pm

Hi.
Here is my config:
VyOS 1.4.0 local build.
eth0 - WAN1
eth1 - LAN
eth2 - WAN2
eth3 - WAN3
vtun1194 - OpenVPN Clients. Server listen on WAN2.

My goal is:
OpenVPN Clients connects through WAN2.
OpenVPN Clients takes ip address and other settings including default route to OpenVPN server.
On VyOS policy based routing named VPN is configured. So - some internal routes goes to LAN. All other goes to WAN3.

The result:
Internal routes to LAN works fine.
Routes to WAN3 not working initially, after reboot for example. But if i temporary disable rule 1000 in PBR and enable it back - all starts working fine.

Is this is bug or wrong configuration?

Here is some set of configurations.

set interfaces ethernet eth0 address 'xxx.xxx.4.254/29'
set interfaces ethernet eth0 hw-id 'xx:xx:xx:xx:xx:7b'
set interfaces ethernet eth1 address 'xxx.xxx.5.254/24'
set interfaces ethernet eth1 hw-id 'xx:xx:xx:xx:xx:8e'
set interfaces ethernet eth2 address 'xxx.xxx.36.84/29'
set interfaces ethernet eth2 hw-id 'xx:xx:xx:xx:xx:fa'
set interfaces ethernet eth3 address 'xxx.xxx.182.54/28'
set interfaces ethernet eth3 hw-id 'xx:xx:xx:xx:xx:54'
set interfaces openvpn vtun1194 device-type 'tun'
set interfaces openvpn vtun1194 encryption cipher 'aes256'
set interfaces openvpn vtun1194 encryption ncp-ciphers 'aes256'
set interfaces openvpn vtun1194 encryption ncp-ciphers 'aes256gcm'
set interfaces openvpn vtun1194 hash 'sha256'
set interfaces openvpn vtun1194 keep-alive failure-count '6'
set interfaces openvpn vtun1194 keep-alive interval '10'
set interfaces openvpn vtun1194 local-host 'xxx.xxx.4.254'
set interfaces openvpn vtun1194 local-port '1194'
set interfaces openvpn vtun1194 mode 'server'
set interfaces openvpn vtun1194 openvpn-option 'xxx'
set interfaces openvpn vtun1194 openvpn-option 'xxx'
set interfaces openvpn vtun1194 openvpn-option 'xxx'
set interfaces openvpn vtun1194 openvpn-option 'xxx'
set interfaces openvpn vtun1194 persistent-tunnel
set interfaces openvpn vtun1194 protocol 'udp'
set interfaces openvpn vtun1194 server domain-name xxxxxx
set interfaces openvpn vtun1194 server max-connections '253'
set interfaces openvpn vtun1194 server name-server 'xxx.xxx.3.21'
set interfaces openvpn vtun1194 server name-server 'xxx.xxx.0.51'
set interfaces openvpn vtun1194 server reject-unconfigured-clients
set interfaces openvpn vtun1194 server subnet 'xxx.xxx.131.0/24'
set interfaces openvpn vtun1194 server topology 'subnet'
set interfaces openvpn vtun1194 tls auth-key 'xxx'
set interfaces openvpn vtun1194 tls ca-certificate 'xxx'
set interfaces openvpn vtun1194 tls certificate 'xxx'
set interfaces openvpn vtun1194 tls dh-params 'xxx'
set interfaces openvpn vtun1194 tls tls-version-min '1.3'
set interfaces loopback lo
set nat source rule 100 outbound-interface name 'eth0'
set nat source rule 100 source address 'xxx.xxx.128.0/21'
set nat source rule 100 translation address 'xxx.xxx.4.254'
set nat source rule 110 outbound-interface name 'eth2'
set nat source rule 110 source address 'xxx.xxx.128.0/21'
set nat source rule 110 translation address 'xxx.xxx.36.84'
set nat source rule 115 outbound-interface name 'eth3'
set nat source rule 115 source address 'xxx.xxx.128.0/21'
set nat source rule 115 translation address 'xxx.xxx.182.54'
set nat source rule 120 outbound-interface name 'eth1'
set nat source rule 120 source address 'xxx.xxx.128.0/21'
set nat source rule 120 translation address 'xxx.xxx.5.254'
set policy local-route rule 10 set table '10'
set policy local-route rule 10 source address 'xxx.xxx.4.254'
set policy local-route rule 15 set table '15'
set policy local-route rule 15 source address 'xxx.xxx.182.54'
set policy local-route rule 20 set table '20'
set policy local-route rule 20 source address 'xxx.xxx.36.84'
set policy route VPN interface 'vtun1194'
set policy route VPN rule 10 destination address 'xxx.xxx.5.0/24'
set policy route VPN rule 10 set table 'main'
set policy route VPN rule 10 source address 'xxx.xxx.128.0/21'
set policy route VPN rule 20 destination address 'xxx.xxx.0.0/21'
set policy route VPN rule 20 set table 'main'
set policy route VPN rule 20 source address 'xxx.xxx.128.0/21'
set policy route VPN rule 1000 destination address 'xxx.xxx.0.0/0'
set policy route VPN rule 1000 set table '20'
set policy route VPN rule 1000 source address 'xxx.xxx.128.0/21'
set protocols static route xxx.xxx.0.0/21 next-hop xxx.xxx.5.1
set protocols static route xxx.xxx.24.0/24 next-hop xxx.xxx.5.1
set protocols static route xxx.xxx.40.0/24 next-hop xxx.xxx.5.1
set protocols static table 10 route xxx.xxx.0.0/0 next-hop xxx.xxx.4.249
set protocols static table 15 route xxx.xxx.0.0/0 next-hop xxx.xxx.182.49
set protocols static table 20 route xxx.xxx.0.0/0 next-hop xxx.xxx.36.81

tjh · April 26, 2024, 11:41pm

This sounds like it might be a bug in the order things are processed on startup. You say a disable/renable fixes it? I’ve seen another user report a similar issue but the problem there was they had a Nat rule that wasn’t being applied (or was being trumped by further config) on startup but then was applied when they committed changes.

I would double check all config to ensure there’s no other rule/policy interfering, and then log a Phabricator bug if you’re sure it’s only an “on reboot” issue.

daddy · April 27, 2024, 10:06am

Yes that’s right. I’ve already checked the configuration and everything else four times.
Steps to reproduce:

Reboot VyOS.
PBR doesn’t work as it should be.
Rule 1000 is present in runtime config.
Disable rule 1000 in PBR. Commit. Enable rule 1000 commit.
Everything works as it should be.

Thanks for your feedback

Viacheslav · April 27, 2024, 10:21am

Rule 1000 and 2000 probably reserved for VRF
Try to check “sudo ip rule show” when it doesn’t work and when it work’s
And you mix policy route and policy local-route
Needs to be careful, as “policy route” uses its own ip rules and such numbers of priority could be overlapped

daddy · April 27, 2024, 12:55pm

Hi.
Thanks for your reply.
Changing rule number in PBR from 1000 to 155 did not gave any result. After reboot PBR not working, disable\enable rule 155 works.
Policy local-route configured for VyOS it self, for traffic, which originated from the router itself. So VyOS will respond from the same interface that the packet was received from.

Reboot VyOS.
PBR doesn’t work as it should be.
Output of “sudo ip rule show”:

0:      from all lookup local
10:     from xxx.xxx.4.254 lookup Default
15:     from xxx.xxx.182.54 lookup Default
20:     from xxx.xxx.36.84 lookup Default
254:    from all fwmark 0x7fffff01 lookup main
32766:  from all lookup main
32767:  from all lookup default

Disable\Enable rule 1000 in PBR.
PBR starts working as it should be.
Output of “sudo ip rule show”:

0:      from all lookup local
10:     from xxx.xxx.4.254 lookup Default
15:     from xxx.xxx.182.54 lookup Default
20:     from xxx.xxx.36.84 lookup Default
20:     from all fwmark 0x7fffffeb lookup Default
254:    from all fwmark 0x7fffff01 lookup main
32766:  from all lookup main
32767:  from all lookup default