Using VRF for source based static routes instead of PBR?

[chase-vgo]

Hey all, I’ve been having some odd issues with Policy Based Routing when paired with static tables. On some occasions, they just simply stop applying until the firewall is rebooted, and on others, I get weird issues when creating.

e.g, a typical PBR for me would be something like:

set policy route PBR8 rule 10 set table '8'
set policy route PBR8 interface 'bond0.8'
set protocols static table 8 route 172.20.192.0/20 next-hop 172.31.5.254

The above would be to make traffic destined for 172.20.192.0/20 within vlan 8 hop to an independent VPN concentrator at 172.31.5.254. The above works, but occasionally doesn’t. Happened both on 1.3.4 and after upgrading to 1.4.0 epa2.

And to make things even weirder, occasionally when creating a PBR/Table, I get the following error, such as when I run the same commands as above, but using ‘PBR08’ and ‘table 08’:

set protocols static table 08 route 172.20.192.0/20 next-hop 172.31.5.254
set policy route PBR08 rule 10 set table '08'

It throws:

Traceback (most recent call last):
  File "/usr/libexec/vyos/conf_mode/policy_route.py", line 196, in <module>
    apply(c)
  File "/usr/libexec/vyos/conf_mode/policy_route.py", line 187, in apply
    apply_table_marks(policy)
  File "/usr/libexec/vyos/conf_mode/policy_route.py", line 163, in apply_table_marks
    cmd(f'{cmd_str} rule add pref {set_table} fwmark {table_mark} table {set_table}')
  File "/usr/lib/python3/dist-packages/vyos/utils/process.py", line 155, in cmd
    raise OSError(code, feedback)
OSError: [Errno 255] failed to run command: ip rule add pref 08 fwmark 2147483639 table 08
returned:
exit code: 255

noteworthy:
cmd 'ip rule add pref 08 fwmark 2147483639 table 08'
returned (out):

returned (err):
Error: argument "08" is wrong: preference value is invalid

[[policy route PBR08]] failed

The only resolution to the above error I have found is just rerun the commands with a different number on the static table. Because of this, I have a slight mix of “09” and “8”, making the naming inconsistent for single digit numbers.

With all of these issues even AFTER upgrading to 1.4, I’m considering moving to VRF based routing instead. e.g:

set vrf name VRF08 protocols static route 172.20.192.0/20 next-hop 172.31.5.254
set interfaces bonding bond0 vif 8 vrf VRF08

Has anyone done static routes specific to a vlan using VRF that can confirm it works well? Alternately, has anyone seen the issues I’ve experienced with PBR that can provide some insight?