How to make wan load balancing between 2 or more wireguard vpn with PBR

It works only for 1.3 1.4

Got it. Thanks! Waiting vyos 1.3 release.:slight_smile:

@echowings It won’t be difficult to add this to the next LTS release.
I can create a .deb pkg for 1.2.7 with this feature. If you can/want to check and test it.
In the local test it works fine

vyos@r2-lts:~$ show conf com | match "policy|bgp"
set policy route-map FOO rule 10 action 'permit'
set policy route-map FOO rule 10 set table '120'
set protocols bgp 65001 neighbor 192.168.122.14 address-family ipv4-unicast route-map import 'FOO'
set protocols bgp 65001 neighbor 192.168.122.14 remote-as '65002'
vyos@r2-lts:~$ 
vyos@r2-lts:~$ show ip bgp sum

IPv4 Unicast Summary:
BGP router identifier 192.168.122.12, local AS number 65001 vrf-id 0
BGP table version 4
RIB entries 3, using 552 bytes of memory
Peers 1, using 20 KiB of memory

Neighbor        V         AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd
192.168.122.14  4      65002       7       8        0    0    0 00:03:00            2

Total number of neighbors 1
vyos@r2-lts:~$ 
vyos@r2-lts:~$ show ip route table 120
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued route, r - rejected route

VRF default table 120:
B>* 192.0.2.0/24 [20/0] via 192.168.122.14, eth0, 00:00:36
B>* 203.0.113.0/24 [20/0] via 192.168.122.14, eth0, 00:00:36
vyos@r2-lts:~$ 
vyos@r2-lts:~$ show version 
Version:          VyOS 1.2.7

when you’re up and running could you share how you’ve managed to LB across 2 (or more) wg interfaces please?

It will be in the next LTS release.

@Viacheslav Could you please share with me about set protocols static table 100 settings? I can’t see details about table 100

For set protocols bgp 65001 maximum-paths ibgp '2'

There isn’t any maximu-paths

vyos@vyos# set protocols bgp 65001
Possible completions:
 > address-family
                BGP address-family parameters
+> neighbor     BGP neighbor
 > parameters   BGP parameters
+> peer-group   BGP peer-group
   route-map    Filter routes installed in local route map
 > timers       BGP protocol timers

It was moved to address-family

1 Like

Thanks
set protocols bgp 65001 address-family ipv4-unicast maximum-paths ibgp '2'

@Viacheslav Here is my configuration .but it isn’t works for me.

set interfaces ethernet eth0 address '100.64.0.2/30'
set interfaces ethernet eth0 hw-id '50:00:00:02:00:00'
set interfaces ethernet eth1 address '192.168.2.1/24'
set interfaces ethernet eth1 hw-id '50:00:00:02:00:01'
set interfaces ethernet eth2 hw-id '50:00:00:02:00:02'
set interfaces ethernet eth3 hw-id '50:00:00:02:00:03'
set interfaces loopback lo
set interfaces wireguard wg01 address '10.10.10.254/32'
set interfaces wireguard wg01 peer cloud1 address '203.0.113.254'
set interfaces wireguard wg01 peer cloud1 allowed-ips '0.0.0.0/0'
set interfaces wireguard wg01 peer cloud1 port '8000'
set interfaces wireguard wg01 peer cloud1 pubkey 'ugeluxTvZkvRt6Py1SJDX0XRfxrF/ifZ0muh9GStH1c='
set interfaces wireguard wg02 address '10.20.20.254/32'
set interfaces wireguard wg02 peer cloud2 address '198.51.100.254'
set interfaces wireguard wg02 peer cloud2 allowed-ips '0.0.0.0/0'
set interfaces wireguard wg02 peer cloud2 port '8000'
set interfaces wireguard wg02 peer cloud2 pubkey 'XGBDUlgTZ+lgCgPR+YB9kQxtuG3Hxt5mEktIeVFgNAc='
set nat source rule 100 outbound-interface 'wg01'
set nat source rule 100 translation address 'masquerade'
set nat source rule 110 outbound-interface 'wg02'
set nat source rule 110 translation address 'masquerade'
set policy route PBR rule 10 destination address '!192.168.2.0/24'
set policy route PBR rule 10 set table '100'
set policy route PBR rule 10 source address '192.168.2.0/24'
set policy route-map RMAP-IN rule 100 action 'permit'
set policy route-map RMAP-IN rule 100 set table '100'
set protocols bgp 65001 address-family ipv4-unicast maximum-paths ibgp '2'
set protocols bgp 65001 neighbor 10.10.10.1 remote-as '65001'
set protocols bgp 65001 neighbor 10.10.10.1 timers holdtime '30'
set protocols bgp 65001 neighbor 10.10.10.1 timers keepalive '30'
set protocols bgp 65001 neighbor 10.10.10.1 update-source '10.10.10.254'
set protocols bgp 65001 neighbor 10.20.20.1 remote-as '65001'
set protocols bgp 65001 neighbor 10.20.20.1 timers holdtime '30'
set protocols bgp 65001 neighbor 10.20.20.1 timers keepalive '10'
set protocols bgp 65001 neighbor 10.20.20.1 update-source '10.20.20.254'
set protocols rip network '100.64.0.0/30'
set protocols rip network '192.168.2.0/24'
set protocols static interface-route 0.0.0.0/0 next-hop-interface wg01
set protocols static interface-route 0.0.0.0/0 next-hop-interface wg02
set protocols static interface-route 10.10.10.0/24 next-hop-interface wg01
set protocols static interface-route 10.20.20.0/24 next-hop-interface wg02
set service dhcp-server shared-network-name LAN-DHCP subnet 192.168.2.0/24 default-router '192.168.2.1'
server '192.168.2.1'
set service dhcp-server shared-network-name LAN-DHCP subnet 192.168.2.0/24 range 0 start '192.168.2.11'
set service dhcp-server shared-network-name LAN-DHCP subnet 192.168.2.0/24 range 0 stop '192.168.2.12'
set system config-management commit-revisions '100'
set system conntrack modules ftp
set system conntrack modules h323
set system conntrack modules nfs
set system conntrack modules pptp
set system conntrack modules sip
set system conntrack modules sqlnet
set system conntrack modules tftp
set system console device ttyS0 speed '115200'
set system host-name 'vyos'
set system login user vyos authentication encrypted-password '$6$p96vCymIVf.pAsYW$KWwo.AM/PF7psfABBHWwAeryXwB0YE74Ns/wK6VhKf0OtKpWgejY6m/ZTGlJAjg9s9ZTQFV.kF91r7KzQULCq0'
set system login user vyos authentication plaintext-password ''
set system ntp server time1.vyos.net
set system ntp server time2.vyos.net
set system ntp server time3.vyos.net
set system syslog global facility all level 'info'
set system syslog global facility protocols level 'debug'

BGP are fine

yos@vyos# run show bgp summary

IPv4 Unicast Summary:
BGP router identifier 192.168.2.1, local AS number 65001 vrf-id 0
BGP table version 0
RIB entries 0, using 0 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt
10.10.10.1      4      65001       523       525        0    0    0 01:26:42            0        0
10.20.20.1      4      65001       116       123        0    0    0 00:00:27            0        0

Total number of neighbors 2

After run command

sudo vtysh -c "conf t" -c "route-map RMAP-IN permit 100" -c " set table 100"
sudo vtysh -c "conf t" -c "router bgp 65001" -c "address-family ipv4 unicast" -c "neighbor 10.10.10.1 route-map RMAP-IN in" -c "neighbor 10.20.20.1 route-map RMAP-IN in"

then to check route table is empty.

vyos@vyos# run show ip route table 100
[edit]
vyos@vyos#

Could you please tell me how to figure out this issue. Thanks a lot!

Hi @Viacheslav ,
Would you kindly tell me how to set route table like this?

vyos@vyos:~$ show ip route table 100

VRF default table 100:
B>* 0.0.0.0/0 [200/0] via 10.10.10.1, wg0, 00:00:10
  *                   via 10.20.20.1, wg1, 00:00:10
vyos@vyos:~$ 

Sorry, I can’t find the routing table on the configure you provide to me.
And this is the most import setting in the solution. So would do me a favor?

FRR setting

vyos# show running-config
Building configuration...

Current configuration:
!
frr version 7.5.1-20211211-00-g294410782
frr defaults traditional
hostname vyos
log syslog
log facility local7
service integrated-vtysh-config
!
router rip
 network 100.64.0.0/30
 network 192.168.2.0/24
!
router bgp 65001
 no bgp ebgp-requires-policy
 no bgp network import-check
 neighbor 10.10.10.1 remote-as 65001
 neighbor 10.10.10.1 update-source 10.10.10.254
 neighbor 10.10.10.1 timers 10 30
 neighbor 10.20.20.1 remote-as 65001
 neighbor 10.20.20.1 update-source 10.20.20.254
 neighbor 10.20.20.1 timers 10 30
 !
 address-family ipv4 unicast
  neighbor 10.10.10.1 route-map RMAP-IN in
  neighbor 10.20.20.1 route-map RMAP-IN in
  maximum-paths ibgp 2
 exit-address-family
!
route-map RMAP-IN permit 100
 set table 100
!
line vty
!
vyos@vyos# run show version

Version:          VyOS 1.3.0
Release train:    equuleus

Built by:         Sentrium S.L.
Built on:         Sun 19 Dec 2021 12:59 UTC
Build UUID:       ff458f8a-3ef7-453c-a7f9-4aeb6d03012e
Build commit ID:  2f691bb2f61e96-dirty

Architecture:     x86_64
Boot via:         installed image
System type:      KVM guest

Hardware vendor:  QEMU
Hardware model:   Standard PC (i440FX + PIIX, 1996)
Hardware S/N:
Hardware UUID:    abe8f380-36c1-401b-ad82-68f0bddef5fb

Copyright:        VyOS maintainers and contributors
[edit]
run show ip bgp sum

IPv4 Unicast Summary:
BGP router identifier 192.168.2.1, local AS number 65001 vrf-id 0or 10.20.20.1 ro
BGP table version 0
RIB entries 0, using 0 bytes of memory
Peers 2, using 43 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt
10.10.10.1      4      65001        28        29        0    0    0 00:04:17            0        0
10.20.20.1      4      65001        28        29        0    0    0 00:04:16            0        0

Total number of neighbors 2

Neighbours should sent you default routes

Actually, I build the same lab environment as you did.

run show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

S>* 10.10.10.0/24 [1/0] is directly connected, wg01, weight 1, 00:04:42
C>* 10.10.10.254/32 is directly connected, wg01, 00:04:42
S>* 10.20.20.0/24 [1/0] is directly connected, wg02, weight 1, 00:04:42
C>* 10.20.20.254/32 is directly connected, wg02, 00:04:43
C>* 100.64.0.0/30 is directly connected, eth0, 00:04:45
C>* 192.168.2.0/24 is directly connected, eth1, 00:04:46
S>* 198.51.100.254/32 [1/0] via 100.64.0.1, eth0, weight 1, 00:04:41
S>* 203.0.113.254/32 [1/0] via 100.64.0.1, eth0, weight 1, 00:04:41
vyos@vyos# run show bgp neighbors
BGP neighbor is 10.10.10.1, remote AS 65001, local AS 65001, internal link
Hostname: cloud1
  BGP version 4, remote router ID 203.0.113.254, local router ID 192.168.2.1
  BGP state = Established, up for 00:08:12
  Last read 00:00:02, Last write 00:00:02
  Hold time is 30, keepalive interval is 10 seconds
  Configured hold time is 30, keepalive interval is 10 seconds
  Neighbor capabilities:
    4 Byte AS: advertised and received
    AddPath:
      IPv4 Unicast: RX advertised IPv4 Unicast and received
    Route refresh: advertised and received(old & new)
    Address Family IPv4 Unicast: advertised and received
    Hostname Capability: advertised (name: vyos,domain name: n/a) received (name: cloud1,domain name: n/a)
    Graceful Restart Capability: advertised and received
      Remote Restart timer is 120 seconds
      Address families by peer:
        none
  Graceful restart information:
    End-of-RIB send: IPv4 Unicast
    End-of-RIB received: IPv4 Unicast
    Local GR Mode: Helper*
    Remote GR Mode: Helper
    R bit: False
    Timers:
      Configured Restart Time(sec): 120
      Received Restart Time(sec): 120
    IPv4 Unicast:
      F bit: False
      End-of-RIB sent: Yes
      End-of-RIB sent after update: Yes
      End-of-RIB received: Yes
      Timers:
        Configured Stale Path Time(sec): 360
  Message statistics:
    Inq depth is 0
    Outq depth is 0
                         Sent       Rcvd
    Opens:                  1          1
    Notifications:          0          0
    Updates:                1          1
    Keepalives:            50         50
    Route Refresh:          0          0
    Capability:             0          0
    Total:                 52         52
  Minimum time between advertisement runs is 0 seconds
  Update source is 10.10.10.254

 For address family: IPv4 Unicast
  Update group 1, subgroup 1
  Packet Queue length 0
  Community attribute sent to this neighbor(all)
  0 accepted prefixes

  Connections established 1; dropped 0
  Last reset 00:08:14,  Waiting for peer OPEN
Local host: 10.10.10.254, Local port: 45143
Foreign host: 10.10.10.1, Foreign port: 179
Nexthop: 10.10.10.254
Nexthop global: fe80::f8f5:cbff:fe74:cf72
Nexthop local: fe80::f8f5:cbff:fe74:cf72
BGP connection: non shared network
BGP Connect Retry Timer in Seconds: 120
Estimated round trip time: 3 ms
Read thread: on  Write thread: on  FD used: 28

BGP neighbor is 10.20.20.1, remote AS 65001, local AS 65001, internal link
Hostname: cloud2
  BGP version 4, remote router ID 198.51.100.254, local router ID 192.168.2.1
  BGP state = Established, up for 00:08:12
  Last read 00:00:02, Last write 00:00:02
  Hold time is 30, keepalive interval is 10 seconds
  Configured hold time is 30, keepalive interval is 10 seconds
  Neighbor capabilities:
    4 Byte AS: advertised and received
    AddPath:
      IPv4 Unicast: RX advertised IPv4 Unicast and received
    Route refresh: advertised and received(old & new)
    Address Family IPv4 Unicast: advertised and received
    Hostname Capability: advertised (name: vyos,domain name: n/a) received (name: cloud2,domain name: n/a)
    Graceful Restart Capability: advertised and received
      Remote Restart timer is 120 seconds
      Address families by peer:
        none
  Graceful restart information:
    End-of-RIB send: IPv4 Unicast
    End-of-RIB received: IPv4 Unicast
    Local GR Mode: Helper*
    Remote GR Mode: Helper
    R bit: False
    Timers:
      Configured Restart Time(sec): 120
      Received Restart Time(sec): 120
    IPv4 Unicast:
      F bit: False
      End-of-RIB sent: Yes
      End-of-RIB sent after update: Yes
      End-of-RIB received: Yes
      Timers:
        Configured Stale Path Time(sec): 360
  Message statistics:
    Inq depth is 0
    Outq depth is 0
                         Sent       Rcvd
    Opens:                  1          1
    Notifications:          0          0
    Updates:                1          1
    Keepalives:            50         50
    Route Refresh:          0          0
    Capability:             0          0
    Total:                 52         52
  Minimum time between advertisement runs is 0 seconds
  Update source is 10.20.20.254

 For address family: IPv4 Unicast
  Update group 1, subgroup 1
  Packet Queue length 0
  Community attribute sent to this neighbor(all)
  0 accepted prefixes

  Connections established 1; dropped 0
  Last reset 00:08:14,  Waiting for peer OPEN
Local host: 10.20.20.254, Local port: 40905
Foreign host: 10.20.20.1, Foreign port: 179
Nexthop: 10.20.20.254
Nexthop global: fe80::f50f:18ff:fe63:90a1
Nexthop local: fe80::f50f:18ff:fe63:90a1
BGP connection: non shared network
BGP Connect Retry Timer in Seconds: 120
Estimated round trip time: 1 ms
Read thread: on  Write thread: on  FD used: 31

Sorry, I don’t get it. How should I do ?
Do you mean set default rout to each neighbor ?

vyos@vyos# set protocols static route 0.0.0.0/0 next-hop 10.10.10.1
[edit]
vyos@vyos# set protocols static route 0.0.0.0/0 next-hop 10.20.20.1

Neighbours should export default routes
After this you receive/import such routes which will be placed in route table declared match route-map rule

Done!
but the route table is till empty.

vyos@vyos# run show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

S>  0.0.0.0/0 [1/0] via 10.10.10.1 (recursive), weight 1, 00:06:42
  *                   via 10.10.10.1, wg01 onlink, weight 1, 00:06:42
                    via 10.20.20.1 (recursive), weight 1, 00:06:42
  *                   via 10.20.20.1, wg02 onlink, weight 1, 00:06:42
S>* 10.10.10.0/24 [1/0] is directly connected, wg01, weight 1, 00:20:41
C>* 10.10.10.254/32 is directly connected, wg01, 00:20:41
S>* 10.20.20.0/24 [1/0] is directly connected, wg02, weight 1, 00:20:41
C>* 10.20.20.254/32 is directly connected, wg02, 00:20:42
C>* 100.64.0.0/30 is directly connected, eth0, 00:20:44
C>* 192.168.2.0/24 is directly connected, eth1, 00:20:45
S>* 198.51.100.254/32 [1/0] via 100.64.0.1, eth0, weight 1, 00:20:40
S>* 203.0.113.254/32 [1/0] via 100.64.0.1, eth0, weight 1, 00:20:40
[edit]
vyos@vyos# run show ip route table  100
[edit]

my vyos config

vyos@vyos# run show configuration commands
set interfaces ethernet eth0 address '100.64.0.2/30'
set interfaces ethernet eth0 hw-id '50:00:00:02:00:00'
set interfaces ethernet eth1 address '192.168.2.1/24'
set interfaces ethernet eth1 hw-id '50:00:00:02:00:01'
set interfaces ethernet eth1 policy route 'PBR'
set interfaces ethernet eth2 hw-id '50:00:00:02:00:02'
set interfaces ethernet eth3 hw-id '50:00:00:02:00:03'
set interfaces loopback lo
set interfaces wireguard wg01 address '10.10.10.254/32'
set interfaces wireguard wg01 peer cloud1 address '203.0.113.254'
set interfaces wireguard wg01 peer cloud1 allowed-ips '0.0.0.0/0'
set interfaces wireguard wg01 peer cloud1 port '8000'
set interfaces wireguard wg01 peer cloud1 pubkey 'ugeluxTvZkvRt6Py1SJDX0XRfxrF/ifZ0muh9GStH1c='
set interfaces wireguard wg02 address '10.20.20.254/32'
set interfaces wireguard wg02 peer cloud2 address '198.51.100.254'
set interfaces wireguard wg02 peer cloud2 allowed-ips '0.0.0.0/0'
set interfaces wireguard wg02 peer cloud2 port '8000'
set interfaces wireguard wg02 peer cloud2 pubkey 'XGBDUlgTZ+lgCgPR+YB9kQxtuG3Hxt5mEktIeVFgNAc='
set nat source rule 100 outbound-interface 'wg01'
set nat source rule 100 translation address 'masquerade'
set nat source rule 110 outbound-interface 'wg02'
set nat source rule 110 translation address 'masquerade'
set policy route PBR rule 10 destination address '!192.168.2.0/24'
set policy route PBR rule 10 set table '100'
set policy route PBR rule 10 source address '192.168.2.0/24'
set policy route-map RMAP-IN rule 100 action 'permit'
set policy route-map RMAP-IN rule 100 set table '100'
set protocols bgp 65001 address-family ipv4-unicast maximum-paths ibgp '2'
set protocols bgp 65001 neighbor 10.10.10.1 remote-as '65001'
set protocols bgp 65001 neighbor 10.10.10.1 timers holdtime '30'
set protocols bgp 65001 neighbor 10.10.10.1 timers keepalive '10'
set protocols bgp 65001 neighbor 10.10.10.1 update-source '10.10.10.254'
set protocols bgp 65001 neighbor 10.20.20.1 remote-as '65001'
set protocols bgp 65001 neighbor 10.20.20.1 timers holdtime '30'
set protocols bgp 65001 neighbor 10.20.20.1 timers keepalive '10'
set protocols bgp 65001 neighbor 10.20.20.1 update-source '10.20.20.254'
set protocols static interface-route 10.10.10.0/24 next-hop-interface wg01
set protocols static interface-route 10.20.20.0/24 next-hop-interface wg02
set protocols static route 0.0.0.0/0 next-hop 10.10.10.1
set protocols static route 0.0.0.0/0 next-hop 10.20.20.1
set protocols static route 198.51.100.254/32 next-hop 100.64.0.1
set protocols static route 203.0.113.254/32 next-hop 100.64.0.1
set service dhcp-server shared-network-name LAN-DHCP subnet 192.168.2.0/24 default-router '192.168.2.1'
set service dhcp-server shared-network-name LAN-DHCP subnet 192.168.2.0/24 name-server '192.168.2.1'
set service dhcp-server shared-network-name LAN-DHCP subnet 192.168.2.0/24 range 0 start '192.168.2.11'
set service dhcp-server shared-network-name LAN-DHCP subnet 192.168.2.0/24 range 0 stop '192.168.2.254'
set system config-management commit-revisions '100'
set system conntrack modules ftp
set system conntrack modules h323
set system conntrack modules nfs
set system conntrack modules pptp
set system conntrack modules sip
set system conntrack modules sqlnet
set system conntrack modules tftp
set system console device ttyS0 speed '115200'
set system host-name 'vyos'
set system login user vyos authentication encrypted-password '$6$p96vCymIVf.pAsYW$KWwo.AM/PF7psfABBHWwAeryXwB0YE74Ns/wK6VhKf0OtKpWgejYK'
set system login user vyos authentication plaintext-password ''
set system ntp server time1.vyos.net
set system ntp server time2.vyos.net
set system ntp server time3.vyos.net
set system syslog global facility all level 'info'
set system syslog global facility protocols level 'debug'