Just to get this out of the way from the beginning: No firewall/packet filter (yet).
My simple policy routing (partially discussed in my previous questions [1-3]) drives me nuts. With your help (pointing me to binding DHCP instance of DSL to a VRF and using vrf bind-to-all), I am close to where I want to be but for some strange reason, outgoing TCP connections don’t work.
My setup (VyOS 1.3):
- A DSL uplink on eth0.2 with dynamically assigned IP & default route
- A public routed /24 (192.0.2/24) via a wireguard tunnel from a VPS endpoint
- A test network 192.0.2.241/29 assigned to a dummy interface (to test routing and OSPF)
- Any traffic should consult local routing information (tables local, main, incl OSPF) if there is a more specific entry than 0.0.0.0/0
- For any traffic that only matches 0.0.0.0/0 (default route):
- If it has a source address in 192.0.2/24 should be routed over the wireguard tunnel
- If not, it should go over the DSL connection
In order to achieve this, I did the following [1]:
- Create a VRF “vrf_dsl” with table 170 and bind it to eth0.2. As a result, the default route (assigned via DHCP) lands in table 170 and NOT the main table
- I am actually not 100% sure what
vrf bind-to-alldoes but it was suggested in [1] and afterwards, it made SSH and wireguard working - The “local-route” policy consults first local and main (both of which do NOT include a default route). Rule 103 sends all packets from 192.0.2/24 over the wireguard link and rule 104 jumps to the default route (catch-all).
At first glance, everything works: Wireguard tunnel is established, resolver (DNS) works and all ICMP ping works. Example:
vyos@SunGate1:~$ ping www.google.com
PING www.google.com (172.217.12.100) 56(84) bytes of data.
64 bytes from sfo03s33-in-f4.1e100.net (172.217.12.100): icmp_seq=1 ttl=112 time=4.22 ms
64 bytes from sfo03s33-in-f4.1e100.net (172.217.12.100): icmp_seq=2 ttl=112 time=4.26 ms
However, I cannot establish a HTTP connection (or any other, like ssh to any server):
vyos@SunGate1:~$ /bin/telnet www.google.com 80
Trying 172.217.12.100...
^C
But now comes the kicker: If start a bash session within the VRF context I can connect!
vyos@SunGate1:~$ sudo ip vrf exec vrf_dsl bash
root@SunGate1:/home/vyos# /bin/telnet www.google.com 80
Trying 172.217.12.100...
Connected to www.google.com.
Escape character is '^]'.
GET / HTTP/1.0
[...]
</script></body></html>Connection closed by foreign host.
vyos@SunGate1:~$
W-T-F-??
I really don’t get it. Ping works, tcp not. And the policy very clearly has the proper rules set for the default route. I can also confirm:
vyos@SunGate1:~$ sudo ip rule
101: from all lookup local
102: from all lookup main
103: from 192.0.2.0/24 lookup 171
104: from all lookup vrf_dsl
1000: from all lookup [l3mdev-table]
2000: from all lookup [l3mdev-table] unreachable
32765: from all lookup local
32766: from all lookup main
32767: from all lookup default
vyos@SunGate1:~$ sudo ip route show table main | grep '0.0.0.0'
vyos@SunGate1:~$ sudo ip route show table vrf_dsl
default nhid 22 via DSL-PUBLIC-ROUTER dev eth0.2 proto static metric 20
broadcast 127.0.0.0 dev vrf_dsl proto kernel scope link src 127.0.0.1
127.0.0.0/8 dev vrf_dsl proto kernel scope link src 127.0.0.1
local 127.0.0.1 dev vrf_dsl proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev vrf_dsl proto kernel scope link src 127.0.0.1
broadcast DSL-PUBLIC-NETWORK dev eth0.2 proto kernel scope link src DSL-PUBLIC-IP
DSL-PUBLIC-NETWORK/21 dev eth0.2 proto kernel scope link src DSL-PUBLIC-IP
local DSL-PUBLIC-IP dev eth0.2 proto kernel scope host src DSL-PUBLIC-IP
broadcast DSL-PUBLIC-BCAST dev eth0.2 proto kernel scope link src DSL-PUBLIC-IP
Since any TCP connection (like telnet in my example above) runs in the default VRF context, it should normally run through the rules. It would first try 101 and not find a match. Similarly no match for 102 in main. Rule 103 does not match in the first place. Then rule 104 matches, it looks up table vrf_dsl and finds the default route there.
It MUST work. But it doesn’t.
Is anyone able to tell me what the heck is going on here?
Finally, full config for reference.
vyos@SunGate1:~$ show conf com
set interfaces dummy dum0 address ‘192.0.2.241/29'
set interfaces ethernet eth0 vif 2 address 'dhcp'
set interfaces ethernet eth0 vif 2 description ‘dsl’
set interfaces ethernet eth0 vif 2 vrf 'vrf_dsl’
set interfaces ethernet eth0 vif 3 address '10.227.79.2/24'
set interfaces ethernet eth0 vif 4 address '10.227.4.2/24'
set interfaces ethernet eth0 vif 5 address '10.227.1.2/24'
set interfaces ethernet eth0 vif 10 address '10.227.80.2/24'
set interfaces ethernet eth0 vif 11 address '192.168.222.2/24'
set interfaces ethernet eth0 vif 12 address '192.168.223.2/24'
set interfaces loopback lo
set interfaces wireguard wg0 address ‘192.0.2.227/31'
set interfaces wireguard wg0 description 'Vultr uplink'
set interfaces wireguard wg0 ip ospf authentication md5 key-id 1 md5-key ‘*’**********
set interfaces wireguard wg0 ip ospf cost '100'
set interfaces wireguard wg0 ip ospf dead-interval '40'
set interfaces wireguard wg0 ip ospf hello-interval '10'
set interfaces wireguard wg0 ip ospf network 'point-to-point'
set interfaces wireguard wg0 ip ospf priority '1'
set interfaces wireguard wg0 ip ospf retransmit-interval '5'
set interfaces wireguard wg0 ip ospf transmit-delay '1'
set interfaces wireguard wg0 peer vultr0 address ‘*’***********
set interfaces wireguard wg0 peer vultr0 allowed-ips '0.0.0.0/0'
set interfaces wireguard wg0 peer vultr0 port '51821'
set nat source rule 100 outbound-interface 'eth0.2'
set nat source rule 100 translation address 'masquerade'
set policy local-route rule 101 destination '0.0.0.0/0'
set policy local-route rule 101 set table 'local'
set policy local-route rule 102 destination '0.0.0.0/0'
set policy local-route rule 102 set table 'main'
set policy local-route rule 103 destination '0.0.0.0/0'
set policy local-route rule 103 set table '171'
set policy local-route rule 103 source ‘192.0.2.0/24'
set policy local-route rule 104 destination '0.0.0.0/0'
set policy local-route rule 104 set table '170'
set protocols ospf area 0.0.0.0 network ‘192.0.2.0/24'
set protocols ospf parameters abr-type 'cisco'
set protocols ospf parameters router-id '10.227.1.2'
set protocols ospf passive-interface 'eth0.2'
set protocols static table 171 route 0.0.0.0/0 next-hop 192.0.2.226
set service ssh port ’22’
set system conntrack modules ftp
set system conntrack modules h323
set system conntrack modules nfs
set system conntrack modules pptp
set system conntrack modules sip
set system conntrack modules sqlnet
set system conntrack modules tftp
set system console device ttyS0 speed '115200'
set system host-name 'SunGate1'
set system name-server 'eth0.2'
set system ntp server time1.vyos.net
set system ntp server time2.vyos.net
set system ntp server time3.vyos.net
set system syslog global facility all level 'info'
set system syslog global facility protocols level 'debug'
set system time-zone 'America/Los_Angeles'
set vrf bind-to-all
set vrf name vrf_dsl table '170'
[1] Putting DHCP default gateway in different routing table - #7 by exp
[2] How can I make my wireguard tunnel accessible via policy routing?
[3] Simple source routing not so simple? (How to migrate this simple RouterOS example to VyOS)
