Packet being modified while changing interfaces and VRFs

Bugs
, , ,
1

Hello VyOS community!

Recently I was playing with VRFs and route leaking between then.

My goal is to segregate WAN and LAN VRFs, while WAN VRF will be used to do BGP peering with multiple peers and keep a full routing table and LAN VRF will only have a default route pointing to WAN VRF.

vrf-diagram
vrf-diagram853Ă—900 22.9 KB

I was able to configure it as needed using @giuppo77 answer from Vrf: default rout leaking, and this is my current configuration at the moment:

Interfaces:

 interfaces {
     bonding bond0 {
         vif 50 {
             address x.x.x.2/30
             vrf WAN
         }
         vif 100 {
             address y.y.y.3/24
             vrf LAN
         }
     }
     virtual-ethernet veth0 {
         address 10.255.255.1/30
         peer-name veth1
         vrf WAN
     }
     virtual-ethernet veth1 {
         address 10.255.255.2/30
         peer-name veth0
         vrf LAN
     }
 }

VRFs:

 vrf {
     name LAN {
         protocols {
             static {
                 route 0.0.0.0/0 {
                     next-hop 10.255.255.1 {
                         interface veth1
                     }
                 }
             }
         }
         table 1000
     }
     name WAN {
         protocols {
             bgp {
                 neighbor 172.29.21.1 {
                     address-family {
                         ipv4-unicast {
                             nexthop-self {
                             }
                             route-map {
                                 export DENY
                             }
                             soft-reconfiguration {
                                 inbound
                             }
                         }
                     }
                     remote-as 22222
                     update-source 172.29.21.2
                 }
                 system-as 11111
             }
         }
         table 2000
     }
 }

To me that kind of configuration should work. A packet would arrive at bond0.100 and “LAN” VRF, follow the default route to 10.255.255.1 and get into “WAN” VRF, where the full routing table is, but looks like it’s being altered in it’s way:

19:12:34.592302 eth5  In  ifindex 7 cc:cc:cc:cc:cc:cc ethertype IPv4 (0x0800), length 104: y.y.y.119 > z.z.z.z: ICMP echo request, id 56, seq 1, length 64
19:12:34.592302 bond0 In  ifindex 10 cc:cc:cc:cc:cc:cc ethertype IPv4 (0x0800), length 104: y.y.y.119 > z.z.z.z: ICMP echo request, id 56, seq 1, length 64
19:12:34.592302 bond0.100 In  ifindex 11 cc:cc:cc:cc:cc:cc ethertype IPv4 (0x0800), length 104: y.y.y.119 > z.z.z.z: ICMP echo request, id 56, seq 1, length 64
19:12:34.592340 veth1 Out ifindex 15 bb:bb:bb:bb:bb:bb ethertype IPv4 (0x0800), length 104: y.y.y.119 > z.z.z.z: ICMP echo request, id 56, seq 1, length 64
19:12:34.592343 veth0 In  ifindex 16 bb:bb:bb:bb:bb:bb ethertype IPv4 (0x0800), length 104: y.y.y.119 > z.z.z.z: ICMP echo request, id 56, seq 1, length 64
19:12:34.592364 bond0.50 Out ifindex 14 aa:aa:aa:aa:aa:aa ethertype IPv4 (0x0800), length 104: y.y.y.119 > z.z.z.z: ICMP echo request, id 13623, seq 1, length 64
19:12:34.592366 bond0 Out ifindex 10 aa:aa:aa:aa:aa:aa ethertype IPv4 (0x0800), length 104: y.y.y.119 > z.z.z.z: ICMP echo request, id 13623, seq 1, length 64
19:12:34.592369 eth4  Out ifindex 6 aa:aa:aa:aa:aa:aa ethertype IPv4 (0x0800), length 104: y.y.y.119 > z.z.z.z: ICMP echo request, id 13623, seq 1, length 64

Notice how the packet id changes when the packet comes from veth0 to bond0.50

If I set de default route to one my upstreams directly the packet is not modified:

 name LAN {
     protocols {
         static {
             route 0.0.0.0/0 {
                 next-hop 172.29.21.1 {
                     vrf WAN
                 }
             }
         }
     }
     table 1000
 }

19:11:39.697734 eth4  In  ifindex 6 cc:cc:cc:cc:cc:cc ethertype IPv4 (0x0800), length 104: y.y.y.119 > z.z.z.z: ICMP echo request, id 54, seq 1, length 64
19:11:39.697734 bond0 In  ifindex 10 cc:cc:cc:cc:cc:cc ethertype IPv4 (0x0800), length 104: y.y.y.119 > z.z.z.z: ICMP echo request, id 54, seq 1, length 64
19:11:39.697734 bond0.100 In  ifindex 11 cc:cc:cc:cc:cc:cc ethertype IPv4 (0x0800), length 104: y.y.y.119 > z.z.z.z: ICMP echo request, id 54, seq 1, length 64
19:11:39.697773 bond0.50 Out ifindex 14 aa:aa:aa:aa:aa:aa ethertype IPv4 (0x0800), length 104: y.y.y.119 > z.z.z.z: ICMP echo request, id 54, seq 1, length 64
19:11:39.697775 bond0 Out ifindex 10 aa:aa:aa:aa:aa:aa ethertype IPv4 (0x0800), length 104: y.y.y.119 > z.z.z.z: ICMP echo request, id 54, seq 1, length 64
19:11:39.697779 eth4  Out ifindex 6 aa:aa:aa:aa:aa:aa ethertype IPv4 (0x0800), length 104: y.y.y.119 > z.z.z.z: ICMP echo request, id 54, seq 1, length 64

This is the version I’m currently testing with:

vyos@test:~$ show version
Version:          VyOS 1.4-testing
Release train:    current

Built by:         
Built on:         Sun 25 Feb 2024 14:26 UTC
Build UUID:       a4d13248-f9fe-4d3e-8793-2fb4e72edfc4
Build commit ID:  8e8e8e4bc5c5c4

Architecture:     x86_64
Boot via:         installed image
System type:      bare metal

Hardware vendor:  HP
Hardware model:   ProLiant DL360 Gen9
Hardware S/N:     
Hardware UUID:    

Copyright:        VyOS maintainers and contributors

Is this a supported configuration? I see no reason for that kind of packet modification while only going from an interface to another.

Any input is really appreciated!

2

Is there some sort of conntrack going on here?
What does sudo conntrack -C show?

This seems like it should probably be logged as a bug though.

3

This is a clean install, so only the default conntrack modules where enabled.

vyos@test:~$ sudo conntrack -C
75

Disabled all modules and rebooted, but it seems that conntrack is still there, even tho I used del system conntrack.

vyos@test:~$ sudo conntrack -C
21
4

So… I think I found the issue.

Just to remember this is a clean install, so there where no changes to the firewall or anything else besides VRFs, BGP and static routing.

Looks like per default VyOS will create some nftables rules, even tho there is no firewall declared on the configuration.

vyos@test# show firewall
Configuration under specified path is empty
[edit]

vyos@test# show system conntrack
Configuration under specified path is empty
[edit]

vyos@test# sudo nft list ruleset
table ip nat {
	chain VYOS_PRE_SNAT_HOOK {
		type nat hook postrouting priority srcnat - 1; policy accept;
		return
	}
}
table inet mangle {
	chain FORWARD {
		type filter hook forward priority mangle; policy accept;
	}
}
table ip raw {
	chain VYOS_TCP_MSS {
		type filter hook forward priority raw; policy accept;
	}

	chain vyos_global_rpfilter {
		return
	}

	chain vyos_rpfilter {
		type filter hook prerouting priority raw; policy accept;
		counter packets 53756 bytes 62460828 jump vyos_global_rpfilter
	}

	chain VYOS_PREROUTING_HOOK {
		type filter hook prerouting priority raw; policy accept;
	}
}
table ip6 raw {
	chain VYOS_TCP_MSS {
		type filter hook forward priority raw; policy accept;
	}

	chain vyos_global_rpfilter {
		return
	}

	chain vyos_rpfilter {
		type filter hook prerouting priority raw; policy accept;
		counter packets 55 bytes 5144 jump vyos_global_rpfilter
	}

	chain VYOS_PREROUTING_HOOK {
		type filter hook prerouting priority raw; policy accept;
	}
}
table inet vrf_zones {
	map ct_iface_map {
		typeof iifname : ct zone
		elements = { "LAN" : 1000,
			     "WAN" : 2000,
			     "veth0" : 2000,
			     "veth1" : 1000,
			     "bond0.50" : 2000,
			     "bond0.100" : 1000 }
	}

	chain vrf_zones_ct_in {
		type filter hook prerouting priority raw; policy accept;
		counter packets 53811 bytes 62465972 ct original zone set iifname map @ct_iface_map
	}

	chain vrf_zones_ct_out {
		type filter hook output priority raw; policy accept;
		counter packets 15713 bytes 862940 ct original zone set oifname map @ct_iface_map
	}
}
table ip vyos_conntrack {
	chain VYOS_CT_IGNORE {
		return
	}

	chain VYOS_CT_TIMEOUT {
		return
	}

	chain PREROUTING {
		type filter hook prerouting priority raw; policy accept;
		counter packets 53756 bytes 62460828 jump VYOS_CT_IGNORE
		counter packets 53756 bytes 62460828 jump VYOS_CT_TIMEOUT
		counter packets 53756 bytes 62460828 jump FW_CONNTRACK
		counter packets 53756 bytes 62460828 jump NAT_CONNTRACK
		counter packets 53756 bytes 62460828 jump WLB_CONNTRACK
		notrack
	}

	chain OUTPUT {
		type filter hook output priority raw; policy accept;
		counter packets 15666 bytes 858492 jump VYOS_CT_IGNORE
		counter packets 15666 bytes 858492 jump VYOS_CT_TIMEOUT
		counter packets 15666 bytes 858492 jump FW_CONNTRACK
		counter packets 15666 bytes 858492 jump NAT_CONNTRACK
		notrack
	}

	chain VYOS_CT_HELPER {
		return
	}

	chain FW_CONNTRACK {
		return
	}

	chain NAT_CONNTRACK {
		return
	}

	chain WLB_CONNTRACK {
		return
	}
}
table ip6 vyos_conntrack {
	chain VYOS_CT_IGNORE {
		return
	}

	chain VYOS_CT_TIMEOUT {
		return
	}

	chain PREROUTING {
		type filter hook prerouting priority raw; policy accept;
		counter packets 55 bytes 5144 jump VYOS_CT_IGNORE
		counter packets 55 bytes 5144 jump VYOS_CT_TIMEOUT
		counter packets 55 bytes 5144 jump FW_CONNTRACK
		counter packets 55 bytes 5144 jump NAT_CONNTRACK
		notrack
	}

	chain OUTPUT {
		type filter hook output priority raw; policy accept;
		counter packets 47 bytes 4448 jump VYOS_CT_IGNORE
		counter packets 47 bytes 4448 jump VYOS_CT_TIMEOUT
		counter packets 47 bytes 4448 jump FW_CONNTRACK
		counter packets 47 bytes 4448 jump NAT_CONNTRACK
		notrack
	}

	chain VYOS_CT_HELPER {
		return
	}

	chain FW_CONNTRACK {
		return
	}

	chain NAT_CONNTRACK {
		return
	}
}

sudo nft flush ruleset did solve the issue, but I as need firewall and tcp-mss changes it’s not a viable solution.

Is it the expected behavior? To me looks like when we use del system conntrack all kinds of conntrack should be completely disabled, and even if it was enabled, it should not modify any packets that are only being forwarded.

5

nftables being populated is expected. The rules present on the vyos_conntrack table are correct and fall through to the notrack rule at the end which should prevent conntrack from taking place.

I’ll do some tests in my lab and see if I can produce the same result.

2 Likes
6

If you remove only the VRF zone table, does it correct the issue?

sudo nft delete table inet vrf_zones

If so, then the issue is narrowed down and I can work on submitting a fix.

7

Yes, it does solve the issue.

After sudo nft delete table inet vrf_zones everything was working as expected.

8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.