Nat66 Seems to be broken in rolling build

davespc · May 27, 2024, 9:58pm

Hello,

I updated my install from 1.5-rolling-202312040024 to 1.5-rolling-202405270020 and noticed that NAT66 is no longer working. Things work as expected after rolling back to 1.5-rolling-202312040024 with the same config.

Relevant config:

set nat66 destination rule 15 destination address 'xxxx:xxxx:0:9c:15::/80'
set nat66 destination rule 15 inbound-interface name 'eth0'
set nat66 destination rule 15 translation address 'xxxx:xxxx:e857:6601:15::/80'
set nat66 destination rule 16 destination address 'xxxx:xxxx:0:9c:16::/80'
set nat66 destination rule 16 inbound-interface name 'eth0'
set nat66 destination rule 16 translation address 'xxxx:xxxx:e857:6601:16::/80'
set nat66 destination rule 80 destination address 'xxxx:xxxx:0:9c:80::/80'
set nat66 destination rule 80 inbound-interface name 'eth0'
set nat66 destination rule 80 translation address 'xxxx:xxxx:e857:6601:80::/80'
set nat66 destination rule 198 destination address 'xxxx:xxxx:0:9c:e198::/80'
set nat66 destination rule 198 inbound-interface name 'eth0'
set nat66 destination rule 198 translation address 'xxxx:xxxx:e857:6601:e198::/80'
set nat66 source rule 1 outbound-interface name 'eth0'
set nat66 source rule 1 source prefix 'xxxx:xxxx:e857:6601:15::/80'
set nat66 source rule 1 translation address 'xxxx:xxxx:0:9c:15::/80'
set nat66 source rule 16 outbound-interface name 'eth0'
set nat66 source rule 16 source prefix 'xxxx:xxxx:e857:6601:16::/80'
set nat66 source rule 16 translation address 'xxxx:xxxx:0:9c:16::/80'
set nat66 source rule 80 outbound-interface name 'eth0'
set nat66 source rule 80 source prefix 'xxxx:xxxx:e857:6601:80::/80'
set nat66 source rule 80 translation address 'xxxx:xxxx:0:9c:80::/80'
set nat66 source rule 198 outbound-interface name 'eth0'
set nat66 source rule 198 source prefix 'xxxx:xxxx:e857:6601:e198::/80'
set nat66 source rule 198 translation address 'xxxx:xxxx:0:9c:e198::/80'

I noticed with 1.5-rolling-202405270020 the translations timeout was relatively short:

$ show nat66 source translations
Pre-NAT                           Post-NAT                     Proto    Timeout    Mark    Zone
--------------------------------  ---------------------------  -------  ---------  ------  ------
xxxx:xxxx:e857:6601:e198::4:62474  xxxx:xxxx:0:9c:e198::4:62474  tcp      88         0
xxxx:xxxx:e857:6601:e198::4:62477  xxxx:xxxx:0:9c:e198::4:62477  tcp      107        0

With 1.5-rolling-202312040024, the timeout is relatively long:

$ show nat66 source translations
Pre-NAT                           Post-NAT                     Proto    Timeout    Mark    Zone
--------------------------------  ---------------------------  -------  ---------  ------  ------
xxxx:xxxx:e857:6601:e198::4:62428  xxxx:xxxx:0:9c:e198::4:62428  tcp      431978     0
xxxx:xxxx:e857:6601:e198::4:62435  xxxx:xxxx:0:9c:e198::4:62435  tcp      431915     0

The other thing I noticed was that the show ipv6 neighbors with 1.5-rolling-202312040024 showed the pre-NAT address with a state of FAILED which would seem to be expected.

Any idea what could be going on here? Would any other show command offer additional clues?

Thanks in advance!

Dave

c-po · May 28, 2024, 3:37am

Please also attach a topology diagram

davespc · May 28, 2024, 4:46pm

Sure!
Vyos eth0 eth1

davespc · February 16, 2025, 3:56pm

Hello VyOS community, following up on this.

NAT66 still seems to be broken. Attached is my configuration which is based on NAT66(NPTv6) — VyOS 1.4.x (sagitta) documentation which only has a few NAT66 rules and no ACLs to make things more simple. This configuration worked as of 1.5-rolling-202312040024.

Topology:

The packet captures referenced and list rules are here: vyos

Config:
vyos@redvyostest1:~$ show configuration commands
set interfaces ethernet eth0 address ‘2001:470:e857:207::1/64’
set interfaces ethernet eth0 hw-id ‘00:15:5d:54:24:46’
set interfaces ethernet eth1 address ‘fc00:4ee5:6c26:1c1d::defa/64’
set interfaces ethernet eth1 hw-id ‘00:15:5d:54:24:47’
set interfaces loopback lo
set nat66 destination rule 2001 destination address ‘2001:470:e857:207::/64’
set nat66 destination rule 2001 inbound-interface name ‘eth0’
set nat66 destination rule 2001 translation address ‘fc00:4ee5:6c26:1c1d::/64’
set nat66 source rule 2001 outbound-interface name ‘eth0’
set nat66 source rule 2001 source prefix ‘fc00:4ee5:6c26:1c1d::/64’
set nat66 source rule 2001 translation address ‘2001:470:e857:207::/64’
set protocols static route6 ::/0 next-hop 2001:470:e857:207::defa
set service ndp-proxy interface eth0 prefix fc00:4ee5:6c26:1c1d::/64 mode ‘static’
set service ssh
set system config-management commit-revisions ‘100’
set system conntrack modules ftp
set system conntrack modules h323
set system conntrack modules nfs
set system conntrack modules pptp
set system conntrack modules sip
set system conntrack modules sqlnet
set system conntrack modules tftp
set system host-name ‘redvyostest1’

When attempting to ping 2607:f8b0:400a:80a::200e from RouteLab1 (fc00:4ee5:6c26:1c1d::4) I see in the packet capture vyostest1-icmp6-eth0-working.pcap from VyosTest1 interface eth0 that the ICMP ping request is NATed to source address as expected:

2025-01-26 20:22:21.592610 1 2001:470:e857:207::4 2607:f8b0:400a:80a::200e ICMPv6 118 Echo (ping) request id=0xf331, seq=1, hop limit=63 (reply in 4)

When the upstream router gets the ping reply, it sends a neighbor solicitation for 2001:470:e857:207::4 as expected and is seen by eth0 on VyosTest1 (packet 2 in vyostest1-icmp6-eth0-working.pcap). VyosTest1 in turn sends a neighbor solicitation for fc00:4ee5:6c26:1c1d::4 out eth1 which is seen in packet 2 in vyostest1-icmp6-eth1-working.pcap. It gets the response and MAC address in packet 4.

2025-01-26 20:22:21.599315 2 fe80::215:5dff:fe54:2447 ff02::1:ff00:4 ICMPv6 86 Neighbor Solicitation for fc00:4ee5:6c26:1c1d::4 from 00:15:5d:54:24:47
2025-01-26 20:22:21.599527 4 fc00:4ee5:6c26:1c1d::4 fe80::215:5dff:fe54:2447 ICMPv6 86 Neighbor Advertisement fc00:4ee5:6c26:1c1d::4 (sol, ovr) is at 00:15:5d:35:56:19

VyosTest1 then sends a neighbor advertisement for IP 2001:470:e857:207::4 out eth0 with the eth0 MAC 00:15:5d:54:24:46 as expected:

2025-01-26 20:22:21.599071 3 fe80::215:5dff:fe54:2446 fe80::62be:b4ff:fe01:f9b9 ICMPv6 86 Neighbor Advertisement 2001:470:e857:207::4 (rtr, sol) is at 00:15:5d:54:24:46

The upstream router then forwards the ping reply which gets translated by VyosTest1 to fc00:4ee5:6c26:1c1d::4 as expected and the pings work:

2025-01-26 20:22:21.599278 4 2607:f8b0:400a:80a::200e 2001:470:e857:207::4 ICMPv6 118 Echo (ping) reply id=0xf331, seq=1, hop limit=120 (request in 1)

When VyosTest1 is upgraded to 1.5-rolling-202501270007, the ICMP ping to 2607:f8b0:400a:80a::200e gets translated to 2001:470:e857:207::4 as expected. This can be seen in packet #4 in vyostest1-icmp6-eth0-notworking.pcap.

VyosTest1 eth0 gets neighbor solicitation requests for 2001:470:e857:207::4 in packets 5, 7, and 8:
Time No. Source Destination Protocol Identification Length Info
2025-01-27 04:47:27.989031 4 2001:470:e857:207::4 2607:f8b0:400a:80a::200e ICMPv6 118 Echo (ping) request id=0xf395, seq=1, hop limit=63 (no response found!)
2025-01-27 04:47:27.994299 5 fe80::62be:b4ff:fe01:f9b9 ff02::1:ff00:4 ICMPv6 86 Neighbor Solicitation for 2001:470:e857:207::4 from 60:be:b4:01:f9:b9
2025-01-27 04:47:28.998393 6 2001:470:e857:207::4 2607:f8b0:400a:80a::200e ICMPv6 118 Echo (ping) request id=0xf395, seq=2, hop limit=63 (no response found!)
2025-01-27 04:47:29.015390 7 fe80::62be:b4ff:fe01:f9b9 ff02::1:ff00:4 ICMPv6 86 Neighbor Solicitation for 2001:470:e857:207::4 from 60:be:b4:01:f9:b9
2025-01-27 04:47:30.040479 8 fe80::62be:b4ff:fe01:f9b9 ff02::1:ff00:4 ICMPv6 86 Neighbor Solicitation for 2001:470:e857:207::4 from 60:be:b4:01:f9:b9

However, this does not result in VyosTest1 interface eth1 immediately generating neighbor solicitation packets out of eth1 for fc00:4ee5:6c26:1c1d::4 as seen in vyostest1-icmp6-eth1-notworking.pcap. Eventually some are received 6 seconds after the first ping request, but it doesn’t appear to be due to the ping response.

Time No. Source Destination Protocol Identification Length Info
2025-01-27 04:47:27.988639 5 fc00:4ee5:6c26:1c1d::4 2607:f8b0:400a:80a::200e ICMPv6 118 Echo (ping) request id=0xf395, seq=1, hop limit=64 (no response found!)
2025-01-27 04:47:28.998352 6 fc00:4ee5:6c26:1c1d::4 2607:f8b0:400a:80a::200e ICMPv6 118 Echo (ping) request id=0xf395, seq=2, hop limit=64 (no response found!)
2025-01-27 04:47:33.311673 7 fe80::215:5dff:fe54:2447 fc00:4ee5:6c26:1c1d::4 ICMPv6 86 Neighbor Solicitation for fc00:4ee5:6c26:1c1d::4 from 00:15:5d:54:24:47
2025-01-27 04:47:33.311689 8 fe80::215:5dff:fe54:2447 fe80::215:5dff:fe35:5619 ICMPv6 86 Neighbor Solicitation for fe80::215:5dff:fe35:5619 from 00:15:5d:54:24:47
2025-01-27 04:47:33.312170 9 fc00:4ee5:6c26:1c1d::4 fe80::215:5dff:fe54:2447 ICMPv6 78 Neighbor Advertisement fc00:4ee5:6c26:1c1d::4 (sol)
2025-01-27 04:47:33.312199 10 fe80::215:5dff:fe35:5619 fe80::215:5dff:fe54:2447 ICMPv6 78 Neighbor Advertisement fe80::215:5dff:fe35:5619 (sol)

The upstream router then sees 2001:470:e857:207::4 as FAILED:
duass@edge1d:~$ show ipv6 neighbors | grep eth2.607
2001:470:e857:207::4 eth2.607 FAILED
fe80::215:5dff:fe54:2446 eth2.607 00:15:5d:54:24:46 STALE

I compared the output of “sudo nft list ruleset” and outside of rule counts, I don’t see any differences. The running kernel version of 1.5-rolling-202312040024 is 6.1.65-amd64-vyos while 1.5-rolling-202501270007 is 6.6.69-vyos. I assume the kernel handles this functionality and not Vyos directly?

Does anyone know what could be causing this to fail? I don’t see any new configuration in the CLI that I’m obviously missing? Does anyone have NAT66 working in VyOS rolling 1.5-rolling-2024* or 1.5-rolling-2025*?

Thanks,
Dave