I can't get NAT64 + NAT44 working

I have a test network setup with VMs: “test-end3” (ubuntu 24.04lts), core-dev (VyOS rolling), dummy-tubes (VyOS rolling)

test-end3 <—> core-dev <—> dummy-tubes

I’m trying to setup an IPv6 only test network. I have this test network setup to figure out some problems. It doesn’t seem that the NAT44 is picking up after the NAT64. I’m not running any firewall to make sure that is not the problem.

Here are the relevant config commands for core-dev:

set interfaces ethernet eth0 address 'dhcp'
set interfaces ethernet eth0 address 'dhcpv6'
set interfaces ethernet eth0 description 'WAN'
set interfaces ethernet eth0 dhcpv6-options pd 0 interface eth1 address '1'
set interfaces ethernet eth0 dhcpv6-options pd 0 interface eth1 sla-id '62'
set interfaces ethernet eth0 dhcpv6-options pd 0 length '48'
set interfaces ethernet eth0 dhcpv6-options rapid-commit
set interfaces dummy dum0 address '10.0.62.1/32'
set interfaces dummy dum0 address 'fd87:c6ff:14b4::62/128'
set nat source rule 102 outbound-interface name 'eth0'
set nat source rule 102 source address '10.0.0.0/8'
set nat source rule 102 translation address 'masquerade'
set nat64 source rule 10 source prefix '64:ff9b::/96'
set nat64 source rule 10 translation pool 1 address '10.0.62.1'
set nat64 source rule 10 translation pool 1 port '1-65535'
set service dns forwarding allow-from 'fd87:c6ff:14b4::/48'
set service dns forwarding dns64-prefix '64:ff9b::/96'
set service dns forwarding listen-address 'fd87:c6ff:14b4::62'
set service dns forwarding name-server 10.0.60.2
set service dns forwarding name-server fd87:c6ff:14b4:3c::2
set service router-advert interface eth1 default-lifetime '300'
set service router-advert interface eth1 default-preference 'high'
set service router-advert interface eth1 dnssl 'unaen.org'
set service router-advert interface eth1 hop-limit '64'
set service router-advert interface eth1 interval max '30'
set service router-advert interface eth1 link-mtu '1500'
set service router-advert interface eth1 name-server 'fd87:c6ff:14b4::62'
set service router-advert interface eth1 other-config-flag
set service router-advert interface eth1 prefix ::/64 preferred-lifetime '300'
set service router-advert interface eth1 prefix ::/64 valid-lifetime '900'
set service router-advert interface eth1 reachable-time '900000'
set service router-advert interface eth1 retrans-timer '0'

Here are the relevant config commands for dummy-tubes:

set interfaces ethernet eth1 address '100.100.0.1/24'
set interfaces ethernet eth1 address '100::1/64'
set interfaces ethernet eth1 description 'tubes'
set service dhcp-server hostfile-update
set service dhcp-server shared-network-name tubes authoritative
set service dhcp-server shared-network-name tubes subnet 100.100.0.0/24 option default-router '100.100.0.1'
set service dhcp-server shared-network-name tubes subnet 100.100.0.0/24 option domain-name 'unaen.org'
set service dhcp-server shared-network-name tubes subnet 100.100.0.0/24 option name-server '100.100.0.1'
set service dhcp-server shared-network-name tubes subnet 100.100.0.0/24 range 0 start '100.100.0.2'
set service dhcp-server shared-network-name tubes subnet 100.100.0.0/24 range 0 stop '100.100.0.254'
set service dhcp-server shared-network-name tubes subnet 100.100.0.0/24 subnet-id '1'
set service dhcpv6-server preference '100'
set service dhcpv6-server shared-network-name tubes subnet 100::/64 interface 'eth1'
set service dhcpv6-server shared-network-name tubes subnet 100::/64 option domain-search 'unaen.org'
set service dhcpv6-server shared-network-name tubes subnet 100::/64 option name-server '100::1'
set service dhcpv6-server shared-network-name tubes subnet 100::/64 prefix-delegation prefix 100:0:100:: delegated-length '48'
set service dhcpv6-server shared-network-name tubes subnet 100::/64 prefix-delegation prefix 100:0:100:: prefix-length '40'
set service dhcpv6-server shared-network-name tubes subnet 100::/64 range 1 start '100::2'
set service dhcpv6-server shared-network-name tubes subnet 100::/64 range 1 stop '100::254'
set service dhcpv6-server shared-network-name tubes subnet 100::/64 subnet-id '1'

The interfaces status of core-dev:

copec@core-dev:~$ show interfaces
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface    IP Address               MAC                VRF        MTU  S/L    Description
-----------  -----------------------  -----------------  -------  -----  -----  ----------------
dum0         10.0.62.1/32             5e:30:12:87:8a:22  default   1500  u/u
             fd87:c6ff:14b4::62/128
eth0         100.100.0.2/24           bc:24:11:b8:ca:8c  default   1500  u/u    WAN
             100::4/128
eth1         100:0:102:3e::1/64       bc:24:11:78:6e:22  default   1500  u/u    LAN test vxlan62
             fd87:c6ff:14b4:3e::1/64
eth2         10.0.61.1/24             bc:24:11:84:7c:6a  default   1500  u/u    LAN test vxlan61
             100:0:102:3d::1/64
             fd87:c6ff:14b4:3d::1/64
eth3         10.0.60.1/24             bc:24:11:5a:d1:99  default   1500  u/u    ADMIN vxlan60
             fd87:c6ff:14b4:3c::1/64
lo           127.0.0.1/8              00:00:00:00:00:00  default  65536  u/u
             ::1/128

I can communicate from test-end3 to any of the interfaces on core-dev, but nothing passes through core-dev:

root@test-end3:~# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether bc:24:11:0f:8e:16 brd ff:ff:ff:ff:ff:ff
    altname enp0s18
    inet 10.0.60.73/24 brd 10.0.60.255 scope global ens18
       valid_lft forever preferred_lft forever
    inet6 fd87:c6ff:14b4:3c::73/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::be24:11ff:fe0f:8e16/64 scope link
       valid_lft forever preferred_lft forever
3: ens19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether bc:24:11:ac:d9:f1 brd ff:ff:ff:ff:ff:ff
    altname enp0s19
    inet6 fd87:c6ff:14b4:3e:be24:11ff:feac:d9f1/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 882sec preferred_lft 282sec
    inet6 100:0:102:3e:be24:11ff:feac:d9f1/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 882sec preferred_lft 282sec
    inet6 fe80::be24:11ff:feac:d9f1/64 scope link
       valid_lft forever preferred_lft forever

root@test-end3:~# ping -I 100:0:102:3e:be24:11ff:feac:d9f1 64:ff9b::10.0.62.1
PING 64:ff9b::10.0.62.1 (64:ff9b::a00:3e01) from 100:0:102:3e:be24:11ff:feac:d9f1 : 56 data bytes
64 bytes from 64:ff9b::a00:3e01: icmp_seq=1 ttl=63 time=0.382 ms
64 bytes from 64:ff9b::a00:3e01: icmp_seq=2 ttl=63 time=0.314 ms
^C
--- 64:ff9b::10.0.62.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1060ms
rtt min/avg/max/mdev = 0.314/0.348/0.382/0.034 ms

root@test-end3:~# ping -I 100:0:102:3e:be24:11ff:feac:d9f1 64:ff9b::100.100.0.2
PING 64:ff9b::100.100.0.2 (64:ff9b::6464:2) from 100:0:102:3e:be24:11ff:feac:d9f1 : 56 data bytes
64 bytes from 64:ff9b::6464:2: icmp_seq=1 ttl=63 time=0.253 ms
64 bytes from 64:ff9b::6464:2: icmp_seq=2 ttl=63 time=0.256 ms
^C
--- 64:ff9b::100.100.0.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1050ms
rtt min/avg/max/mdev = 0.253/0.254/0.256/0.001 ms

root@test-end3:~# ping -I 100:0:102:3e:be24:11ff:feac:d9f1 64:ff9b::100.100.0.1
PING 64:ff9b::100.100.0.1 (64:ff9b::6464:1) from 100:0:102:3e:be24:11ff:feac:d9f1 : 56 data bytes
^C
--- 64:ff9b::100.100.0.1 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4102ms

root@test-end3:~# ping -I 100:0:102:3e:be24:11ff:feac:d9f1 64:ff9b::100.100.0.3
PING 64:ff9b::100.100.0.3 (64:ff9b::6464:3) from 100:0:102:3e:be24:11ff:feac:d9f1 : 56 data bytes
^C
--- 64:ff9b::100.100.0.3 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1052ms

root@test-end3:~# ping -I 100:0:102:3e:be24:11ff:feac:d9f1 64:ff9b::100.100.0.4
PING 64:ff9b::100.100.0.4 (64:ff9b::6464:4) from 100:0:102:3e:be24:11ff:feac:d9f1 : 56 data bytes
^C
--- 64:ff9b::100.100.0.4 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2079ms

Pure IPv6 traffic is passing through, and pure ipv4 traffic from my other test vlan also passes through fine.

root@test-end3:~# ping -I 100:0:102:3e:be24:11ff:feac:d9f1 100::1
PING 100::1 (100::1) from 100:0:102:3e:be24:11ff:feac:d9f1 : 56 data bytes
64 bytes from 100::1: icmp_seq=1 ttl=63 time=1.05 ms
copec@test-end2:~$ ping 100.100.0.1
PING 100.100.0.1 (100.100.0.1) 56(84) bytes of data.
64 bytes from 100.100.0.1: icmp_seq=1 ttl=63 time=0.532 ms

I apologize for the dump, but after trying to figure it out for sometime, I figure someone else will probably run into this issue too.

Doing some more digging, it appears that anything from the NAT64 exits the external interface as the v4 IP set in the NAT64:

root@core-dev:~# tcpdump --interface eth0                                                                                                                                                                                       
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode                                                                                                                                                       
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes                                                                                                                                                    
17:40:03.323804 IP 10.0.62.1 > 100.100.0.1: ICMP echo request, id 23522, seq 1, length 64                                                                                                                                       
17:40:04.377847 IP 10.0.62.1 > 100.100.0.1: ICMP echo request, id 23522, seq 2, length 64                                                                                                                                       
17:40:05.401803 IP 10.0.62.1 > 100.100.0.1: ICMP echo request, id 23522, seq 3, length 64                                                                                                                                       
17:40:06.425798 IP 10.0.62.1 > 100.100.0.1: ICMP echo request, id 23522, seq 4, length 64                                                                                                                                       

That is pinging from:

root@test-end3:~# ping 64:ff9b::100.100.0.1
PING 64:ff9b::100.100.0.1 (64:ff9b::6464:1) 56 data bytes
^C
--- 64:ff9b::100.100.0.1 ping statistics ---
23 packets transmitted, 0 received, 100% packet loss, time 22535ms

that is going through the NAT64 first.