Conntrac tftp appears to be failing (and udp conntrack in general)

Bugs
1

I’m setting up a new vyos install to handle some lab machines that will need to talk to an upstream tftp server for pxeboot handling.

vyos version: VyOS 1.4-rolling-202306200647
bare metal install

I’ve confirmed nf_conntrack_tftp and nf_nat_tftp are both loaded:

lsmod | grep conntrack
nf_conntrack_tftp      20480  1 nf_nat_tftp
nf_conntrack_sip       45056  1 nf_nat_sip
nf_conntrack_pptp      24576  1 nf_nat_pptp
nf_conntrack_h323      77824  1 nf_nat_h323
nf_conntrack_ftp       24576  1 nf_nat_ftp
nf_conntrack          180224  14 nf_nat,nf_conntrack_tftp,nfnetlink_cthelper,nft_ct,nf_nat_ftp,nf_conntrack_pptp,nf_nat_tftp,nf_conntrack_sip,nf_conntrack_h323,nf_nat_pptp,nf_conntrack_ftp,nft_masq,nf_nat_h323,nf_nat_sip
nf_defrag_ipv6         24576  1 nf_conntrack
nf_defrag_ipv4         16384  1 nf_conntrack

lsmod | grep tftp
nf_nat_tftp            16384  0
nf_conntrack_tftp      20480  1 nf_nat_tftp
nf_nat                 57344  7 nf_nat_ftp,nf_nat_tftp,nf_nat_pptp,nft_masq,nf_nat_h323,nft_chain_nat,nf_nat_sip
nf_conntrack          180224  14 nf_nat,nf_conntrack_tftp,nfnetlink_cthelper,nft_ct,nf_nat_ftp,nf_conntrack_pptp,nf_nat_tftp,nf_conntrack_sip,nf_conntrack_h323,nf_nat_pptp,nf_conntrack_ftp,nft_masq,nf_nat_h323,nf_nat_sip

I’ve configured both subnets to be masquerade nat’d:

show nat
 source {
     rule 10 {
         log
         outbound-interface eth0
         source {
             address 172.17.32.0/20
         }
         translation {
             address masquerade
         }
     }
     rule 11 {
         log
         outbound-interface eth0
         source {
             address 172.17.16.0/20
         }
         translation {
             address masquerade
         }
     }
 }

However even thou the nat seems to be working in other respects you end up with issues when udp services attempt to respond.

The nat entry gets created:

Id          Original src           Original dst           Reply src              Reply dst              Protocol    State        Timeout    Mark    Zone
----------  ---------------------  ---------------------  ---------------------  ---------------------  ----------  -----------  ---------  ------  ------
1178614433  172.17.16.15:5726      <tftp-server>:69     <tftp-server>:69     <router>:5726   udp                      28         0

but the tcp dump shows that the router is responding that the router is failing to deliver the response to the port:

16:49:08.755080 IP <router-ip>.5726 > <tftp-serverip>.tftp:  53 RRQ "pxelinux.0" octet tsize 0 blksize 1468 windowsize 4
16:49:08.755107 IP <router-ip>.5726 > <tftp-server-contiainer>.tftp:  53 RRQ "pxelinux.0" octet tsize 0 blksize 1468 windowsize 4
16:49:08.755113 IP <router-ip>.5726 > <tftp-server-contiainer>.tftp:  53 RRQ "pxelinux.0" octet tsize 0 blksize 1468 windowsize 4
16:49:08.758876 IP <tftp-server-contiainer>.43941 > <router-ip>.5726: UDP, length 27
16:49:08.758876 IP <tftp-server-contiainer>.43941 > <router-ip>.5726: UDP, length 27
16:49:08.758904 IP <tftp-serverip>.43941 > <router-ip>.5726: UDP, length 27
16:49:08.759125 IP <router-ip> > <tftp-serverip>.110: ICMP <router-ip> udp port 5726 unreachable, length 63
16:49:08.759136 IP <router-ip> > <tftp-server-contiainer>: ICMP <router-ip> udp port 5726 unreachable, length 63
16:49:08.759139 IP <router-ip> > <tftp-server-contiainer>: ICMP <router-ip> udp port 5726 unreachable, length 63

I’ve also tried replacing the nat with a port forward so when clients attempt to hit their gateway IP at port 69 it forwards to the appropriate server. this unfortunately has also failed.

This system is using zoned firewalls and outside-lan has a specific carveout to allow all traffic from the imaging server in, related and established are also enabled. I can provide sanitized firewall rules if that will help as well.

The tftp server being talked to is tftpd running in centos 9 stream to support a cobblerd instance.

side notes:

  • clients behind both masquerade ports can’t make tftp connections
  • traceroute fails from behind these ports as well
  • ping works just fine
  • tftp gets from the local router to the tftp server work
  • tftp gets from devices behind the router to a tftp server on the router work

Any idea how to make this work?

2

This seems to be an issue with conntrack helpers in general with VyOS 1.4. I’m experiencing the same issue with TFTP, and there’s another thread regarding similar issue with FTP (which also requires a helper).

Unfortunately, that’s not how TFTP works. You can’t just forward it as is because in addition to UDP/69 it uses a random port for the data channel - which is why the helper module is needed.

AFAIK, helper modules add their temporary entries in the conntrack “expectations” table which you can view using conntrack -L expect. Unfortunately, I wasn’t able to see anything, indicating again that something is wrong with the conntrack helpers.

3

Yea the port forward was a shot in the dark after exhausting a bunch of other attempts. I’ve confirmed that conntrac -L expect is indeed showing nothing despite the show conntrac table ipv4 showing a number of connections, and sudo conntrack -L showing similarly those connections with [UNREPLIED] for all udp connections.

4

Just a quick followup, i ended up down reving to 1.3, luckily we don’t have any hardware in that machine that requires 1.4 so everything worked as desired. I did of course have to rewrite much of the config as it can’t all be imported accurately.