No TTL exceeded reply / ICMP

Hey,

I have discovered a wired behavior on my VyOS setup.
First I’ve seen that MTR and Traceroute don’t work.

I’ve tried to manually send a ICMP request with a low TTL so see if I get a response.
But nope, VyOS seems to either just ignore it or is unable to send me the answer.

The same behavior also happens, for internal routes between VLANs.
So for example

VLAN100-Client (172.20.100.50) -> VyOS (172.20.100.1 / 172.20.30.1) -> VLAN104-Client (172.20.30.103)

In this example there is no firewall or NAT involved. Just plain routing where VyOS acts as a gateway.

Hi @Mr_Funken,

I don’t have an answer, but some questions.

Are you pinging from VyOS of from a workstation?

Have you tried pinging from different interfaces?
ping -I eth0 1.1.1.1

Have you tried pinging 127.0.0.1?

Do you get the same results?

Hey @s.lorente

The issues occur when I try to ping from a client.
MTRs and Pings from the VyOS System itself work flawlessly.
(At least to public IPs. Can’t really test internal IPs from the VyOS System, as it has a direct route to each internal VLAN/IP.)

In general, when I have for example an internal route, that should look like that:

Hop 0 - 172.20.100.67 - Client
Hop 1 - 172.20.100.1 - VyOS Router
Hop 2 - 127.20.20.81 - Client on different VLAN

I can manually ping each hop without an issue.
But when I try to do a traceroute to Hop 2 I only see the Hop 2 but not the VyOS router in between.

It seems like my VyOS gateway just filters /ignores the “Time to live exceeded” message that I should get in a return of a TTL that is to short.

maybe this from the traceroute manpage helps:

 A more interesting example is:

 [yak 72]% traceroute allspice.lcs.mit.edu.
 traceroute to allspice.lcs.mit.edu (18.26.0.115), 64 hops max
 1  helios.ee.lbl.gov (128.3.112.1)  0 ms  0 ms  0 ms
 2  lilac-dmc.Berkeley.EDU (128.32.216.1)  19 ms  19 ms  19 ms
 3  lilac-dmc.Berkeley.EDU (128.32.216.1)  39 ms  19 ms  19 ms
 4  ccngw-ner-cc.Berkeley.EDU (128.32.136.23)  19 ms  39 ms  39 ms
 5  ccn-nerif22.Berkeley.EDU (128.32.168.22)  20 ms  39 ms  39 ms
 6  128.32.197.4 (128.32.197.4)  59 ms  119 ms  39 ms
 7  131.119.2.5 (131.119.2.5)  59 ms  59 ms  39 ms
 8  129.140.70.13 (129.140.70.13)  80 ms  79 ms  99 ms
 9  129.140.71.6 (129.140.71.6)  139 ms  139 ms  159 ms
 10  129.140.81.7 (129.140.81.7)  199 ms  180 ms  300 ms
 11  129.140.72.17 (129.140.72.17)  300 ms  239 ms  239 ms
 12  * * *
 13  128.121.54.72 (128.121.54.72)  259 ms  499 ms  279 ms
 14  * * *
 15  * * *
 16  * * *
 17  * * *
 18  ALLSPICE.LCS.MIT.EDU (18.26.0.115)  339 ms  279 ms  279 ms

 Note that the gateways 12, 14, 15, 16 & 17 hops away either don't send
 ICMP "time exceeded" messages or send them with a ttl too small to
 reach us.  14 - 17 are running the MIT C Gateway code that doesn't
 send "time exceeded"s.  God only knows what's going on with 12.

Is there any firewall config on the VLAN at all (specifically a local one).

Traceroute uses both ICMP and UDP Ports, below is an example config (incomplete only showing relevant parts)…

firewall {
    name ROUTER-LOCAL-ACL {
        default-action drop
        rule 10 {
            action accept
            state {
                established enable
                related enable
            }
        }
        rule 20 {
            action accept
            description "Allow ICMP"
            icmp {
                type-name echo-request
            }
            protocol icmp
            state {
                new enable
            }
        }
        rule 21 {
            action reject
            description "Allow Traceroute"
            destination {
                port 33434-33523
            }
            protocol udp
        }
        [...] Other rules for remote access/BGP/SNMP Etc
    }
}
interfaces {
    ethernet eth1 {
        firewall {
            local {
                name ROUTER-LOCAL-ACL
            }
        }
    }
}