Time to live exceeded

muhammadtoffaha · April 3, 2022, 12:35pm

Hi All,
I do have a VyOS setup for my VMware lab as the following:-

Physical server with GW 172.16.36.1 contains DNS & vCenter server
additional interface for Nested ESXi
Interface IP Address S/L Description

eth0 172.16.36.230/24 u/u Outside
eth1 - u/u Internal
eth1.1610 172.16.10.1/24 u/u VLAN 1610 for Management
eth1.1620 172.16.20.1/24 u/u VLAN 1620 for Servers-1
eth1.1630 172.16.30.1/24 u/u VLAN 1630 for Servers-2
eth1.1698 - u/u VLAN 1698 for vSAN
eth1.1699 - u/u VLAN 1699 for vMotion
lo 127.0.0.1/8 u/u
::1/128

my issue is
when I try to ping 172.16.10.x from my physical network 172.16.36.x I got the following

ping 172.16.10.21
PING 172.16.10.21 (172.16.10.21) 56(84) bytes of data.
From 172.16.6.2 icmp_seq=1 Time to live exceeded
From 172.16.6.2 icmp_seq=2 Time to live exceeded
From 172.16.6.2 icmp_seq=3 Time to live exceeded

any clue to fix this issue?

muhammadtoffaha · April 3, 2022, 12:53pm

I am following the same configurations

Lean · April 3, 2022, 12:53pm

Hi @muhammadtoffaha !

Can you ping from VyOS to 172.16.36.x network but using eth0 as source?

ping 172.16.10.21 source-adress 172.16.36.230

muhammadtoffaha · April 3, 2022, 12:58pm

Hi @Lean
Yes, I can.

vyos@vyos:~$ ping 172.16.10.21 source-address 172.16.36.230
PING 172.16.10.21 (172.16.10.21) from 172.16.36.230 : 56(84) bytes of data.
64 bytes from 172.16.10.21: icmp_seq=1 ttl=64 time=0.385 ms
64 bytes from 172.16.10.21: icmp_seq=2 ttl=64 time=0.325 ms
64 bytes from 172.16.10.21: icmp_seq=3 ttl=64 time=0.376 ms

Lean · April 3, 2022, 1:07pm

Good, so routing is not an issue!

did you enable promiscuous mode and forged transmits on the vSwitch?

i would also test to allow MAC Changes (last item in the image).

on the other hand, are you using 172.16.36.230 as your Gateway in the device you are pinging from? if not… did you add a static route to 172.16.10.0/24 and use 172.16.36.230 as your next-hop?

maybe you can share a ip route or route print (if using windows).

muhammadtoffaha · April 3, 2022, 1:14pm

Yes, I have all security policies allowed

No, I am using 172.16.36.1 as Gateway not 172.16.36.230
I tried to add static route, but still same issue
vyos@vyos# set protocols static route 172.16.10.0/24 next-hop 172.16.36.230

Windows Route Print

C:\Users\Administrator>route print
===========================================================================
Interface List
 25...00 0c 29 b5 14 ad ......vmxnet3 Ethernet Adapter
 11...00 0c 29 b5 14 a3 ......Intel(R) 82574L Gigabit Network Connection
  1...........................Software Loopback Interface 1
===========================================================================

IPv4 Route Table
===========================================================================
Active Routes:
Network Destination        Netmask          Gateway       Interface  Metric
          0.0.0.0          0.0.0.0      172.16.36.1    172.16.36.228    281
        127.0.0.0        255.0.0.0         On-link         127.0.0.1    331
        127.0.0.1  255.255.255.255         On-link         127.0.0.1    331
  127.255.255.255  255.255.255.255         On-link         127.0.0.1    331
      172.16.10.0    255.255.255.0         On-link     172.16.10.228    271
    172.16.10.228  255.255.255.255         On-link     172.16.10.228    271
    172.16.10.255  255.255.255.255         On-link     172.16.10.228    271
      172.16.36.0    255.255.255.0         On-link     172.16.36.228    281
    172.16.36.228  255.255.255.255         On-link     172.16.36.228    281
    172.16.36.255  255.255.255.255         On-link     172.16.36.228    281
        224.0.0.0        240.0.0.0         On-link         127.0.0.1    331
        224.0.0.0        240.0.0.0         On-link     172.16.36.228    281
        224.0.0.0        240.0.0.0         On-link     172.16.10.228    271
  255.255.255.255  255.255.255.255         On-link         127.0.0.1    331
  255.255.255.255  255.255.255.255         On-link     172.16.36.228    281
  255.255.255.255  255.255.255.255         On-link     172.16.10.228    271
===========================================================================
Persistent Routes:
  Network Address          Netmask  Gateway Address  Metric
          0.0.0.0          0.0.0.0      172.16.36.1  Default
===========================================================================

Lean · April 3, 2022, 1:32pm

I see you have 2 connections in that route print, one interface to 172.16.36.X and other to 172.16.10.X, is that correct?

on the other hand, do you have a static route IN your default-gateway that has 172.16.36.230 as next-hop to 172.16.10.X?

n.fort · April 3, 2022, 1:38pm

Problem seems to be routing and miss-configuration of gateways. Using traceroute (tracert) may help to find out where packets are going

muhammadtoffaha · April 3, 2022, 1:38pm

yes, I added the second connection 172.16.10.x, just for testing!
Should I remove it?

Yes I added static route to be pointed to 172.16.36.230 but still the same

vyos@vyos# set protocols static route 172.16.10.0/24 next-hop 172.16.36.230

muhammadtoffaha · April 3, 2022, 1:40pm

Hi @n.fort

This is a traceroute from my vCenter server (172.16.36.229)

r

oot@vcsa [ ~ ]# traceroute 172.16.10.21
traceroute to 172.16.10.21 (172.16.10.21), 30 hops max, 60 byte packets
 1  _gateway (172.16.36.1)  0.682 ms  0.854 ms  1.138 ms
 2  172.16.6.2 (172.16.6.2)  0.629 ms  0.589 ms  0.975 ms
 3  _gateway (172.16.36.1)  1.944 ms  2.232 ms  2.499 ms
 4  172.16.6.2 (172.16.6.2)  1.250 ms  1.296 ms  1.261 ms
 5  _gateway (172.16.36.1)  2.635 ms  3.257 ms  2.891 ms
 6  172.16.6.2 (172.16.6.2)  1.361 ms  0.822 ms  1.914 ms
 7  _gateway (172.16.36.1)  3.950 ms  3.333 ms  4.207 ms
 8  172.16.6.2 (172.16.6.2)  2.583 ms  2.564 ms  2.538 ms
 9  _gateway (172.16.36.1)  4.350 ms  5.005 ms  5.313 ms
10  172.16.6.2 (172.16.6.2)  2.656 ms  2.841 ms  2.910 ms
11  _gateway (172.16.36.1)  5.439 ms  5.731 ms  5.941 ms
12  172.16.6.2 (172.16.6.2)  2.219 ms  1.861 ms  1.930 ms
13  _gateway (172.16.36.1)  4.004 ms  4.131 ms  4.214 ms
14  172.16.6.2 (172.16.6.2)  1.784 ms  1.762 ms  1.461 ms
15  _gateway (172.16.36.1)  3.275 ms  3.574 ms  3.843 ms
16  172.16.6.2 (172.16.6.2)  2.419 ms  2.485 ms  2.402 ms
17  _gateway (172.16.36.1)  4.015 ms  4.276 ms  4.552 ms
18  172.16.6.2 (172.16.6.2)  2.195 ms  2.272 ms  2.170 ms
19  _gateway (172.16.36.1)  3.037 ms  2.687 ms  3.263 ms
20  172.16.6.2 (172.16.6.2)  2.335 ms  3.049 ms  3.013 ms
21  _gateway (172.16.36.1)  5.447 ms  6.483 ms  7.536 ms
22  172.16.6.2 (172.16.6.2)  3.106 ms  3.094 ms  3.085 ms
23  _gateway (172.16.36.1)  8.777 ms  8.561 ms  7.378 ms
24  172.16.6.2 (172.16.6.2)  4.024 ms  4.023 ms  4.088 ms
25  _gateway (172.16.36.1)  8.709 ms  8.974 ms  7.642 ms
26  172.16.6.2 (172.16.6.2)  4.101 ms  4.331 ms  4.193 ms
27  _gateway (172.16.36.1)  8.116 ms  8.450 ms  6.894 ms
28  172.16.6.2 (172.16.6.2)  4.076 ms  4.030 ms  3.934 ms
29  _gateway (172.16.36.1)  5.430 ms  5.686 ms  5.700 ms
30  172.16.6.2 (172.16.6.2)  3.767 ms  3.716 ms  3.248 ms

n.fort · April 3, 2022, 1:42pm

Please provide interface and routing info from vyos:

show interfaces
show ip route
show config commands | grep protocol

n.fort · April 3, 2022, 1:50pm

So packet is not even reaching VyOS router! You should fix internal routing

Lean · April 3, 2022, 2:00pm

As Nicolas is saying and I was suggesting the problem is how you set up your network to work:

in this scenario if a packet is sent to anything outside 172.16.36.0/24 it will go to your default Gateway. Since your Gateway do not know how to reach 172.16.10.0/24 it will send it to his gateway (172.16.6.2) but again, since 172.16.6.2 does not know go to reach 172.16.36.0/24 it will follow its routing table (in this case, sent it back to 172.16.6.1)

So my suggestion is:

add a route in you laptop/workstation to sedn traffic to 172.16.10.0/24 using 172.16.36.230 as next hop
(route add 172.16.10.0 MASK 255.255.255.0 172.16.36.230)
set you network to use VyOS router as gateway and add a default route in your VyOS to use 172.16.36.1 as next hop.

you can do 1) to test

please disable the interface on 172.6.10.0/24.

let us know how it goes

muhammadtoffaha · April 3, 2022, 3:05pm

vyos@vyos:~$ show interfaces
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface        IP Address                        S/L  Description
---------        ----------                        ---  -----------
eth0             172.16.36.230/24                  u/u  Outside
eth1             -                                 u/u  Internal
eth1.1610        172.16.10.1/24                    u/u  VLAN 1610 for Management
eth1.1620        172.16.20.1/24                    u/u  VLAN 1620 for Servers-1
eth1.1630        172.16.30.1/24                    u/u  VLAN 1630 for Servers-2
eth1.1698        -                                 u/u  VLAN 1698 for vSAN
eth1.1699        -                                 u/u  VLAN 1699 for vMotion
lo               127.0.0.1/8                       u/u
                 ::1/128

=========

vyos@vyos:~$ show ip route
S>* 0.0.0.0/0 [1/0] via 172.16.36.1, eth0, weight 1, 02:04:51
C>* 172.16.10.0/24 is directly connected, eth1.1610, 3d03h08m
C>* 172.16.20.0/24 is directly connected, eth1.1620, 3d03h07m
C>* 172.16.30.0/24 is directly connected, eth1.1630, 3d03h07m
C>* 172.16.36.0/24 is directly connected, eth0, 3d21h16m

=======

vyos@vyos:~$ show config commands | grep protocol
set protocols static route 0.0.0.0/0 next-hop 172.16.36.1
set protocols static route 172.16.10.0/24 next-hop 172.16.36.230
set system syslog global facility protocols level 'debug'

muhammadtoffaha · April 3, 2022, 3:08pm

Option 1 => works fine from my workstation

C:\Users\Administrator>route add 172.16.10.0 MASK 255.255.255.0 172.16.36.230
 OK!

C:\Users\Administrator>ping 172.16.10.21

Pinging 172.16.10.21 with 32 bytes of data:
Reply from 172.16.10.21: bytes=32 time<1ms TTL=63
Reply from 172.16.10.21: bytes=32 time<1ms TTL=63
Reply from 172.16.10.21: bytes=32 time<1ms TTL=63

how can I set option number 2 as I have my vCenter server with GW 172.16.36.1?

muhammadtoffaha · April 3, 2022, 3:27pm

static route from vCenter works fine as well

root@vcsa [ /etc/systemd/network ]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.16.36.1     0.0.0.0         UG    0      0        0 eth0
172.16.10.0     172.16.36.230   255.255.255.0   UG    0      0        0 eth0
172.16.36.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0
root@vcsa [ /etc/systemd/network ]# ping 172.16.10.21
PING 172.16.10.21 (172.16.10.21) 56(84) bytes of data.
64 bytes from 172.16.10.21: icmp_seq=1 ttl=63 time=0.860 ms
64 bytes from 172.16.10.21: icmp_seq=2 ttl=63 time=0.655 ms
64 bytes from 172.16.10.21: icmp_seq=3 ttl=63 time=0.537 ms

Lean · April 3, 2022, 4:43pm

Great!

you can keep option 1 for as long as you need, any device on the 172.16.36.0/24 network that need to access the 172.16.10.0/24 (or any network that is connected to the VyOS router) would need a static route to do it.

To go with option two you would need to change any existing device to have 172.16.34.230 as gateway. If you use DHCP over that network would you need to change the settings as well and renew the leases.

you can also mix both options as need it.

Have a great sunday!

muhammadtoffaha · April 3, 2022, 10:11pm

Thanks a lot @Lean
Have a great day

system · April 5, 2022, 10:11pm

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.