What's the setting to use to allow ICMP echo via GRE from an ISP or cloud provider?

harguna · March 17, 2022, 8:16am

I see this could work since it’s applying the setting using the kernel but is there a specific setting on VyOS to allow this?

set system sysctl custom net.ipv4.conf.all.accept_local value '1'

Nikolay · March 19, 2022, 1:28am

By default VyOS does not block ICMP traffic
If ICMP are blocked, most likely a firewall is configured
Do you have any firewall rules?

harguna · March 19, 2022, 1:40am

Yeah I do but ICMP is allowed though:

rule 20 {
            action accept
            icmp {
                type-name echo-request
            }
            log enable
            protocol icmp
            state {
                new enable
            }
        }

Nikolay · March 19, 2022, 1:57am

Could you attach the complete configuration?
sh config comm | strip-private

harguna · March 19, 2022, 1:59am

set firewall all-ping 'enable'
set firewall broadcast-ping 'disable'
set firewall config-trap 'disable'
set firewall group ipv6-network-group cf-ipv6 network 'xxxx:xxxx::/32'
set firewall group ipv6-network-group cf-ipv6 network 'xxxx:xxxx::/32'
set firewall group ipv6-network-group cf-ipv6 network 'xxxx:xxxx::/32'
set firewall group ipv6-network-group cf-ipv6 network 'xxxx:xxxx::/32'
set firewall group ipv6-network-group cf-ipv6 network 'xxxx:xxxx::/32'
set firewall group ipv6-network-group cf-ipv6 network 'xxxx:xxxx::/29'
set firewall group ipv6-network-group cf-ipv6 network 'xxxx:xxxx::/32'
set firewall group network-group cf-ipv4 network 'xxx.xxx.48.0/20'
set firewall group network-group cf-ipv4 network 'xxx.xxx.244.0/22'
set firewall group network-group cf-ipv4 network 'xxx.xxx.200.0/22'
set firewall group network-group cf-ipv4 network 'xxx.xxx.4.0/22'
set firewall group network-group cf-ipv4 network 'xxx.xxx.64.0/18'
set firewall group network-group cf-ipv4 network 'xxx.xxx.192.0/18'
set firewall group network-group cf-ipv4 network 'xxx.xxx.240.0/20'
set firewall group network-group cf-ipv4 network 'xxx.xxx.96.0/20'
set firewall group network-group cf-ipv4 network 'xxx.xxx.240.0/22'
set firewall group network-group cf-ipv4 network 'xxx.xxx.128.0/17'
set firewall group network-group cf-ipv4 network 'xxx.xxx.0.0/15'
set firewall group network-group cf-ipv4 network 'xxx.xxx.0.0/13'
set firewall group network-group cf-ipv4 network 'xxx.xxx.0.0/14'
set firewall group network-group cf-ipv4 network 'xxx.xxx.0.0/13'
set firewall group network-group cf-ipv4 network 'xxx.xxx.72.0/22'
set firewall ipv6-name EXTERNAL-IN-v6 default-action 'drop'
set firewall ipv6-name EXTERNAL-IN-v6 enable-default-log
set firewall ipv6-name EXTERNAL-IN-v6 rule 10 action 'accept'
set firewall ipv6-name EXTERNAL-IN-v6 rule 10 log 'enable'
set firewall ipv6-name EXTERNAL-IN-v6 rule 10 state established 'enable'
set firewall ipv6-name EXTERNAL-IN-v6 rule 10 state related 'enable'
set firewall ipv6-name EXTERNAL-IN-v6 rule 20 action 'accept'
set firewall ipv6-name EXTERNAL-IN-v6 rule 20 destination port '80,443'
set firewall ipv6-name EXTERNAL-IN-v6 rule 20 log 'enable'
set firewall ipv6-name EXTERNAL-IN-v6 rule 20 protocol 'tcp_udp'
set firewall ipv6-name EXTERNAL-IN-v6 rule 20 source group network-group 'cf-ipv6'
set firewall ipv6-name EXTERNAL-IN-v6 rule 20 state new 'enable'
set firewall ipv6-name EXTERNAL-LOCAL-v6 default-action 'drop'
set firewall ipv6-name EXTERNAL-LOCAL-v6 enable-default-log
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 10 action 'accept'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 10 log 'enable'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 10 state established 'enable'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 10 state related 'enable'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 20 action 'accept'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 20 icmpv6 type 'echo-request'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 20 log 'enable'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 20 protocol 'icmpv6'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 20 state new 'enable'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 30 action 'drop'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 30 destination port '22'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 30 log 'enable'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 30 protocol 'tcp'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 30 recent count '15'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 30 recent time '60'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 30 state new 'enable'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 31 action 'accept'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 31 destination port '22'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 31 log 'enable'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 31 protocol 'tcp'
set firewall ipv6-name EXTERNAL-LOCAL-v6 rule 31 state new 'enable'
set firewall ipv6-receive-redirects 'disable'
set firewall ipv6-src-route 'disable'
set firewall ip-src-route 'disable'
set firewall log-martians 'enable'
set firewall name EXTERNAL-IN default-action 'drop'
set firewall name EXTERNAL-IN enable-default-log
set firewall name EXTERNAL-IN rule 10 action 'accept'
set firewall name EXTERNAL-IN rule 10 log 'enable'
set firewall name EXTERNAL-IN rule 10 state established 'enable'
set firewall name EXTERNAL-IN rule 10 state related 'enable'
set firewall name EXTERNAL-IN rule 20 action 'accept'
set firewall name EXTERNAL-IN rule 20 destination address 'xxx.xxx.69.8'
set firewall name EXTERNAL-IN rule 20 destination port '80,443'
set firewall name EXTERNAL-IN rule 20 log 'enable'
set firewall name EXTERNAL-IN rule 20 protocol 'tcp_udp'
set firewall name EXTERNAL-IN rule 20 source group network-group 'cf-ipv4'
set firewall name EXTERNAL-IN rule 20 state new 'enable'
set firewall name EXTERNAL-IN rule 25 action 'accept'
set firewall name EXTERNAL-IN rule 25 description 'bind'
set firewall name EXTERNAL-IN rule 25 destination address 'xxx.xxx.69.2'
set firewall name EXTERNAL-IN rule 25 destination port '5053'
set firewall name EXTERNAL-IN rule 25 log 'enable'
set firewall name EXTERNAL-IN rule 25 protocol 'tcp_udp'
set firewall name EXTERNAL-IN rule 25 state new 'enable'
set firewall name EXTERNAL-LOCAL default-action 'drop'
set firewall name EXTERNAL-LOCAL enable-default-log
set firewall name EXTERNAL-LOCAL rule 10 action 'accept'
set firewall name EXTERNAL-LOCAL rule 10 log 'enable'
set firewall name EXTERNAL-LOCAL rule 10 state established 'enable'
set firewall name EXTERNAL-LOCAL rule 10 state related 'enable'
set firewall name EXTERNAL-LOCAL rule 20 action 'accept'
set firewall name EXTERNAL-LOCAL rule 20 icmp type-name 'echo-request'
set firewall name EXTERNAL-LOCAL rule 20 log 'enable'
set firewall name EXTERNAL-LOCAL rule 20 protocol 'icmp'
set firewall name EXTERNAL-LOCAL rule 20 state new 'enable'
set firewall name EXTERNAL-LOCAL rule 30 action 'drop'
set firewall name EXTERNAL-LOCAL rule 30 destination port '22'
set firewall name EXTERNAL-LOCAL rule 30 log 'enable'
set firewall name EXTERNAL-LOCAL rule 30 protocol 'tcp'
set firewall name EXTERNAL-LOCAL rule 30 recent count '15'
set firewall name EXTERNAL-LOCAL rule 30 recent time '60'
set firewall name EXTERNAL-LOCAL rule 30 state new 'enable'
set firewall name EXTERNAL-LOCAL rule 31 action 'accept'
set firewall name EXTERNAL-LOCAL rule 31 destination port '22'
set firewall name EXTERNAL-LOCAL rule 31 log 'enable'
set firewall name EXTERNAL-LOCAL rule 31 protocol 'tcp'
set firewall name EXTERNAL-LOCAL rule 31 state new 'enable'
set firewall options interface eth0
set firewall options interface eth3
set firewall options interface tun0 adjust-mss '1436'
set firewall receive-redirects 'disable'
set firewall send-redirects 'enable'
set firewall source-validation 'disable'
set firewall syn-cookies 'enable'
set firewall twa-hazards-protection 'disable'
set interfaces bonding bond0 address 'dhcp'
set interfaces bonding bond0 description 'M1-WAN'
set interfaces bonding bond0 firewall in ipv6-name 'EXTERNAL-IN-v6'
set interfaces bonding bond0 firewall in name 'EXTERNAL-IN'
set interfaces bonding bond0 firewall local ipv6-name 'EXTERNAL-LOCAL-v6'
set interfaces bonding bond0 firewall local name 'EXTERNAL-LOCAL'
set interfaces bonding bond0 ip source-validation 'loose'
set interfaces bonding bond0 lacp-rate 'fast'
set interfaces bonding bond0 member interface 'eth0'
set interfaces bonding bond0 member interface 'eth1'
set interfaces bonding bond0 mode '802.3ad'
set interfaces ethernet eth0 description 'EXTERNAL1'
set interfaces ethernet eth0 duplex 'auto'
set interfaces ethernet eth0 firewall in ipv6-name 'EXTERNAL-IN-v6'
set interfaces ethernet eth0 firewall in name 'EXTERNAL-IN'
set interfaces ethernet eth0 firewall local ipv6-name 'EXTERNAL-LOCAL-v6'
set interfaces ethernet eth0 firewall local name 'EXTERNAL-LOCAL'
set interfaces ethernet eth0 hw-id 'xx:xx:xx:xx:xx:de'
set interfaces ethernet eth0 speed 'auto'
set interfaces ethernet eth1 description 'EXTERNAL2'
set interfaces ethernet eth1 duplex 'auto'
set interfaces ethernet eth1 firewall in ipv6-name 'EXTERNAL-IN-v6'
set interfaces ethernet eth1 firewall in name 'EXTERNAL-IN'
set interfaces ethernet eth1 firewall local ipv6-name 'EXTERNAL-LOCAL-v6'
set interfaces ethernet eth1 firewall local name 'EXTERNAL-LOCAL'
set interfaces ethernet eth1 hw-id 'xx:xx:xx:xx:xx:df'
set interfaces ethernet eth1 speed 'auto'
set interfaces ethernet eth2 address 'xxx.xxx.69.1/24'
set interfaces ethernet eth2 description 'INTERNAL1'
set interfaces ethernet eth2 duplex 'auto'
set interfaces ethernet eth2 hw-id 'xx:xx:xx:xx:xx:e0'
set interfaces ethernet eth2 speed 'auto'
set interfaces ethernet eth3 address 'xxx.xxx.70.1/24'
set interfaces ethernet eth3 description 'INTERNAL2'
set interfaces ethernet eth3 duplex 'auto'
set interfaces ethernet eth3 hw-id 'xx:xx:xx:xx:xx:e1'
set interfaces ethernet eth3 policy route 'magic-wan'
set interfaces ethernet eth3 speed 'auto'
set interfaces loopback lo
set interfaces tunnel tun0 address 'xxx.xxx.72.20/31'
set interfaces tunnel tun0 description 'Magic-WAN'
set interfaces tunnel tun0 encapsulation 'gre'
set interfaces tunnel tun0 ip source-validation 'loose'
set interfaces tunnel tun0 mtu '1476'
set interfaces tunnel tun0 remote 'xxx.xxx.66.5'
set interfaces tunnel tun0 source-address 'xxx.xxx.189.102'
set nat destination rule 10 description 'servarr'
set nat destination rule 10 destination port '80,443'
set nat destination rule 10 inbound-interface 'bond0'
set nat destination rule 10 protocol 'tcp_udp'
set nat destination rule 10 translation address 'xxx.xxx.69.8'
set nat source rule 100 outbound-interface 'bond0'
set nat source rule 100 source address 'xxx.xxx.0.0/16'
set nat source rule 100 translation address 'masquerade'
set policy route magic-wan rule 100 set table '100'
set protocols static table 100 route xxx.xxx.0.0/0 next-hop xxx.xxx.72.21
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.69.0/24 default-router 'xxx.xxx.69.1'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.69.0/24 domain-name xxxxxx
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.69.0/24 lease '300'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.69.0/24 name-server 'xxx.xxx.69.1'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.69.0/24 range 0 start 'xxx.xxx.69.2'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.69.0/24 range 0 stop 'xxx.xxx.69.254'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.69.0/24 static-mapping xxxxxx ip-address 'xxx.xxx.69.5'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.69.0/24 static-mapping xxxxxx mac-address 'xx:xx:xx:xx:xx:b6'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.69.0/24 static-mapping xxxxxx ip-address 'xxx.xxx.69.6'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.69.0/24 static-mapping xxxxxx mac-address 'xx:xx:xx:xx:xx:33'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.69.0/24 static-mapping xxxxxx ip-address 'xxx.xxx.69.7'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.69.0/24 static-mapping xxxxxx mac-address 'xx:xx:xx:xx:xx:64'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.69.0/24 static-mapping xxxxxx ip-address 'xxx.xxx.69.8'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.69.0/24 static-mapping xxxxxx mac-address 'xx:xx:xx:xx:xx:07'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.69.0/24 static-mapping xxxxxx ip-address 'xxx.xxx.69.4'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.69.0/24 static-mapping xxxxxx mac-address 'xx:xx:xx:xx:xx:28'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.69.0/24 static-mapping xxxxxx ip-address 'xxx.xxx.69.2'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.69.0/24 static-mapping xxxxxx mac-address 'xx:xx:xx:xx:xx:d8'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.70.0/24 default-router 'xxx.xxx.70.1'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.70.0/24 domain-name xxxxxx
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.70.0/24 lease '300'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.70.0/24 name-server 'xxx.xxx.70.1'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.70.0/24 range 0 start 'xxx.xxx.70.2'
set service dhcp-server shared-network-name xxxxxx subnet xxx.xxx.70.0/24 range 0 stop 'xxx.xxx.70.254'
set service dns forwarding allow-from 'xxx.xxx.0.0/16'
set service dns forwarding cache-size '0'
set service dns forwarding listen-address 'xxx.xxx.69.1'
set service dns forwarding listen-address 'xxx.xxx.70.1'
set service dns forwarding name-server 'xxx.xxx.69.7'
set service ssh port '22'
set system config-management commit-revisions '100'
set system conntrack modules ftp
set system conntrack modules h323
set system conntrack modules nfs
set system conntrack modules pptp
set system conntrack modules sip
set system conntrack modules sqlnet
set system conntrack modules tftp
set system console device ttyS0 speed '115200'
set system host-name xxxxxx
set system login user xxxxxx authentication encrypted-password xxxxxx
set system login user xxxxxx authentication public-keys xxxx@xxx.xxx key xxxxxx
set system login user xxxxxx authentication public-keys xxxx@xxx.xxx type 'ecdsa-sha2-nistp256'
set system name-server 'xxx.xxx.69.1'
set system ntp server xxxxx.tld
set system ntp server xxxxx.tld
set system ntp server xxxxx.tld
set system ntp server xxxxx.tld
set system static-host-mapping host-name xxxxxx inet 'xxx.xxx.69.5'
set system static-host-mapping host-name xxxxxx inet 'xxx.xxx.69.6'
set system static-host-mapping host-name xxxxxx inet 'xxx.xxx.69.7'
set system static-host-mapping host-name xxxxxx inet 'xxx.xxx.69.8'
set system static-host-mapping host-name xxxxxx inet 'xxx.xxx.69.4'
set system static-host-mapping host-name xxxxxx inet 'xxx.xxx.69.2'
set system sysctl custom net.ipv4.conf.all.accept_local value '1'
set system syslog global facility all level 'debug'
set system syslog global facility protocols level 'debug'

Nikolay · March 19, 2022, 2:52am

There is no GRE protocol listed in the firewall rules.
Try adding something like this:

set firewall name EXTERNAL-LOCAL rule 40 action 'accept'
set firewall name EXTERNAL-LOCAL rule 40 protocol gre

harguna · March 19, 2022, 3:08am

Ah, d’oh. You’re right. Also question, do you have to specify protocols in a firewall rule? Or will it implicit allow any protocol?

Nikolay · March 19, 2022, 3:21am

Of course a stricter rule on the firewall is better.
It depends on your goals and needs
If you’re asking specifically about this GRE rule, that’s a protocol in itself. But you can specify source/destination IPs (the ends of GRE tunnel) for security

harguna · March 19, 2022, 4:06am

Ah, actually, once removed the sysctl config, the GRE tunnel healthchecks start to fail again, so it’s not a firewall thing. The tunnel healthcheck is an icmp echo-reply not echo-request.

Nikolay · March 19, 2022, 4:35am

Do you have it working now?
Do you mean to change this rule to “echo-reply”?
set firewall name EXTERNAL-LOCAL rule 20 icmp type-name 'echo-request'

harguna · March 19, 2022, 4:38am

I don’t. I removed the sysctl rule and immediately the healthchecks starts to fail.

Would it work?

Do you mean to change this rule to “echo-reply”?
set firewall name EXTERNAL-LOCAL rule 20 icmp type-name 'echo-request'

Yeah just tried, it doesn’t do anything.

Nikolay · March 19, 2022, 4:51am

OK
Did I get it right, you are running Ping checks from this VyOS router?
Let’s look at the routing table, the interfaces and the test command itself:

sh ip route
sh interfaces
ping {dest_IP}

harguna · March 19, 2022, 4:56am

Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

S>* 0.0.0.0/0 [210/0] via x.x.184.1, bond0, weight 1, 00:00:37
C>* x.x.69.0/24 is directly connected, eth2, 00:54:16
C>* x.x.72.20/31 is directly connected, tun0, 00:54:17
C>* x.x.184.0/21 is directly connected, bond0, 00:54:15

Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface        IP Address                        S/L  Description
---------        ----------                        ---  -----------
bond0            x.x.189.102/21                u/u  M1-WAN
eth0             -                                 u/u  EXTERNAL1
eth1             -                                 u/u  EXTERNAL2
eth2             x.x.69.1/24                     u/u  INTERNAL1
eth3             x.x.70.1/24                     u/D  INTERNAL2
lo               127.0.0.1/8                       u/u
                 ::1/128
tun0             x.x.72.20/31                    u/u  Magic-WAN

04:54:27.009780 IP x.x.66.5 > x.x.189.118.static.m1net.com.sg: GREv0, length 88: IP x.x.189.118.static.m1net.com.sg > x.x.253.86: ICMP echo reply, id 58573, seq 0, length 64

 ping x.x.66.5
PING x.x.66.5 (162.159.66.5) 56(84) bytes of data.
64 bytes from x.x.66.5: icmp_seq=1 ttl=56 time=2.34 ms
64 bytes from x.x.66.5: icmp_seq=2 ttl=56 time=2.28 ms
64 bytes from x.x.66.5: icmp_seq=3 ttl=56 time=2.39 ms

harguna · March 19, 2022, 4:58am

I think the source is my own router to itself?

Nikolay · March 19, 2022, 5:08am

Maybe I didn’t understand your problem correctly at all.
Right now your ping (toward 162.159.66.5) is going according to the default route from the bond0 interface
Packets will go through tun0 only towards the x.x.72.20/31 network (say xxx.xxx.72.21)

harguna · March 19, 2022, 5:27am

Yep, that’s right, the tunnel itself is working fine. Just not the health-check, in this case, from Cloudlfare on my tunnel. Somehow only that sysctl command will work.

Nikolay · March 19, 2022, 5:59am

Thanks, I understand, the health check is not done through the GRE tunnel.
Then you need to add rules like this:

set firewall name EXTERNAL-LOCAL rule 50 action ‘accept’
set firewall name EXTERNAL-LOCAL rule 50 icmp type-name 'echo-reply
set firewall name EXTERNAL-LOCAL rule 50 log ‘enable’
set firewall name EXTERNAL-LOCAL rule 50 protocol ‘icmp’

Attention, you do NOT need to do this line:

set firewall name EXTERNAL-LOCAL rule 50 state new ‘enable’

harguna · March 19, 2022, 6:06am

Yeah I already tried that, doesn’t work. Failing from all the data centers. A colleague of mine said he had to enable the sysctl rule in the kernel, he didn’t have any firewall rules on. Still can’t wrap my head around why. The echo reply is sent via GRE.

Sounds like a common GRE thing? Gentoo Forums :: View topic - question for kernel experts

Nikolay · March 19, 2022, 6:23am

This is very weird. So the requests are sent through bond but the answers come through GRE?
Let’s try to capture traffic with an healthcheck’ IP address:
tcpdump host 162.159.66.5

harguna · March 19, 2022, 6:34am

06:33:27.406949 IP (tos 0x0, ttl 51, id 12364, offset 0, flags [DF], proto GRE (47), length 108)
    xx.xx.66.5 > 102.189.xx.xx.static.m1net.com.sg: GREv0, Flags [none], length 88
        IP (tos 0x0, ttl 64, id 0, offset 0, flags [none], proto ICMP (1), length 84)
    102.189.xx.xx.static.m1net.com.sg > xx.xx.77.184: ICMP echo reply, id 37477, seq 0, length 64