Here again. I have another issue that has me stumped.
I have a public facing eth0 interface. It has most of two /28 and two /29 networks assigned to it, about 28’ish IPv4 addresses in total.
I have a caddy docker image running in a container, which this config:
set container name REVERSEPROXY description 'Caddy Reverse Proxy'
set container name REVERSEPROXY memory 2048
set container name REVERSEPROXY cpu-quota 2
set container name REVERSEPROXY image 'caddy'
set container name REVERSEPROXY restart always
set container name REVERSEPROXY allow-host-networks
set container name REVERSEPROXY volume config source '/config/reverseproxy/config'
set container name REVERSEPROXY volume config destination '/etc/caddy'
set container name REVERSEPROXY volume config mode ro
set container name REVERSEPROXY volume data source '/config/reverseproxy/data'
set container name REVERSEPROXY volume data destination '/data'
set container name REVERSEPROXY volume data mode rw
set container name REVERSEPROXY volume log source '/var/log/reverseproxy'
set container name REVERSEPROXY volume log destination '/log'
set container name REVERSEPROXY volume log mode rw
Caddy binds to ports 80 and 443 of 7 of these IPs.
This works fine on VyOS instances where the publc interface has only one IP.
On this instance however, so soon as Caddy binds those IPs, they become unreachable. I can ping them from the router itself, but I can’t ping them from the internet. I also don’t get a connection on them on ports 80 or 443, it looks like they don’t exist.
All other IP’s on eth0 that are not bound by Caddy in the container respond fine to a ping, and other services on those IP’s work too.
The weird thing is, when I delete the container, and the caddy bindings are gone, the problem remains.
edit: All our VyOS instances are dual-stack, so I have an IPv6 address for every IPv4 address on this interface, and the corresponding IPv6 adresses are also bound by Caddy. These do not have this issue, I can ping them, and Caddy proxies the requet to the backend if I make a v6 request to a website via cURL.
Considering you are trying to bind on privileged ports, you probably need to add a kernel capability to your container, such as: set container name REVERSEPROXY capability net-bind-service
What I personally do is create a container network, assign the container one or more addresses on that network, and then create a DNAT rule for the public IP to the container IP.
set container name REVERSEPROXY network <networkname> address <address>
Also VyOS doesn’t always nicely destroy/rebuild container network configs. So not surprised that the problem remains even after deleting the container. Normally a reboot fixes it.
Considering you are trying to bind on privileged ports, you probably need to add a kernel capability to your container,
I doubt that, as netstat shows it binds just fine, and the same config also works on other routers.
Problem is that only the bind on the primary IP works, IP’s listed as “global secondary” have the problem (which is only IPv4, the none of the IPv6 addresses are shown as secondary in the interface details).
Since you already have “allow-host-networks” you can try to add this to VyOS and see if the container behaves better regarding IP-addresses?
Probably couldn’t hurt, but I doubt that is the issue, as the adresses are all added on boot, before the container starts. Also, a restart of the container doesn’t fix it, and the adresses are absolutely there at that point.
The weird thing is that handling ICMP is a kernel thing, whether or not something is bound to the IP on a port is not relevant for that.
And I can ping the address from the VyOS console, so from a kernel standpoint is is active. So what would cause the kernel not to respond to the ping anymore?
I’ll try a tcpdump tonight to check if the problem is the ping doesn’t arrive, or the answer isn’t send back.
Just had a chat with a network engineer of my colocation provider.
Chances are it is an ARP cache problem, and the link with Caddy a coincidence as those adresses are the onces recieving hunderds of requests a second, and definitely in the ARP cache of the DC core router. The other addresses are used either infrequently or some not at all, and can be pinged because they don’t have ARP cache entries still pointing to the wrong MAC.
He suggested doing a ping <broadcast IP> -I <source IP> for all addresses on the interface to force an ARP cache update on the router, so I’m going to try that tonight.
As you explained used to force remote hosts to update their ARP-tables.
The receiving part must also accept incoming GARPs so there is that to add to the mix.
Common issue when you have a L2-switch between you and the L3-gateway.
Since the link between the L3-switch and the L2-switch never goes down it wont clear its ARP.
So when you as customer (common ISP-design) change your broadbandrouter it will take the ARP timeout (unless your router sends out GARP) for the L3-switch to pick up the changed ARP.
Back in the days the default for ARP timeout was 4 hours so that was… fun.
Today its recommended to set ARP timeout to lower than MAC timeout.
So normally ARP timeout should be at 240 seconds (4 min) and MAC timeout be at 300 seconds (5 min).
VyOS have garp settings available when configuring VRRP:
vyos@vyos# set high-availability vrrp group FOO garp
Possible completions:
interval Interval between Gratuitous ARP (default: 0)
master-delay Delay for second set of gratuitous ARPs after transition to
master (default: 5)
master-refresh Minimum time interval for refreshing gratuitous ARPs while
beeing master (default: 5)
master-refresh-repeat
Number of gratuitous ARP messages to send at a time while beeing
master (default: 1)
master-repeat Number of gratuitous ARP messages to send at a time after
transition to master (default: 5)
There is also this case to add this capability for IPv6:
This is a virtualized environment, so even the switch ports will never go down.
The DC network staff were willing to flush the ARP cache, but in order to keep downtime to a minimum that had be done at exactly the same time we would switch over, which was difficult to plan.
So in the end we’ve opted to clone the MAC from the old firewall onto the VyOS interface, and that solved this problem.