Container and IPv6 NDP

When I create a container with an IPv6 address it behaves really weirdly.

The address is not accessible from the outside.
However, as soon as I ping the address from the VyOS it responds and then becomes also available from the outside.
When I stop pinging the container, it stays available for a couple of seconds and then becomes unresponsive again.

show ipv6 neighbors show it as FAILED and changes to REACHABLE while I ping the container.
When I stop the ping it goes back to FAILED after some seconds.

Does anyone have an idea what the issue could be?

Container config:

 name nginx {
     image nginx:stable
     network homelab {
         address 172.18.10.10
         address fd9e:63ac:6dcd::10
     }
     restart always
     volume data {
         destination /etc/nginx/conf.d
         source /config/docker/nginx/data
     }
 }

 network homelab {
     description "Docker Network"
     prefix 172.18.10.0/24
     prefix fd9e:63ac:6dcd::/48
 }

(fd9e:63ac:6dcd::/48 placeholder for public IPv6 prefix)

Docker has always been funny for me with IPv6 NDP, even when I had a vultr instance. On vultr I landed up installing the proxy ndp daemon (ndppd) to keep the neighbor table decently updated.