Hence using dhcp-relay with option82.
This way through ACL and configuration whatever ends up at SW1-INT5 will always use the same IP.
I fired up the old EdgeRouter 4 to look at how it was handling DNS and SLAAC. It turns out that ER4 is using dnsmasq for both DHCP, RA and DNS. dnsmasq has a nifty configuration option (ra-names) where it is using hostname knowledge from DHCPv4 to create AAAA DNS entries for SLAAC configured IPv6 devices.
In the /etc/dnsmasq.conf file on the router, I find the following IPv6 configuration:
interface=eth1
dhcp-option=option6:dns-server,[fe80::]
dhcp-option=option6:domain-search,home.arpa
enable-ra
dhcp-range=::,constructor:eth1,slaac,ra-names
The dnsmasq documentation says the following about the ra-names option:
ra-names enables a mode which gives DNS names to dual-stack hosts which do SLAAC for IPv6. Dnsmasq uses the host’s IPv4 lease to derive the name, network segment and MAC address and assumes that the host will also have an IPv6 address calculated using the SLAAC algorithm, on the same network segment. The address is pinged, and if a reply is received, an AAAA record is added to the DNS for this IPv6 address. Note that this is only happens for directly-connected networks, (not one doing DHCP via a relay) and it will not work if a host is using privacy extensions. ra-names can be combined with ra-stateless and slaac.
This is a pretty creative and ad-hoc way to solve this problem that I was asking for. It solves it completely without mDNS. dnsmasq is a remarkably featureful piece of software, especially taking into account that it is basically maintained by one single person.
At least I understand why VyOS and EdgeRouter behaves differently now. DHCP, RA and DNS are split into three separate services in VyOS. And I guess each of these services would never attempt to implement the kind of ad-hoc feature that dnsmasq provides. It probably needs to be formalized through some sort of RFC first.
So my next step would perhaps be to make a stab at some sort of mDNS-to-DNS service. Or just make sure that all my devices have mDNS enabled/configured and try to use mDNS consistently across the network. Alternatively, I could perhaps set some static IPv6 ULA addresses on the most important machines and add those in the VyOS DNS.