Need L2TP to listen on secondary WAN IP

Hello. I have VyOS as main router, it serves both site-to-site and “road warriors” L2TP VPNs.
However, due to need of creating one site-to-site that has dynamic ip on the other end of the internet, I have to move L2TP users on secondary IP (as IPSec implementation in VyOS will not pick up proper PSK when secondary selector is %any%). So I did:

set interface ethernet eth0 vif 8 address a.b.c.e/27  // primary one is a.b.c.d
set vpn l2tp remote-access outside-address a.b.c.e

but none of clients can reach the VPN - VyOS is not responding on secondary address. I’ve tried regular linux way with rt_tables, but without luck. Is it even possible, if so - how?