It seems Conntrack is enabled by default on 1.4 RR. I also tested by starting a fully clean image and it seems even then the nat/conntrack kernel modules are loaded (and active).
This seem to be caused by default rules provided by VyOS in nft, which are not configured within our VyOS config.
We temporarily fixed this by flushing all nft rules and unloading the kernel modules, as we do not want conntrack to be enabled. It seems to be a regression, as I only find documentation that conntrack should only be enabled when we configured NAT and/or any stateful rules.