Conntrack ignore

I have two pairs of vyos routers, in seperate DC’s. They were deployed last month.
Before then we just had the edge routers with very simple filters.

We only need stateful firewalls on 3 of 15 VLAN’s that go through the firewalls, so I was wondering if I should have conntrack ignore all traffic except the 3 VLANS that have stateful firewalls.

If I had ignore rules for each subnet (source and destination), am I right that the traffic would not create records in the conntrack table, and it would not be affected if the conntrack table became full?

Any downside except not being able to NAT or use stateful firewall?