Frr loses routing info after 5-12k l2tp subs connected

Im thinking could it be that you for whatever reason are running out of conntrack table space?

One way if you dont use the firewall settings is to disable conntrack (trying to figure out how to do that in VyOS 1.4) the other is to tweak the timers to something like so (dont just copy the values, depends on how much RAM you got etc but you get a hint of which settings to dig into further):

system {
    conntrack {
        expect-table-size 10485760
        hash-size 10485760
        table-size 10485760
        timeout {
            icmp 10
            other 600
            tcp {
                close 10
                close-wait 30
                established 600
                fin-wait 30
                last-ack 30
                syn-recv 30
                syn-sent 30
                time-wait 30
            }
            udp {
                other 600
                stream 600
            }
        }
    }
    option {
        performance throughput
        reboot-on-panic
        startup-beep
    }
}

Ref: Bgpd 10Gbps nf_conntrack: table full, dropping packet - #4 by sidnei