OK, here is what I finally came up with to block clear text gre outbound using nftables on VyOS 1.4. This seems to work in my virtual lab. Normal VyOS firewall rules can be used to allow ipsec/gre inbound and block all other gre clear text inbound.
# Allow GRE outbound if attached to ipsec policy and block all other clear text GRE
nft add table ip custom
nft add chain ip custom output '{ type filter hook output priority -10 ; }'
nft add rule ip custom output ip protocol gre ipsec out reqid * counter accept
nft add rule ip custom output ip protocol gre counter drop
I am not sure if matching ipsec reqid with a wildcard is really proper but it seems to work (though it looks weird in nft list output). Another option is to match any reqid that is not 0 (i.e. anything above 0). E.g.
# Allow GRE outbound if attached to ipsec policy and block all other clear text GRE
nft add table ip custom
nft add chain ip custom output '{ type filter hook output priority -10 ; }'
nft add rule ip custom output ip protocol gre ipsec out reqid != 0 counter accept
nft add rule ip custom output ip protocol gre counter drop
This also seems to work in my virtual lab.