I figured out the key to getting this to work was changing my tunnel local IP address from being the one exposed on the external pfSense firewall to instead the private IPV4 address of the vyos node itself. That allowed vyos to figure out the ipv4 routing off the device for the tunnel.
I also modified pfSense to check “Enable IPv6 over IPv4 tunneling” under System->Advanced->Networking.
Lastly I created a floating rule that was above my traffic shaping rule to pass IPv6 protocol packets (which I believe is how pfSense represents protocol 41 per List of IP protocol numbers - Wikipedia). I also set “Apply the action immediately on match.” To the rule as it’s not clear to me from documentation I read whether the traffic shaper I’m using would eat the packets.
I’m not 100% sure that I have the pfSense configuration correct yet. But I am successfully pinging and tracerouting from vyos over the tunnel now.