I temporary fixed by switching switch ssh type from paramiko to libssh and now it is working.
It looks there might be bug for paramiko.
<10.41.170.53> attempting to start connection
<10.41.170.53> using connection plugin ansible.netcommon.network_cli
Found ansible-connection at path /usr/local/bin/ansible-connection
<10.41.170.53> found existing local domain socket, using it!
**<10.41.170.53> invoked shell using ssh_type: libssh**
<10.41.170.53> ssh connection done, setting terminal
<10.41.170.53> loaded terminal plugin for network_os vyos.vyos.vyos
<10.41.170.53> firing event: on_open_shell()
<10.41.170.53> ssh connection has completed successfully
<10.41.170.53> updating play_context for connection
<10.41.170.53>
<10.41.170.53> local domain socket path is /root/.ansible/pc/6b5c2da01c
redirecting (type: action) vyos.vyos.vyos_vlan to vyos.vyos.vyos
redirecting (type: action) vyos.vyos.vyos_vlan to vyos.vyos.vyos
[WARNING]: provider is unnecessary when using network_cli and will be ignored
<10.41.170.53> ANSIBLE_NETWORK_IMPORT_MODULES: disabled
<10.41.170.53> ANSIBLE_NETWORK_IMPORT_MODULES: module execution time may be extended
<10.41.170.53> ESTABLISH LOCAL CONNECTION FOR USER: root