Adjust commit timeout with Ansible?

I’m using VyOS on a quite slow machine (CPU is Intel J1900), and I’m getting frequent commit timeouts when running an Ansible playbook to setup the machine. Adjusting the Ansible task timeout makes no difference.

If I run the same set of commands manually, it’s all fine.

Typical error message:

failed: [liberal] (item={'name': 'wan-in', 'default-action': 'drop', 'description': 'Chain WAN', 'rules': {100: ["action 'drop'", "recent count '4'", "recent time 'minute'", "state new 'enable'", "destination port '22'", "protocol 'tcp'"], 101: ["action 'accept'", "destination port '22'", "protocol 'tcp'", "state new 'enable'"]}}) => {"ansible_loop_var": "item", "changed": false, "item": {"default-action": "drop", "description": "Chain WAN", "name": "wan-in", "rules": {"100": ["action 'drop'", "recent count '4'", "recent time 'minute'", "state new 'enable'", "destination port '22'", "protocol 'tcp'"], "101": ["action 'accept'", "destination port '22'", "protocol 'tcp'", "state new 'enable'"]}}, "msg": "commit failed: timeout value 30 seconds reached while trying to send command: b'commit comment \"configured by vyos_config\"'"}
 [ERROR]: Traceback (most recent call last):   File "/nix/store/hxhb2al3zgvzlgz0s1vpxzfqhvh9yzhj-
python3.10-paramiko-2.11.0/lib/python3.10/site-packages/paramiko/channel.py", line 699, in recv     out = self.in_buffer.read(nbytes,
self.timeout)   File "/nix/store/hxhb2al3zgvzlgz0s1vpxzfqhvh9yzhj-python3.10-paramiko-2.11.0/lib/python3.10/site-
packages/paramiko/buffered_pipe.py", line 164, in read     raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout  During handling of the
above exception, another exception occurred:  Traceback (most recent call last):   File
"/home/oscar/.ansible/collections/ansible_collections/ansible/netcommon/plugins/connection/network_cli.py", line 1056, in send
response = self.receive(   File
"/home/oscar/.ansible/collections/ansible_collections/ansible/netcommon/plugins/connection/network_cli.py", line 1005, in receive
response = self.receive_paramiko(   File
"/home/oscar/.ansible/collections/ansible_collections/ansible/netcommon/plugins/connection/network_cli.py", line 813, in receive_paramiko
data = self._ssh_shell.recv(256)   File "/nix/store/hxhb2al3zgvzlgz0s1vpxzfqhvh9yzhj-python3.10-paramiko-2.11.0/lib/python3.10/site-
packages/paramiko/channel.py", line 701, in recv     raise socket.timeout() TimeoutError

My datastructure is quirky but easy to write, used together with a custom filter which should output the following lines based on the payload in the error message above:

set firewall ipv4 name wan-in default-action 'drop'
set firewall ipv4 name wan-in description 'Chain WAN'
set firewall ipv4 name wan-in rule 100 action 'drop'
set firewall ipv4 name wan-in rule 100 recent count '4'
set firewall ipv4 name wan-in rule 100 recent time 'minute'
set firewall ipv4 name wan-in rule 100 state new 'enable'
set firewall ipv4 name wan-in rule 100 destination port '22'
set firewall ipv4 name wan-in rule 100 protocol 'tcp'
set firewall ipv4 name wan-in rule 101 action 'accept'
set firewall ipv4 name wan-in rule 101 destination port '22'
set firewall ipv4 name wan-in rule 101 protocol 'tcp'
set firewall ipv4 name wan-in rule 101 state new 'enable'

This is the task in my playbook:

- name: Setup custom chains
  vyos.vyos.vyos_config:
    lines: "{{ item | vyos_split_chain }}"
  when: vyos_firewall_chains is defined
  loop: "{{ vyos_firewall_chains }}"
  tags: firewall_chains

My VyOS version:

Version:          VyOS 1.5-rolling-202310010025
Release train:    current

Built by:         autobuild@vyos.net
Built on:         Sun 01 Oct 2023 01:49 UTC
Build UUID:       faa9f6a7-91d3-4674-9230-57c4b4ab5d79
Build commit ID:  352cfe5761c92b

Architecture:     x86_64
Boot via:         installed image
System type:      bare metal

Hardware vendor:  To be filled by O.E.M.
Hardware model:   To be filled by O.E.M.
Hardware S/N:     To be filled by O.E.M.
Hardware UUID:

Copyright:        VyOS maintainers and contributors

I thought sending a list of commands (like the vyos_split_chain outputs) should make them go in one commit, but it’s either done in multiple commits, or my computer is just too slow? Timeout is 30 s.

Can I increase the timeout? It doesn’t seem to be an Ansible timeout in this case, as adjusting it in the task makes no difference.

Also - having netfilter style filters in VyOS is awesome, finally! :smiley:

Try disables host validation. We’ve seen this behavior before:

set service ssh disable-host-validation

Then re-run ansible script and check if you get timeout.

Another issue are the commit times in VyOS where just a few hundred static routes can make the commit times skyrocket to several minutes (similar case with longer firewall rulesets).

Reported in: ⚓ T5388 Something is fishy with commit and boot times when more than a few hundred static routes are being used

Disabled host validation, still having the same issue.

It’s enough to change one field to trigger the issue again, in this case I just edited the description for a rule.

If I run the playbook again it looks like the previous run made the changes as expected, though.

# show | commands > /tmp/derp
# wc -l /tmp/derp 
451 derp

Most of the configuration are address/network groups, I haven’t even translated the current ruleset to this configuration. Then it will increase in size…

When you have saved your config whats the output of the console during boot?

That is the final lines before login prompt shows up:

[21.170614] vyos-router[835]: Waiting for NICs to settle down: settled in 0sec..
[26.898939] vyos-router[835]: Mounting VyOS Config...done.
[58.927104] vyos-router[835]: Starting VyOS router: migrate configure.
[59.238917] vyos-config[844]: Configuration success

Welcome to VyOS - vyos tty1

vyos login:

You can also clock it when you do some minor change when in conf-mode (login through console or ssh) and then hit “commit” (which is the actual commit time since the above does other things aswell).

Im thinking that you have enough of config to make it take longer to commit than the API timeout so your Ansible gets sad.