I’m experimenting with Vyos before swapping over from a Unifi USG and in doing so I am wanting to use terraform to codify changes. By default terraform will perform several things in parallel and as I have been adding more code blocks in the terraform provider, i’ve started to notice frequently I am getting 502 Bad Gateway responses from the http api. To try to reproduce this outside of terraform I am using a node program called “autocannon” to allow making concurrent requests.
When running a single connection it all seems to be fine:
$ npx autocannon --duration 10 --connections 1 --renderStatusCodes --method POST --form '{"data": {"type":"text", "value": "{\"op\": \"showConfig\", \"path\": []}"}, "key": {"type":"text","value":"my-secret-key"}}' 'https://vyos:8443/retrieve'
Running 10s test @ https://vyos:8443/retrieve
1 connections
┌─────────┬────────┬────────┬────────┬────────┬───────────┬──────────┬────────┐
│ Stat │ 2.5% │ 50% │ 97.5% │ 99% │ Avg │ Stdev │ Max │
├─────────┼────────┼────────┼────────┼────────┼───────────┼──────────┼────────┤
│ Latency │ 545 ms │ 564 ms │ 603 ms │ 603 ms │ 563.48 ms │ 12.98 ms │ 603 ms │
└─────────┴────────┴────────┴────────┴────────┴───────────┴──────────┴────────┘
┌───────────┬──────┬──────┬───────┬───────┬─────────┬─────────┬──────┐
│ Stat │ 1% │ 2.5% │ 50% │ 97.5% │ Avg │ Stdev │ Min │
├───────────┼──────┼──────┼───────┼───────┼─────────┼─────────┼──────┤
│ Req/Sec │ 1 │ 1 │ 2 │ 2 │ 1.7 │ 0.46 │ 1 │
├───────────┼──────┼──────┼───────┼───────┼─────────┼─────────┼──────┤
│ Bytes/Sec │ 8 kB │ 8 kB │ 16 kB │ 16 kB │ 13.6 kB │ 3.67 kB │ 8 kB │
└───────────┴──────┴──────┴───────┴───────┴─────────┴─────────┴──────┘
┌──────┬───────┐
│ Code │ Count │
├──────┼───────┤
│ 200 │ 17 │
└──────┴───────┘
Req/Bytes counts sampled once per second.
# of samples: 10
18 requests in 10.03s, 136 kB read
Bumping that up to just 2 concurrent connections I start getting errors.
$ npx autocannon --duration 10 --connections 2 --renderStatusCodes --method POST --form '{"data": {"type":"text", "value": "{\"op\": \"showConfig\", \"path\": []}"}, "key": {"type":"text","value":"my-secret-key"}}' 'https://vyos:8443/retrieve'
Running 10s test @ https://vyos:8443/retrieve
2 connections
┌─────────┬──────┬───────┬───────┬────────┬──────────┬──────────┬────────┐
│ Stat │ 2.5% │ 50% │ 97.5% │ 99% │ Avg │ Stdev │ Max │
├─────────┼──────┼───────┼───────┼────────┼──────────┼──────────┼────────┤
│ Latency │ 4 ms │ 47 ms │ 98 ms │ 104 ms │ 48.08 ms │ 27.71 ms │ 216 ms │
└─────────┴──────┴───────┴───────┴────────┴──────────┴──────────┴────────┘
┌───────────┬────────┬────────┬────────┬────────┬────────┬─────────┬────────┐
│ Stat │ 1% │ 2.5% │ 50% │ 97.5% │ Avg │ Stdev │ Min │
├───────────┼────────┼────────┼────────┼────────┼────────┼─────────┼────────┤
│ Req/Sec │ 333 │ 333 │ 572 │ 638 │ 557.6 │ 79.96 │ 333 │
├───────────┼────────┼────────┼────────┼────────┼────────┼─────────┼────────┤
│ Bytes/Sec │ 109 kB │ 109 kB │ 180 kB │ 200 kB │ 176 kB │ 23.7 kB │ 109 kB │
└───────────┴────────┴────────┴────────┴────────┴────────┴─────────┴────────┘
┌──────┬───────┐
│ Code │ Count │
├──────┼───────┤
│ 200 │ 2 │
├──────┼───────┤
│ 502 │ 5574 │
└──────┴───────┘
Req/Bytes counts sampled once per second.
# of samples: 10
2 2xx responses, 5574 non 2xx responses
6k requests in 10.02s, 1.76 MB read
Sometimes I’ll get a handful of successful ones and other times I’ll get all failures when using this tool.
In the monitor log I also see the http api is crashing/segfaulting:
Feb 12 22:40:15 kernel: vyos-http-api-s[3490]: segfault at f1f47738 ip 00007fccf226dde3 sp 00007fccf1c89d60 error 6 in libvyosconfig.so.0[7fccf21aa000+10f000] likely on CPU 3 (core 3, socket 0)
Feb 12 22:40:15 kernel: Code: 8b 5f 18 48 c7 03 03 00 00 00 48 8b 5f 10 48 83 fb 01 74 4f 49 83 ef 38 4d 3b 7e 08 0f 82 8c 0e 00 00 49 8d 5f 08 48 83 c3 28 <48> c7 43 f8 0b 04 00 00 48 89 03 48 8d 7b e8 48 c7 47 f8 17 08 00
Feb 12 22:40:15 systemd[1]: vyos-http-api.service: Main process exited, code=killed, status=11/SEGV
Previously I was running an image from June last year when I initially was starting to test out Vyos before getting too busy. I tried upgrading to the latest version and still getting the errors.
1: 1.4-rolling-202302130317 (default boot) (running image)
2: 1.4-rolling-202206250934
Hardware wise, I’m running on an inexpensive Intel N5105 from AliExpress running 16GB RAM, which should be more than sufficient for this, so I doubt it’s a hardware/resources issue.
