VyOS HTTPS API becomes unresponsive during concurrent config-file load operations

navneet51.barwal · February 4, 2026, 4:21pm

Description:

Concurrent /config-file load requests via HTTPS API cause the VyOS HTTP API to become unresponsive.

Environment

VyOS version: 2026.02.03-0027-rolling
Release train: current
Architecture: x86_64
Platform: KVM guest (QEMU, Q35)
Access method: HTTPS API (vyos-http-api-server)
API transport: HTTPS (localhost, port 443)

Version Details

vyos@vyos:~$ show version
Version:          VyOS 2026.02.03-0027-rolling
Release train:    current
Release flavor:   generic

Built by:         autobuild@vyos.net
Built on:         Tue 03 Feb 2026 00:27 UTC
Build UUID:       3e264cbe-8a06-43da-a09c-1abef9dcf34e
Build commit ID:  2c2e8a395073ba

Architecture:     x86_64
Boot via:         installed image
System type:      KVM guest
Secure Boot:      n/a (BIOS)

Hardware vendor:  QEMU
Hardware model:   Standard PC (Q35 + ICH9, 2009)
Hardware S/N:     
Hardware UUID:    1fb150d8-2b91-4051-a1f3-b9f56d2497a0

Copyright:        VyOS maintainers and contributors

Initial Configuration

Baseline system configuration before reproducing the issue:

interfaces {
    ethernet eth0 {
        address dhcp
        hw-id 52:54:00:bd:ab:83
        offload {
            gro
            gso
            sg
            tso
        }
    }
    loopback lo {}
}

service {
    ntp {
        allow-client {
            address 127.0.0.0/8
            address 169.254.0.0/16
            address 10.0.0.0/8
            address 172.16.0.0/12
            address 192.168.0.0/16
            address ::1/128
            address fe80::/10
            address fc00::/7
        }
        server time1.vyos.net {}
        server time2.vyos.net {}
        server time3.vyos.net {}
    }
    ssh {
        port 22
    }
}

system {
    host-name vyos
    login {
        user vyos {
            authentication {
                encrypted-password <redacted>
            }
        }
    }
}

HTTPS API Setup Verification

HTTP API Process Running

ps -ef | grep http

root 4895 1 /usr/share/vyos-http-api-tools/bin/python3 \
/usr/libexec/vyos/services/vyos-http-api-server

API Health Check

curl -k https://127.0.0.1/info

{
  "success": true,
  "data": {
    "banner": "Welcome to VyOS",
    "hostname": "vyos",
    "version": "2026.02.03-0027-rolling"
  },
  "error": null
}

Expected Behavior

The HTTPS API should remain responsive even when multiple configuration load operations are triggered.
Concurrent /config-file load requests should be safely serialized.
Other API endpoints such as /info and /retrieve should continue responding normally.

Observed Behavior

When two /config-file load operations are triggered concurrently, the HTTPS API:
- Stops responding to requests.
- /info endpoint hangs indefinitely.
The vyos-http-api-server process remains running but becomes unresponsive.
No clear error is returned to the client.
Manual interruption (Ctrl+C) is required on the client side.

Steps to Reproduce

1. Load configuration file A

curl -k https://127.0.0.1/config-file \
  --form data='{"op": "load", "file": "/home/vyos/config-pki-1.boot"}' \
  --form key='MY-HTTPS-API-PLAINTEXT-KEY'

Result: Success
Configuration updated as expected.

2. Load configuration file B

curl -k https://127.0.0.1/config-file \
  --form data='{"op": "load", "file": "/home/vyos/config-pki.boot"}' \
  --form key='MY-HTTPS-API-PLAINTEXT-KEY'

Result: Success
Configuration reverts correctly.

3. Trigger concurrent loads (bug trigger)

curl -k https://127.0.0.1/config-file \
  --form data='{"op": "load", "file": "/home/vyos/config-pki-1.boot"}' \
  --form key='MY-HTTPS-API-PLAINTEXT-KEY' &

curl -k https://127.0.0.1/config-file \
  --form data='{"op": "load", "file": "/home/vyos/config-pki.boot"}' \
  --form key='MY-HTTPS-API-PLAINTEXT-KEY'

4. Observe API hang

curl -k https://127.0.0.1/info

Result: No response (hangs indefinitely)

Process State During Failure

ps -ef | grep http

		vyos@vyos:~$ ps -ef | grep http
		root        4895       1  0 12:57 ?        00:00:04 /usr/share/vyos-http-api-tools/bin/python3 /usr/libexec/vyos/services/vyos-http-api-server
		vyos        6493    3589  0 13:12 pts/0    00:00:00 curl -k https://127.0.0.1/config-file --form data={"op": "load", "file": "/home/vyos/config-pki-1.boot"} --form key=MY-HTTPS-API-PLAINTEXT-KEY
vyos        6494    3589  0 13:12 pts/0    00:00:00 curl -k https://127.0.0.1/config-file --form data={"op": "load", "file": "/home/vyos/config-pki.boot"} --form key=MY-HTTPS-API-PLAINTEXT-KEY

The HTTP API server process remains alive but is unresponsive.

Logs During Issue

vyos@vyos:~$ monitor log
Feb 04 13:11:28 vyos-hostsd[748]: Writing /run/pdns-recursor/recursor.vyos-hostsd.conf.lua
Feb 04 13:11:28 vyos-hostsd[748]: Writing /run/pdns-recursor/recursor.forward-zones.conf
Feb 04 13:11:28 vyos-hostsd[748]: pdns_recursor not running, not sending "reload-lua-config"
Feb 04 13:11:28 vyos-hostsd[748]: pdns_recursor not running, not sending "reload-zones"
Feb 04 13:11:28 vyos-hostsd[748]: Success
Feb 04 13:11:28 vyos-hostsd[748]: Saving state to /run/vyos-hostsd/vyos-hostsd.state
Feb 04 13:11:28 vyos-hostsd[748]: Sent response: {'data': {'message': 'Applied 2 changes'}}
Feb 04 13:11:28 dhclient-script-vyos[6351]: No changes to apply via vyos-hostsd-client
Feb 04 13:11:28 dhclient[6245]: bound to 192.168.122.103 -- renewal in 1532 seconds.
Feb 04 13:11:28 dhclient[6245]: bound to 192.168.122.103 -- renewal in 1532 seconds.
Feb 04 13:12:56 vyos-http-api[4895]: processing form data
Feb 04 13:12:56 vyos-http-api[4895]: processing form data

Additional Notes

The issue appears to be a race condition or missing locking mechanism in the HTTPS API when handling concurrent /config-file load operations.
The system itself does not crash, but the HTTPS API enters a non-recoverable hung state until restarted.
This behavior makes automation using the HTTPS API unreliable under parallel or concurrent workloads.

Issue Reason and Suggested Fix

Issue Reason

The /config-file API handler in routers.py is implemented as an asynchronous FastAPI endpoint (async def), but it uses a blocking threading.Lock (lock.acquire()) to serialize configuration operations.

Under concurrent /config-file load requests, this can lead to an event-loop deadlock:

One request acquires the blocking lock and awaits work in a threadpool.
A subsequent request attempts to acquire the same lock.
Because lock.acquire() blocks the event loop, the first coroutine cannot resume and release the lock.
This results in the HTTPS API becoming unresponsive while the process itself remains running.

Suggested Fix

It is recommended to replace the blocking threading.Lock with an asynchronous lock (such as asyncio.Lock) and serialize configuration operations using non-blocking async synchronization.

This approach avoids blocking the event loop, prevents deadlock under concurrent requests, and allows the HTTPS API to remain responsive during parallel configuration operations.
config-pki.boot.txt (8.3 KB)
config-pki-1.boot.txt (8.4 KB)

jestabro · February 4, 2026, 5:50pm

@navneet51.barwal Thank you for the detailed report. I’ve opened ⚓ T8235 VyOS HTTPS API becomes unresponsive during concurrent config-file load operations to resolve.

navneet51.barwal · February 5, 2026, 7:28am

@jestabro , Thank you for acknowledging . We do have a fix for this , and we have validated it as well with image build. Please let me know the process briefly to make my submissions.

jestabro · February 5, 2026, 2:36pm

Thanks, @navneet51.barwal
The submission process is here:

Note also, that the rolling release mentioned in your report preceded (by a few hours) the development merged at the link below, which likely pertain to a global solution — note, however, that the mechanism introduced there has not yet been extended to the config-file endpoint, though that is a straightforward addition I believe:

github.com/vyos/vyos-1x

http-api: T7090: Implement background configure operations for REST API

current ← alexandr-san4ez:T7090-current

opened 03:47PM - 23 Jan 26 UTC

alexandr-san4ez

+463 -45

## Change summary Large config commits (`service config-sync`) can block the RE…ST API request path and sometimes must be deferred (e.g., when changing `service https`). This commit introduces an in-memory background operation manager that queues (FIFO) full configure operations (commands + commit/commit-confirm) as single jobs, tracks status/result, and exposes active operations via `/retrieve/background-operations`. ## Types of changes - [ ] Bug fix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Code style update (formatting, renaming) - [ ] Refactoring (no functional changes) - [ ] Migration from an old Vyatta component to vyos-1x, please link to related PR inside obsoleted component - [ ] Other (please describe): ## Related Task(s) * https://vyos.dev/T7090 ## How to test / Smoketest result ### Manual test 1. Configure two routers and sync of `firewall` section: ``` # Router 1 set service config-sync mode 'load' set service config-sync secondary address '172.168.99.3' set service config-sync secondary key 'NOT_SECRET_KEY' set service config-sync secondary port '443' set service config-sync section firewall # Router 2 set service https api rest set service https api keys id KEY key 'NOT_SECRET_KEY' set service https listen-address '0.0.0.0' ``` 2. Using `add_fw_config_primary.sh` script from [this task](https://vyos.dev/T7090) generate firewall rules: ``` vyos@1e1e72e3b3d7# ./add_fw_config_primary.sh How many lines do you want to add to the config? \nEnter a number between 10 and 20000: 10 Valid input: 10 Adding firewall rules from: 2 to 4 Starting with set commands at Fri Jan 23 14:13:54 UTC 2026 Do you want to commit the changes? (yes/no): yes Starting commit at Fri Jan 23 14:13:57 UTC 2026 INFO:vyos_config_sync:Config synchronization: Mode=load, Secondary=172.168.99.3 Commit completed at Fri Jan 23 14:14:00 UTC 2026 [edit] ``` 4. Verify that firewall rules applied on **the second** router: ``` vyos@e89322c6a5c6# run sh conf comm | match firewall set firewall ipv4 name client_to_dmz rule 2 action 'accept' set firewall ipv4 name client_to_dmz rule 2 destination port '82' set firewall ipv4 name client_to_dmz rule 2 protocol 'tcp' set firewall ipv4 name client_to_dmz rule 3 action 'accept' set firewall ipv4 name client_to_dmz rule 3 destination port '83' set firewall ipv4 name client_to_dmz rule 3 protocol 'tcp' set firewall ipv4 name client_to_dmz rule 4 action 'accept' set firewall ipv4 name client_to_dmz rule 4 destination port '84' set firewall ipv4 name client_to_dmz rule 4 protocol 'tcp' ``` 5. Get list of background operations by REST API: ``` vyos@1e1e72e3b3d7:~$ curl -k -X 'POST' \ 'https://172.168.99.3:443/retrieve/background-operations' \ -H 'accept: application/json' \ -H 'Content-Type: application/json' \ -d '{ "key": "NOT_SECRET_KEY" }' | jq . {"success": true, "data": {"operations": [ { "op_id": "2b1773ba-aa35-4c54-af09-e98c923e2571", "created_at": 1769156996, "started_at": 1769157004, "finished_at": 1769157007, "status": "succeeded", "result": "", "error": null } ]}, "error": null} ``` ### Smoketest ``` vyos@vyos:~$ /usr/libexec/vyos/tests/smoke/cli/test_service_https.py ... test_api_configure_background (__main__.TestHTTPSService.test_api_configure_background) ... ok test_api_configure_background_ops_over_max (__main__.TestHTTPSService.test_api_configure_background_ops_over_max) ... ok test_api_configure_section_background (__main__.TestHTTPSService.test_api_configure_section_background) ... ok ... ``` ## Checklist: - [x] I have read the [**CONTRIBUTING**](https://github.com/vyos/vyos-1x/blob/current/CONTRIBUTING.md) document - [x] I have linked this PR to one or more Phabricator Task(s) - [x] I have run the components [**SMOKETESTS**](https://github.com/vyos/vyos-1x/tree/current/smoketest/scripts/cli) if applicable - [x] My commit headlines contain a valid Task id - [ ] My change requires a change to the documentation - [ ] I have updated the documentation accordingly

navneet51.barwal · February 9, 2026, 1:12pm

Hi @jestabro , I have opened a PR . http-api: T8235:Use asyncio lock to prevent deadlock in config-file handler by NavneetBarwal-RA · Pull Request #4976 · vyos/vyos-1x · GitHub for solution. Though my PR title have a slight mistake of a missed space after “:”. And i am unable to edit it. Please suggest how to resolve this to pass the check.

jestabro · February 9, 2026, 3:06pm

Thanks @navneet51.barwal Regarding the typo in PR title, I’ve gone ahead and added the space. Thanks again for the report and the PR.

echowings · February 10, 2026, 5:31am

I met the same issue. request api with concurrent will make the api dead and need to reboot to fix it.