VyOS build failing smoketests

Development
1

Hello everyone,

I’m again facing some issues with the smoketests. After fixing an issue with the test-no-interfaces target, I’m left with either broken tests or some broken code for login related stuff. You can check what I get by looking it up in the most recent file uploaded to ⚓ T7215 VRF smoke tests failing, around line 2737, testfile test_system_login.py. Is this a known issue that’s being worked on? Or is this an issue on my side? A quick search didn’t reveal any obvious tickets, so it might be the latter. I’m still in the dark on what actual test targets are used for the nightly builds, as whatever they test must go through just fine. So maybe I’m in the wrong by trying to use test-no-interfaces in the first place. The target frequently breaking due to missing parameter adaptions (such as the fix I pushed) hints toward this target not being in regular use.

What I try to accomplish is building custom ISOs for internal use, based on the state of nightlies (i.e. passing all relevant smoke tests). I’m resorting to building it from scratch, as I had issues with reusing the nightly ISO.

Best regards

2

Do not see any issues with login:

DEBUG - Running Testcase: /usr/libexec/vyos/tests/smoke/cli/test_system_login.py
DEBUG - test_add_linux_system_user (__main__.TestSystemLogin.test_add_linux_system_user) ... ok
DEBUG - test_delete_current_user (__main__.TestSystemLogin.test_delete_current_user) ... ok
DEBUG - test_pam_nologin (__main__.TestSystemLogin.test_pam_nologin) ...
DEBUG - test_radius_kernel_features (__main__.TestSystemLogin.test_radius_kernel_features) ... ok
DEBUG - test_system_login_max_login_session (__main__.TestSystemLogin.test_system_login_max_login_session) ... ok
DEBUG - test_system_login_otp (__main__.TestSystemLogin.test_system_login_otp) ... ok
DEBUG - test_system_login_radius_ipv4 (__main__.TestSystemLogin.test_system_login_radius_ipv4) ... ok
DEBUG - test_system_login_radius_ipv6 (__main__.TestSystemLogin.test_system_login_radius_ipv6) ... ok
DEBUG - test_system_login_tacacs (__main__.TestSystemLogin.test_system_login_tacacs) ... ok
DEBUG - test_system_login_user (__main__.TestSystemLogin.test_system_login_user) ... ok
DEBUG - test_system_login_weak_password_warning (__main__.TestSystemLogin.test_system_login_weak_password_warning) ... ok
DEBUG - test_system_user_ssh_key (__main__.TestSystemLogin.test_system_user_ssh_key) ... ok
DEBUG - 
DEBUG - ----------------------------------------------------------------------
DEBUG - Ran 12 tests in 103.049s

Build system at least does these steps:

sudo make test-no-interfaces-no-vpp | tee smoketest_make_test_no_interfaces_no_vpp.log
sudo make test-vpp | tee smoketest_make_test_vpp.log
sudo make test-interfaces | tee smoketest_make_test_interfaces.log
sudo make testc | tee smoketest_make_testc.log
sudo make testcvpp | tee smoketest_make_testcvpp.log
sudo make testraid | tee smoketest_make_testraid.log
sudo make testtpm | tee smoketest_make_testtpm.log

(VPP split was added recently)

I’m thinking of extending it with the container lab or robot framework for the topology tests in the containers/VMs; these are only thoughts.

3

I think you’re looking at the wrong file. Might not have been my best idea to link to that ticket, as it contains the logs multiple test runs :grimacing:

Anyway, here is the output I meant:

DEBUG - Running Testcase: /usr/libexec/vyos/tests/smoke/cli/test_system_login.py
DEBUG - test_add_linux_system_user (__main__.TestSystemLogin.test_add_linux_system_user) ... ok
DEBUG - test_delete_current_user (__main__.TestSystemLogin.test_delete_current_user) ... ok
DEBUG - test_pam_nologin (__main__.TestSystemLogin.test_pam_nologin) ... ERROR
DEBUG - test_radius_kernel_features (__main__.TestSystemLogin.test_radius_kernel_features) ... ok
DEBUG - test_system_login_max_login_session (__main__.TestSystemLogin.test_system_login_max_login_session) ... ok
DEBUG - test_system_login_otp (__main__.TestSystemLogin.test_system_login_otp) ... ok
DEBUG - test_system_login_radius_ipv4 (__main__.TestSystemLogin.test_system_login_radius_ipv4) ... ERROR
DEBUG - test_system_login_radius_ipv6 (__main__.TestSystemLogin.test_system_login_radius_ipv6) ... ERROR
DEBUG - test_system_login_tacacs (__main__.TestSystemLogin.test_system_login_tacacs) ... ERROR
DEBUG - test_system_login_user (__main__.TestSystemLogin.test_system_login_user) ... ok
DEBUG - test_system_login_weak_password_warning (__main__.TestSystemLogin.test_system_login_weak_password_warning) ... ok
DEBUG - test_system_user_ssh_key (__main__.TestSystemLogin.test_system_user_ssh_key) ... ok
DEBUG - 
DEBUG - ======================================================================
DEBUG - ERROR: test_pam_nologin (__main__.TestSystemLogin.test_pam_nologin)
DEBUG - ----------------------------------------------------------------------
DEBUG - Traceback (most recent call last):
DEBUG -   File "/usr/libexec/vyos/tests/smoke/cli/test_system_login.py", line 561, in test_pam_nologin
DEBUG -     out, err = self.ssh_send_cmd(ssh_test_command, username, password)
DEBUG -                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DEBUG -   File "/usr/libexec/vyos/tests/smoke/cli/base_vyostest_shim.py", line 161, in ssh_send_cmd
DEBUG -     ssh_client.connect(hostname=hostname, username=username,
DEBUG -   File "/usr/lib/python3/dist-packages/paramiko/client.py", line 450, in connect
DEBUG -     self._auth(
DEBUG -   File "/usr/lib/python3/dist-packages/paramiko/client.py", line 781, in _auth
DEBUG -     raise saved_exception
DEBUG -   File "/usr/lib/python3/dist-packages/paramiko/client.py", line 768, in _auth
DEBUG -     self._transport.auth_password(username, password)
DEBUG -   File "/usr/lib/python3/dist-packages/paramiko/transport.py", line 1564, in auth_password
DEBUG -     return self.auth_handler.wait_for_response(my_event)
DEBUG -            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DEBUG -   File "/usr/lib/python3/dist-packages/paramiko/auth_handler.py", line 259, in wait_for_response
DEBUG -     raise e
DEBUG - paramiko.ssh_exception.BadAuthenticationType: Bad authentication type; allowed types: ['publickey']
DEBUG - 
DEBUG - ======================================================================
DEBUG - ERROR: test_system_login_radius_ipv4 (__main__.TestSystemLogin.test_system_login_radius_ipv4)
DEBUG - ----------------------------------------------------------------------
DEBUG - Traceback (most recent call last):
DEBUG -   File "/usr/libexec/vyos/tests/smoke/cli/test_system_login.py", line 296, in test_system_login_radius_ipv4
DEBUG -     self._system_login_radius_test_helper(radius_servers, radius_source)
DEBUG -   File "/usr/libexec/vyos/tests/smoke/cli/test_system_login.py", line 405, in _system_login_radius_test_helper
DEBUG -     out, err = self.ssh_send_cmd(ssh_test_command, username, password)
DEBUG -                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DEBUG -   File "/usr/libexec/vyos/tests/smoke/cli/base_vyostest_shim.py", line 161, in ssh_send_cmd
DEBUG -     ssh_client.connect(hostname=hostname, username=username,
DEBUG -   File "/usr/lib/python3/dist-packages/paramiko/client.py", line 450, in connect
DEBUG -     self._auth(
DEBUG -   File "/usr/lib/python3/dist-packages/paramiko/client.py", line 781, in _auth
DEBUG -     raise saved_exception
DEBUG -   File "/usr/lib/python3/dist-packages/paramiko/client.py", line 768, in _auth
DEBUG -     self._transport.auth_password(username, password)
DEBUG -   File "/usr/lib/python3/dist-packages/paramiko/transport.py", line 1564, in auth_password
DEBUG -     return self.auth_handler.wait_for_response(my_event)
DEBUG -            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DEBUG -   File "/usr/lib/python3/dist-packages/paramiko/auth_handler.py", line 259, in wait_for_response
DEBUG -     raise e
DEBUG - paramiko.ssh_exception.BadAuthenticationType: Bad authentication type; allowed types: ['publickey']
DEBUG - 
DEBUG - ======================================================================
DEBUG - ERROR: test_system_login_radius_ipv6 (__main__.TestSystemLogin.test_system_login_radius_ipv6)
DEBUG - ----------------------------------------------------------------------
DEBUG - Traceback (most recent call last):
DEBUG -   File "/usr/libexec/vyos/tests/smoke/cli/test_system_login.py", line 301, in test_system_login_radius_ipv6
DEBUG -     self._system_login_radius_test_helper(radius_servers, radius_source)
DEBUG -   File "/usr/libexec/vyos/tests/smoke/cli/test_system_login.py", line 405, in _system_login_radius_test_helper
DEBUG -     out, err = self.ssh_send_cmd(ssh_test_command, username, password)
DEBUG -                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DEBUG -   File "/usr/libexec/vyos/tests/smoke/cli/base_vyostest_shim.py", line 161, in ssh_send_cmd
DEBUG -     ssh_client.connect(hostname=hostname, username=username,
DEBUG -   File "/usr/lib/python3/dist-packages/paramiko/client.py", line 450, in connect
DEBUG -     self._auth(
DEBUG -   File "/usr/lib/python3/dist-packages/paramiko/client.py", line 781, in _auth
DEBUG -     raise saved_exception
DEBUG -   File "/usr/lib/python3/dist-packages/paramiko/client.py", line 768, in _auth
DEBUG -     self._transport.auth_password(username, password)
DEBUG -   File "/usr/lib/python3/dist-packages/paramiko/transport.py", line 1564, in auth_password
DEBUG -     return self.auth_handler.wait_for_response(my_event)
DEBUG -            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DEBUG -   File "/usr/lib/python3/dist-packages/paramiko/auth_handler.py", line 259, in wait_for_response
DEBUG -     raise e
DEBUG - paramiko.ssh_exception.BadAuthenticationType: Bad authentication type; allowed types: ['publickey']
DEBUG - 
DEBUG - ======================================================================
DEBUG - ERROR: test_system_login_tacacs (__main__.TestSystemLogin.test_system_login_tacacs)
DEBUG - ----------------------------------------------------------------------
DEBUG - Traceback (most recent call last):
DEBUG -   File "/usr/libexec/vyos/tests/smoke/cli/test_system_login.py", line 522, in test_system_login_tacacs
DEBUG -     out, err = self.ssh_send_cmd(ssh_test_command, username, password)
DEBUG -                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DEBUG -   File "/usr/libexec/vyos/tests/smoke/cli/base_vyostest_shim.py", line 161, in ssh_send_cmd
DEBUG -     ssh_client.connect(hostname=hostname, username=username,
DEBUG -   File "/usr/lib/python3/dist-packages/paramiko/client.py", line 450, in connect
DEBUG -     self._auth(
DEBUG -   File "/usr/lib/python3/dist-packages/paramiko/client.py", line 781, in _auth
DEBUG -     raise saved_exception
DEBUG -   File "/usr/lib/python3/dist-packages/paramiko/client.py", line 768, in _auth
DEBUG -     self._transport.auth_password(username, password)
DEBUG -   File "/usr/lib/python3/dist-packages/paramiko/transport.py", line 1564, in auth_password
DEBUG -     return self.auth_handler.wait_for_response(my_event)
DEBUG -            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DEBUG -   File "/usr/lib/python3/dist-packages/paramiko/auth_handler.py", line 259, in wait_for_response
DEBUG -     raise e
DEBUG - paramiko.ssh_exception.BadAuthenticationType: Bad authentication type; allowed types: ['publickey']
DEBUG - 
DEBUG - ----------------------------------------------------------------------
DEBUG - Ran 12 tests in 69.508s

I just gave it another shot, still fails on these three with freshly pulled repos.

Build system at least does these steps
…

Thank you a lot! So I after all wasn’t that far off with what I’m doing, though test-no-interfaces is actually not in use. How is that? What is the advantage of calling each test separately? Especially as AFAIR test-no-interfaces-no-vpp should cover e.g. test-c (not sure right now, always aborted after I encountered the first failure), so those tests run twice?

I’m thinking of extending it with the container lab or robot framework for the topology tests in the containers/VMs; these are only thoughts.

Sounds like an interesting approach! Though I’m not that familiar with these solutions. The closest I came was some barely automated gns3 setup, that got thrown away after not being that useful.

4

Check my log, the test name is the same.

You can start them in parallel and decrease the test time.
Also, it is convenient to check the VPP PHONY only without other tests. This is great and saves a lot of time for VPP development. But you can use test-no-interface if you do not need parallel execution.

5

Ah, so I’ve got you wrong. This is the output on your end? Which target do you call to execute the test? Or do you call that file only? If the latter, could you give me the necessary command to run that single file, so I can try that as well. Besides some more unknown shenanigans with test-no-interfaces I have no idea why the test could fail on my side, but succeed at yours.

You can start them in parallel and decrease the test time.

I feel kinda stupid for not seeing the obvious. That makes a lot of sense :sweat_smile:

Also, it is convenient to check the VPP PHONY only without other tests. This is great and saves a lot of time for VPP development. But you can use test-no-interface if you do not need parallel execution.

Yeah, that came into my mind already and is perfectly sensible as well. I was just wondering about having the “test everything” target but it seemed to be used nowhere. Thinking about it, would you accept a PR that changes the test-no-interfaces target to make use of the other existing targets? That way these accidental breakages I fixed could be avoided.

6

I still can’t get these three login tests to pass. I made sure to delete my local clone of the build repo and start from scratch. I’m using all targets you mentioned above, all pass except test-no-interface-no-vpp. See the attached logfile of the run. I’m sincerely out of ideas what could be wrong on my side.

test-no-interfaces-no-vpp.log (214,0 KB)

7

Try official published release and execute

/usr/libexec/vyos/tests/smoke/cli/test_system_login.py
2 Likes
8

Another thing I find useful is you can do this to call just an indiviudal test:

python3 /usr/libexec/vyos/tests/smoke/cli/test_system_login.py -k test_pam_nologin
3 Likes
9

I’ll do so ASAP next week!

EDIT: Unfortunately it doesn’t look like I’ll be able to re-test it before my vacation. If that’s the case, don’t expect a message from me until the last week of August :confused: On the other hand, until then it might have fixed itself xD

1 Like