Nightly smoketests don't fail when they probably should

[Buzoth]

The nightly build actions don’t seem to honor the exit codes generated from the smoketest system. This causes images that fail the smoketest(s) to be released and published.

For example, build vyos-1.5-rolling-202407070023-amd64.iso:

1.5-rolling-202407060019

This particular test case seems to fail > 90% of the time, but when executed manually on the live image from the same failing iso, works without issue.

out: DEBUG - ======================================================================
out: DEBUG - FAIL: test_ospf_15_ldp_sync (__main__.TestProtocolsOSPF.test_ospf_15_ldp_sync)
out: DEBUG - ----------------------------------------------------------------------
out: DEBUG - Traceback (most recent call last):
out: DEBUG -   File "/usr/libexec/vyos/tests/smoke/cli/test_protocols_ospf.py", line 488, in test_ospf_15_ldp_sync
out: DEBUG -     self.assertIn(f'router ospf', frrconfig)
out: DEBUG - AssertionError: 'router ospf' not found in ''
out: DEBUG - 
out: DEBUG - ----------------------------------------------------------------------
out: DEBUG - Ran 16 tests in 100.229s
...
<stripped>
...
out: ERROR - Hmm... system got an exception while processing.
out: ERROR - The ISO image is not considered usable!
err: make: *** [Makefile:24: test] Error 1
==============================================
✅ Successfully executed commands to all host.
==============================================

I see that there’s work going on around improving this all together, but for the current script, I believe simply adding script_stop: true could help, though I don’t really have the ability to test this.
Documentation: GitHub - appleboy/ssh-action: GitHub Actions for executing remote ssh commands.

[Buzoth]

It actually seems like the new, more parallel build process suffers the same failure and continues to publish as well:
From github:
vyos/vyos-rolling-nightly-builds/actions/runs/9784664621/job/27016348015#step:5:2326

[c-po]

@Buzoth

script_stop

According to the Stop script after first failure this is not what we wan’t we wan’t to see everything that fails. Nevertheless you are right @Buzoth and that image should not have been published - I do not know, why.

[c-po]

This is a draft implementation by by to simply the Pipeline that currently runs on the main branch

[yevhen]

@Buzoth it should be fixed now, thanks for pointing out.