Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lots of twister errors in CI that don't get reported as failures #72807

Closed
arbrauns opened this issue May 15, 2024 · 6 comments · Fixed by #72893
Closed

Lots of twister errors in CI that don't get reported as failures #72807

arbrauns opened this issue May 15, 2024 · 6 comments · Fixed by #72893
Assignees
Labels
area: Twister Twister bug The issue is a bug, or the PR is fixing a bug priority: high High impact/importance bug

Comments

@arbrauns
Copy link
Contributor

There are a lot of python exceptions being thrown in twister CI runs, but the run doesn't get marked as failed:

Process Process-17:
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/__w/zephyr/zephyr/scripts/pylib/twister/twisterlib/runner.py", line [132](https://github.com/zephyrproject-rtos/zephyr/actions/runs/9094695276/job/24996411498#step:11:134)5, in pipeline_mgr
    pb.process(pipeline, done_queue, task, lock, results)
  File "/__w/zephyr/zephyr/scripts/pylib/twister/twisterlib/runner.py", line 677, in process
    self.run()
  File "/__w/zephyr/zephyr/scripts/pylib/twister/twisterlib/runner.py", line 1[134](https://github.com/zephyrproject-rtos/zephyr/actions/runs/9094695276/job/24996411498#step:11:136), in run
    instance.handler.handle(harness)
  File "/__w/zephyr/zephyr/scripts/pylib/twister/twisterlib/handlers.py", line 306, in handle
    with subprocess.Popen(command, stdout=subprocess.PIPE,
  File "/usr/lib/python3.10/subprocess.py", line 971, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.10/subprocess.py", line 1863, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/__w/zephyr/zephyr/twister-out/unit_testing/tests/unit/winstream/utilities.winstream/zephyr/zephyr.exe'

Example: https://github.com/zephyrproject-rtos/zephyr/actions/runs/9094695276/job/24996411498#step:11:131

As far as I can tell, this kills the twister process and half the tests aren't even being run, yet no failure is reported in the GitHub action.

@arbrauns arbrauns added the bug The issue is a bug, or the PR is fixing a bug label May 15, 2024
@PerMac
Copy link
Member

PerMac commented May 16, 2024

I found the root cause of this. Will provide a fix soon

@PerMac
Copy link
Member

PerMac commented May 16, 2024

the above link fixes "unit" type tests. Next PR will fix the exception handling so such errors are correctly reported

@aescolar aescolar added the priority: high High impact/importance bug label May 16, 2024
@nashif
Copy link
Member

nashif commented May 16, 2024

the above link fixes "unit" type tests. Next PR will fix the exception handling so such errors are correctly reported

what next PR? where is it?

@PerMac
Copy link
Member

PerMac commented May 16, 2024

WIP. Probably won't be able to push it today. The error with failing test is fixed. IMO failing unit tests were high priority. However, the fact that twister was not able to properly handle those issue is not that high, especially that this was already like this and the recent unit test errors just exposed this

@nashif
Copy link
Member

nashif commented May 16, 2024

ok, I will try and push something to address this

@PerMac
Copy link
Member

PerMac commented May 16, 2024

I just opened a PR with proposition @nashif #72892

PerMac added a commit to PerMac/zephyr that referenced this issue May 16, 2024
If the binary is not existing twister won't report such case as error.
This patch fixes this.

fixes: zephyrproject-rtos#72807

Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
nashif added a commit to nashif/zephyr that referenced this issue May 16, 2024
Deal with binaries not found. Also set default return code to something
other than zero to catch issues and report correct status.

Fixes zephyrproject-rtos#72807

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
nashif added a commit to nashif/zephyr that referenced this issue May 17, 2024
Handle all exceptions when running tests.

Fixes zephyrproject-rtos#72807

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
nashif added a commit to nashif/zephyr that referenced this issue May 17, 2024
Abort execution in case of exceptions.

Fixes zephyrproject-rtos#72807

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
nashif added a commit to nashif/zephyr that referenced this issue May 18, 2024
Abort execution in case of exceptions.

Fixes zephyrproject-rtos#72807

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
nashif added a commit to nashif/zephyr that referenced this issue May 23, 2024
When something goes wrong with execution, due to twister issues and
bugs, do not continue with execution and abort the process and report
and return an error code.

Fixes zephyrproject-rtos#72807

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
nashif added a commit to nashif/zephyr that referenced this issue May 29, 2024
When something goes wrong with execution, due to twister issues and
bugs, do not continue with execution and abort the process and report
and return an error code.

Fixes zephyrproject-rtos#72807

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
henrikbrixandersen pushed a commit that referenced this issue Jun 4, 2024
When something goes wrong with execution, due to twister issues and
bugs, do not continue with execution and abort the process and report
and return an error code.

Fixes #72807

Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: Twister Twister bug The issue is a bug, or the PR is fixing a bug priority: high High impact/importance bug
Projects
6 participants