Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Re-enable and stabilise system test PythonChannels.PythonChannelTest #37366

Closed
SilkeSchomann opened this issue May 14, 2024 · 2 comments · Fixed by #37427
Closed

Re-enable and stabilise system test PythonChannels.PythonChannelTest #37366

SilkeSchomann opened this issue May 14, 2024 · 2 comments · Fixed by #37427
Assignees
Labels
Maintenance Unassigned issues to be addressed in the next maintenance period.
Milestone

Comments

@SilkeSchomann
Copy link
Contributor

Describe the outcome that is desired.
In #37364 the PythonChannels.PythonChannelTest was disabled after several random failures in the last few days.

This problem is not new: #36326, #36358 and #36362.

Describe any solutions you are considering
The aim is to find a permanent solution. The default behaviour of multiprocessing will change in Python 3.14, so it would be good to keep that in mind.

Additional context
Further information can be found here: https://docs.python.org/3/library/multiprocessing.html

@SilkeSchomann SilkeSchomann added the Maintenance Unassigned issues to be addressed in the next maintenance period. label May 14, 2024
@SilkeSchomann SilkeSchomann changed the title Investigate ... Re-enable and stabilise system test PythonChannels.PythonChannelTest May 14, 2024
@SilkeSchomann SilkeSchomann added this to the Release 6.10 milestone May 14, 2024
@rosswhitfield
Copy link
Member

rosswhitfield commented May 17, 2024

Since no one linked to the actual test failure in this issue or #37364, here is the failure as seen on the build server from 7377

======================================================================
ERROR: test_PythonStdoutChannel_update_instruments (PythonChannels.PythonLoggingTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/jenkins_workdir/workspace/pull_requests-conda-linux/Testing/SystemTests/tests/framework/PythonChannels.py", line 84, in test_PythonStdoutChannel_update_instruments
    self._create_script_and_execute(LoggingClass="PythonStdoutChannel", UpdateInstrumentDefinitions=1)
  File "/jenkins_workdir/workspace/pull_requests-conda-linux/Testing/SystemTests/tests/framework/PythonChannels.py", line 99, in _create_script_and_execute
    result = subprocess.run([sys.executable, TEST_SCRIPT_NAME], timeout=60)  # This will raise subprocess.TimeoutExpired if deadlocke
  File "/jenkins_workdir/workspace/pull_requests-conda-linux/mambaforge/envs/mantid-developer/lib/python3.10/subprocess.py", line 505, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "/jenkins_workdir/workspace/pull_requests-conda-linux/mambaforge/envs/mantid-developer/lib/python3.10/subprocess.py", line 1154, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/jenkins_workdir/workspace/pull_requests-conda-linux/mambaforge/envs/mantid-developer/lib/python3.10/subprocess.py", line 2047, in _communicate
    self.wait(timeout=self._remaining_time(endtime))
  File "/jenkins_workdir/workspace/pull_requests-conda-linux/mambaforge/envs/mantid-developer/lib/python3.10/subprocess.py", line 1209, in wait
    return self._wait(timeout=timeout)
  File "/jenkins_workdir/workspace/pull_requests-conda-linux/mambaforge/envs/mantid-developer/lib/python3.10/subprocess.py", line 1951, in _wait
    raise TimeoutExpired(self.args, timeout)
subprocess.TimeoutExpired: Command '['/jenkins_workdir/workspace/pull_requests-conda-linux/mambaforge/envs/mantid-developer/bin/python3', 'pythonloggingtestscript.py']' timed out after 59.999970824457705 seconds

----------------------------------------------------------------------
Ran 4 tests in 99.908s

FAILED (errors=1)
Traceback (most recent call last):
  File "/tmp/tmpqlr5dqn1", line 12, in <module>
    systest.execute()
  File "/jenkins_workdir/workspace/pull_requests-conda-linux/Testing/SystemTests/lib/systemtests/systemtesting.py", line 248, in execute
    self.runTest()
  File "/jenkins_workdir/workspace/pull_requests-conda-linux/Testing/SystemTests/tests/framework/PythonChannels.py", line 119, in runTest
    self.assertTrue(runner.run(suite).wasSuccessful())
  File "/jenkins_workdir/workspace/pull_requests-conda-linux/mambaforge/envs/mantid-developer/lib/python3.10/unittest/case.py", line 687, in assertTrue
    raise self.failureException(msg)
AssertionError: False is not true

This problem is not new: #36326, #36358 and #36362.

It is unrelated to the issue that OSX was having with this test.

I looked through 3 months of builds on pull_requests-conda-linux and the earliest instance of this test failing was 6932 (1 month and 8 days ago), after which it was failing somewhat often. So this is actually a new issue that didn't exist before that.

I don't know what changed that could be causing the issue and I am unable to reproduce the issue locally.

My suggestion is to remove the test completely. The code comments in the actual code fixes is hopefully enough to prevent someone accidentally removing them. This is a really hard thing to write a reliable test for, so we can leave this for manual testing and the users that pick up the issue originally will quickly find them again if they come back.

But if you do feel like we should try to stabilize the tests have 2 thoughts.

  1. You can remove the 10 second sleep as that was only included to get around a bug that was later fixed by Clone extracted IFunction to separate from python object #36798
  2. Increase the timeout, I am only guessing that something is slowing down this test to the point that the test fails, if you increase it to 600 seconds and it still randomly fails then there must be some other issue.

@SilkeSchomann SilkeSchomann self-assigned this May 24, 2024
@sf1919 sf1919 linked a pull request May 28, 2024 that will close this issue
@sf1919
Copy link
Contributor

sf1919 commented May 30, 2024

@SilkeSchomann can this now be closed?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Maintenance Unassigned issues to be addressed in the next maintenance period.
Projects
Status: Done
Status: Done
Status: Done
Development

Successfully merging a pull request may close this issue.

3 participants