You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This is an umbrella issue to track progress making the Sphinx test suite reliable. That means:
No flaky tests.
Tests can run in parallel.
Tests are independent of each other.
Regarding the last item: test independence is important to ensure that each test is checking the behaviours it claims to, and does not inadvertently rely on the side-effects of some other test. Similarly, a test must not begin failing if any other test is run before it -- if it does, that could mean (but does not definitely mean) that the application's behaviour itself may vary based on the order in which code is evaluated.
It might seem like there are a large number of items here! However, please bear in mind that we have a total of more than 2000 individual pytest test cases that run.
picnixz
changed the title
[meta-issue] [tests] Python test suite reliability: striving for non-flaky, parallelizable, random-orderable tests.
[tracker] [tests] Python test suite reliability: striving for non-flaky, parallelizable, random-orderable tests.
Mar 23, 2024
This is an umbrella issue to track progress making the Sphinx test suite reliable. That means:
Regarding the last item: test independence is important to ensure that each test is checking the behaviours it claims to, and does not inadvertently rely on the side-effects of some other test. Similarly, a test must not begin failing if any other test is run before it -- if it does, that could mean (but does not definitely mean) that the application's behaviour itself may vary based on the order in which code is evaluated.
It might seem like there are a large number of items here! However, please bear in mind that we have a total of more than 2000 individual
pytest
test cases that run.Test flakiness
test_build_linkcheck.test_connect_to_selfsigned_fails
(pr: [tests] Increase timeouts for linkcheck build tests. #12166)test_build_linkcheck.test_defaults
(pr: [tests] Increase timeouts for linkcheck build tests. #12166)test_build_linkcheck.test_linkcheck_exclude_documents
(pr: [tests] Attempt to uncover and resolve test flakiness in 'test_linkcheck_exclude_documents'. #12189)Test parallelization
tests.test_builders.test_build_linkcheck
(pr: [tests] linkcheck: bind each test HTTP server to a unique port per-testcase. #12126)Test independence
tests.test_builders.test_build_text.py
(pr: none)tests.test_domains.test_domain_cpp
(pr: [test] fix side-effect due to repeated fields #12187)tests.test_builders.test_build_html
(pr: [tests] Make the 'test_html_copy_source' case agnostic of test evaluation order. #12118)tests.test_builders.test_build_html_download
(pr: Remove test result-sharing between 'test_html5_output' and 'test_html5_download' #12119)The text was updated successfully, but these errors were encountered: