CI fails due to timing issue occasionally

[BenediktBurger]

Quite often one test fails due to timing issues making the whole CI pipeline fail, probably the time limit should be reduced slightly:

    def test_check_query_delay():
        with expected_protocol(ATSBase, [("TTIM?", "7")]) as inst:
            start = perf_counter()
            assert inst.maximum_test_time == 7
            delay = perf_counter() - start
>           assert delay > 0.05
E           assert 0.0491846000000038 > 0.05

tests\instruments\temptronic\test_temptronic_base.py:36: AssertionError

[CasperSchippers]

It sounds a bit like a floating-point errors: the start and delay might be too large (numericall) such that that difference cannot be calculated accurately. Not sure if this is indeed the case, but something to consider. If this is the case, it could be circumvented by using perf_counter_ns, which returnes the time as an int (in nanoseconds) in stead of float.

[BenediktBurger]

Another solution could be pytest.approx, however, we should check first, what is actually tested, maybe the test is ok, but the code under test not.

[BenediktBurger]

Having merged #1109, CI runs succeed. After the previous two merges, they always failed.

Seems to work, we should keep it open though for some more time.

[BenediktBurger]

It did not work in CI run:

    def test_check_query_delay():
        with expected_protocol(ATSBase, [("TTIM?", "7")]) as inst:
            start = perf_counter_ns()
            assert inst.maximum_test_time == 7
            delay = perf_counter_ns() - start
>           assert delay > 0.05 * 1e9
E           assert 47828800 > (0.05 * 1000000000.0)

[BenediktBurger]

Interestingly, it is always Python 3.8 or Python 3.9 on windows, which fails.

It's crazy, but it seems, that sleep sleeps only 95% of the specified time.

Last resort: using an error margin of 5%