TESTed: Can't get evaluation right for an exercise with output>input>output

[marcothijs]

Hi,

Please see exercise https://dodona.be/nl/activities/1160199994/
The idea is to get some output (print intro), then ask for an input (input) and end with an output (print result). It is an excercise on functions and I want to test both main script and functions (you already helped me with getting that right with TESTed).
The problem I have now is that de first output, doesn't show in the expected output, so the first test fails.

Thanks for looking into it.
Marco

[niknetniko]

We don't support duplicate attributes, so your second stdout will overwrite the first. You can get this exercise to work, by putting all stdout togheter:

- stdout: |
            ****************************************
            * hexadecimaal <-> decimaal convertor  *
            ****************************************
            * Kies 1 voor hexadecimaal -> decimaal *
            * Kies 2 voor decimaal -> Hexadecimaal *
            ****************************************
            De decimale waarde is 15.

This is with the big caveat that this won't actually test the order of stdout -> stdin -> stdout. Students could thus ask for input first, and then print the all stdout in one go, and it would pass this test.

TESTed currently has no way of knowing the relative order in which stdout/stdin are used. We run the submission, provide it with all stdin at the start and collect all stdout at the end. Unfortunately, I don't immediately see a solution or workaround.

[marcothijs]

Thanks for your quick response. I had simular cases in the past with the "pythia_judge" where the order wasn't actually tested. At least the evaluation gets tested OK. I tell the pupils always that they should code the excercise, not the Dodona tests...
Thanks again.
Marco

[pdawyndt]

Actually, at process level, stdin and stdout are deconnected. So there is no way to check proper synchronisation between read from stdin and write to stdout. This could only be check through static source code analysis (hard) or dynamic analysis of runtime behaviour (easy), but both are not supported by TESTed, nor by the old Python judge.