devenv test reporting success even when errors and failures occur

[kourtni]

Describe the bug
When running devenv test, failures and errors can get swallowed resulting in devenv test reporting that "Tests passed".

To reproduce
This gist, https://gist.github.com/kourtni/7ee2749b0bb75253e0d29d2deab1b4fb, is a complete example that reproduces the issue. I expect that running devenv test from that environment should fail, however it currently passes.

Version
devenv 1.0.5 (x86_64-linux)

[domenkozar]

Could you also attach the python files so I can reproduce the failure?

[kourtni]

Hi,

The issue is that devenv test swallows errors and test failures if enterTest ends with a command that exits normally.

The screenshot is from tests run in a private repo, so I can't share those files. But as I indicated in the Gist, https://gist.github.com/kourtni/7ee2749b0bb75253e0d29d2deab1b4fb, this behavior is completely reproducible without actually requiring any python files. In fact the point of that Gist is to show that devenv test passes even without ANY of the required packages installed or source files available. I am able to reproduce the issue with only the devenv.nix and devenv.yaml files included in the Gist, and then simply running devenv test.

But if you absolutely want to include a dummy test for some reason, you could create a file with a simple test where you assert 1 == 2 or something that will obviously fail.

As long as the final command in enterTest is an echo or some other command guaranteed to succeed, then any failures that occurred prior will be swallowed and the devenv test will complete in a Tests passed state.