Tests on Windows require Git Bash or a similar environment

[EliahKagan]

Current behavior 😯

In this project, tests are typically run on Windows in a Git Bash environment. On CI in Windows, bash is Git Bash. This is also the environment from which I have typically run tests. This is a native environment; it should not be confused with WSL or even something like Cygwin. Furthermore, tools maintained by rustup are not specifically connected to this environment, though they are run from it.

All tests are able to pass when run in Git Bash on Windows, and likely would pass if run in similar environments that provide a POSIX-compatible shell for Windows, set environment variables accordingly, and provide access to POSIX versions of common tools.

But running the tests from PowerShell instead produces 627 test failures. This presumably relates to the associated environment rather than the shell itself, since however the tests are run, a test runner subprocess is controlling the run.

The main problem seems to be the use of paths with backslashes in them, and the way that interacts with fixture scripts. Note, however, that this is not specific to the situation where GIX_TEST_IGNORE_ARCHIVES is set. That is to say that this is a separate issue from #1358 (which also involves far fewer failures) and occurs even when GIX_TEST_IGNORE_ARCHIVES is unset.

Even the test summary showing one failure per line is too long for GitHub to allow me to include it in this issue description. This gist has it. The full output can be seen in this other gist.

As of now, I have not figured out the cause, which I think is a prerequisite for making a decision about whether to try to support such environments or to instead document them as unsupported.

Expected behavior 🤔

Either all tests should pass even when run from PowerShell, or the need to run them in a Git Bash environment (or whatever other more precise requirement is known) should be documented.

I think which is better depends on how cumbersome it would be to enable the tests to pass when run from PowerShell and, more importantly, how cumbersome it would be to maintain this state. I suspect it may be feasible, though.

Git behavior

A direct comparison is difficult in that this is about running the tests rather than the functionality of Git and gitoxide. However, it is worth noting that Git for Windows has its own SDK environment that is used for building it and running its tests, and this environment is separate even from the more minimal Git Bash environment.

Thus, if it is infeasible to support running tests from PowerShell, documenting that would still not be very restrictive, even compared to what is needed to develop and test Git for Windows, since the non-SDK "Git Bash" environment works fine for running gitoxide's tests.

Steps to reproduce 🕹

Using Windows, make sure the tests can pass when run from Git Bash using the first of two cargo test-runner commands shown below.

Optionally clean the build with cargo clean or, if preferred, clean everything with gix clean -xde.

There are two approaches. The first approach is to display the output in the console: To do that, in PowerShell, run:

cargo nextest --all --no-fail-fast

However, that is hard to read because of the very large volume of output combined with the staggering of output lines suggesting a problem identifying the terminal width or computing the width of text written.

Therefore, you may instead wish to write both stdout and stderr to a file, and inspect the file during and/or after the run.

cargo nextest --all --no-fail-fast *> ../output.txt

This writes all output to output.txt in the directory above the current directory. This is to avoid creating a new file in the repository while running the tests, in case that were to interfere with something, though it shouldn't since that should only be able to affect a small number of journey test runs, which are not included when running cargo nextest.

The *> operator in PowerShell is similar to the &> or >& operators in some shells and to the effect of 2>&1 > in most shells.

Although Windows comes with Windows PowerShell, I suggest against using it for this, since is somewhat less intuitive, and has fewer convenience features, compared to the newer PowerShell (which is sometimes called PowerShell Core). Windows PowerShell is also less likely to be used by developers on Windows, and thus probably less important to support than recent versions of PowerShell. I used PowerShell 7.4.2 on Windows 10.

[Byron]

Thanks for the detailed report!

It appears that gix-testtools are unable to execute any script with backslashes in its path as these will count as escape-character. So for this to work (better), all it might take is to properly escape such paths. There might be more to it though. Maybe a Windows build of gix-testtools could have special mitigations for that built in at some point.