Make contribution easier in the Reqnroll repo #72
Replies: 4 comments 21 replies
-
Definitely not being able to run the test suite. Even after the changes made in #64 I cannot get all of the tests to succeed, which makes making changes to the core a lot more difficult. There is also a dependency on having Visual Studio installed (through vswhere) which I think we might be able to eliminate as well. |
Beta Was this translation helpful? Give feedback.
-
One of my issues is/was the error reporting on CI. Initially we used Now my next attempt is to try https://github.com/Tyrrrz/GitHubActionsTestLogger: I have introduced that to the Reqnroll.VisualStudio repo and pretty happy with it:
If you are happy with this, I would try to apply it to the main repo as well. CC @reqnroll/core-team |
Beta Was this translation helpful? Give feedback.
-
We all see that testing is a crucial part of improving the contribution experience. As fas a I see one of the problems that currently almost all of the tests work in a combinatorial fashion: they target multiple frameworks and the Specs project even targets multiple test runners. The reasons are clear: we would like to make sure that Reqnroll works on all supported platforms with all test runners, and this is a valid expectation. But with the current approach this generates too many overlaps, too many tests and slow test execution. In fact it is even controversial, because due to the complexity and the slow execution of test, we don't actually run them in multiple targets - the CI currently runs them only with .NET 6.0 and probably no one does it locally either regularly. My imagined test strategy is the following:
I have started to work on the "system tests" concept (it's in this PR) and will work on the rework of the "specs" project as these are quite connected. (This is why I asked @Tiberriver256 to skip Specs in #76.) Any comments are welcome. |
Beta Was this translation helpful? Give feedback.
-
Another issues for contribution were the problem of cached packages. This issue we have discovered with @livioc on #54: If you modify the generation part (e.g. MsTestV2GeneratorProvider) and rerun the related "specs" test, the test will pick up the previously generated nuget package from the package cache and therefore the test will not reflect your changes (you still see the old behavior). The reason for this is that these E2E test use the generated NuGet packages for generating the sample projects. Between two compilations these packages have the same version number, therefore if a version has been used, nuget automatically caches that in the local nuget cache (C:\Users<your-user>.nuget\packages), so the subsequent execution will re-use the cached version. The solution was to manually clean the Reqnroll related packages from the local nuget cache. There has been tries to avoid this problem: in SpecFlow we tried to generate commit specific nuget versions (this caused other problems and did not solve the changes within commits) and also there was a code that tried to delete the packages from the local nuget cache (most of the cases did not work because of file locking issues). I have made a PR to address this (#77). This PR uses a different approach: it reconfigures nuget for the test projects to use an alternative local nuget cache folder somewhere in $TEMP, so that the execution always starts with an empty nuget cache. (There is a NuGet/Home#5619 in nuget where this idea was mentioned.) The nuget cache has several megabytes, so instead of making a local nuget cache for every sample project, we introduce a "Test Run" folder, that contains all sample project folders and the local nuget cache. Each time a new test run folder is created in $TEMP, but we have a cleanup mechanism that removes old test run folders. I have tested that by replaying the changes of #54 and it works fine, all changes of the code are immediately reflected in tests. @reqnroll/core-team: Please send a few 👍 on the PR or here if you are OK with this or make your comment to the PR if not. Would like to merge this tomorrow if there is no disagreement. |
Beta Was this translation helpful? Give feedback.
-
I guess anyone who tried to contribute to the main repo faced different issues that made the work harder.
I would like to take it as a top priority for myself to improve this. I have seen several problems myself as well, but I am interested to your comments as well:
What was hindering you when working with the Reqnroll solution?
Note: The compilation / locking issue seems to be fixed now by #64 (see #38)
CC @reqnroll/core-team
Beta Was this translation helpful? Give feedback.
All reactions