Roslyn Source Generator

[Code-Grump]

Should Reqnroll's code-generation be moved to a Roslyn source generator? It was mentioned a few times on the SpecFlow project and wondered if it was relevant to Reqnroll's future plans.

I think there are numerous minor benefits to a source generator over the existing code-generator:

• Easier integration into the build process
• Faster to execute because it doesn't require launching a separate executable
• Code is generated in IDEs without needing to run a build
• Faster builds due to built-in incremental support

I have been experimenting with writing a source generator for this purpose and got as far as getting a generator to produce a simple output every time I changed a feature file.

[gasparnagy]

I would love this. I tried once, but realized that it needs quite much experties to get it working and I did not have time to dig into that.

If you have expereicnce with it, especially with the incremental generators, I would be happy to discuss this in detail, so that maybe you can do a POC for it. (Others can of course help too.)

[Code-Grump]

It's surprisingly simple, there's just a lot of nuance around how things like configuration options are handled.

Very happy to discuss further and contribute this feature.

[gasparnagy]

Super. I will collect my ideas I had and summarize them here (next week) and then we can see how we could make something out of it.

[SabotageAndi]

As far as I know, source generators are not available for .NET Framework.
So as long Reqnroll isn't dropping support for it, I think it is not possible to use source generators.

[Code-Grump]

I can confirm that source-generation works for modern .NET Framework projects.

A simple-ish demo is to:

1. Create a new C# project targeting .NET Framework 4.7.2
2. Install the ResX Source Generator package https://www.nuget.org/packages/Microsoft.CodeAnalysis.ResxSourceGenerator
3. Add a ResX file to the project

The package installs as an analyser and has no runtime assets.

Roslyn itself was incorporated into the tooling that shipped with Visual Studio 2015. I would be interested to know what the earliest SDK is that supports analysers, but I've not been able to find that information yet.

[gasparnagy]

Good news! But just to be clear. Is this also true for incremental source code generators? (Because for Reqnroll those are the onces that would really make a boost.)

[ajeckmans]

It is a Roslyn feature so not tied to the framework explicitly. And seeing as you need to target .net standard 2.0 to be able to create incremental source generators I fully expect it to work on any platform that also supports .net standard 2.0.

[Code-Grump]

Is this also true for incremental source code generators?

I can confirm this is an example of an incremental source generator.

[gasparnagy]

[removed] placed comment as a reply instead of a comment, see it below

[gasparnagy]

As promised, here is my brain dump of my ideas about using the Roslyn code generators. I was thinking about that a year ago, so some things might have changed.

Looking forward to your thoughts.

I have seen two levels of implementation.

Level 1

On this level, basically we would generate more or less the same code what is generated through MsBuild right now, but with Roslyn code generators. What we would win:

• simplicity (for the users and tooling)
• with incremental generators, we might get improved compilation time (to be checked!)
• provides a baseline for Level 2

As part of this level, we have to consider the following questions:

• How does the debugging support work with Roslyn code generators. So far we included #line directives to the generated code to refer to the source lines of the feature file and the debugger handled that automatically. Does this method work with the new infrastructure as well? If not, what can we do?
• How the actual code generation will be done. Currently we use the obsolete "CodeDom" technology that is obsolete and does not support new code constructs. On the other hand, we cannot use a simple text writer, because different components of Reqnroll and the plugins need to extend the generated code: add additional classes, add additional fields/properties/methods, add statements to methods. So at the end, we may need to build up a simplified CodeDom that fits to our purpose but far less complex than CodeDom is.
• What should happen with Visual Basic code generation? I think we should not invest making the new infrastructure work with VB as well, but rather direct the users to a path, where they have a C# project with only the feature files and a VB project with all the step definitions.
• What should happen with the "generator plugins"? I think we will not be able to keep compatibility here, but the fact that you can customize the generation behaviour with a plugin is important, so I would still allow generator plugins, but with the new infrastructure. As now all this is in .NET Standard and does not need MsBuild magic, hopefully it will be easier to implement generator plugins.
• Do we want to provide backwards compatibility? Currently the MsBuild based code generation is in a separate package (Reqnroll.Tools.MsBuild.Generation) that is automatically included by the Reqnroll.MsTest, etc packages. With the new infra, we will remove this dependency, but we can still keep it for a while for those users who have existing generator plugins that they cannot yet upgrade. The only thing we need for this is a global switch that switches off the Roslyn based generation, so that the classes are not generated twice. Does not sound very complex, so probably would be worth considering it.

Level 2

So once we cleared a couple of questions with Level 1, we can move on to level 2 where the things get even more exciting in my opinion.

In level 2, we would not only generate the "code-behind" files for the feature files, but we would also process the step definitions during compile time (not easy, but maybe somehow possible). With that information, we would already be able to have a good guess which step definition method will be called if a particular step is going to be executed (for most of the cases we will now it 100%). Knowing that, we can even include lines to the generated code that actually invoke that step definition method. There are two key benefits of this

1. No reflection is needed to call these methods
2. Static code analyzers start to work (e.g. you will see in Visual Studio if a particular step definition method is used)

So for example if there is a step When I do something and a related step definition MyStepDefinitions.WhenIDoSomething(), the generated code would be something like:

testRunner.When<MyStepDefinitions>("When I do something", stepDef => stepDef.WhenIDoSomething());

Of course this is just a simple case, but maybe the concept is visible. We would still need to handle multi-matches (for scenario outlines) and step definition parameters and probably a couple of other tricks as well.

Also we need to handle step definitions from external assemblies. For that my idea would be that once we compile and discover the step definitions anyway, we keep this discovery info and save it as a separate file or as an embedded resource. So if you consume an external step definition assembly, you immediately get the step definition information as well. This could be used by the IDE integration as well BTW, currently the VS integration is doing something similar.

[gasparnagy]

What is complicated is anything that requires runtime knowledge, like which assemblies will be loaded into the application. There may be something we can do based on conventions, but we can't know the runtime environment. To be really valuable, we'd have to constrain our system to only use information available to the compilation, which might break some user's complex cases.

@Code-Grump Can you give a brief summary on how these generators are loaded and what restrictions do we have during generator code execution? E.g. can we load classes (e.g. Gherkin parser) from external assemblies? Can we do dynamic assembly loading (e.g. a plugin) with Assembly.Load and use reflection? Can we load the configuration for reqnroll.json?

[Code-Grump]

I haven't been able to find any documentation on how generators are loaded. Like all analysers, they are loaded both by the IDE the compiler. The IDE-hosted version is triggered to run any time any of the compilation inputs change, where the compiler version executes only when csc.exe is called as part of MSBuild.

This document is probably the best explanation of the structure of incremental source generators https://github.com/dotnet/roslyn/blob/main/docs/features/incremental-generators.md

There are no enforced limitations on what you can do, but the model of an incremental generator requires it be deterministic from its inputs. The design optimises for cheaply detecting when an input has changed in a meaningful way and only performing expensive generation operations when required.

You can add references to NuGet packages and they'll be loaded by the analyser system. I had some challenges around breaking down an analyser into multiple assemblies without making them NuGet packages and getting them all loaded. This was especially annoying trying to test using project-references only.

We can absolutely do dynamic loading and reflection; the problems are:

• How frequently do you do these - an analyser will be invoked in an IDE on almost every keystroke.
• How do you want to deal with their dependencies and conflicts with assemblies already loaded by the host? There are more IDEs than Visual Studio too.
• How do you safely integrate third-party code (and potentially their additional inputs) into a process that must be deterministic?

[gasparnagy]

@Code-Grump Thanks. Good input. At some point we should decide when to stop thinking and go for a POC. Currently I don't see any blocking problem, just a couple of challenges that we could better address with a POC.

[Code-Grump]

If you could give me some specifics on what you'd want to see achieved and where to put it, I'll put something together.

[gasparnagy]

@Code-Grump I collected my thoughts as a top-level post, see below.

[gasparnagy]

Cross-linking the discussion Directly provide ITestDiscoverer and ITestExecutor, instead of a host test framework that is basically trying to achieve a similar goal with a different approach. But some of the questions (e.g. the necessity of generator plugins) is the same.

[gasparnagy]

Ideas for the proof of concept (POC):

The best would be to have a POC that is somewhat integrates to the Reqnroll infrastructure, so that we can see the impact, but uses as many shortcuts as possible, so that we can get a quick validation. I'm not hoping for this, but it is even possible that we need to cancel this, so keep it simple.

Here are my thoughts, everything is flexible:

Integrated to Reqnroll infrastructure

• Is in the Reqnroll repo (separate branch of course)
• It generates code that can be executed by Reqnroll runtime
• It uses the Gherkin parser to process the feature files (to be able to assess the speed)

Shortcuts

Basically it is enough if it is able to run super basic scenarios from a simple Reqnroll project.

• Should be a new project, no need to use anything from the current Generator project
• Only support NUnit (or any single other, but I think NUnit is the simplest)
• Ignore configuration
• Ignore specific Gherkin elements (ie. does not need to support), like Background, Rule, Scenario Outlines, Data Tables, DocStrings, maybe even tags
• Use the simplest possible code generation for now: basic strings, only C#
• Ignore hooks & test execution framework specifics, like TestContext
• Ignore generator plugins
• Ignore any special namespace collision or class name collision problems. It can even generate everything to a hardcoded namespace for now
• If helps, we can even ignore realistic scenario names, enough if it works with scenarios whose name is a valid C# method name (Sceanrio: MySceanrio1)

[Code-Grump]

I've created a prototype branch for a generator here: https://github.com/reqnroll/Reqnroll/tree/source-generator-poc

Two projects have been added:

1. Reqnroll.FeatureSourceGenerator.NUnit - produces a NuGet package containing the source generator.
2. Reqnroll.FeatureSourceGenerator.NUnit.Example - consumes the NuGet package to host the source generator in a sample project.

The following functions are demonstrated:

• Invoking the Gherkin parser to process feature files into generated source.
• Gherkin syntax errors added to compiler output as errors.
• NUnit test-fixture generation based on scenario names.

Not demonstrated:

• Unit testing the generator using Microsoft's Roslyn testing libraries.
• Hooking into the Reqnroll infrastructure to execute scenario steps.

---

Apologies for the delay in getting this out there; my life has been hectic for the last week. Hopefully this branch will serve as a basis for further conversation and investigation.

[gasparnagy]

@Code-Grump no need to apologize, this is all good. I will do courses next week, so probably can play with it only in the week after, but I'm eager to see this growing.

[gasparnagy]

I created a draft PR to better track the changes of this branch and to later see the impact of it on testing.

#65

[Code-Grump]

Regarding Generator Plug-Ins

Plugins remain my biggest concern.

The Easy Path
The easiest path is to not support any kind of extension mechanism - if we need to allow code customisation, it has to be baked into the generator. We don't need to take a direct dependency on any third-party libraries to emit the required strings and we can use the presence of the referenced assembly in the compilation as a signal to drive whether and how we activate any particular customization.

We gain a great deal of simplicity, we lose the ability for users to customize their test-generation without going through an existing mechanism. For simple scenarios that just want to include an attribute, I can imagine generic options driven through config or user code. For more complex things (like scenarios driven by external data sources) we'd have to directly incorporate that feature into the generator.

The Dangerous Path
If we must offer an extension mechanism, we have to solve considerable problems:

• work out how users can package their code in a way that we will be able to load during execution without slowing down the critical path
• put in effort to trap and report type-loading errors within our generator -or- accept users will need to live with Roslyn's error output
• build a system that encourages/enforces determinism within plugins

---

I quite like being able to write plug-ins for the generator. I've found them to be quite difficult to get right. Very, incredibly difficult. Obviously it's quite a niche space - most users just want something they can install that works and does what they need.

I could not find any examples of Roslyn components doing this - the ones I see being produced are designed to do very specific, very narrow tasks: create a strongly-typed ResX wrapper, create native Regex matcher, create a JSON serializer for a type.

My gut feeling is to narrow the scope to what Roslyn seems best suited for and take the easy path. What do your respective guts have to say?

[Code-Grump]

I think that puts us back at the original position. Either find a mechanism to load executable additions into the generator, or bake in everything the generator can support into the generator assembly with only minimal extension-points (e.g. a way to add custom attributes)

[gasparnagy]

Ok. No worries. I also thought about that and here is my proposal for a plan "B":

• we should incorporate our own plugins into the main generator (e.g. external data or verify, and of course the test runners): this can be either a full implementation that is enabled automatically based on the existence of the appropriate runtime plugin or by providing infrastructure that can be enabled with config settings.
• common, simple external pluging (e.g. ones that decorate generated test methods with additional attributes based on the existence of a tag) can be supported by infrastructure that can be enabled with config settings.
• in-house, customer or project specific generation needs can be supported by letting them replace our generator with their custom one (that derives from ours). Of course in this case the "composition" is their responsibility.
• any other need (hopefully not many) can be solved with either a completly separate generator that extens the generated partial class or by IL waving.

We can provide examples for these.

With this we can continue on the easy path and can get rid of the dynamic generator plugin loading.

What do you think?

[Code-Grump]

I'll continue with the assumption that we're not going to allow add-ins to run code.

[clrudolphi]

Would you consider either of the following ideas as feasible?

• Expose the Roslyn AST of the generated code to the plugins, allowing them to modify it, and then complete the generation of the code-behind
• Use a multi-pass AST approach. Eg, Parse and process Gherkin into an intermediate representation that begins to model the structure of the test classes and methods that are to be generated (but much simpler than a Roslyn C# AST and purpose-built for describing RnR tests), run that through a pipeline of AST visitors (each plug-in would implement a visitor), and then use the resulting representation to generate C# source

[Code-Grump]

Those are both reasonable approaches, however they do need a mechanism for loading plug-in types into the Roslyn execution context.

It's a solvable problem - you can package any assembly as an analyser and have it loaded. The tricky bit is all the type-loading problems that have to be navigated, like finding a different version of an assembly already loaded by something else.

Existing within a shared runtime is a complex business that'd be nice to sidestep if we can.

[Code-Grump]

Implementation is progressing, and I thought I'd try integration with our Spec project as a way to prove out the Roslyn implementation, first as a way to test within the test project as an alternative configuration (using FeatureSourceGenerator within the generated projects) and then to use to build the spec project itself.

It was at this point I noticed that we have a plug-in for generating different flavours of each feature: target framework, test framework, etc. This has long been something I think is broadly useful and have often wanted to implement: a way to have a whole test suite run with multiple base configurations. It comes up very often dealing with cross-browser compatibility testing, but I'm sure there are many opportunities this would be valuable.

I propose we bake this into FeatureSourcdGenerator: we can drive it from user-code (Roslyn gives us the whole source-tree after all) and make it as simple as having a parameter on a step-definition class that we recognise as a "flavor" of test.

public record BrowserDescription(
    string Name,
    string Version);

[Binding]
public class BrowserSteps(
    [Flavoring] BrowserDescription browser)
{
    ...
}

We'd need to determine how to name the generated classes based on the values of the flavor parameters. It could just be a case of calling ToString() and mangling the result into an identifier. Certainly it's an easy extension-point for customisation.

We'd also need to determine how an end-user could supply all the permutations in a way that is visible at generation-time. Attributes are easiest, but have the most limitations (can't do complex types):

[Binding]
public class BrowserSteps(
    [Flavor("ms-edge", "current")]
    [Flavor("chrome", "current")]
    BrowserDescription browser)
{
    ...
}

There's probably some additional effort in supporting this for bindings outside of the feature assembly, but I think that can come later.

[gasparnagy]

Depends on what you want. If you want to have a "test" that verifies whether the Roslyn code gen works and compatible with the msbuild one, you should add one more test class to the generation folder of the system tests project (so there the tests will run both on msbuild and Roslyn).

If you just want to have a real life experience with the Roslyn generator in a larger project, feel free to reconf the specs project itself to use that (this is anyway what we will need to do). What you should not bother with is to reconfigure the tests run by the specs project to use the new generator, becase from those I would like to remove the whole generation anyway and run them in-memory.

Does that answer your question?

[Code-Grump]

I'm looking for test-cases to prove that the Roslyn generator works for all the same core scenarios that the MSBuild-based one does, that it can handle tags and rules and scenario outlines, and hook into all the right parts of each test framework's lifecycles to trigger the expected events within Reqnroll.

I have already configured the specs project to run double the number of tests, by running the same set again for the Roslyn generator. I assumed we would have all these things already defined for the existing generator, but I confess I have not checked that in detail. If they need writing in a fresh project, I shall write them!

[gasparnagy]

These tests are verifying those: https://github.com/reqnroll/Reqnroll/blob/main/Tests/Reqnroll.SystemTests/Generation/GenerationTestBase.cs

To use them, you just need to create a new subclass and configure the test framework and the generation behavior, similar to https://github.com/reqnroll/Reqnroll/blob/main/Tests/Reqnroll.SystemTests/Generation/MsTestGenerationTest.cs

[gasparnagy]

Or maybe make a subclass of the MsTestGenerationTest class so that the MsTest specific details are also verified.

[Code-Grump]

Subclassing each of the test-framework suites is working well so far. If we had to add a third vector, I think this approach would start to hurt. The test-coverage has me a little concerned: would a successful test-run give you sufficient confidence in this generator, or would you need to see some additional scenarios?

[Code-Grump]

Just been looking at adding scenario outline support. The current implementation leverages parameterised tests to achieve this and I wanted to know if there was any specific reason for this approach? I was contemplating just emitting additional methods for each example and giving them display names to organise them accordingly, as this would avoid the need to handle inline test data and lets us put tag attributes on individual examples.

The only immediate downside I am aware of is an increase in the size output assembly (since it has additional methods) but I could personally live with that. 😅

[ajeckmans]

I don't know how this is in nunit, but with mstest datarow tests are grouped together visually in the test explorers of rider and visual studio. I think the parameterized tests of xunit do the same.

[gasparnagy]

Yes. That's true for all three. They are visually grouped if we use the data driven test approach (in runners and also in reporting tools), this is why we use that.

So it would make sense to keep it.

[Code-Grump]

In Visual Studio, when running NUnit, the grouping is driven by naming of the test. If you override the test name with a custom description, it drops out the grouping, unless you prefix it with the name of the test.

This might just be a Visual Studio thing and other runners respect the hierarchy regardless of what you name your test. Probably surest to keep producing data-driven tests.