Request for information on LivingDoc support

[ptsumbe]

Hi @gasparnagy, Really delighted to see new offering started by you. Currently we use SpecFlow with LivingDoc. What we have seen so far Reqnroll does not support it, can you please help comment when will it be available ? Do we have any prospect dates in mind ? #LivingDoc support

[gasparnagy]

Thank you @ptsumbe for your support.

I think the problem you facing is a common problem for many of us, so solving it should be a priority problem. (Especially because the SpecFlow.Plus.LivingDoc.CLI supports only .NET 3.1 and .NET 5, both are out of support already by Microsoft.)

We are currently busy with stabilizing the code base, but the next thing after should be the support for living doc. We track the efforts at #37. As we all work on this on a voluntary basis I think it would not be fair to set any deadline, but I hope to have something available before summer. Also our I got the information from the Allure team that they are also working on their Reqnroll integration that will be ready in the next weeks, so Allure reports will become a good alternative / workaround until the living doc support is implemented.

[ptsumbe]

Thank you @gasparnagy for response. It really helped!

[fkurzyk]

Thanks you @gasparnagy for this answer and for all your current and past efforts on .NET BDD.
I would like to extend ptsumbe's question about LivingDoc support and ask specifically about planned scope of LivingDoc replacement. Do you plan to have an Azure DevOps integration? Currently it is possible to view LivingDoc natively in AzDO, it would be great is it was also possible with Reqnroll's version.

[gasparnagy]

@fkurzyk The short answer would be that making the LivingDoc integrated to Azure DevOps is not fully out of scope, but obviously we first need the solution to generate HTML from the feature files & result. After that, we should re-evaluate if/how we could host that on Azure DevOps.

[ThomasHeijtink]

Just to mention it here as well. You can easily create a TestExecution.json file yourself in order to make the LivingDocs work again. So this doesn't have to be a deal breaker anymore to switch to Reqnroll.

[gasparnagy]

@ThomasHeijtink Could you share a bit more info about that? What does you build script uses as an input to generate the TestExecution.json?

[ThomasHeijtink]

@gasparnagy I hope this is what you are after.

I've created a MSBuild task which splits up all our feature files for the ultimate scenario level parallelisation. In that code I loop through all the feature files and its scenarios, and in case of a scenario outline I also make a separate feature file per example. Coincidentally, this is also the level of input required by the LivingDocs.

I have a .props file like this

<Project TreatAsLocalProperty="TaskFolder;TaskAssembly">
    <PropertyGroup>
        <CustomTasksFolder>..\tasks\net8.0</CustomTasksFolder>
        <CustomTasksAssembly>$(CustomTasksFolder)\Common.Interface.Testing.MSBuild.dll</CustomTasksAssembly>
    </PropertyGroup>

    <UsingTask TaskName="Common.Interface.Testing.MSBuild.SplitFeatureFiles" AssemblyFile="$(CustomTasksAssembly)"/>

    <PropertyGroup>
        <RootFolder Condition="'$(RootFolder)' == ''">$(MSBuildProjectDirectory)</RootFolder>
    </PropertyGroup>
</Project>

A .targets file like this

<Project>
    <Import Project="Common.Interface.Testing.MSBuild.props" />
        
    <ItemGroup>
        <FeatureFiles Include="$(RootFolder)\Features\*.feature" />
    </ItemGroup>

    <Target Name="SplitFeatureFiles" BeforeTargets="UpdateFeatureFilesInProject" DependsOnTargets="BeforeUpdateFeatureFilesInProject">
        <ItemGroup>
            <ReqnrollFeatureFiles Remove="SplitFeatures\*.feature"/>
            <Compile Remove="SplitFeatures\*.feature.cs"/>
        </ItemGroup>
        <SplitFeatureFiles FeatureFiles="@(FeatureFiles)" OutputPath="$(OutputPath)" />
        <ItemGroup>
            <ReqnrollFeatureFiles Include="SplitFeatures\*.feature"/>
            <Compile Include="SplitFeatures\*.feature.cs"/>
        </ItemGroup>
    </Target>
</Project>

My task looks something like this:

public class SplitFeatureFiles : Task {

    private readonly string _newLine = Environment.NewLine;
    private readonly string _directory = "SplitFeatures";
    private readonly List<string> _splitFeatureFileNames = new();

    public string OutputPath { get; set; } = string.Empty;
    
    [Required]
    public ITaskItem[] FeatureFiles { get; set; }
    
    public override bool Execute() {

        var testResult = new TestExecution(DateTime.Now);
        
        if (Directory.Exists($"./{_directory}") == false)
            Directory.CreateDirectory($"./{_directory}");
        
        foreach (var featureFile in FeatureFiles) {
            SplitFeatureFile(featureFile.GetMetadata("FullPath"), testResult);
        }

        var filesToDelete = Directory.GetFiles($"./{_directory}", "*.feature")
                                     .Where(f => _splitFeatureFileNames.Contains(Path.GetFileName(f)) == false)
                                     .ToArray();
        
        foreach (var fileToDelete in filesToDelete) {
            try {
                File.Delete(fileToDelete);
                File.Delete($"{fileToDelete}.cs");
            }
            catch (Exception e) when (e is IOException or UnauthorizedAccessException) {
                Log.LogErrorFromException(e, showStackTrace: true);
            }
            catch {
                //Probably already gone
            }
        }
        
        var serializerOptions = new JsonSerializerOptions
        {
            Encoder       = JavaScriptEncoder.UnsafeRelaxedJsonEscaping,
            WriteIndented = true
        };

        if (string.IsNullOrEmpty(OutputPath) == false && 
            Directory.Exists(OutputPath) == false) {
            Directory.CreateDirectory(OutputPath);
        }
        
        File.WriteAllText($"{OutputPath}{nameof(TestExecution)}.json", JsonSerializer.Serialize(testResult, serializerOptions), Encoding.UTF8);
        
        return Log.HasLoggedErrors == false;
    }

    private void SplitFeatureFile(string filePath, TestExecution testExecution) {

        var lines = File.ReadAllLines(filePath);
        
        var splitSettings = lines.Any(l => l.Contains("language: en"))
            ? SplitSettings.GetEnglish()
            : SplitSettings.GetDutch();

        var featureTitle = lines.FirstOrDefault(x => x.Contains(splitSettings.FeatureKeyword))?.Trim();

        if (featureTitle == null) {
            return;
        }
        
        var featureTitleWithTags = featureTitle;
        featureTitle = string.Join(':', featureTitle.Split(':', StringSplitOptions.TrimEntries).Skip(1));
        
        var content = string.Join(
            _newLine, 
            lines.Select(l => l.Trim()
                               .Replace(splitSettings.ScenarioOutlineKeyword, splitSettings.ScenarioOutlineMockKeyword))
                 .Where(l => l.StartsWith('#') == false));

        var sectionsAndTags = Regex.Split(content, splitSettings.SplitRegexExpression)
                                   .Where(s => string.IsNullOrEmpty(s) == false)
                                   .Select(s => s.Replace(splitSettings.ScenarioOutlineMockKeyword,
                                       splitSettings.ScenarioOutlineKeyword)).ToArray();
        
        var tags       = new HashSet<string>(){$"@split{_newLine}"};
        var background = string.Empty;
        var scenarios  = new List<string>();
        foreach (var sectionOrTag in sectionsAndTags) {
            if (sectionOrTag.StartsWith('@')) {
                tags.Add(sectionOrTag);
                continue;
            }

            if (sectionOrTag.StartsWith(splitSettings.FeatureKeyword)) {
                featureTitleWithTags = tags.Count > 0 ? $"{string.Join(_newLine, tags)}{featureTitleWithTags}" : featureTitleWithTags ;
            }
            else if (sectionOrTag.StartsWith(splitSettings.BackgroundKeyword)) {
                background = $"{sectionOrTag}";
            }
            else {
                scenarios.Add($"{string.Join(_newLine, tags)}{sectionOrTag}");
            }
            
            tags.Clear();
        }

        var scenarioCount = 1;
        foreach (var scenario in scenarios) {
            var examples        = new []{""};
            var scenarioSection = scenario;
            var exampleHeaders  = string.Empty;
            var scenarioTitle   = string.Join(':', scenario.Split(_newLine)[0].Trim().Split(':').Skip(1)).Trim();
            if (scenario.StartsWith(splitSettings.ScenarioOutlineKeyword)) {
                var scenarioSections = scenario.Split(splitSettings.ExampleKeyword, StringSplitOptions.TrimEntries);
                scenarioSection  = scenarioSections.First();
                var exampleSection   = scenarioSections.Last();
                examples = exampleSection.Split(_newLine, StringSplitOptions.TrimEntries | StringSplitOptions.RemoveEmptyEntries); 
                exampleHeaders = $"{splitSettings.ExampleKeyword}{_newLine}{examples[0]}";
                examples = examples.Skip(1).ToArray();
            }

            foreach (var (count, example) in examples.Select((e, i) => (i+1, e))) {

                var numberedScenario = scenarioSection.Replace(
                    $"{splitSettings.ScenarioOutlineKeyword} ",
                    $"{splitSettings.ScenarioOutlineKeyword} {count}_");

                var featureContent = $"""
                                      {splitSettings.LanguageIndicator}{featureTitleWithTags} {scenarioCount++}

                                      {background}{numberedScenario}
                                      """;

                if (example != string.Empty) {
                    featureContent += $"{_newLine}{_newLine}{exampleHeaders}{_newLine}{example}";
                }

                var newFileName = Path.GetFileNameWithoutExtension(filePath) + $"{scenarioCount - 1}.feature";

                File.WriteAllText($"./{_directory}/{newFileName}", featureContent);

                var stepArguments = example.Split('|', StringSplitOptions.TrimEntries);
                stepArguments = stepArguments.Skip(1).Take(stepArguments.Length - 2).ToArray();
                
                _splitFeatureFileNames.Add(newFileName);
                var executionResult = new ExecutionResult(
                    featureTitle, 
                    scenarioTitle,
                    stepArguments);
                
                testExecution.ExecutionResults.Add(executionResult);
            }
        }
    }
}

I use this parent class:

internal class TestExecution(DateTime executionTime) {
    public string[] Nodes => [];
    public DateTime ExecutionTime => executionTime;
    public DateTime GenerationTime { get; }
    public Guid PluginUserSpecFlowId => new Guid("36bde1dc-fd2c-45e7-a4f2-fae8799bfad4"); //This is some sort of Guid which I could find on our build server. It was always the same in all test runs but differed from the Guid value that was used on my local machine.
    public string? CLIUserSpecFlowId { get; }

    public List<ExecutionResult> ExecutionResults { get; } = new();

    public string? StepReports { get; }
}

And each scenario and example represent an ExecutionResult like this:

internal class ExecutionResult(
    string featureTitle,
    string scenarioTitle,
    string[] stepArguments) {
    public string ContextType => "Scenario";
    public string FeatureFolderPath => "Features";

    public string FeatureTitle => featureTitle;
    public string ScenarioTitle => scenarioTitle;
    public string[] StepArguments => stepArguments;
    
    public string Status => "OK";
    public string[] StepResults => [];
    public string? Outputs { get; }
}

When you write this to a TestExecution.json file in a directory depending on your configuration of your SpecFlowPlus@0 task. It's normally generated in the OutputDir for the build.

It will be picked up and used for the LivingDoc.

This will result in something like this:

{
  "Nodes": [],
  "ExecutionTime": "2024-04-23T14:37:07.1205458+02:00",
  "GenerationTime": "0001-01-01T00:00:00",
  "PluginUserSpecFlowId": "36bde1dc-fd2c-45e7-a4f2-fae8799bfad4",
  "CLIUserSpecFlowId": null,
  "ExecutionResults": [
    {
      "ContextType": "Scenario",
      "FeatureFolderPath": "Features",
      "FeatureTitle": "Example feature",
      "ScenarioTitle": "Example scenario",
      "StepArguments": [],
      "Status": "OK",
      "StepResults": [],
      "Outputs": null
    },
    {
      "ContextType": "Scenario",
      "FeatureFolderPath": "Features",
      "FeatureTitle": "Example feature",
      "ScenarioTitle": "Example scenario outline",
      "StepArguments": [
        "example value 1",
        "example value 2",
        "example value 3",
        "example value 4",
        "example value 5"
      ],
      "Status": "OK",
      "StepResults": [],
      "Outputs": null
    }
  ],
  "StepReports": null
}

I our case we weren't interested in the actual test results as this is part of our pipeline. So I didn't care about it that much. Only the regular scenarios, as opposed the scenario outlines, were marked as passed. Even though I reported everything was OK. So maybe the scenario outlines require something extra which I haven't found yet. Maybe they require at least a single step result. But in the StepResults you can add objects which represent these values.

The step results look like this:

{"Duration":"00:00:00.0053099","Status":"OK","Error":null,"Outputs":null}

I hope it helps someone to make the step to Reqnroll!