Documentation/Sample for xUnit v2 tests programs as exes

[davidmatson]

The v3 support for tests programs as exes is really useful - is there documentation or a sample for how to do this for v2?

Here's what I ended up doing, but I'm not sure if this anything close to a recommended approach:

using Xunit.Abstractions;
using Xunit;

class Program
{
    public static void Main()
    {
        // Change the following line if a different output format is needed:
        IRunnerReporter runnerReporter = new DefaultRunnerReporterWithTypes();
        IMessageSink messageSink = runnerReporter.CreateMessageHandler(new ConsoleRunnerLogger(true));

        using (IMessageSinkWithTypes executionMessageSink = MessageSinkWithTypesAdapter.Wrap(messageSink))
        using (ExecutionSink executionSink = new ExecutionSink(executionMessageSink, new ExecutionSinkOptions { DiagnosticMessageSink = messageSink }))
        using (XunitFrontController controller = new XunitFrontController(AppDomainSupport.IfAvailable, typeof(Program).Assembly.Location, diagnosticMessageSink: messageSink))
        {
            ITestFrameworkExecutionOptions executionOptions = TestFrameworkOptions.ForExecution();
            ITestFrameworkDiscoveryOptions discoveryOptions = TestFrameworkOptions.ForDiscovery();
            controller.RunAll(executionSink, discoveryOptions, executionOptions);
            executionSink.Finished.WaitOne();
        }
    }
}

This is code obviously limited in terms of harness functionality (doesn't support filtering or anything fancy - it just runs all the tests), but is this a reasonable approach, or could some documentation/a sample be added for getting "test as exe" for v2?

(Or maybe there's already something, and I just didn't find it.)

[bradwilson]

Yes, what you're doing is the core of what we do. Our version is 10-20x more complex mostly because of options that you aren't doing here (like reporter choice, support for output files, etc.).

This is the full version from the v2 console runner:

xunit/src/xunit.console/ConsoleRunner.cs

Lines 342 to 462 in 57af1d9

	XElement ExecuteAssembly(object consoleLock,
	XunitProjectAssembly assembly,
	bool serialize,
	bool needsXml,
	bool? parallelizeTestCollections,
	int? maxThreadCount,
	bool diagnosticMessages,
	bool noColor,
	AppDomainSupport? appDomains,
	bool failSkips,
	bool stopOnFail,
	XunitFilters filters,
	bool internalDiagnosticMessages)
	{
	foreach (var warning in assembly.ConfigWarnings)
	logger.LogWarning(warning);
	if (cancel)
	return null;
	failSkips = failSkips || assembly.Configuration.FailSkipsOrDefault;
	var assemblyElement = needsXml ? new XElement("assembly") : null;
	try
	{
	if (!ValidateFileExists(consoleLock, assembly.AssemblyFilename) || !ValidateFileExists(consoleLock, assembly.ConfigFilename))
	return null;
	// Turn off pre-enumeration of theories, since there is no theory selection UI in this runner
	assembly.Configuration.PreEnumerateTheories = false;
	assembly.Configuration.DiagnosticMessages |= diagnosticMessages;
	assembly.Configuration.InternalDiagnosticMessages |= internalDiagnosticMessages;
	if (appDomains.HasValue)
	assembly.Configuration.AppDomain = appDomains;
	// Setup discovery and execution options with command-line overrides
	var discoveryOptions = TestFrameworkOptions.ForDiscovery(assembly.Configuration);
	var executionOptions = TestFrameworkOptions.ForExecution(assembly.Configuration);
	if (maxThreadCount.HasValue)
	executionOptions.SetMaxParallelThreads(maxThreadCount);
	if (parallelizeTestCollections.HasValue)
	executionOptions.SetDisableParallelization(!parallelizeTestCollections.GetValueOrDefault());
	if (stopOnFail)
	executionOptions.SetStopOnTestFail(stopOnFail);
	var assemblyDisplayName = Path.GetFileNameWithoutExtension(assembly.AssemblyFilename);
	var diagnosticMessageSink = DiagnosticMessageSink.ForDiagnostics(consoleLock, assemblyDisplayName, assembly.Configuration.DiagnosticMessagesOrDefault, noColor);
	var internalDiagnosticsMessageSink = DiagnosticMessageSink.ForInternalDiagnostics(consoleLock, assemblyDisplayName, assembly.Configuration.InternalDiagnosticMessagesOrDefault, noColor);
	var appDomainSupport = assembly.Configuration.AppDomainOrDefault;
	var shadowCopy = assembly.Configuration.ShadowCopyOrDefault;
	var longRunningSeconds = assembly.Configuration.LongRunningTestSecondsOrDefault;
	using (AssemblyHelper.SubscribeResolveForAssembly(assembly.AssemblyFilename, internalDiagnosticsMessageSink))
	using (var controller = new XunitFrontController(appDomainSupport, assembly.AssemblyFilename, assembly.ConfigFilename, shadowCopy, diagnosticMessageSink: diagnosticMessageSink))
	using (var discoverySink = new TestDiscoverySink(() => cancel))
	{
	// Discover & filter the tests
	reporterMessageHandler.OnMessage(new TestAssemblyDiscoveryStarting(assembly, controller.CanUseAppDomains && appDomainSupport != AppDomainSupport.Denied, shadowCopy, discoveryOptions));
	controller.Find(false, discoverySink, discoveryOptions);
	discoverySink.Finished.WaitOne();
	var testCasesDiscovered = discoverySink.TestCases.Count;
	var filteredTestCases = discoverySink.TestCases.Where(filters.Filter).ToList();
	var testCasesToRun = filteredTestCases.Count;
	reporterMessageHandler.OnMessage(new TestAssemblyDiscoveryFinished(assembly, discoveryOptions, testCasesDiscovered, testCasesToRun));
	// Run the filtered tests
	if (testCasesToRun == 0)
	completionMessages.TryAdd(Path.GetFileName(assembly.AssemblyFilename), new ExecutionSummary());
	else
	{
	if (serialize)
	filteredTestCases = filteredTestCases.Select(controller.Serialize).Select(controller.Deserialize).ToList();
	reporterMessageHandler.OnMessage(new TestAssemblyExecutionStarting(assembly, executionOptions));
	var resultsOptions = new ExecutionSinkOptions
	{
	AssemblyElement = assemblyElement,
	CancelThunk = () => cancel,
	FinishedCallback = summary => completionMessages.TryAdd(assemblyDisplayName, summary),
	DiagnosticMessageSink = diagnosticMessageSink,
	FailSkips = failSkips,
	LongRunningTestTime = TimeSpan.FromSeconds(longRunningSeconds),
	};
	var resultsSink = new ExecutionSink(reporterMessageHandler, resultsOptions);
	controller.RunTests(filteredTestCases, resultsSink, executionOptions);
	resultsSink.Finished.WaitOne();
	reporterMessageHandler.OnMessage(new TestAssemblyExecutionFinished(assembly, executionOptions, resultsSink.ExecutionSummary));
	if ((resultsSink.ExecutionSummary.Failed != 0 || resultsSink.ExecutionSummary.Errors != 0) && executionOptions.GetStopOnTestFailOrDefault())
	{
	Console.WriteLine("Canceling due to test failure...");
	cancel = true;
	}
	}
	}
	}
	catch (Exception ex)
	{
	failed = true;
	var e = ex;
	while (e != null)
	{
	Console.WriteLine("{0}: {1}", e.GetType().FullName, e.Message);
	if (internalDiagnosticMessages)
	Console.WriteLine(e.StackTrace);
	e = e.InnerException;
	}
	}
	return assemblyElement;
	}

We haven't documented this because we don't expect average users to want to do this. We chose stand alone executables because it's impossible to rationalize linking in .NET Core (which is why we ended up abandoning dotnet xunit). The model where you load the unit test like it's a "plugin" into the actual runner was clever but wrong from the start in retrospect.

The complexity is why there is the new in-process console runner in v3. But there shouldn't be anything hidden away that you can't use, if you choose to dig into the code yourself.

[davidmatson]

Thanks, Brad.

Do you think that, once the v3 exe contract is settled, it would make sense to provide a one-line helper class for v2 that allows turning a v2 test project into an exe, so that some of those benefits are easily avaliable in v2 as well? (not that the helper would run the test internally as v3, just that it would support the overall similar command-line switches and output, perhaps compatible from the standpoint of an outside consumer)

If not, please feel free to close this issue.