Replacing XtextSmokeTestRunner in JUnit 5

[trancexpress]

Hi,

since generally replacing XtextSmokeTestRunner seems difficult, I've opted to adjust our tests to recognize smoke testing mode and act accordingly. I'm doing so based on this comment: #2883 (comment)

Unfortunately, I don't understand the entire state machinations that surround WrappingInjectorProvider.

	private final IInjectorProvider delegate;
	private final Injector injector;
	private GlobalStateMemento stateBeforeInjectorCreation;
	private final GlobalStateMemento stateAfterInjectorCreation;

	public WrappingInjectorProvider(IInjectorProvider delegate) {
		this.delegate = delegate;
		stateBeforeInjectorCreation = GlobalRegistries.makeCopyOfGlobalState();
		this.injector = createInjector();
		if (delegate instanceof IRegistryConfigurator) {
			((IRegistryConfigurator) delegate).setupRegistry();
		}
		registerFactory(injector);
		stateAfterInjectorCreation = GlobalRegistries.makeCopyOfGlobalState();
		if (delegate instanceof IRegistryConfigurator) {
			((IRegistryConfigurator) delegate).restoreRegistry();
		}
		stateBeforeInjectorCreation.restoreGlobalState();
	}

	@Override
	public void restoreRegistry() {
		stateBeforeInjectorCreation.restoreGlobalState();
		stateBeforeInjectorCreation = null;
	}

	@Override
	public void setupRegistry() {
		stateBeforeInjectorCreation = GlobalRegistries.makeCopyOfGlobalState();
		stateAfterInjectorCreation.restoreGlobalState();
	}

Then the smoke test runner itself will do state machinations in AbstractScenarioRunner.methodBlock():

	@Override
	protected Statement methodBlock(final FrameworkMethod method) {
		IInjectorProvider injectorProvider = getOrCreateInjectorProvider();
		final IRegistryConfigurator registryConfigurator = (IRegistryConfigurator) injectorProvider;
		registryConfigurator.setupRegistry();
		final Statement methodBlock = superMethodBlock(method);
		return new Statement() {
			@Override
			public void evaluate() throws Throwable {
				try {
					try {
						methodBlock.evaluate();
						throw new AssumptionViolatedException("Method " + method.getName() + " did parse any input");
					} finally {
						registryConfigurator.restoreRegistry();
					}
				} catch(TestDataCarrier testData) {
					process(testData.getData());
				}
			}
		};
	}
	
	protected void process(String data) throws Exception {
		IInjectorProvider delegate = getOrCreateInjectorProvider().getDelegate();
		if (delegate instanceof IRegistryConfigurator) {
			IRegistryConfigurator registryConfigurator = (IRegistryConfigurator) delegate;
			registryConfigurator.setupRegistry();
			try {
				ScenarioProcessor processor = delegate.getInjector().getInstance(processorClass);
				String preProcessed = processor.preProcess(data);
				if (preProcessed == null) {
					throw new AssumptionViolatedException("Input is filtered by the pre processing step: " + data);
				}
				doProcess(preProcessed, processor);
			} finally {
				registryConfigurator.restoreRegistry();
			}
		}
	}

I don't follow what state is being set to what. It looks like the wrapping IInjectorProvider is being setup and restored, and within that block, the actual IInjectorProvider is being setup and restored. However, I wrap the injector itself, as I cannot control what IInjectorProvider is being used by the tests (that is specified via an annotation).

The errors I get during the extra smoke test validation look e.g. like this:

1567 [main] ERROR text.validation.CompositeEValidator  - Error executing EValidator
java.lang.NullPointerException: Cannot invoke "String.equals(Object)" because the return value of "org.eclipse.emf.common.util.URI.scheme()" is null

That particular error seems to go away if I call setupRegistry() on the IInjectorProvider used in the tests. But that causes failures later on, in other tests.

What state exactly do I need to set and restore here, prior to the extra validation we have in smoke tests (i.e. the code that runs during doProcess())? And what state needs to be set and restored, when I wrap the injector?

Best regards and thanks,
Simeon

[trancexpress]

Hi @szarnekow ,

could you help here? I'm struggling to understand the memento logic.

Best regards and thanks,
Simeon

[trancexpress]

OK, I tried to stay as close as possible to the setup/restore logic in XtextSmokeTestRunner and that seems to work.

To do so, I replaced the @InjectWith IInjectorProvider of all tests used in a smoke test, with an IInjectorProvider that does the following:

If smoke tests are not running (ENV variable for us), delegate to an instance of the original IInjectorProvider type.
If smoke tests are running, use InjectorCache to wrap the original IInjectorProvider.

Then, implement an exception handler that does this:

public class SmokeTestExecutionHandler implements TestExecutionExceptionHandler, BeforeTestExecutionCallback {

    /* The TestDataCarrier class is package protected, so we cannot do an instanceof check with it... */
    private static final String TEST_DATA_CARRIER_CLASS_NAME = "org.eclipse.xtext.testing.smoketest.internal.TestDataCarrier";
    private static final Class<? extends ScenarioProcessor> TESTER_CLASS = SmokeTestProcessor.class;

    @Override
    public void handleTestExecutionException(ExtensionContext context, Throwable throwable) throws Throwable {
        if (isTestDataCarrierException(throwable)) {
            IInjectorProvider injectorProvider = ExposeInjectionExtensionInternals.getInjectorProvider(context);
            InjectorProviderForSmokeTests ip = (InjectorProviderForSmokeTests) injectorProvider;
            InjectorProviderForTests delegate = ip.getDelegate();
            /*
             * NOTE:
             *
             * The following restore and setup logic follows the logic found in XtextSmokeTestRunner.
             * Without this sequence of restoring and setting up, on the exact same injector providers,
             * smoke test validation will fail silently with logged errors.
             *
             * XtextSmokeTestRunner essentially surrounds the test method with a setup and restore block,
             * using the WrappingInjectorProvider. The smoke test validation is done outside of this block,
             * surrounded by a setup and restore block of the *delegate* IInjectorProvider.
             * So we replicate this logic as closely as possible below.
             */
            /*
             * The xtext JUnit 5 InjectWith logic has a setup and restore block around each test method.
             * The JUnit 5 listener for exceptions (which is used to handle the TestDataCarrier exceptions in this method)
             * is notified within this setup and restore block. So we must restore and finally setup, to ensure
             * smoke test validation is done "outside" of the setup and restore block.
             */
            ip.restoreRegistry();
            try {
                /*
                 * We have cleared the state, so now prepare the delegate IInjectorProvider for smoke test validation.
                 */
                delegate.setupRegistry();
                try {
                    ScenarioProcessor tester = injectorProvider.getInjector().getInstance(TESTER_CLASS);
                    String data = getData(throwable);
                    String preProcessed = tester.preProcess(data);
                    if (preProcessed == null) {
                        throw new AssumptionViolatedException("Input is filtered by the pre processing step: " + data);
                    }
                    tester.processFile(data);
                } finally {
                    delegate.restoreRegistry();
                }
            } finally {
                /*
                 * Bring back the state as it was before smoke test validation, since the InjectWith logic
                 * will try to do a restore after the test method is done. If there is no setup here, that restore will fail.
                 */
                ip.setupRegistry();
            }
        } else {
            throw throwable;
        }
    }

    private static boolean isTestDataCarrierException(Throwable throwable) {
        return throwable != null && TEST_DATA_CARRIER_CLASS_NAME.equals(throwable.getClass().getName());
    }

    private static String getData(Throwable throwable) {
        // use reflection, since TestDataCarrier is package private...
        try {
            Field field = throwable.getClass().getDeclaredField("data");
            boolean wasAccessible = field.canAccess(throwable);
            try {
                field.setAccessible(true);
                Object data = field.get(throwable);
                return data.toString();
            } finally {
                field.setAccessible(wasAccessible);
            }
        } catch (Exception e) {
            throw new AssertionError("Failed to retrieve field 'data' of exception type: " + e.getClass(), e);
        }
    }

    @Override
    public void beforeTestExecution(ExtensionContext context) throws Exception {
        if (SmokeTestSession.isRunning()) {
            Optional<AnnotatedElement> optional = context.getElement();
            if (optional.isPresent()) {
                AnnotatedElement element = optional.get();
                IgnoredBySmokeTest ignoredBySmokeTest = element.getAnnotation(IgnoredBySmokeTest.class);
                if (ignoredBySmokeTest != null) {
                    throw new AssumptionViolatedException("Test is ignored during smoke testing");
                }
            }
        }
    }

    private static class ExposeInjectionExtensionInternals extends InjectionExtension {
        static IInjectorProvider getInjectorProvider(ExtensionContext context) {
            return InjectionExtension.getOrCreateInjectorProvider(context);
        }
    }
}

The wrapper essentially looks like this:

**
 * This class replaces {@link XtextSmokeTestRunner} for JUnit 5, as there is no replacement from xtext.
 * We wrap the {@link IInjectorProvider} we use for the smoke tests - {@link InjectorProviderForTests}.
 * If smoke tests are not running, we delegate to the wrapped {@link InjectorProviderForTests}.
 * If smoke tests are running, we delegate to a {@link WrappingInjectorProvider} version of {@link InjectorProviderForTests}
 *
 * The {@link WrappingInjectorProvider} will insert a {@link TestDataProvider},
 * that throws a {@code TestDataCarrier} exception whenever a model is loaded. This exception is used
 * to stop the running test, do extra validation on the model and base the test result on the validation result.
 *
 * What we are missing from the original {@link XtextSmokeTestRunner} implementation,
 * is that tests that don't run in a {@code TestDataCarrier} exception are marked as skipped.
 * Its unclear how to alter the test result in JUnit 5 based on the absence of the exception.
 * In addition, whether the tests is marked as passing or skipped is mostly cosmetic for our testing purposes.
 */
public class InjectorProviderForSmokeTests implements IInjectorProvider, IRegistryConfigurator {

    private final IInjectorProvider injectorProvider;

    public InjectorProviderForSmokeTests() {
        if (SmokeTestSession.isRunning()) {
            IInjectorProvider delegate = new InjectorProviderForTests();
            injectorProvider =  InjectorCache.wrap(delegate);
        } else {
            injectorProvider = new InjectorProviderForTests();
        }
    }

    @Override
    public void setupRegistry() {
        ((IRegistryConfigurator) injectorProvider).setupRegistry();
    }

    @Override
    public void restoreRegistry() {
        ((IRegistryConfigurator) injectorProvider).restoreRegistry();
    }

    @Override
    public Injector getInjector() {
        return injectorProvider.getInjector();
    }

    public InjectorProviderForTests getDelegate() {
        if (SmokeTestSession.isRunning()) {
            return (InjectorProviderForTests) ((WrappingInjectorProvider) injectorProvider).getDelegate();
        }
        throw new AssertionError("Smoke tests are not running, there is no delegate");
    }
}

With our custom ScenarioProcessor, this is enough to replace our use of XtextSmokeTestRunner...

[trancexpress]

I don't know how to do the above without changing the annotation value of @InjectWith for all the tests that are smoke tested. So I don't know how to actually generically replace XtextSmokeTestRunner, in the way its provided in JUnit 4 (just extend a test suite and its smoke tested, no test case changes).