TestBed passes even without any declarations in NG9 and NG10

[smzelek]

🐞 bug report

Affected Package

The issue seems to be in @angular/core/testing or the new bazel/ivy build rig.

Is this a regression?

Yes, this behavior worked in Angular 8.
Angular 8 reports the following error...

Error: Illegal state: Could not load the summary for directive AppComponent.
error properties: Object({ ngSyntaxError: true })

Description

Angular's TestBed no longer works in any kind of logical way.

You can completely remove calls to TestBed.configureTestingModule and the unit tests still run, compile, and succeed. This is totally unexpected, as you are supposed to be able to pass declarations or mock declarations for every token. It's now completely unclear what environment is being used when the test runs, whereas before you explicitly provided every dependency.

🔬 Minimal Reproduction

https://github.com/smzelek/minimal-repro-ng10-testbed-bug
Simply run the unit tests:

1. npm run test

Notice that all 3 tests pass, the component is successfully rendered in the Karma runner window, variable templating succeeds..., even though there is no TestBed configuration at all.

🌍 Your Environment

Angular Version:

     _                      _                 ____ _     ___ 
    / \   _ __   __ _ _   _| | __ _ _ __     / ___| |   |_ _|
   / △ \ | '_ \ / _` | | | | |/ _` | '__|   | |   | |    | | 
  / ___ \| | | | (_| | |_| | | (_| | |      | |___| |___ | | 
 /_/   \_\_| |_|\__, |\__,_|_|\__,_|_|       \____|_____|___|
                |___/
    

Angular CLI: 10.0.7
Node: 12.18.2
OS: win32 x64

Angular: 10.0.12
... animations, common, compiler, compiler-cli, core, forms
... platform-browser, platform-browser-dynamic, router
Ivy Workspace: Yes

Package                           Version
-----------------------------------------------------------
@angular-devkit/architect         0.1000.7
@angular-devkit/build-angular     0.1000.7
@angular-devkit/build-optimizer   0.1000.7
@angular-devkit/build-webpack     0.1000.7
@angular-devkit/core              10.0.7
@angular-devkit/schematics        10.0.7
@angular/cli                      10.0.7
@ngtools/webpack                  10.0.7
@schematics/angular               10.0.7
@schematics/update                0.1000.7
rxjs                              6.5.5
typescript                        3.9.7
webpack                           4.43.0

[smzelek]

I believe that Issue #38373 may also be related to this regression, which I guess could have been caused when the cached dependencies in tests feature was added.

[AndrewKushnir]

Hi @smzelek,

Thanks for providing an example that illustrates the setup of the described use-case. This is indeed a known difference between View Engine and Ivy - in View Engine, components are compiled in the context of some environment - either a platform browser instance, or a test. Therefore each test must declare its own testing module and import what it wants to compile. In Ivy, there is only one compiled version of a component at any time, and this compilation happens independently of the TestBed. Therefore TestBed.configureTestingModule is not necessary if there are no changes being made to the component's "production" configuration.

Currently we are in a transition period where both View Engine and Ivy renderers exist in the codebase and are available to developers. In order to provide backwards compatibility for most common use-cases, we retained existing TestBed API semantics (which are designed around View Engine's behavior) and made it compatible with Ivy. Once View Engine is removed from the codebase, we plan to revisit the TestBed API and redesign it to better reflect Ivy semantics.

Thank you.

[smzelek]

@AndrewKushnir Thank you for your detailed response! It would be great if there was any mention of this in https://angular.io/guide/testing-components-basics or...anywhere, I guess. Is this documented somewhere that I don't know? We couldn't find anything about it.

Plus, I'm not sure you're right anyway. I have made another reproduction repo.
https://github.com/smzelek/minimal-repro-ng10-testbed-bug-2

When you run the app with npm start you will see that Component1 which depends on DependencyComponent is rendered. Both are declared in the AppModule, all is well.

Now, let's look at the tests. I have 3 tests in tests.spec.ts: 1, 2, and 3. Whether these tests pass or fail completely depends upon the order that you run the tests in. This is... really unfortunate behavior to say the least. Test 1 declares no module configuration. Test 2 only declares the DependencyComponent. Test 3 declares Component1 and the DependencyComponent.

From your answer, one would expect all three tests to pass. This is not true. When you run them individually, Test 1 fails, Test 2 fails, and Test 3 passes.

In another regression, none of these failing tests correctly complain about their missing dependencies. They successfully render the root component of the Test (Component1), the <app-dependency-component> tag is rendered with no content. I suppose you could call this behavior "graceful" if it wasn't so obfuscating for developers trying to debug a failing test.

The real trouble comes when you run the tests together as a suite, because their pass/fail completely depends upon order, like so....

// 1 2 3 -> XX✓
// 2 1 3 -> XX✓
// 1 3 2 -> X✓✓
// 2 3 1 -> X✓✓
// 3 2 1 -> ✓✓✓

In short, this should not be tagged as "state: works-as-expected". This entire behavior is mystifying for a developer.

[AndrewKushnir]

Hi @smzelek,

Thanks for providing a new repro, the described scenario is not handled by Ivy TestBed and this is a bug. I performed additional investigation and found out that there is a piece of state that is preserved between tests in case a component is present in the declarations field while calling TestBed.configureTestingModule in one test, but it's absent (or the whole TestBed.configureTestingModule call is omitted) in the next test. I've created a PR that should address this issue, see #38659. Running tests from your repro with the changes from the mentioned PR make them pass/fail consistently independent of the order.

You can also test the behavior using an updated version of the @angular/core package using this instruction and the core package from the mentioned PR.

Thank you.

[ahnpnl]

This is indeed a known difference between View Engine and Ivy - in View Engine, components are compiled in the context of some environment - either a platform browser instance, or a test. Therefore each test must declare its own testing module and import what it wants to compile. In Ivy, there is only one compiled version of a component at any time, and this compilation happens independently of the TestBed.

Hi @AndrewKushnir,

Sorry to jump in this because I need a bit help in understanding what TestBed does, for Ivy vs VE in combination with component compliation.

Does it mean in Ivy, component compilation is done somewhere else before tests execute while TestBed only set up test environment ? In VE, TestBed also does compilation ?

It would be very appreciate for your help to explain a bit more about that. I am working for jest-preset-angular and we are struggling with Ivy compatibility.

[AndrewKushnir]

Hi @ahnpnl,

Components might be pre-compiled (in AOT mode) before tests are executed by TestBed, in which case TestBed uses compiled components and only re-compile them (in JIT mode) when there is an override which can not be applied as a patch (i.e. without triggering a re-compilation), for example if a template is overwritten in a test. In View Engine's version of TestBed components were always re-compiled for each set of tests.

Looking at the issue that you refer to (thymikee/jest-preset-angular#409), I think it might make sense to open a new ticket and describe questions/issues that you have, so that the team can provide some additional insights.

Thank you.

[angular-automatic-lock-bot]

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{This action has been performed automatically by a bot.}