Module-oriented test-utils for module authors

[lihbr]

Overview

This document presents additions/feature requests to @nuxt/test-utils. They are aimed at simplifying module (unit) testing for module authors, hopefully incentivizing modules to be tested more and allowing for fewer regressions through the ecosystem in the long run.

Background Information

Testing a Nuxt application is not a trivial task as you need a lot of systems to be running correctly in order to do so. To ease that process the Nuxt team made available @nuxt/test-utils in 2020: https://github.com/nuxt/test-utils (afaik, as a rebrand of @nuxt/module-test-utils by @ricardogobbosouza created in 2019: https://github.com/nuxt/test-utils/tree/1.x)

@nuxt/test-utils has been maintained alongside Nuxt 2 since then. It focuses on end-to-end (e2e) testing, basically launching a test Nuxt application (a.K.a. fixture) and allowing you to interact with it through a minimal API and playwright.

With Nuxt 3, @nuxt/test-utils was moved to the framework repository, allowing for tighter integration with the framework's core: https://github.com/nuxt/framework/tree/main/packages/test-utils As of writing this, Nuxt 3 test utils are mainly a port of Nuxt 2's to the new framework, providing roughly the same features, and focus on end-to-end testing.

This focus on end-to-end testing is great as it represents really the hard part of testing Nuxt applications. However, it is draining when it comes to unit-testing applications:

• From the performance perspective, it impacts pretty much anyone: Nuxt test utils are "slow" for unit tests as they, by design, have to launch the Full Real Deal™, but you don't want to launch the Full Real Deal™ each time in a unit test context;
• From the DX perspective, it impacts mainly module authors (I think? Assuming they are the ones that really need to unit test their module with different Nuxt-related test scenarios), forcing them to create a fixture for almost every test scenario they have.

As the author of @nuxtjs/prismic module for Nuxt 3 (the only module I'm authoring at the moment for Nuxt 3) I wanted to test my module and ultimately faced the above issues. Instead of writing endless fixtures for that sake, I tried to explore alternative approaches with a focus on performance, test relevance, and DX. This work is available here: https://github.com/nuxt-community/prismic-module/tree/v3/test

Proposal

Please bear in mind that while I'm familiar with writing tests, I'm by far not an expert in doing it and pretty ignorant of advanced/powerful test patterns.

The following users stories are the ones I'd like to be addressed with this proposal:

1. As a module author, I want to run my module inside unit tests
2. As a module author, I want to test my module with different Nuxt configs
   A module can behave differently based on its own config or the overall Nuxt config
3. As a module author, I want to test my module's impact on Nuxt config
   A module can alter Nuxt config in pretty much any way
4. As a module author, I want to make sure my module performs the actions it is supposed to perform correctly
   It's none of the module's business if utils from @nuxt/kit indeed perform what they say they perform, however, it's important for the module to call them correctly/when appropriate

Providing a "runner for Nuxt modules"

The idea with a runner would be to allow modules to be run inside tests easily. Developers would then be able to assert if they performed the expected tasks. Such a runner would allow developers to run the module with different Nuxt config and module options sets.

Here's a hypothetical interface the runner could sport:

type UseModule<TModule extends NuxtModule>(
	module: TModule,
): {
	run: (
		moduleOptions?: Awaited<ReturnType<Required<TModule>["getOptions"]>>
		nuxtConfig?: NuxtOptions;
	) => Promise<{ nuxt: Nuxt }>
};

While technically, developers could just provide a Nuxt config embedding the module options, I'd vote to keep a pattern to only provide such module options and no Nuxt config as it's (in my opinion) quite a common pattern to just test the module with different options & same default Nuxt config.

Example usage:

import { it, expect } from "vitest";
import { useModule } from "@nuxt/test-utils";

import myModule from "../src/module";

const { run: runModule } = useModule(myModule)

it("does something when SSR is disabled", async () => {
	const { nuxt } = await runModule({ foo: "bar" }, { ssr: false });

	expect(...).toHaveBeenCalled();
	expect(nuxt.options.build.whatever).toBe("bar");
	/* ... */
});

This is currently hard to achieve because the defineNuxtModule() function from @nuxt/kit, while returning a function, relies on Nuxt internals/context to be available when called (that's the main issue).

I worked that around in my tests by creating some sort of a minimal module runner of mine, and injecting it as a mock of the defineNuxtModule(): https://github.com/nuxt-community/prismic-module/blob/v3/test/__testutils__/mockedNuxtKit.ts#L10-L31 (it works, but takes some shortcuts I was fine taking, like not caring about options merging much, only caring about the part of Nuxt config I was relying on, etc.)

That module runner would be more versatile:

• It would work off a regular NuxtModule function, not forcing users to mock defineNuxtModule() (working around the function internal requirements);
• It would provide a full default Nuxt config for the module to run with;
• It would perform some minimal merging of module options (nothing fancy, just enough not to create confusion/be confident)

I'd expect it, however, not to rely on anything file system-wise (e.g. not using C12 to create the default config). Returning the full mocked/dummy nuxt object would allow users to test hooks for example ("Does my module call this hook when Y?") as well as the module's impact on Nuxt config.

Mocking Nuxt kit

As stated in user story n°4 ("As a module author, I want to make sure my module performs the actions it is supposed to perform correctly"), users shouldn't need to test whether @nuxt/kit is working.

In my test, I worked that around thanks to Vitest powerful mock API here: https://github.com/nuxt-community/prismic-module/blob/v3/test/__testutils__/mockedNuxtKit.ts#L32-L42, allowing me to test that utils were called correctly depending on the context (example)

I'm unsure if there's a better pattern than that, but I'd like us to explore potential improvements we could provide through @nuxt/test-utils for that, if any (my lack of testing experience got me there ^^')

How it could be implemented

/

To Be Discussed

I'm not sure I'd have time/the knowledge to work on it myself yet, will see, but any guidance is very welcome!

How to provide feedback

If you would like to provide feedback, please reply to this discussion. As the above proposal is quite loose/high level, everything in it is open for discussion.

Thanks! ⛰️

[pi0]

Hi Lucie and thanks again for making detailed written RFC like always 💯

As you mentioned, testing Nuxt modules which are integrations is mostly end-to-end behavior testing. Since they can virtually extend any part of the Nuxt lifecycle, testing them with different configurations is necessary for a module author to ensure they work perfectly as well.

Currently, one can run setupTest multiple times with different configurations to ensure Nuxt app with module builds and works as expected with different setups. and thanks to the Vite and Vitest performances, the cost of running them is significantly less for Nuxt 3.

I think unit kind of testing of Modules with different configurations still makes sense as of this proposal. In scenarios, a Nuxt module is not extending builder or changing build behavior by different configs, we only need to unit test module effect on Nuxt interface and ensure it goes as expected.

Since Nuxt 3, we have a special mode called prepare enabled by _prepare flag. It is essentially a dry run that initializes the Nuxt instance and Modules, generates virtual templates and stops (not building cost). I believe this mode is ideal for unit testing modules because it remove build costs yet is identical (actually same) behavior of a normal Nuxt app without mocking duplicates.

To extend to mocking kit/nuxt idea, all kit utils are composables depending on unctx for Nuxt. Smartly there is one point to change that essentially mocks all kit utils: useNuxt(). It can return a proxy emulating simple Nuxt interface and hooks and enabled with setupMockedTest() (instead of setupTest()) I think this worth a try to try at least but we have to keep in mind and document it comes with it's own costs. For example not all hooks are available. Would be more than happy to discuss more about implementing this.