No error thrown on extensions not loading.

[u12206050]

Describe the Bug

After updating to 9.21.0 everything seemed to be fine, (ie. was able to build extensions etc) so we pushed into production, for about 72 hours now. Only now realised that none of the hooks (actions, filters) and custom endpoints have been working due to changes in the code structure. So even though there is a warning for each extension that couldn't load, the main app still works, but without any of the validation or custom logic implemented in code.

Believe it has something to do with #16374

To Reproduce

Try using import { Accountability } from '@directus/shared/types'; in a custom extension and starting directus.

Errors Shown

require() of ES Module /usr/node/app/node_modules/@directus/shared/node_modules/nanoid/index.js from /usr/node/app/node_modules/@directus/shared/dist/cjs/composables/use-custom-selection.js not supported.

What version of Directus are you using?

9.21.0

What version of Node.js are you using?

16

What database are you using?

mysql 8

What browser are you using?

chrome

How are you deploying Directus?

locally and gcp

[br41nslug]

Mind sharing a bit more specific reproduction steps to get to this situation? Just adding import { Accountability } from '@directus/shared/types'; is not enough to reproduce and considering the error is about the nanoid dependency that seems to come from a non-type import instead. I've been able to get similar errors when importing types in a JS only extension which makes sense as JS has no use for these types. However in a TS extension im unable to get reproduce this problem so far.

Also make sure the extension-sdk version is in sync with directus for your extension

[u12206050]

Here is one file that I had to update after installing 9.21.0

BEFORE:

import { BaseException } from '@directus/shared/exceptions';
export default class LogicError extends BaseException {
   constructor(name: string, message: string, code: number = 400) {
       super(message, code, name);
   }
}

AFTER:

import { BaseException } from '@directus/shared/dist/esm/exceptions';
export default class LogicError extends BaseException {
   constructor(name: string, message: string, code: number = 400) {
       super(message, code, name);
   }
}

Unfortunately I have a lot of extensions, so not sure exactly which one breaks, but had to do similar changes as above in all of them.

[paescuj]

I don't think that this is related to #16374. As you know from #16606 the "only" problem with this pull request is that the types aren't exposed anymore. While some type errors might be spit out during the build process of an extension, this has no effect on extension errors during runtime.

That being said, I'm able to reproduce the error you've mentioned above and I'm looking into this right now.

[u12206050]

But I got this error on existing extensions. Haven't created a new extension just yet.

[paescuj]

Situation

It seems like the issue is a result of "a series of unfortunate events":

1) Old create-directus-extension version

Starting with the fact that npm init directus-extension doesn't necessarily uses the latest version ¹ of the create-directus-extension package. To ensure that the latest version is being used, one might use npm init directus-extension@latest. When omitted, in my case, it was using version 9.3.0 (!) of create-directus-extension.

Note: This doesn't seem to be the case with pnpm create.

2) Externalized extensions-sdk dependency

Before version 9.5.0 resp. 9.5.1 (fixed with #11099 / #11329) the dependency extensions-sdk was marked as external which means if it has been used in an extension it was getting requireed in the resulting bundle (e.g. [...] require("@directus/extensions-sdk").defineHook [...]).

3) Upgrade to ESM version of nanoid

In version 9.17.0 (via #15094) the dependency nanoid in the shared package has been upgraded to version 4.0.0 which is ESM. This doesn't pose a problem per se, but in combination with the two facts from above.

Outcome

Due to 1) the chance is high that you (unknowingly) end up using an outdated version of create-directus-extension / extensions-sdk and as described in 2) if this version is < 9.5.1 then you will have that require in the bundle. In combination with a version >= 9.17.0 of the Directus api this means that the shipped version of extensions-sdk (which in turn imports the shared package) is requireed and this fails because of 3).

Similar things happen if you're using something like import { BaseException } from '@directus/shared/dist/esm/exceptions'; in your extension, because of this:

"@directus/shared/dist/esm/exceptions" is imported by "src/index.ts", but could not be resolved – treating it as an external dependency.

Conclusion

Short-term

First of all, please make sure you're using the latest version of extensions-sdk and don't forget to rebuild your extensions.
Secondly, change the import statement back to import { BaseException } from '@directus/shared/exceptions'; (your IDE should no longer complain with version 9.21.2 as the types are exposed again via #16641).

@u12206050 Have you made a bigger version jump? Otherwise, I don't see why these problems have suddenly appeared...

Long-term

• I suggest that we add a warning to create-directus-extension when npm init is using an outdated version, similar to the notification in create-directus-project.
  (Happy to submit a pull request for this)
• @nickrum What's the purpose of the "host" field in the package.json file of an extension? Could this be used to show a warning if it doesn't match the current version of the Directus api?

    "directus:extension": {
    	"type": "hook",
    	"path": "dist/index.js",
    	"source": "src/index.ts",
    	"host": "^9.21.2"
    },

• We might want to add the possibility to fail / shutdown the server when an extension couldn't be loaded, e.g. via an environment variable (REQUIRE_EXTENSION_SUCCESS, ...). However, this might be worth a separate discussion.

Footnotes

1. https://docs.npmjs.com/cli/v9/commands/npm-init#description, probably related: https://github.com/npm/cli/issues/2395 ↩

[u12206050]

No haven't been jumping versions. Just updated from 9.20.4

Will try 9.21.2

[paescuj]

No haven't been jumping versions. Just updated from 9.20.4

Will try 9.21.2

Okay, what was the version of extensions-sdk you were using then? And did you rebuild the extensions?

[u12206050]

9.20.4

[paescuj]

9.20.4

Hmm okay, in that case I don't know why it has worked before resp. why it stopped working after.
Does it work now with 9.21.2?

[u12206050]

The most seems to work, although I have one class that is extending on the BaseException
import { BaseException } from "@directus/shared/exceptions";

I get the following error

Class extends value undefined is not a constructor or null
    err: {
      "type": "TypeError",
      "message": "Class extends value undefined is not a constructor or null",

[u12206050]

Works if I don't extend on BaseException used to work before.

[paescuj]

Glad to hear that it's working so far!

I cannot reproduce this error on my end. Does it show up when you're running npm run build?

[u12206050]

No it succeeds to build but comes up when starting directus.

[paescuj]

Strange! And you're using the official extension template (npm init directus-extension@latest)? Any special configurations? No warnings either during npm run build?
Would you be able to share an excerpt of how you're using BaseException in your extension?

[rijkvanzanten]

Linear: ENG-127

[nickrum]

I suggest that we add a warning to create-directus-extension when npm init is using an outdated version, similar to the notification in create-directus-project.
(Happy to submit a pull request for this)

Great idea! I think I even had this on my todo list at some point. We might also want to update the docs recommend using npm init directus-extension@latest.

@nickrum What's the purpose of the "host" field in the package.json file of an extension? Could this be used to show a warning if it doesn't match the current version of the Directus api?

  "directus:extension": {
  	"type": "hook",
  	"path": "dist/index.js",
  	"source": "src/index.ts",
  	"host": "^9.21.2"
  },

That's exactly what it was meant for (I simply haven't had the time to implement it yet).

We might want to add the possibility to fail / shutdown the server when an extension couldn't be loaded, e.g. via an environment variable (REQUIRE_EXTENSION_SUCCESS, ...). However, this might be worth a separate discussion.

Another great idea 👍

[WoLfulus]

We might want to add the possibility to fail / shutdown the server when an extension couldn't be loaded, e.g. via an environment variable (REQUIRE_EXTENSION_SUCCESS, ...). However, this might be worth a separate discussion.

I've been doing this through monkey patches. TBH I think this should be the default behavior in production, because if it's a core extension (for example something that processes events and does additional work) it does a lot of damage if they are not loaded and Directus still serves the requests.

[u12206050]

Fully agree, in gcp I added an alert to monitor for such error logs as extensions not loading so I can at least roll back asap