Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support serialising reserved NodeJS identifiers #15879

Merged
merged 1 commit into from
Apr 10, 2024
Merged

Conversation

lunaris
Copy link
Contributor

@lunaris lunaris commented Apr 9, 2024

This commit addresses part of #11942, in which we fail to serialise closures whose code would use reserved identifiers like exports. This is due to a change we made where module imports are hoisted to avoid importing the same module multiple times. Previously, code adopted a strategy of passing all dependencies using a with statement, viz.:

function x() {
  return (function () {
    with({ fooBarBaz: require("foo/bar/baz"), ... }) {
      // Use of fooBarBaz
    }
  }).apply(...)
}

function y() {
  return (function () {
    with({ fooBarBaz: require("foo/bar/baz"), ... }) {
      // Use of fooBarBaz
    }
  }).apply(...)
}

This was changed to remove the duplicate imports, yielding code like:

const fooBarBaz = require("foo/bar/baz")

function x() {
  return (function () {
    with({ ... }) {
      // Use of fooBarBaz
    }
  }).apply(...)
}

function y() {
  return (function () {
    with({ ... }) {
      // Use of fooBarBaz
    }
  }).apply(...)
}

However, while the previous approach would work with reserved identifiers such as exports (with({ exports, ... }) { ... } is perfectly acceptable), the new one does not (const exports = ... is not acceptable since NodeJS will not allow redeclaration of the exports global).

This commit combines the two approaches. Modules are only imported once, but if an import would use a reserved identifier, we generate a fresh non-conflicting identifier and alias this using a with statement. For example:

const fooBarBaz = require("foo/bar/baz")

// __pulumi_closure_import_exports is generated to avoid shadowing the reserved "exports"
const __pulumi_closure_import_exports = require("some/other/module")

function x() {
  return (function () {
    with({ exports: __pulumi_closure_import_exports, ... }) {
      // Use of fooBarBaz and exports
    }
  }).apply(...)
}

Note that it is not expected that #11942 will be solved in its entirety. While this commit fixes code that introduces identifiers like exports, the introduction in question in that issue is caused by the use of pulumi.output(...) in the constructor of a dynamic resource provider. Since dynamic resource providers are implemented under the hood by serialising their code, we attempt to serialise pulumi.output and its dependency chain. This commit allows us to make more progress in that regard, but other things go wrong thereafter. Fixing the deeper issue that underpins #11942 (and likely other challenges with dynamic providers) is probably a more involved piece of work.

@lunaris lunaris requested a review from a team as a code owner April 9, 2024 14:19
@pulumi-bot
Copy link
Contributor

pulumi-bot commented Apr 9, 2024

Changelog

[uncommitted] (2024-04-10)

Bug Fixes

  • [sdk/nodejs] Correctly serialise functions whose code would make use of reserved identifiers
    #15879

@lunaris lunaris force-pushed the wjones/11942 branch 2 times, most recently from e762940 to 0c505de Compare April 10, 2024 08:49
@lunaris lunaris force-pushed the wjones/11942 branch 2 times, most recently from 0ca8c79 to 8b89ebe Compare April 10, 2024 10:06
Copy link
Contributor

@julienp julienp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉

@lunaris lunaris added this pull request to the merge queue Apr 10, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Apr 10, 2024
This commit addresses part of #11942, in which we fail to serialise
closures whose code would use reserved identifiers like `exports`. This
is due to a change we made where module imports are hoisted to avoid
importing the same module multiple times. Previously, code adopted a
strategy of passing all dependencies using a `with` statement, viz.:

```typescript
function x() {
  return (function () {
    with({ fooBarBaz: require("foo/bar/baz"), ... }) {
      // Use of fooBarBaz
    }
  }).apply(...)
}

function y() {
  return (function () {
    with({ fooBarBaz: require("foo/bar/baz"), ... }) {
      // Use of fooBarBaz
    }
  }).apply(...)
}
```

This was changed to remove the duplicate imports, yielding code like:

```typescript
const fooBarBaz = require("foo/bar/baz")

function x() {
  return (function () {
    with({ ... }) {
      // Use of fooBarBaz
    }
  }).apply(...)
}

function y() {
  return (function () {
    with({ ... }) {
      // Use of fooBarBaz
    }
  }).apply(...)
}
```

However, while the previous approach would work with reserved
identifiers such as `exports` (`with({ exports, ... }) { ... }` is
perfectly acceptable), the new one does not (`const exports = ...` is
not acceptable since NodeJS will not allow redeclaration of the
`exports` global).

This commit combines the two approaches. Modules are only imported once,
but if an import would use a reserved identifier, we generate a fresh
non-conflicting identifier and alias this using a `with` statement. For
example:

```typescript
const fooBarBaz = require("foo/bar/baz")

// __pulumi_closure_import_exports is generated to avoid shadowing the
// reserved "exports"
const __pulumi_closure_import_exports = require("some/other/module")

function x() {
  return (function () {
    with({ exports: __pulumi_closure_import_exports, ... }) {
      // Use of fooBarBaz and exports
    }
  }).apply(...)
}
```

Note that it is not expected that #11942 will be solved in its entirety.
While this commit fixes code that introduces identifiers like `exports`,
the introduction in question in that issue is caused by the use of
`pulumi.output(...)` in the constructor of a dynamic resource provider.
Since dynamic resource providers are implemented under the hood by
serialising their code, we attempt to serialise `pulumi.output` and its
dependency chain. This commit allows us to make more progress in that
regard, but other things go wrong thereafter. Fixing the deeper issue
that underpins #11942 (and likely other challenges with dynamic
providers) is probably a more involved piece of work.
@lunaris lunaris added this pull request to the merge queue Apr 10, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Apr 10, 2024
@lunaris lunaris added this pull request to the merge queue Apr 10, 2024
Merged via the queue into master with commit 0a6876c Apr 10, 2024
49 checks passed
@lunaris lunaris deleted the wjones/11942 branch April 10, 2024 16:33
github-merge-queue bot pushed a commit that referenced this pull request Apr 11, 2024
Tentative changelog:


### Features

- [cli] Support always fully qualifying stack names in CLI output
  [#15857](#15857)

- [cli] Add --no-plugins and --no-dependencies to `pulumi install`
  [#15910](#15910)

- [docs] Implement Java constructor syntax examples
  [#15805](#15805)

- [sdk/nodejs] Make TypeScript and ts-node optional peer dependencies to
allow using user specified versions
  [#15622](#15622)

- [sdk/{nodejs,python}] Allow apply to have unknown values during
updates
  [#15898](#15898)

- [sdk/python] Adds 'typeChecker' runtime option to the Python language
host
  [#15725](#15725)


### Bug Fixes

- [auto] Tolerate missing stack and bookkeeping files in ProgramTest.
  [#15863](#15863)

- [cli] Fix a panic when user's home directory could not be looked up
  [#15872](#15872)

- [cli] Fix some commands that didn't respect
`--disable-integrity-checking`

- [programgen/dotnet] Removes trailing whitespace from emitted DependsOn
resource option expressions
  [#15892](#15892)

- [auto/go] Avoid flakyness when reading the event log from pulumi
commands
  [#15856](#15856)

- [sdk/go] Fix Provider and Providers options in Go transform functions
  [#15885](#15885)

- [sdk/nodejs] Handle serialization of aliases for well known native
functions
  [#15873](#15873)

- [sdk/nodejs] Correctly serialise functions whose code would make use
of reserved identifiers
  [#15879](#15879)

- [sdk/nodejs] Serialize function values obtained from Function.bind
  [#15887](#15887)

- [sdk/python] Improve types of getters in Python SDK
  [#15865](#15865)

- [sdkgen/{dotnet,go}] Fixes SDK-generation when referencing shared
types in config variables
  [#15772](#15772)


### Miscellaneous

- [sdk/nodejs] Update builtin module list for function serialization
  [#15830](#15830)

- [sdk/nodejs] Set package.json engines to node >= 18
  [#15845](#15845)
@justinvp justinvp mentioned this pull request Apr 11, 2024
github-merge-queue bot pushed a commit that referenced this pull request Apr 12, 2024
### Features

- [cli] Support always fully qualifying stack names in CLI output
  [#15857](#15857)

- [cli] Add --no-plugins and --no-dependencies to `pulumi install`
  [#15910](#15910)

- [docs] Implement Java constructor syntax examples
  [#15805](#15805)

- [sdk/nodejs] Make TypeScript and ts-node optional peer dependencies to
allow using user specified versions
  [#15622](#15622)

- [sdk/{nodejs,python}] Allow apply to have unknown values during
updates
  [#15898](#15898)

- [sdk/python] Adds 'typeChecker' runtime option to the Python language
host
  [#15725](#15725)


### Bug Fixes

- [auto] Tolerate missing stack and bookkeeping files in ProgramTest.
  [#15863](#15863)

- [cli] Fix a panic when user's home directory could not be looked up
  [#15872](#15872)

- [cli] Fix some commands that didn't respect
`--disable-integrity-checking`
  [#15911](#15911)

- [programgen/dotnet] Removes trailing whitespace from emitted DependsOn
resource option expressions
  [#15892](#15892)

- [auto/go] Avoid flakyness when reading the event log from pulumi
commands
  [#15856](#15856)

- [sdk/go] Fix Provider and Providers options in Go transform functions
  [#15885](#15885)

- [sdk/nodejs] Handle serialization of aliases for well known native
functions
  [#15873](#15873)

- [sdk/nodejs] Correctly serialise functions whose code would make use
of reserved identifiers
  [#15879](#15879)

- [sdk/nodejs] Serialize function values obtained from Function.bind
  [#15887](#15887)

- [sdk/python] Improve types of getters in Python SDK
  [#15865](#15865)

- [sdkgen/{dotnet,go}] Fixes SDK-generation when referencing shared
types in config variables
  [#15772](#15772)


### Miscellaneous

- [sdk/nodejs] Update builtin module list for function serialization
  [#15830](#15830)

- [sdk/nodejs] Set package.json engines to node >= 18
  [#15845](#15845)
@justinvp justinvp mentioned this pull request Apr 12, 2024
github-merge-queue bot pushed a commit that referenced this pull request Apr 15, 2024
### Features

- [auto/{go,nodejs,python}] Add support for continue-on-error parameter
of the destroy command to the Automation API
  [#15921](#15921)

- [cli] Support always fully qualifying stack names in CLI output
  [#15857](#15857)

- [cli] Add --no-plugins and --no-dependencies to `pulumi install`
  [#15910](#15910)

- [docs] Implement Java constructor syntax examples
  [#15805](#15805)

- [sdk/nodejs] Make TypeScript and ts-node optional peer dependencies to
allow using user specified versions
  [#15622](#15622)

- [sdk/{nodejs,python}] Allow apply to have unknown values during
updates
  [#15898](#15898)

- [sdk/python] Add 'typeChecker' runtime option to the Python language
host
  [#15922](#15922)
  [#15725](#15725)


### Bug Fixes

- [auto] Tolerate missing stack and bookkeeping files in ProgramTest
  [#15922](#15922)
  [#15863](#15863)

- [cli] Fix a panic when user's home directory could not be looked up
  [#15872](#15872)

- [cli] Fix some commands that didn't respect
`--disable-integrity-checking`
  [#15911](#15911)

- [programgen/dotnet] Remove trailing whitespace from emitted DependsOn
resource option expressions
  [#15922](#15922)
  [#15892](#15892)

- [auto/go] Avoid flakyness when reading the event log from pulumi
commands
  [#15856](#15856)

- [sdk/go] Fix Provider and Providers options in Go transform functions
  [#15885](#15885)

- [sdk/nodejs] Handle serialization of aliases for well known native
functions
  [#15873](#15873)

- [sdk/nodejs] Correctly serialise functions whose code would make use
of reserved identifiers
  [#15879](#15879)

- [sdk/nodejs] Serialize function values obtained from Function.bind
  [#15887](#15887)

- [sdk/python] Improve types of getters in Python SDK
  [#15865](#15865)

- [sdkgen/{dotnet,go}] Fix SDK-generation when referencing shared types
in config variables
  [#15772](#15772)
  [#15922](#15922)


### Miscellaneous

- [sdk/nodejs] Update builtin module list for function serialization
  [#15830](#15830)

- [sdk/nodejs] Set package.json engines to node >= 18
  [#15845](#15845)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants