Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support baseUrl/path import aliases (and other newish ts/ts-node features) #8787

Open
tenwit opened this issue Jan 19, 2022 · 5 comments
Open
Labels
area/sdks Pulumi language SDKs impact/usability Something that impacts users' ability to use the product easily and intuitively kind/enhancement Improvements or new features language/javascript

Comments

@tenwit
Copy link

tenwit commented Jan 19, 2022

Hello!

  • Vote on this issue by adding a 馃憤 reaction
  • If you want to implement this feature, comment to let us know (we'll work with you on design, scheduling, etc.)

Issue details

Some of my typescript Pulumi codebases are monorepos, and suffer from relative path hell. Typescript provides the path mapping sugar to help with this.

I'd also like to use ESM throughout my codebases. ts-node supports this, though currently only via experimental settings.

Recent changes in Pulumi seem to allow supporting these features much simpler (e.g. pulumi/templates#244, #8655), though it might not be possible yet (#4876).

Requests:

Affected area/feature

Typescript support.

@tenwit tenwit added the kind/enhancement Improvements or new features label Jan 19, 2022
@mikhailshilkov mikhailshilkov added area/sdks Pulumi language SDKs impact/usability Something that impacts users' ability to use the product easily and intuitively language/javascript labels Jan 20, 2022
@tenwit
Copy link
Author

tenwit commented May 26, 2022

Overlaps with #3061

@RobbieMcKinstry RobbieMcKinstry self-assigned this Jan 27, 2023
@RobbieMcKinstry
Copy link
Contributor

Hello @tenwit ! Coming across this issue just now, and I wanted to give a little more context into how I'm currently thinking about this issue and its blockers.

I'm reading this issue as having two main feature requests (plus documentation):

  1. Can users pick a better version of TS-Node?
  2. Can Pulumi improve support for monorepos by allowing them to use one tsconfig file at the project root?
  3. More generally, can users have greater control over how the runtime executes their programs (via Allow passing configuration/arguments to underlying runtime engine聽#3061)?

TL;DR: IMO these three items are separate requests. I also think these items are tracked in other open issues. This is really exciting for me, because it indicates that these are features people really want, if users are independently arriving at the same conclusions. I'm going to type up an explanation of the state of the world for the features as described above.

More Flexible Dependencies

Can new ts/ts-node features be turned on through suitable configuration of Pulumi.yaml

This one has been on my radar for six months now, and I really want to improve things here. 馃 The short answer is "no, that's currently a breaking change, but I wish it weren't so!". The long answer is under discussion here. I'm hopeful we can come up with a slick solution. For example:

  1. Import the current version of ts-node under an alias.
  2. Use a dynamic import to see if the user has ts-node installed as a peer dependency. If so, use the version provided by the user.
  3. If not, fallback to the aliased version.

Another approach that's rattled around in my mind, but I don't have a good design for, is some kind of "lifecycle hooks" for the Node runtime. A lot of the motivation for upgrading TS-Node and TypeScript versions is using one's own typechecker or transpiler for faster builds. I'd like to consider adding a configuration option to the TypeScript runtime that allows users to specify their own build step. For example, putting this in your Pulumi.yaml file:

runtime:
  name: nodejs
  options:
    typescript: true
    buildStep: build

...where buildStep: myCustomBuildStep means "look in my package.json file for a script named myCustomBuildStep and use that instead of TS-Node."

Better Monorepo Support

Typescript provides the path mapping sugar to help with this.

RE: path mapping, I recently used path mapping in our generated Node code, but I came across this open bug in the TypeScript compiler which makes path mapping unusable in libraries. I suppose that as long as your Pulumi program as tsconfig available during runtime, you should still be able to use path mapping, right? I think you should be able to do this right now, though perhaps not in a monorepo with only one tsconfig file above the Pulumi project in the directory hierarchy. However, we're actively working on #2619 which will fix this.

I expect once #2619 is closed, you'll be able to use path mapping without issue. Does that sound right to you?

Pass Args to Runtimes

Overlaps with #3061

Right now, the Pulumi engine executes a Go binary called the "language host", which is responsible for spinning up each "language runtime" and providing the runtime with the appropriate flags. The good news is that way flags are passed between the engine => lang host and lang host => lang runtime is pretty well defined, so I don't think this issue has any obvious blockers. I think it'd be relatively straightforward to implement, but it requires a degree of design work, since it's not obvious to me what the CLI UX would look like for passing those flags. My point of reference is Cargo, which does something like this:

$ cargo run -- custom args --and -subcommands=true

The bad news is that this requires plumbing into all six of our language runtimes: NodeJS, Go, Python, Dotnet, YAML, and Java. That makes a little daunting of a task for one person, but because it's parallelizable and support for each language can be implemented in isolation of the others, maybe it's something we can collaborate on with a community contributor.


If I understand your ask correctly, this issue is tracked in #10122, #2619, and #3061. Would it be fair to close this issue as a duplicate of those?

I'm happy to continue the conversation here or in those issues. I'm actively working on #2619 , passively working on #10122, and unfortunately #3061 is backlogged for now.

@RobbieMcKinstry RobbieMcKinstry added the awaiting-feedback Blocked on input from the author label Jan 27, 2023
@cody2094
Copy link

cody2094 commented Mar 3, 2023

EDIT - I am using pulumi version 3.33.2

@RobbieMcKinstry So I'm currently trying to figure out an issue around the following :

Better Monorepo Support
Typescript provides the path mapping sugar to help with this.

RE: path mapping, I recently used path mapping in our generated Node code, but I came across microsoft/TypeScript#25677 in the TypeScript compiler which makes path mapping unusable in libraries. I suppose that as long as your Pulumi program as tsconfig available during runtime, you should still be able to use path mapping, right? I think you should be able to do this right now, though perhaps not in a monorepo with only one tsconfig file above the Pulumi project in the directory hierarchy. However, we're actively working on #2619 which will fix this.

I expect once #2619 is closed, you'll be able to use path mapping without issue.

I have a monorepo setup with NX, but we don't use one central tsconfig/package.json file. We have a tsconfig file defined in the root, and then in our pulumi directories we extend our base tsconfig file. I'm having trouble making use of path mapping in my pulumi program. When I'm in the code, Typescript/ESLint seems to be happy with my path mapping reference (import * from @mypackage/infra, however, when i run pulumi up I get an exception : Error: Cannot find module '@mypackage/infra'.

My repo setup is :

~
 - tsconfig.base.json
 - libs
   - infra
     - src
       - index.ts 
 - apps
   - pulumi-app-1
     - tsconfig.json 
     - src
       - index.ts

And my tsconfig.base.json file has :

....
"compilerOptions": {
    ....
    "baseUrl": ".",
    "paths": {
       "@mypackage/infra": ["libs/infra/src/index.ts"]
    ....

My Pulumi project has the following tsconfig :

{
  "extends": "../../../tsconfig.base.json",
  "compilerOptions": {
    "outDir": "./bin",
    "types": ["node"],  },
  "include": ["src/**/*.ts", "src/**/*.tsx"]
}

I've attempted moving the path mapping for mypackage to the pulumi apps tsconfig file to no avail there either. Other apps seem to recognize my packages through this way, so it seems like something that Pulumi isn't doing correctly. Here is the full exception :

 error: Running program '/path/to/repo/apps/infra-api/ecs' failed with an unhandled exception:
    Error: Cannot find module '@mypackage/infra'
    Require stack:
    - /path/to/repo/apps/infra-api/ecs/src/index.ts
    - /path/to/repo/node_modules/@pulumi/pulumi/cmd/run/run.js
    - /path/to/repo/node_modules/@pulumi/pulumi/cmd/run/index.js
        at Function.Module._resolveFilename (node:internal/modules/cjs/loader:933:15)
        at Function.Module._load (node:internal/modules/cjs/loader:778:27)
        at Module.require (node:internal/modules/cjs/loader:1005:19)
        at require (node:internal/modules/cjs/helpers:102:18)
        at Object.<anonymous> (/path/to/repo/apps/infra-api/ecs/src/index.ts:4:1)
        at Module._compile (node:internal/modules/cjs/loader:1101:14)
        at Module.m._compile (/path/to/repo/node_modules/@pulumi/pulumi/node_modules/ts-node/src/index.ts:439:23)
        at Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
        at Object.require.extensions.<computed> [as .ts] (/path/to/repo/node_modules/@pulumi/pulumi/node_modules/ts-node/src/index.ts:442:12)
        at Module.load (node:internal/modules/cjs/loader:981:32)

@omercnet
Copy link

would love to see support for paths
it works when using pulumi automations, but not when running the project directly with pulumi

@cnunciato cnunciato removed the awaiting-feedback Blocked on input from the author label Mar 6, 2024
@cnunciato
Copy link
Member

cnunciato commented Mar 6, 2024

Removed the awaiting-feedback label as we probably don't need to wait for feedback before at least investigating this issue. (I just hit it myself looking into pulumi/pulumi-hugo#1504.)

I expect once #2619 is closed, you'll be able to use path mapping without issue.

#2619 has been closed, but this is indeed still an issue. Using paths does not work when running either pulumi up or with a local program using stack.up() with Automation API. (I still get MODULE_NOT_FOUND, despite the TypeScript compiler being happy.)

{
    "compilerOptions": {
        "strict": true,
        "outDir": "bin",
        "target": "es2016",
        "module": "commonjs",
        "moduleResolution": "node",
        "sourceMap": true,
        "experimentalDecorators": true,
        "pretty": true,
        "noFallthroughCasesInSwitch": true,
        "noImplicitReturns": true,
        "forceConsistentCasingInFileNames": true,
        "baseUrl": "../",
        "paths": {
            "myapp": [
                "app"
            ]
        }
    },
    "files": [
        "index.ts"
    ]
}
> start
> ts-node index.ts

Error: Cannot find module 'myapp'
Require stack:
- /Users/cnunciato/Projects/dev/pulumi/aws-typescript-38c78f1/runner/index.ts
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:1048:15)
    at Function.Module._resolveFilename.sharedData.moduleResolveFilenameHook.installedValue [as _resolveFilename] (/Users/cnunciato/Projects/dev/pulumi/aws-typescript-38c78f1/runner/node_modules/@cspotcode/source-map-support/source-map-support.js:811:30)
    at Function.Module._load (node:internal/modules/cjs/loader:901:27)
    at Module.require (node:internal/modules/cjs/loader:1115:19)
    at require (node:internal/modules/helpers:130:18)
    at Object.<anonymous> (/Users/cnunciato/Projects/dev/pulumi/aws-typescript-38c78f1/runner/index.ts:4:1)
    at Module._compile (node:internal/modules/cjs/loader:1241:14)
    at Module.m._compile (/Users/cnunciato/Projects/dev/pulumi/aws-typescript-38c78f1/runner/node_modules/ts-node/src/index.ts:1618:23)
    at Module._extensions..js (node:internal/modules/cjs/loader:1295:10)
    at Object.require.extensions.<computed> [as .ts] (/Users/cnunciato/Projects/dev/pulumi/aws-typescript-38c78f1/runner/node_modules/ts-node/src/index.ts:1621:12) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/Users/cnunciato/Projects/dev/pulumi/aws-typescript-38c78f1/runner/index.ts'
  ]
}

The current workaround (via #3061) seems to be to use tsconfig-paths, either by adding a ts-node directive to tsconfig.json (which is all that's needed for an Automation API program):

{
    "compilerOptions": {
        "strict": true,
        "outDir": "bin",
        "target": "es2016",
        "module": "commonjs",
        "moduleResolution": "node",
        "sourceMap": true,
        "experimentalDecorators": true,
        "pretty": true,
        "noFallthroughCasesInSwitch": true,
        "noImplicitReturns": true,
        "forceConsistentCasingInFileNames": true,

        // 馃憞馃憞馃憞
        "baseUrl": "../",
        "paths": {
            "my-alias": [
                "./my-local-path"
            ]
        }
    },
    "files": [
        "index.ts"
    ],

    // 馃憞馃憞馃憞
    "ts-node": {
        "require": ["tsconfig-paths/register"]
    }
}

...or adding a block to the TypeScript program entrypoint (e.g., index.ts) to apply the baseUrl and paths settings:

import * as tsConfigPaths from "tsconfig-paths";
const config = tsConfigPaths.loadConfig("./");

if (config.resultType === "success") {
    tsConfigPaths.register({
        baseUrl: config.absoluteBaseUrl,
        paths: config.paths,
    });
}

// ... the rest of your index.ts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/sdks Pulumi language SDKs impact/usability Something that impacts users' ability to use the product easily and intuitively kind/enhancement Improvements or new features language/javascript
Projects
Status: backlog
Development

No branches or pull requests

6 participants