Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cli: overrideTemplate is stored in working directory rather than assembly directory (i.e. import does not work) #22928

Closed
saltman424 opened this issue Nov 16, 2022 · 8 comments · Fixed by #29830
Assignees
Labels
bug This issue is a bug. effort/small Small work item – less than a day of effort p1 package/tools Related to AWS CDK Tools or CLI

Comments

@saltman424
Copy link
Contributor

saltman424 commented Nov 16, 2022

Describe the bug

Recently, I have had issues using cdk import that seem to stem from the overridden template being generated in my working directory rather than the assembly directory. Obviously, when deploy tries to use the generated template, it is expecting it to come from my assembly directory. Please see the Possible Solution section below for details.

Expected Behavior

cdk import should work without any issues

Current Behavior

Error message from cdk import:

[0%] start: Building and publishing 04926632678b4ce87690ab5dff6a51bed012b313a6dbc96b8be43c23aba2a6b1:current
[100%] fail: ENOENT: no such file or directory, open 'C:\Users\Me\my-repo\cdk.out\assembly-MyStage\MyStack1234ABCD.template.json-04926632678b4ce87690ab5dff6a51bed012b313a6dbc96b8be43c23aba2a6b1.yaml'

 ❌  MyStage/MyStack (MyStack) failed: Error: Failed to publish one or more assets. See the error messages above for more information.

Note that the MyStack1234ABCD.template.json-04926632678b4ce87690ab5dff6a51bed012b313a6dbc96b8be43c23aba2a6b1.yaml file is added to my working directory just before it gets to this stage of the process.

Reproduction Steps

I don't have a simple repo to reproduce the issue, but in the Possible Solution section below, I believe I identified the line of code that needs to be updated.

Possible Solution

I believe this line:

await fs.writeFile(templateFile, templateJson, { encoding: 'utf-8' });

Should be changed to:

 await fs.writeFile(path.join(stack.assembly.directory, templateFile), templateJson, { encoding: 'utf-8' }); 

Additional Information/Context

For some reason, I did not used to get this issue. I have been using cdk import since shortly after it was released and only recently did I start having issues. I update cdk regularly, so I don't know exactly what version introduced the issue, or if there is something different in my configuration that is causing the issue to surface, but I know that when I go back to 2.46.0 I still have the issue. Also, when I go into node_modules and manually update the line mentioned above, everything works as it is supposed to.

CDK CLI Version

2.50.0 (build 4c11af6)

Framework Version

2.50.0

Node.js Version

16.15.1

OS

Windows 10 Enterprise

Language

Typescript

Language Version

4.8.4

Other information

No response

@saltman424 saltman424 added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Nov 16, 2022
@github-actions github-actions bot added the package/tools Related to AWS CDK Tools or CLI label Nov 16, 2022
@peterwoodworth peterwoodworth added p1 effort/small Small work item – less than a day of effort and removed needs-triage This issue or PR still needs to be triaged. labels Nov 30, 2022
@peterwoodworth peterwoodworth added the needs-reproduction This issue needs reproduction. label Nov 30, 2022
@peterwoodworth
Copy link
Contributor

peterwoodworth commented Nov 30, 2022

Thanks for reporting this @saltman424,

I'm unsure what is causing you to run into this. I'm also unable to reproduce this. However, I wonder if it has something to do with the stack being in a stage, or if you're on windows. I'm going to leave this open, it would be really helpful if we have clear reproduction steps if anyone else runs into this in the future.

Thanks for the proposed solution by the way, if necessary we can use your suggestion 🙂

@fusion27
Copy link

fusion27 commented Dec 5, 2022

Experiencing this issue as well, I'm also on Windows 10. Running NodeJS v18.2.0

@tlrdstd
Copy link

tlrdstd commented Dec 9, 2022

I am also experiencing this issue, running on Amazon Linux 2. CDK version 2.53.0 (build 7690f43).

I confirm this only started recently - worked fine ~2 months ago, and without changing anything specific, it's stopped working.

My stacktrace below. I tried briefly to use the fix suggested above, but the error references a non-existent .ts file, and editing the .js equivalent at node_modules/aws-cdk/lib/api/deploy-stack.js didn't seem to help. So I just created a symlink from the package build directory back up to my current working directory where the override template was incorrectly created:

ln -s $PWD/AmiBuilderBaseStack.template.json-ffefca5f96d16aad7d75e248477cf23371def5c11773d7e4bc776c297f618f66.yaml /workplace/me/MyPackageCDK/src/MyPackageCDK/build/cdk.out/AmiBuilderBaseStack.template.json-ffefca5f96d16aad7d75e248477cf23371def5c11773d7e4bc776c297f618f66.yaml
AmiBuilderBaseStack
AmiBuilderBaseStack/RegionalRpmBucket/RegionalRpmBucket-RpmBucket/Resource (AWS::S3::Bucket): import with BucketName=my-regional-bucket-name (yes/no) [default: yes]? y
AmiBuilderBaseStack/RegionalRpmBucket/RegionalRpmBucket-RpmBucket/Policy/Resource: unsupported resource type AWS::S3::BucketPolicy, skipping import.
AmiBuilderBaseStack: importing resources into stack...
[0%] start: Building and publishing ffefca5f96d16aad7d75e248477cf23371def5c11773d7e4bc776c297f618f66:current
[100%] fail: ENOENT: no such file or directory, open '/workplace/me/MyPackageCDK/src/MyPackageCDK/build/cdk.out/AmiBuilderBaseStack.template.json-ffefca5f96d16aad7d75e248477cf23371def5c11773d7e4bc776c297f618f66.yaml'

 ❌  AmiBuilderBaseStack failed: Error: Failed to publish one or more assets. See the error messages above for more information.
    at publishAssets (/workplace/me/MyPackageCDK/src/MyPackageCDK/node_modules/aws-cdk/lib/util/asset-publishing.ts:60:11)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at deployStack (/workplace/me/MyPackageCDK/src/MyPackageCDK/node_modules/aws-cdk/lib/api/deploy-stack.ts:297:3)
    at ResourceImporter.importResources (/workplace/me/MyPackageCDK/src/MyPackageCDK/node_modules/aws-cdk/lib/import.ts:130:22)
    at CdkToolkit.import (/workplace/me/MyPackageCDK/src/MyPackageCDK/node_modules/aws-cdk/lib/cdk-toolkit.ts:485:5)
    at initCommandLine (/workplace/me/MyPackageCDK/src/MyPackageCDK/node_modules/aws-cdk/lib/cli.ts:364:12)

@saltman424
Copy link
Contributor Author

@tlrdstd in order to update it in node_modules you actually need to update it in node_modules/aws-cdk/lib/index.js since that seems to be a bundled file. In my current version, the line that needs to be modified is 943611. You can search for if (overrideTemplate) {.

@peterwoodworth is it possible that CDK previously changed the working directory to the assembly directory, but is no longer doing so?

@peterwoodworth peterwoodworth removed the needs-reproduction This issue needs reproduction. label Dec 9, 2022
kzym-w added a commit to kzym-w/aws-cdk that referenced this issue Dec 26, 2022
@turacma
Copy link

turacma commented Dec 1, 2023

I ran into this issue as well, though it was the first time using cdk import so I cannot speak to any change in behavior. Not sure if it's relevant or similar for others on the thread, but our cdk projects contain multiple stacks (for different stages of a release pipeline).

@alFReD-NSH
Copy link

I had the same issue as well and the mentioned fix worked for me as well.

@thake
Copy link

thake commented Feb 6, 2024

Ran into the same issue. I called it using the following arguments:
npx cdk import my-stack -c environment=develop -f. CDK version was 2.125.0.

@bergjaak bergjaak self-assigned this May 8, 2024
@mergify mergify bot closed this as completed in #29830 May 10, 2024
mergify bot pushed a commit that referenced this issue May 10, 2024
…folder (#29830)

### Issue # (if applicable)

Closes #22928 #22530 

I'm reopening as the original was closed due to inactivity.

### Reason for this change

When the template is synthesized, if it is larger than 50kb and cannot be inlined in the CFN request, CLI writes it to the filesystem to be uploaded to S3. Unlike regular deployments `import` overrides the template, when it does so the code responsible for writing the large template to the filesystem writes to the project root. 

This code reproduces the issue:
```
import * as cdk from "aws-cdk-lib";
import { Construct } from "constructs";
import * as sqs from "aws-cdk-lib/aws-sqs";
import * as iam from "aws-cdk-lib/aws-iam";

export class AwsCdkImportBugStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    for (let i = 0; i < 70; i++) {
      const q = new sqs.Queue(this, `AwsCdkImportBugQueue${i}`, {
        visibilityTimeout: cdk.Duration.seconds(300),
        enforceSSL: true,
      });
    }

    // new sqs.Queue(this, `AwsCdkImportBugQueue`, {
    //   visibilityTimeout: cdk.Duration.seconds(300),
    //   enforceSSL: true,
    //   removalPolicy: cdk.RemovalPolicy.RETAIN,
    // });
  }
}

```

Deploy the stack, uncomment the queue, and try to import. This error happens: 
```
$ npx cdk import                                  
AwsCdkImportBugStack
AwsCdkImportBugStack/AwsCdkImportBugQueue/Resource (AWS::SQS::Queue): enter QueueUrl (empty to skip): https://sqs.eu-central-1.amazonaws.com/<cut>/AwsCdkBugStack-keep186A2ECA-IznVNwytuY1b
AwsCdkImportBugStack/AwsCdkImportBugQueue/Policy/Resource (AWS::SQS::QueuePolicy): enter Id (empty to skip): 
Skipping import of AwsCdkImportBugStack/AwsCdkImportBugQueue/Policy/Resource
AwsCdkImportBugStack: importing resources into stack...
[0%] start: Publishing 06a2c6415fd2982e3285e9d615e5f9b99d1d795a9064479fbe6b88455ac62f6c:current
[100%] fail: ENOENT: no such file or directory, open '/home/nburtsev/projects/aws-cdk-import-bug/cdk.out/AwsCdkImportBugStack.template.json-06a2c6415fd2982e3285e9d615e5f9b99d1d795a9064479fbe6b88455ac62f6c.yaml'

 ❌  AwsCdkImportBugStack failed: Error: Failed to publish one or more assets. See the error messages above for more information.
    at publishAssets (/home/nburtsev/projects/aws-cdk-import-bug/node_modules/aws-cdk/lib/index.js:420:84876)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async deployStack (/home/nburtsev/projects/aws-cdk-import-bug/node_modules/aws-cdk/lib/index.js:430:391)
    at async ResourceImporter.importResources (/home/nburtsev/projects/aws-cdk-import-bug/node_modules/aws-cdk/lib/index.js:433:171694)
    at async ResourceImporter.importResourcesFromMap (/home/nburtsev/projects/aws-cdk-import-bug/node_modules/aws-cdk/lib/index.js:433:171357)
    at async CdkToolkit.import (/home/nburtsev/projects/aws-cdk-import-bug/node_modules/aws-cdk/lib/index.js:433:205058)
    at async exec4 (/home/nburtsev/projects/aws-cdk-import-bug/node_modules/aws-cdk/lib/index.js:488:54378)

Failed to publish one or more assets. See the error messages above for more information.
```

The file is created, it's just in the project root not in cdk.out:
```
~/projects/aws-cdk-import-bug $ ls -la         
total 284
drwxr-xr-x   7 nburtsev users   4096 Mar 22 10:48 .
drwxr-xr-x   8 nburtsev users   4096 Mar 22 10:11 ..
-rw-r--r--   1 nburtsev users  64131 Mar 22 10:48 AwsCdkImportBugStack.template.json-06a2c6415fd2982e3285e9d615e5f9b99d1d795a9064479fbe6b88455ac62f6c.yaml
drwxr-xr-x   2 nburtsev users   4096 Mar 22 10:12 bin
-rw-r--r--   1 nburtsev users   3185 Mar 22 10:12 cdk.json
drwxr-xr-x   2 nburtsev users   4096 Mar 22 10:48 cdk.out
<cut>
```

### Description of changes

I've just added a path.join similar to how it is done in regular template generation.

### Description of how you validated changes

I've added a test case but I believe tests are broken globally as per @TheRealAmazonKendra 

### Checklist
- [x] My code adheres to the [CONTRIBUTING GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and [DESIGN GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md)

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue is a bug. effort/small Small work item – less than a day of effort p1 package/tools Related to AWS CDK Tools or CLI
Projects
None yet
Development

Successfully merging a pull request may close this issue.

9 participants