feat(assets): support drop-in docker replacements by setting `$CDK_DOCKER` #21838

misterjoshua · 2022-08-30T23:56:50Z

This PR introduces the ability for users to change the command used for building and publishing Docker images to a drop-in replacement like the Depot Docker shim. To change the docker command, the user provides the alternative command via an environment variable named CDK_DOCKER.

There's already something similar in core's asset bundling, but it applies only to bundling and not to building and publishing Docker image assets.

aws-cdk/packages/@aws-cdk/core/lib/bundling.ts

Line 523 in 3e55092

const prog = process.env.CDK_DOCKER ?? 'docker';

Fixes #21836

All Submissions:

Have you followed the guidelines in our Contributing guide?

Adding new Unconventional Dependencies:

This PR adds new unconventional dependencies following the process described here

New Features

Have you added the new feature to an integration test?
- Did you use yarn integ to deploy the infrastructure and generate the snapshot (i.e. yarn integ without --dry-run)?

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

Depends-On: #22194

gitpod-io · 2022-08-30T23:56:58Z

Hi-Fi · 2022-08-31T05:22:41Z

How about getting that custom executable to replace docker to asset step's image(s)? Should custom image be used, some installation added to steps or what? I think this should be mentioned in documentation.

misterjoshua · 2022-08-31T17:40:45Z

How about getting that custom executable to replace docker to asset step's image(s)? Should custom image be used, some installation added to steps or what? I think this should be mentioned in documentation.

@Hi-Fi Thanks for your comments, but I need more clarification. Do you mean to suggest we include documentation on installing third-party tools alongside the AWS CDK and instructions on using those tools with the AWS CDK? i.e., a tutorial on installing and using Depot to build Docker image assets? (curl https://... | sh && env CDK_DOCKER=depot-shim cdk deploy)

Hi-Fi · 2022-08-31T18:01:28Z

How about getting that custom executable to replace docker to asset step's image(s)? Should custom image be used, some installation added to steps or what? I think this should be mentioned in documentation.

@Hi-Fi Thanks for your comments, but I need more clarification. Do you mean to suggest we include documentation on installing third-party tools alongside the AWS CDK and instructions on using those tools with the AWS CDK? i.e., a tutorial on installing and using Depot to build Docker image assets? (curl https://... | sh && env CDK_DOCKER=depot-shim cdk deploy)

As asset stage is a bit special in pipelines, and it's configuration is just one that is offered. So documentation should have recommendation if to use custom build image with executable included or add installation command to some assetPublishingCodeBuildDefaults.

Using cdk deploy is different thing, as with that there's no need to worry about stages. And in that I don't think it's needed to tell how to install tool needed.

misterjoshua · 2022-08-31T19:29:35Z

How about getting that custom executable to replace docker to asset step's image(s)? Should custom image be used, some installation added to steps or what? I think this should be mentioned in documentation.

@Hi-Fi Thanks for your comments, but I need more clarification. Do you mean to suggest we include documentation on installing third-party tools alongside the AWS CDK and instructions on using those tools with the AWS CDK? i.e., a tutorial on installing and using Depot to build Docker image assets? (curl https://... | sh && env CDK_DOCKER=depot-shim cdk deploy)

As asset stage is a bit special in pipelines, and it's configuration is just one that is offered. So documentation should have recommendation if to use custom build image with executable included or add installation command to some assetPublishingCodeBuildDefaults.

Using cdk deploy is different thing, as with that there's no need to worry about stages. And in that I don't think it's needed to tell how to install tool needed.

@Hi-Fi Thanks for clarifying! I'll take a crack at adding documentation to the CDK Pipelines README as well.

misterjoshua · 2022-09-06T18:15:41Z

@Mergifyio update

mergify · 2022-09-06T18:16:17Z

update

✅ Branch has been successfully updated

Hi-Fi · 2022-09-03T12:15:48Z

packages/@aws-cdk/pipelines/README.md

+  }),
+
+  // Configure CodeBuild to use a drop-in Docker replacement.
+  codeBuildDefaults: {


Better to use assetPublishingCodeBuildDefaults at all cases, as that's where assets are built.

True that the prop you mention applies only to the publishing steps and that's where docker image assets are built. But, I'm inclined to think that if a customer is trying to use a drop-in Docker replacement like the Depot Docker shim, they'd first think of using it throughout their CDK app, not just in the asset publishing stage.

For instance, it's not obvious to the caller that aws_lambda.Code.fromDockerBuild() builds a Docker image in synth and not during asset publishing - an example with assetPublishingCodeBuildDefaults fails to cover this case and could result in unexpected behaviour in this light.

That said, assetPublishingCodeBuildDefaults is still available if the customer has a preference.

rix0rrr

I think the CDK_DOCKER feature was mostly intended for testing, but I guess I don't mind supporting this. A couple of doc updates and then this is good to go.

(Sidebar: the more I see and learn about Docker, the less I like supporting it. It seems everyone has very specific opinions on how it should be used in their specific use cases, leading to a proliferation of flags and tweaks that are tough to maintain... 😔)

rix0rrr · 2022-09-07T08:30:05Z

packages/@aws-cdk/pipelines/README.md

+In CDK Pipelines, the drop-in replacement for the `docker` command must be
+included in the CodeBuild environment and configured for your pipeline:
+
+**Adding to the default CodeBuild image**


Suggested change

**Adding to the default CodeBuild image**

### Adding to the default CodeBuild image

rix0rrr · 2022-09-07T08:34:17Z

packages/@aws-cdk/pipelines/README.md

+});
+```
+
+**Using a custom build image**


👍 on including an example for this, but it feels repetitive and doesn't have a paragraph explaining what it is you're doing or why.

Can you add a descriptive paragraph, and cut out nonessentials from the example? For example, declare the source with declarse const source: CodePipelineSource, pass a symbolic commands array, that kind of thing. Let's try to leave only the most important parts to highlight them.

rix0rrr · 2022-09-07T08:34:30Z

packages/@aws-cdk/pipelines/README.md

+
+```ts
+const pipeline = new pipelines.CodePipeline(this, 'Pipeline', {
+  selfMutation: false,


Please don't recommend this :).

misterjoshua · 2022-09-07T15:09:12Z

I think the CDK_DOCKER feature was mostly intended for testing, but I guess I don't mind supporting this. A couple of doc updates and then this is good to go.

Sounds good. I'll get to these changes this afternoon or evening after work at the latest.

(Sidebar: the more I see and learn about Docker, the less I like supporting it. It seems everyone has very specific opinions on how it should be used in their specific use cases, leading to a proliferation of flags and tweaks that are tough to maintain... 😔)

Agreed. The CDK really took the bull by the horns by tackling the issue of building/publishing containers. I think a lot of users appreciate the help though, myself included.

Pull request has been modified.

misterjoshua · 2022-09-12T20:07:25Z

@Mergifyio update

mergify · 2022-09-12T20:08:35Z

update

✅ Branch has been successfully updated

misterjoshua · 2022-09-21T16:11:38Z

@Mergifyio update

mergify · 2022-09-21T16:12:20Z

update

✅ Branch has been successfully updated

rix0rrr · 2022-09-22T15:48:40Z

Needs #22194 to be merged first

rix0rrr · 2022-09-23T07:49:18Z

@Mergifyio update

mergify · 2022-09-23T07:49:53Z

update

✅ Branch has been successfully updated

github-actions

The Pull Request Linter fails with the following errors:

❌ Features must contain a change to an integration test file and the resulting snapshot.

PRs must pass status checks before we can provide a meaningful review.

Pull Request updated. Dissmissing previous PRLinter Review.

aws-cdk-automation · 2022-09-23T08:27:52Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
Commit ID: 241b324
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mergify · 2022-09-23T08:28:08Z

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

…CKER` (aws#21838) This PR introduces the ability for users to change the command used for building and publishing Docker images to a drop-in replacement like the [Depot](https://depot.dev) Docker shim. To change the docker command, the user provides the alternative command via an environment variable named `CDK_DOCKER`. There's already something similar in `core`'s asset bundling, but it applies only to bundling and not to building and publishing Docker image assets. https://github.com/aws/aws-cdk/blob/3e55092fb70e0ec74ee7c4144d6e39a29d8757ae/packages/%40aws-cdk/core/lib/bundling.ts#L523 Fixes aws#21836 ---- ### All Submissions: * [x] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) ### Adding new Unconventional Dependencies: * [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies) ### New Features * [ ] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)? * [ ] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)? *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license* Depends-On: aws#22194

feat(assets): support drop-in docker replacements

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.

GPG key ID: 4AEE18F83AFDEB23
Expired

Verified
Learn about vigilant mode

5ae49bf

aws-cdk-automation requested a review from a team August 30, 2022 23:57

github-actions bot added feature-request p2 labels Aug 30, 2022

misterjoshua marked this pull request as ready for review August 30, 2022 23:59

misterjoshua added 2 commits August 30, 2022 18:17

fix the test

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.

GPG key ID: 4AEE18F83AFDEB23
Expired

Verified
Learn about vigilant mode

81197ef

change env var to CDK_DOCKER to align with core

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.

GPG key ID: 4AEE18F83AFDEB23
Expired

Verified
Learn about vigilant mode

3773d32

misterjoshua mentioned this pull request Aug 31, 2022

(assets): support drop-in docker replacements #21836

Closed

2 tasks

update docs after first review

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.

GPG key ID: 4AEE18F83AFDEB23
Expired

Verified
Learn about vigilant mode

a87da4c

Merge branch 'main' into main

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.

GPG key ID: 4AEE18F83AFDEB23
Expired

Verified
Learn about vigilant mode

d66d686

Hi-Fi reviewed Sep 7, 2022

View reviewed changes

rix0rrr previously requested changes Sep 7, 2022

View reviewed changes

review changes to docs

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.

GPG key ID: 4AEE18F83AFDEB23
Expired

Verified
Learn about vigilant mode

b3bdbbd

Merge branch 'main' into main

3abe75a

Merge branch 'main' into main

13bd72b

misterjoshua requested a review from rix0rrr September 21, 2022 16:55

rix0rrr approved these changes Sep 22, 2022

View reviewed changes

rix0rrr changed the title ~~feat(assets): support drop-in docker replacements~~ feat(assets): support drop-in docker replacements by setting $CDK_DOCKER Sep 22, 2022

github-actions bot added effort/small p1 and removed p2 labels Sep 22, 2022

Merge branch 'main' into main

241b324

github-actions bot previously requested changes Sep 23, 2022

View reviewed changes

rix0rrr added the pr-linter/exempt-integ-test label Sep 23, 2022

mergify bot merged commit d52310e into aws:main Sep 23, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(assets): support drop-in docker replacements by setting `$CDK_DOCKER` #21838

feat(assets): support drop-in docker replacements by setting `$CDK_DOCKER` #21838

misterjoshua commented Aug 30, 2022 •

edited by rix0rrr

Loading

gitpod-io bot commented Aug 30, 2022

Hi-Fi commented Aug 31, 2022 •

edited

Loading

misterjoshua commented Aug 31, 2022

Hi-Fi commented Aug 31, 2022

misterjoshua commented Aug 31, 2022

misterjoshua commented Sep 6, 2022

mergify bot commented Sep 6, 2022

Hi-Fi Sep 3, 2022

misterjoshua Sep 7, 2022 •

edited

Loading

rix0rrr left a comment

rix0rrr Sep 7, 2022

rix0rrr Sep 7, 2022

rix0rrr Sep 7, 2022

misterjoshua commented Sep 7, 2022

misterjoshua commented Sep 12, 2022

mergify bot commented Sep 12, 2022

misterjoshua commented Sep 21, 2022

mergify bot commented Sep 21, 2022

rix0rrr commented Sep 22, 2022

rix0rrr commented Sep 23, 2022

mergify bot commented Sep 23, 2022

github-actions bot left a comment

aws-cdk-automation commented Sep 23, 2022

mergify bot commented Sep 23, 2022

	Adding to the default CodeBuild image
	### Adding to the default CodeBuild image

feat(assets): support drop-in docker replacements by setting $CDK_DOCKER #21838

feat(assets): support drop-in docker replacements by setting $CDK_DOCKER #21838

Conversation

misterjoshua commented Aug 30, 2022 • edited by rix0rrr Loading

All Submissions:

Adding new Unconventional Dependencies:

New Features

gitpod-io bot commented Aug 30, 2022

Hi-Fi commented Aug 31, 2022 • edited Loading

misterjoshua commented Aug 31, 2022

Hi-Fi commented Aug 31, 2022

misterjoshua commented Aug 31, 2022

misterjoshua commented Sep 6, 2022

mergify bot commented Sep 6, 2022

✅ Branch has been successfully updated

Hi-Fi Sep 3, 2022

Choose a reason for hiding this comment

misterjoshua Sep 7, 2022 • edited Loading

Choose a reason for hiding this comment

rix0rrr left a comment

Choose a reason for hiding this comment

rix0rrr Sep 7, 2022

Choose a reason for hiding this comment

rix0rrr Sep 7, 2022

Choose a reason for hiding this comment

rix0rrr Sep 7, 2022

Choose a reason for hiding this comment

misterjoshua commented Sep 7, 2022

misterjoshua commented Sep 12, 2022

mergify bot commented Sep 12, 2022

✅ Branch has been successfully updated

misterjoshua commented Sep 21, 2022

mergify bot commented Sep 21, 2022

✅ Branch has been successfully updated

rix0rrr commented Sep 22, 2022

rix0rrr commented Sep 23, 2022

mergify bot commented Sep 23, 2022

✅ Branch has been successfully updated

github-actions bot left a comment

Choose a reason for hiding this comment

aws-cdk-automation commented Sep 23, 2022

AWS CodeBuild CI Report

mergify bot commented Sep 23, 2022

feat(assets): support drop-in docker replacements by setting `$CDK_DOCKER` #21838

feat(assets): support drop-in docker replacements by setting `$CDK_DOCKER` #21838

misterjoshua commented Aug 30, 2022 •

edited by rix0rrr

Loading

Hi-Fi commented Aug 31, 2022 •

edited

Loading

misterjoshua Sep 7, 2022 •

edited

Loading