Improve build performance of NodejsFunction on Mac (@aws-cdk/aws-lambda-nodejs package) #8544
Comments
Dzhuneyt
added
feature-request
|
@Dzhuneyt would this apply to other OS as well? Can you propose the change in aws-cdk/packages/@aws-cdk/core/lib/bundling.ts Lines 128 to 131 in 4521ae3 cc @eladb |
|
Other OSs don't suffer from this performance issue and the ":cached" or ":delegated" flag is a no-op for them. |
|
@jogold could you point me to another GitHub issue or a place where I can learn more about this upcoming refactoring? Regardless, I think this fix should go forward with the current implementation until such refactoring happens (because it's tiny and the benefit for developers that use MacOS as a dev env of choice are pretty obvious). |
Not opened yet, will try to start the discussion in an issue/PR this week or next week. |
|
Is there a reason why it runs inside a Docker container anyways? Seems to be a bit heavy handed. I think there should be an option to run it locally. |
See Why we use Docker? in #8460 |
SomayaB
added
os/mac
|
@jogold I plan on working on the implementation in "core" at some point. Does this mean this PR is unnecessary? Personally, I still see value in it being pushed live until such refactoring comes into effect, but the decision is with you. |
Has any work started on this part "lambda-nodejs is going to be refactored to use the bundling in core"? |
|
@Dzhuneyt do you still want to propose this change? Let met know if you need some help. |
|
Yes, I will try to implement this. It's my first major contribution to AWS CDK, so I might need some guidance. I'll write here if i need any help. |
First time contributor to AWS CDK here. Let me know if I've missed something. As a developer who uses macOS for development purposes on a daily basis and AWS CDK for an enterprise project, I am really impacted by the official and [well known Docker volumes performance issue](https://docs.docker.com/docker-for-mac/osxfs-caching/). Since the project that we are working on, relies heavily on Lambdas that are primarly written in TypeScript, the mass deployment of these Lambdas using AWS CDK CLI and the (recently introduced) NodejsFunction construct takes a significant amount of time and causes a CPU overload. This is mainly due to the fact that the construct internally does the following things: 1. Starting a docker container with Parcel 2. Mounting the root of the closest folder that contains .git (this is the root of the whole CDK app for us, that includes the CDK stacks, the Lambdas and any other utility functions and node_modules folders, since we opted for the "monorepo" structural pattern) 3. Runs the parcel bundler inside the container, which emits a single .js file as an asset that will serve as the Lambda's "code" 4. Deploys the .js file using the standard lambda.Function construct (NodejsFunction is just a wrapper of lambda.Function with @aws-cdk/core bundling and Parcel involved). For our project, this means CDK deployment spawns dozens of containers (one per Lambda), that all mount (using Docker volumes), the whole project. The frequent IO operations by Parcel within the container cause a massive CPU spike in all of these containers, causing each Lambda compilation to take 1-2 seconds. This latency is quickly magnified as the number of Lambdas grow within the project. The latency and CPU spike is mainly a side effect of how Docker and volumes work in macOS for which you can read more on the above link. This is a common pain point for all developers who use these tools, even outside the AWS CDK world. Here are some examples and further reading, including further reading on how flags like ":cached" or ":delegated" help: - docker/for-mac#1759 (comment) - https://stackoverflow.com/questions/58277794/diagnosing-high-cpu-usage-on-docker-for-mac/58293240#58293240 - https://stackoverflow.com/questions/51694789/macbook-pro-2018-high-temperature-and-cpu-usage-with-docker/51698665#51698665 - https://stackoverflow.com/questions/60878918/docker-hyperkit-process-cpu-usage-going-crazy-how-to-keep-it-under-control#comment107712547_60878918 - https://stackoverflow.com/questions/55951014/docker-in-macos-is-very-slow/55953023#55953023 I've decided to use ":delegated" here since the bundling mechanism generates files inside the container that need replication on the host machine with small tolerable delay, not the other way around (where ":cached" would have been useful). The addition of the flag is non-conditional based on the current OS, because the existence of the flag is a NO-OP in all other operating systems (tested) and is only taken into account by Docker when the volume driver is detected to be osxfs. Closes #8544 ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Using volumes in Docker on Mac specifically is a huge pain point due to performance issues. CPU often spikes to above 100% utilization when there is heavy read or write operations inside the container (as it happens with the
parcel-builderDocker container that the CDK construct NodejsFunction uses internally.This has been very well documented in https://docs.docker.com/docker-for-mac/osxfs-caching/ and the workarounds are also described.
I suggest we force Docker volume mounts within this construct to start using Docker volume in "delegated" mode, to improve build performance (speed) on Mac based deployment environments.
I think a good starting point is:
https://github.com/aws/aws-cdk/blob/master/packages/%40aws-cdk/aws-lambda-nodejs/lib/builder.ts#L118
Use Case
Developing and deploying CDK stacks with many lambdas on a Mac machine.
Proposed Solution
Replace
-v sourcevolume:targetvolumewith-v sourcevolume:targetvolume:delegatedOther
This is a 🚀 Feature Request
The text was updated successfully, but these errors were encountered: