Skip to content

Commit

Permalink
fix(lambda-python): root-owned cache items not cleaned up after insta…
Browse files Browse the repository at this point in the history
…ll (aws#22512)

This is another attempt at fixing aws#22012. aws#22398 intended to fix it, but the issue still occurs when building on arm64 machines with both `compatible_runtimes=[lambdas.Runtime.PYTHON_3_9]` and `compatible_architectures=[lambdas.Architecture.ARM_64]`

The core problem is that the build leaves root-owned files under `/tmp/pip-cache`, which then cause permission problems when subsequent layers calling `pip install` are unable to create temporary directories. Example error:

```
WARNING: Building wheel for alembic-utils failed: [Errno 13] Permission denied: '/tmp/pip-cache/wheels/5c'
```

Expanding on the idea within aws#22398, this PR adds one more step to the chain of operations, removing any temporary files from the cache that are no longer needed (because the packages are now installed), and ensuring that subsequent layers are able to create temporary files inside the cache directories.

Comparing the contents of the Docker image before and after this change...

Before:
```
$ ls -aFl /tmp/*cache
/tmp/pip-cache:
total 16
drwxrwxrwx  4 root root 4096 Oct 15 00:57 ./
drwxrwxrwt  1 root root 4096 Oct 15 00:58 ../
drwxr-xr-x 18 root root 4096 Oct 15 00:57 http/
drwxr-xr-x  2 root root 4096 Oct 15 00:57 selfcheck/

/tmp/poetry-cache:
total 8
drwxrwxrwx 2 root root 4096 Oct 15 00:57 ./
drwxrwxrwt 1 root root 4096 Oct 15 00:58 ../
```

After:
```
$ ls -aFl /tmp/*cache
/tmp/pip-cache:
total 8
drwxrwxrwx 2 root root 4096 Oct 15 01:00 ./
drwxrwxrwt 1 root root 4096 Oct 15 01:00 ../

/tmp/poetry-cache:
total 8
drwxrwxrwx 2 root root 4096 Oct 15 00:59 ./
drwxrwxrwt 1 root root 4096 Oct 15 01:00 ../
```

----

### All Submissions:

* [x] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md)

### Adding new Unconventional Dependencies:

* [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies)

### New Features

* [x] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)?
	* [x] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)?

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
  • Loading branch information
varju authored and mrgrain committed Oct 24, 2022
1 parent be36d1e commit 0096f47
Show file tree
Hide file tree
Showing 14 changed files with 1,788 additions and 5 deletions.
4 changes: 3 additions & 1 deletion packages/@aws-cdk/aws-lambda-python/lib/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,8 @@ RUN \
# Ensure all users can write to poetry cache
chmod -R 777 /tmp/poetry-cache && \
# pipenv 2022.4.8 is the last version with Python 3.6 support
pip install pipenv==2022.4.8 poetry
pip install pipenv==2022.4.8 poetry && \
# Ensure no temporary files remain in the caches
rm -rf /tmp/pip-cache/* /tmp/poetry-cache/*

CMD [ "python" ]

0 comments on commit 0096f47

Please sign in to comment.