(@aws-cdk/aws-lambda-python): BundlingOptions for PythonLayerVersion are ignored

[cponfick]

What is the problem?

It appears that asset_hash_type and asset_hash are ignored in the bundling parameter of the PythonLayerVersion. The asset_hash_type used is always AssetHashType.SOURCE. This is a problem if the requirement file includes a env variable which changes on every deploy, e.g a token.

Reproduction Steps

# token = required from aws cli (changes every time)
# lock_file_hash = hash from Pipfile.lock (does not change)
# entry = path to the directory containing the Pipfile.lock

self.base_python_layer = PythonLayerVersion(self, "BasePythonLayer",
                                                    entry=entry,
                                                    description="Dependencies from Pipfile.lock.",
                                                    compatible_runtimes=[Runtime.PYTHON_3_8],
                                                    bundling=BundlingOptions(
                                                        environment={"PYPI_MIRROR_TOKEN": token},
                                                        asset_hash_type=AssetHashType.CUSTOM,
                                                        asset_hash=lock_file_hash
                                                    ), compatible_architectures=[Architecture.X86_64])

The token is the only thing, which is changing in the entry repository. On every cdk deploy call.

What did you expect to happen?

Update the layer only when the lock_file_hash variable changes.

What actually happened?

The layer is updated every time.

CDK CLI Version

2.17.0

Framework Version

No response

Node.js Version

v17.7.2

OS

macOS Monterey 12.3

Language

Python

Language Version

Python (3.8.9)

Other information

This issue is somewhat related to #19508, since I tried to create a custom hash as a answer to the missing exclude possibility.

In my opinion it is also a security issue uploading the requirements file to the layer, since it might contain user and password information to private pypi mirrors. In our case a token with a short expiration time was the solution, but it might not be for other users.

[corymhall]

@cponfick-bhs can you try to reproduce this using a very simple hash (e.g. asset_hash="abc"). I've
not been able to reproduce this issue and that should help us narrow down the issue.

[cponfick]

@cponfick-bhs can you try to reproduce this using a very simple hash (e.g. asset_hash="abc"). I've not been able to reproduce this issue and that should help us narrow down the issue.

@corymhall I am able to reproduce it by using asset_hash="1". The layer is updated as long as files in the entry change. Hence, I assume asset_hash_type=AssetHashType.Source is still used despite changing it to Custom.

In my case the changing file is the requirements.txt file, containing the auth token. When I use the same token in sequential deploys. The layer is not updated.

[corymhall]

ah ok it looks like it's because the hash that is calculated includes the BundlingOptions which is
changing every time for you. I think the solution to this should be #19508 so you don't have to try
to hack around it.

[cponfick]

@corymhall do I understand this correctly. When using a custom hash, this hash is used to create another hash based on the BundlingOptions? This does not sound right to me.

[corymhall]

Yes that is correct. We create a new hash using the custom hash and bundling options as inputs. This is because the bundling options instruct CDK how the asset should be bundled, so if that changes then the hash should be invalidated. Imagine you passed a different Docker image or an environment variable that would affect the output of bundling.

aws-cdk/packages/@aws-cdk/core/lib/asset-staging.ts

Lines 487 to 518 in f1c9c4e

	private calculateHash(hashType: AssetHashType, bundling?: BundlingOptions, outputDir?: string): string {
	// When bundling a CUSTOM or SOURCE asset hash type, we want the hash to include
	// the bundling configuration. We handle CUSTOM and bundled SOURCE hash types
	// as a special case to preserve existing user asset hashes in all other cases.
	if (hashType == AssetHashType.CUSTOM || (hashType == AssetHashType.SOURCE && bundling)) {
	const hash = crypto.createHash('sha256');
	// if asset hash is provided by user, use it, otherwise fingerprint the source.
	hash.update(this.customSourceFingerprint ?? FileSystem.fingerprint(this.sourcePath, this.fingerprintOptions));
	// If we're bundling an asset, include the bundling configuration in the hash
	if (bundling) {
	hash.update(JSON.stringify(bundling));
	}
	return hash.digest('hex');
	}
	switch (hashType) {
	case AssetHashType.SOURCE:
	return FileSystem.fingerprint(this.sourcePath, this.fingerprintOptions);
	case AssetHashType.BUNDLE:
	case AssetHashType.OUTPUT:
	if (!outputDir) {
	throw new Error(`Cannot use \`${hashType}\` hash type when \`bundling\` is not specified.`);
	}
	return FileSystem.fingerprint(outputDir, this.fingerprintOptions);
	default:
	throw new Error('Unknown asset hash type.');
	}
	}
	}

[cponfick]

Thanks for your explanation. Shouldn't this then be the fault of the user? As in: "You use a custom hash. Hence, it is your responsibility to provide a sufficient hash." As I see it right now it is not possible to use a token to authenticate to a pypi mirror. Even if I am able to exclude the requirements file, by implementing #19508, the environment will always be different, because the token is different.

If I am not mistaken, the bellow example from the docs should always create a new layer.

from child_process import exec_sync


entry = "/path/to/function"
image = DockerImage.from_build(entry)

domain = "my-domain"
domain_owner = "111122223333"
repo_name = "my_repo"
region = "us-east-1"
code_artifact_auth_token = exec_sync(f"aws codeartifact get-authorization-token --domain {domain} --domain-owner {domainOwner} --query authorizationToken --output text").to_string().trim()

index_url = f"https://aws:{codeArtifactAuthToken}@{domain}-{domainOwner}.d.codeartifact.{region}.amazonaws.com/pypi/{repoName}/simple/"

lambda_.PythonFunction(self, "function",
    entry=entry,
    runtime=Runtime.PYTHON_3_8,
    bundling=lambda.BundlingOptions(
        environment={"PIP_INDEX_URL": index_url}
    )
)

[icj217]

@cponfick-bhs Even if I use a custom asset_hash, caching doesn't work because, as @corymhall stated above, the BundlingOptions(...) is always used to update the hash, and since we're passing a CodeArtifact auth token via an environment variable, that means the contents of BundlingOptions changes on every build, hence the cache never being leveraged.

[cponfick]

@icj217 What are you using as a dependency management tool?
We use pipenv. With pipenv lock your are able to inject the token into the requirements file. This makes it possible to build the layer without the environment variable.

Sorry, this was the actual solution I found for the problem.

[icj217]

@cponfick-bhs We're just using pip + requirements.txt. How does pipenv inject the token into the requirements file, specifically in the context of defining the lambda layer resource? Thanks in advance for the help!

[cponfick]

@icj217 You can set your source in the Pipfile:

[[source]]
name = 'your name'
url = '${ENV_VAR_NAME}my-url' 
verify_ssl = true

Described here.

Then just use pipenv lock --requirements to generate your requirements.txt before you build your layer. This will put the token directly into the requirements.txt. Hence, no need to pass it in the PythonLayerVersion. This will make it possible to use a custom hash. I usually use the hash in the Pipfile.lock.

Note, you should set a expiration time for your token, since the requirements.txt file will be uploaded as part of your generated layer.

[icj217]

@cponfick-bhs Ok thanks so much for explaining in detail! We've ended up writing a construct that handles the CA token retrieval and injection into pip config, but this definitely seems like a better long-term solution given the other benefits of using pipenv 🙂

[ppena-LiveData]

It would be nice if this could take a DockerImageAssetInvalidationOptions (or something like it), so we could tell it to ignore the constantly changing build_args.

[diranged]

I was surprised to find that this is still a problem - and that the documentation around using CodeArtifact with the PythonFunction construct is invalid:

Specifically - the second paragraph indicates that using bundling.buildArgs. PIP_INDEX_URL=... will cause your artifacts to be rebuilt every time .. but that implies that using bundling.environment.PIP_INDEX_URL=... would be fine - however it clearly is not based on this thread.

We've had to abandon the @aws-cdk/aws-lambda-python-alpha module entirely and package our libraries up using the Code.fromAsset()... function along with the tryBundling() function (skipping Docker entirely), but that's hardly ideal.

Am I missing something here? Isn't this a clear bug that needs resolving?

[cponfick]

We also moved away from @aws-cdk/aws-lambda-python-alpha. Currently, we are using pantsbuild to build our lambdas without layers. With the 2.18.0 release pants will also support lambda layer creation. Hence, we will probably never create the layer inside cdk again. It was just too much of a struggle, to be honest.

[OrGgamliel]

Hi @cponfick
Can you elaborate more and share how do you use 'pantsbuild' with cdk?
What is your repository structure and how you integrated it together?