Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix lambda logical version id for layers in same stack #8066

Merged
merged 44 commits into from Sep 24, 2020
Merged
Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
3e6fa90
Include layer details in logical id digest
pwithams Aug 5, 2020
34b0877
Enable change by removing empty array
pwithams Aug 6, 2020
91e3077
Add version id change based on layer zip content and non-S3Key proper…
pwithams Aug 8, 2020
f927470
Fix version id creation logic to pass existing tests
pwithams Aug 8, 2020
f93b1aa
Add tests for new lambda version layer logic
pwithams Aug 8, 2020
b4ee79a
Merge branch 'master' into fix-logical-version-id-for-layers
pwithams Aug 8, 2020
3a01dc9
Remove some lodash uses
pwithams Aug 9, 2020
b51b81e
Add test to check for reservedConcurrency version id change
pwithams Aug 9, 2020
6d412bc
Refactor first test to use version id comparisions rather than explic…
pwithams Aug 10, 2020
b5cb580
Update tests to use comparisons rather than explicit checks
pwithams Aug 11, 2020
3080dd3
Use path.join to create test file paths
pwithams Aug 11, 2020
14de011
Refactor compile logic and update tests
pwithams Aug 18, 2020
001f440
Add test to check for version update on key change
pwithams Aug 22, 2020
8678062
Merge branch 'master' into fix-logical-version-id-for-layers
pwithams Aug 22, 2020
619050c
Fix windows tests
pwithams Aug 22, 2020
781aa98
Use correct layer reference syntax
pwithams Aug 24, 2020
a6ccbac
Improve artifact path detection logic
pwithams Aug 26, 2020
3ad5651
Change promise map to all and update reservedConcurrency test
pwithams Aug 26, 2020
bccd773
Clean up function fixture use in reservedConcurrency test
pwithams Aug 26, 2020
997c0f0
Remove duplicate layerConfiguration definitions
pwithams Aug 26, 2020
8df8aac
Remove extra function layer fixture logic into single fixture
pwithams Aug 27, 2020
a6ec21d
Update main logic and test fixture to handle different layer name and…
pwithams Aug 27, 2020
8b21863
Make layer artifact hashing deterministic
pwithams Aug 27, 2020
18e6b1f
Merge branch 'master' into fix-logical-version-id-for-layers
pwithams Aug 27, 2020
231cf14
Add more reliable serverless layer key resolution logic
pwithams Aug 29, 2020
4941a15
Merge branch 'master' into fix-logical-version-id-for-layers and reso…
pwithams Aug 29, 2020
62cd779
Merge branch 'master' into fix-logical-version-id-for-layers
pwithams Aug 29, 2020
625aea8
Fix fallback layer artifact name resolution logic
pwithams Aug 29, 2020
d7704ec
Simplfy naming of hidden serverless layer name object
pwithams Sep 13, 2020
b1571c3
Merge branch 'master' into fix-logical-version-id-for-layers
pwithams Sep 13, 2020
9ee00b4
Move layer artifact name resolution into separate method
pwithams Sep 16, 2020
38c867e
Update tests to use async/await and new fixture engine
pwithams Sep 16, 2020
ff09d04
Merge branch 'master' into fix-logical-version-id-for-layers
pwithams Sep 16, 2020
6e0be19
Use fs-extra for async/await functionality
pwithams Sep 16, 2020
193c516
Add test case for function with arn layer reference
pwithams Sep 16, 2020
7ad4818
Merge branch 'master' into fix-logical-version-id-for-layers
pwithams Sep 19, 2020
9e08cf5
Improve naming and refactor tests
pwithams Sep 19, 2020
c8be293
Fix equality check used in refactored existing test
pwithams Sep 19, 2020
eee300e
Move object clone logic out of function
pwithams Sep 22, 2020
76bf822
Merge branch 'master' into fix-logical-version-id-for-layers
pwithams Sep 22, 2020
a9baf17
Merge master and fix conflicts
pwithams Sep 22, 2020
6967213
Fix naming issue and increase native interface usage
pwithams Sep 23, 2020
b3cea34
Removed refactored logic
pwithams Sep 23, 2020
7c3ac73
Merge branch 'master' into fix-logical-version-id-for-layers
pwithams Sep 23, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
210 changes: 133 additions & 77 deletions lib/plugins/aws/package/compile/functions/index.js
Expand Up @@ -431,6 +431,7 @@ class AwsCompileFunctions {
: this.serverless.service.provider.versionFunctions;

let versionCompilationPromise;

if (shouldVersionFunction || functionObject.provisionedConcurrency) {
// Create hashes for the artifact and the logical id of the version resource
// The one for the version resource must include the function configuration
Expand All @@ -441,98 +442,120 @@ class AwsCompileFunctions {
fileHash.setEncoding('base64');
versionHash.setEncoding('base64');

// Read the file in chunks and add them to the hash (saves memory and performance)
versionCompilationPromise = BbPromise.fromCallback(cb => {
const readStream = fs.createReadStream(artifactFilePath);

readStream
.on('data', chunk => {
fileHash.write(chunk);
versionHash.write(chunk);
})
.on('close', () => {
cb();
})
.on('error', error => {
cb(error);
// Include function and layer configuration in version id hash (without the Code part)
versionCompilationPromise = addFileContentsToHashes(artifactFilePath, [
fileHash,
versionHash,
medikoo marked this conversation as resolved.
Show resolved Hide resolved
])
.then(() => {
// Include all referenced layer code in the version id hash
const layerConfigurations = extractLayerConfigurationsFromFunction(
functionResource.Properties,
cfTemplate
);

const layerZipFilePaths = [];
layerConfigurations.forEach(layer => {
const layerZipFileName = layer.properties.Content.S3Key.split('/').pop();
const layerZipFilePath = path.join(this.packagePath, layerZipFileName);
layerZipFilePaths.push(layerZipFilePath);
});
}).then(() => {
// Include function configuration in version id hash (without the Code part)
const properties = _.omit(_.get(functionResource, 'Properties', {}), 'Code');
_.forOwn(properties, value => {
if (_.isObject(value)) {
versionHash.write(JSON.stringify(value));
} else {
versionHash.write(value != null ? String(value) : '');

return BbPromise.map(layerZipFilePaths, zipFileName => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's not use Bluebird` functions which do not have native counterparts.

We should use BbPromise.all instead

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has been updated now, but I realized that both approaches in their current state will run asynchronously on the same hash object. I'm not sure if there could be a race condition or something if hash.write(chunk) calls on the same object collide.

A potential fix would be to rewrite addFileContentsToHash to return a new string or hash object representing the file. Once all promises finish, the array of strings/hashes could then be combined and added to the versionId hash. What do you think?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has been updated now, but I realized that both approaches in their current state will run asynchronously on the same hash object.

That's true, it won't be deterministic then. We need to run it sequentially, I believe you can achieve it as:

return layerZipFilePaths.reduce(
  (previousHashCalculation, zipFileName) => previousHashCalculation.then(() => addFileContentsToHashes(zipFileName, [versionHash]),
  BbPromise.resolve()
)

Note that also layerZipFilePaths order needs to also be deterministic

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this should be done now.

return addFileContentsToHashes(zipFileName, [versionHash]);
});
})
.then(() => {
// Include function and layer configuration details in the version id hash
const layerConfigurations = extractLayerConfigurationsFromFunction(
functionResource.Properties,
cfTemplate
);

for (const layerConfig of layerConfigurations) {
delete layerConfig.properties.Content.S3Key;
}
});

// Finalize hashes
fileHash.end();
versionHash.end();
// sort the layer conifigurations for hash consistency
const sortedLayerConfigurations = {};
const byKey = ([key1], [key2]) => key1.localeCompare(key2);
for (const { name, properties: layerProperties } of layerConfigurations) {
sortedLayerConfigurations[name] = _.fromPairs(_.entries(layerProperties).sort(byKey));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's use Object.entries instead of _.entries (as in a meantime we've dropped support for Node.js v8)

}

const fileDigest = fileHash.read();
const versionDigest = versionHash.read();
const functionProperties = _.cloneDeep(functionResource.Properties);
delete functionProperties.Code;
functionProperties.layerConfigurations = sortedLayerConfigurations;

const versionResource = this.cfLambdaVersionTemplate();
const sortedFunctionProperties = _.fromPairs(_.entries(functionProperties).sort(byKey));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here

versionHash.write(JSON.stringify(sortedFunctionProperties));

versionResource.Properties.CodeSha256 = fileDigest;
versionResource.Properties.FunctionName = { Ref: functionLogicalId };
if (functionObject.description) {
versionResource.Properties.Description = functionObject.description;
}
// Finalize hashes
fileHash.end();
versionHash.end();

// use the version SHA in the logical resource ID of the version because
// AWS::Lambda::Version resource will not support updates
const versionLogicalId = this.provider.naming.getLambdaVersionLogicalId(
functionName,
versionDigest
);
functionObject.versionLogicalId = versionLogicalId;
const newVersionObject = {
[versionLogicalId]: versionResource,
};
const fileDigest = fileHash.read();
const versionDigest = versionHash.read();

Object.assign(cfTemplate.Resources, newVersionObject);
const versionResource = this.cfLambdaVersionTemplate();

// Add function versions to Outputs section
const functionVersionOutputLogicalId = this.provider.naming.getLambdaVersionOutputLogicalId(
functionName
);
const newVersionOutput = this.cfOutputLatestVersionTemplate();
versionResource.Properties.CodeSha256 = fileDigest;
versionResource.Properties.FunctionName = { Ref: functionLogicalId };
if (functionObject.description) {
versionResource.Properties.Description = functionObject.description;
}

newVersionOutput.Value = { Ref: versionLogicalId };
// use the version SHA in the logical resource ID of the version because
// AWS::Lambda::Version resource will not support updates
const versionLogicalId = this.provider.naming.getLambdaVersionLogicalId(
functionName,
versionDigest
);
functionObject.versionLogicalId = versionLogicalId;
const newVersionObject = {
[versionLogicalId]: versionResource,
};

Object.assign(cfTemplate.Outputs, {
[functionVersionOutputLogicalId]: newVersionOutput,
});
Object.assign(cfTemplate.Resources, newVersionObject);

if (!functionObject.provisionedConcurrency) return;
if (!shouldVersionFunction) delete versionResource.DeletionPolicy;
// Add function versions to Outputs section
const functionVersionOutputLogicalId = this.provider.naming.getLambdaVersionOutputLogicalId(
functionName
);
const newVersionOutput = this.cfOutputLatestVersionTemplate();

const provisionedConcurrency = Number(functionObject.provisionedConcurrency);
const aliasLogicalId = this.provider.naming.getLambdaProvisionedConcurrencyAliasLogicalId(
functionName
);
const aliasName = this.provider.naming.getLambdaProvisionedConcurrencyAliasName();

functionObject.targetAlias = { name: aliasName, logicalId: aliasLogicalId };

const aliasResource = {
Type: 'AWS::Lambda::Alias',
Properties: {
FunctionName: { Ref: functionLogicalId },
FunctionVersion: { 'Fn::GetAtt': [versionLogicalId, 'Version'] },
Name: aliasName,
ProvisionedConcurrencyConfig: {
ProvisionedConcurrentExecutions: provisionedConcurrency,
newVersionOutput.Value = { Ref: versionLogicalId };

Object.assign(cfTemplate.Outputs, {
[functionVersionOutputLogicalId]: newVersionOutput,
});

if (!functionObject.provisionedConcurrency) return;
if (!shouldVersionFunction) delete versionResource.DeletionPolicy;

const provisionedConcurrency = Number(functionObject.provisionedConcurrency);
const aliasLogicalId = this.provider.naming.getLambdaProvisionedConcurrencyAliasLogicalId(
functionName
);
const aliasName = this.provider.naming.getLambdaProvisionedConcurrencyAliasName();

functionObject.targetAlias = { name: aliasName, logicalId: aliasLogicalId };

const aliasResource = {
Type: 'AWS::Lambda::Alias',
Properties: {
FunctionName: { Ref: functionLogicalId },
FunctionVersion: { 'Fn::GetAtt': [versionLogicalId, 'Version'] },
Name: aliasName,
ProvisionedConcurrencyConfig: {
ProvisionedConcurrentExecutions: provisionedConcurrency,
},
},
},
DependsOn: functionLogicalId,
};
DependsOn: functionLogicalId,
};

cfTemplate.Resources[aliasLogicalId] = aliasResource;
});
cfTemplate.Resources[aliasLogicalId] = aliasResource;
});
} else {
versionCompilationPromise = BbPromise.resolve();
}
Expand Down Expand Up @@ -721,4 +744,37 @@ class AwsCompileFunctions {
}
}

function addFileContentsToHashes(filePath, hashes) {
return new BbPromise((cb, reject) => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's use native Promise, and for consistency let's use resolve name instead cb

const readStream = fs.createReadStream(filePath);
readStream
.on('data', chunk => {
hashes.forEach(hash => {
hash.write(chunk);
});
})
.on('close', () => {
cb();
})
.on('error', error => {
reject(new Error(`Could not add file content to hash: ${error}`));
});
});
}

function extractLayerConfigurationsFromFunction(functionProperties, cfTemplate) {
const layerConfigurations = [];
if (!functionProperties.Layers) return layerConfigurations;
functionProperties.Layers.forEach(potentialLocalLayerObject => {
if (potentialLocalLayerObject.Ref) {
const configuration = _.cloneDeep(cfTemplate.Resources[potentialLocalLayerObject.Ref]);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to clone it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Main reason was I assumed this line https://github.com/pwithams/serverless/blob/c8be293c97dd8550f2a5d02ee8d61e6032da3335/lib/plugins/aws/package/compile/functions/index.js#L513 would delete part of the already created cloudformation template if the reference was returned.

Also, in general it seemed like cfTemplate is such an important object that if part of it is being returned then it should be a copy so that anything using the return value doesn't accidentally alter it.

However, if the delete doesn't actually affect anything (I think removing the clone only fails one of my tests?) then it can be taken out, especially if lodash usage is being minimized.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cfTemplate is such an important object that if part of it is being returned then it should be a copy so that anything using the return value doesn't accidentally alter it.

In fact, if we're taking it as reference, then indeed we should never attempt to modify it. However implying a practice that whenever we take it, we copy it (just in case), could be noisy and difficult to follow, it's definitely not natural in JS.
It's more natural to purely stay away from modifying such reference objects.

So I'd say, let's not clone and let's ensure we do not modify it in further processing

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That makes sense. I think it still has to be modified later in some way as S3Key can't be part of the hash otherwise it will change every time. I've now moved the clone out of the function and applied it to the returned result instead - this solves the configuration._serverlessLayerName naming issue below too.

Let me know if you have any other ideas how to handle keeping S3Key out of the hash without a clone.

layerConfigurations.push({
name: potentialLocalLayerObject.Ref,
properties: configuration.Properties,
});
}
});
return layerConfigurations;
}

module.exports = AwsCompileFunctions;