Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix lambda logical version id for layers in same stack #8066

Merged
merged 44 commits into from Sep 24, 2020
Merged
Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
3e6fa90
Include layer details in logical id digest
pwithams Aug 5, 2020
34b0877
Enable change by removing empty array
pwithams Aug 6, 2020
91e3077
Add version id change based on layer zip content and non-S3Key proper…
pwithams Aug 8, 2020
f927470
Fix version id creation logic to pass existing tests
pwithams Aug 8, 2020
f93b1aa
Add tests for new lambda version layer logic
pwithams Aug 8, 2020
b4ee79a
Merge branch 'master' into fix-logical-version-id-for-layers
pwithams Aug 8, 2020
3a01dc9
Remove some lodash uses
pwithams Aug 9, 2020
b51b81e
Add test to check for reservedConcurrency version id change
pwithams Aug 9, 2020
6d412bc
Refactor first test to use version id comparisions rather than explic…
pwithams Aug 10, 2020
b5cb580
Update tests to use comparisons rather than explicit checks
pwithams Aug 11, 2020
3080dd3
Use path.join to create test file paths
pwithams Aug 11, 2020
14de011
Refactor compile logic and update tests
pwithams Aug 18, 2020
001f440
Add test to check for version update on key change
pwithams Aug 22, 2020
8678062
Merge branch 'master' into fix-logical-version-id-for-layers
pwithams Aug 22, 2020
619050c
Fix windows tests
pwithams Aug 22, 2020
781aa98
Use correct layer reference syntax
pwithams Aug 24, 2020
a6ccbac
Improve artifact path detection logic
pwithams Aug 26, 2020
3ad5651
Change promise map to all and update reservedConcurrency test
pwithams Aug 26, 2020
bccd773
Clean up function fixture use in reservedConcurrency test
pwithams Aug 26, 2020
997c0f0
Remove duplicate layerConfiguration definitions
pwithams Aug 26, 2020
8df8aac
Remove extra function layer fixture logic into single fixture
pwithams Aug 27, 2020
a6ec21d
Update main logic and test fixture to handle different layer name and…
pwithams Aug 27, 2020
8b21863
Make layer artifact hashing deterministic
pwithams Aug 27, 2020
18e6b1f
Merge branch 'master' into fix-logical-version-id-for-layers
pwithams Aug 27, 2020
231cf14
Add more reliable serverless layer key resolution logic
pwithams Aug 29, 2020
4941a15
Merge branch 'master' into fix-logical-version-id-for-layers and reso…
pwithams Aug 29, 2020
62cd779
Merge branch 'master' into fix-logical-version-id-for-layers
pwithams Aug 29, 2020
625aea8
Fix fallback layer artifact name resolution logic
pwithams Aug 29, 2020
d7704ec
Simplfy naming of hidden serverless layer name object
pwithams Sep 13, 2020
b1571c3
Merge branch 'master' into fix-logical-version-id-for-layers
pwithams Sep 13, 2020
9ee00b4
Move layer artifact name resolution into separate method
pwithams Sep 16, 2020
38c867e
Update tests to use async/await and new fixture engine
pwithams Sep 16, 2020
ff09d04
Merge branch 'master' into fix-logical-version-id-for-layers
pwithams Sep 16, 2020
6e0be19
Use fs-extra for async/await functionality
pwithams Sep 16, 2020
193c516
Add test case for function with arn layer reference
pwithams Sep 16, 2020
7ad4818
Merge branch 'master' into fix-logical-version-id-for-layers
pwithams Sep 19, 2020
9e08cf5
Improve naming and refactor tests
pwithams Sep 19, 2020
c8be293
Fix equality check used in refactored existing test
pwithams Sep 19, 2020
eee300e
Move object clone logic out of function
pwithams Sep 22, 2020
76bf822
Merge branch 'master' into fix-logical-version-id-for-layers
pwithams Sep 22, 2020
a9baf17
Merge master and fix conflicts
pwithams Sep 22, 2020
6967213
Fix naming issue and increase native interface usage
pwithams Sep 23, 2020
b3cea34
Removed refactored logic
pwithams Sep 23, 2020
7c3ac73
Merge branch 'master' into fix-logical-version-id-for-layers
pwithams Sep 23, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
225 changes: 147 additions & 78 deletions lib/plugins/aws/package/compile/functions/index.js
Expand Up @@ -441,98 +441,125 @@ class AwsCompileFunctions {
fileHash.setEncoding('base64');
versionHash.setEncoding('base64');

// Read the file in chunks and add them to the hash (saves memory and performance)
versionCompilationPromise = BbPromise.fromCallback(cb => {
const readStream = fs.createReadStream(artifactFilePath);

readStream
.on('data', chunk => {
fileHash.write(chunk);
versionHash.write(chunk);
})
.on('close', () => {
cb();
})
.on('error', error => {
cb(error);
// Include function and layer configuration in version id hash (without the Code part)
versionCompilationPromise = this.addFileContentsToHashes(artifactFilePath, [
fileHash,
versionHash,
medikoo marked this conversation as resolved.
Show resolved Hide resolved
])
.then(() => {
// Include all referenced layer code in the version id hash
const functionProperties = _.omit(_.get(functionResource, 'Properties', {}), 'Code');
pwithams marked this conversation as resolved.
Show resolved Hide resolved
const layerProperties = this.extractLayerPropertiesFromFunction(
functionProperties,
cfTemplate
);

const layerZipFilePaths = [];
layerProperties.forEach(layer => {
const layerZipFileName = layer.properties.Properties.Content.S3Key.split(
path.sep
).pop();
const layerZipFilePath = path.join(this.packagePath, layerZipFileName);
layerZipFilePaths.push(layerZipFilePath);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks as not necessarily equivalent to:

const artifactFilePath =
layerObject.package && layerObject.package.artifact
? layerObject.package.artifact
: path.join(this.serverless.config.servicePath, '.serverless', artifactFileName);

I believe safest way, would be to resolve an actual layer configuration object for given layer, and follow above logic to resolve path to layer artifact.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't seem to be addressed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I looked into addressing this but an issue is that in the linked logic it uses layerObject which is defined using:

    const layerObject = this.serverless.service.getLayer(layerName);

where layerName is actually the layer key used in the serverless.yml file, not the specific layer name field.

So the challenge would be how to get the serverless.yml layer key from the Ref key provided under the function layer, i.e. how to get testLayer from { Ref: TestLayerLambdaLayer } and also TestLayer from { Ref: TestLayerLambdaLayer} as the key is capitalized. Any ideas? Obviously could just split by LambdaLayer then if an error is thrown try with lowercase first letter, but it wouldn't be very clean.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually tried the split/lowercase method and it doesn't seem too bad, but let me know what you think: a6ec21d#diff-771a8504e96d8db95f70f2cca6e3a906R458

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably most bulletproof way, would be to introduce a hidden property on layer resource, that references layerObject

We can do following

Object.defineProperty(newLayer, "_config", { value: layerObject });

At

layerObject.package = layerObject.package || {};

It's safe for CF template handling, as hidden properties don't go down to JSON, and in our logic we can access the config as:

cicfTemplate.Resources[potentialLocalLayerObject.Ref]._config

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if this is possible as it looks like most resources are added using _.merge(this.serverless.service.provider.compiledCloudFormationTemplate... which will drop the hidden (enumerable: false) value. If enumerable: true then it seems to go down to JSON.

Is there somewhere else on the main serverless object it could be stored? Could add a method to lib/classes/Service.js but not sure that would make it much better. I wonder if there was an "official" way of resolving between the two then this https://github.com/serverless/serverless/blob/master/lib/plugins/aws/package/compile/functions/index.js#L449 could support just a list of layer object names from serverless.yml, rather than having to use the { Ref: MyLayerLambdaLayer} syntax for layers within the same stack.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_.merge(this.serverless.service.provider.compiledCloudFormationTemplate... which will drop the hidden (enumerable: false)

We may replace _.merge with Object.assign. Note _.merge is used in many places without a valid reasons, and as it's a deep merge occasionally it causes an unexpected issues.

Copy link
Contributor Author

@pwithams pwithams Aug 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is now changed here: 231cf14 (plus this fix for the fallback condition, not sure which cases this would be used: 625aea8)

});
}).then(() => {
// Include function configuration in version id hash (without the Code part)
const properties = _.omit(_.get(functionResource, 'Properties', {}), 'Code');
_.forOwn(properties, value => {
if (_.isObject(value)) {
versionHash.write(JSON.stringify(value));
} else {
versionHash.write(value != null ? String(value) : '');
}
});

// Finalize hashes
fileHash.end();
versionHash.end();
return BbPromise.map(layerZipFilePaths, zipFileName => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's not use Bluebird` functions which do not have native counterparts.

We should use BbPromise.all instead

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has been updated now, but I realized that both approaches in their current state will run asynchronously on the same hash object. I'm not sure if there could be a race condition or something if hash.write(chunk) calls on the same object collide.

A potential fix would be to rewrite addFileContentsToHash to return a new string or hash object representing the file. Once all promises finish, the array of strings/hashes could then be combined and added to the versionId hash. What do you think?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has been updated now, but I realized that both approaches in their current state will run asynchronously on the same hash object.

That's true, it won't be deterministic then. We need to run it sequentially, I believe you can achieve it as:

return layerZipFilePaths.reduce(
  (previousHashCalculation, zipFileName) => previousHashCalculation.then(() => addFileContentsToHashes(zipFileName, [versionHash]),
  BbPromise.resolve()
)

Note that also layerZipFilePaths order needs to also be deterministic

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this should be done now.

return this.addFileContentsToHashes(zipFileName, [versionHash]);
});
})
.then(() => {
// Include function and layer configuration in version id hash (without the Code part)
const functionProperties = _.omit(_.get(functionResource, 'Properties', {}), 'Code');
const layerProperties = this.extractLayerPropertiesFromFunction(
functionProperties,
cfTemplate
);

const fileDigest = fileHash.read();
const versionDigest = versionHash.read();
const mappedLayerFunctionProperties = layerProperties.map(layer => {
const clonedLayer = _.cloneDeep(layer);
clonedLayer.properties.Properties.Content.S3Key = '';
return clonedLayer.properties;
});
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's reuse already resolved layerProperties (nice if named layerConfigurations) in previous block.

Also if I see correctly this logic simply removes S3Key properties, that can be done simpler way via:

for (const layerProperties of layerConfigurations) delete layerProperties.S3Key;

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's reuse already resolved layerProperties (nice if named layerConfigurations) in previous block.

Doesn't seem to be addressed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand correctly I think this should be covered by 14de011#diff-771a8504e96d8db95f70f2cca6e3a906R475

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't seem. Note that layerConfigurations are still resolved twice in code as proposed:

const layerConfigurations = extractLayerConfigurationsFromFunction(
functionResource.Properties,
cfTemplate
);

const layerConfigurations = extractLayerConfigurationsFromFunction(
functionResource.Properties,
cfTemplate
);

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, it is now moved out of the block and used twice.


const versionResource = this.cfLambdaVersionTemplate();
mappedLayerFunctionProperties.push(functionProperties);

mappedLayerFunctionProperties.forEach(property => {
Object.keys(property).forEach(key => {
const value = property[key];
if (_.isObject(value)) {
versionHash.write(JSON.stringify(value));
} else {
versionHash.write(value != null ? String(value) : '');
}
});
});
pwithams marked this conversation as resolved.
Show resolved Hide resolved

versionResource.Properties.CodeSha256 = fileDigest;
versionResource.Properties.FunctionName = { Ref: functionLogicalId };
if (functionObject.description) {
versionResource.Properties.Description = functionObject.description;
}
// Finalize hashes
fileHash.end();
versionHash.end();

// use the version SHA in the logical resource ID of the version because
// AWS::Lambda::Version resource will not support updates
const versionLogicalId = this.provider.naming.getLambdaVersionLogicalId(
functionName,
versionDigest
);
functionObject.versionLogicalId = versionLogicalId;
const newVersionObject = {
[versionLogicalId]: versionResource,
};
const fileDigest = fileHash.read();
const versionDigest = versionHash.read();

Object.assign(cfTemplate.Resources, newVersionObject);
const versionResource = this.cfLambdaVersionTemplate();

// Add function versions to Outputs section
const functionVersionOutputLogicalId = this.provider.naming.getLambdaVersionOutputLogicalId(
functionName
);
const newVersionOutput = this.cfOutputLatestVersionTemplate();
versionResource.Properties.CodeSha256 = fileDigest;
versionResource.Properties.FunctionName = { Ref: functionLogicalId };
if (functionObject.description) {
versionResource.Properties.Description = functionObject.description;
}

// use the version SHA in the logical resource ID of the version because
// AWS::Lambda::Version resource will not support updates
const versionLogicalId = this.provider.naming.getLambdaVersionLogicalId(
functionName,
versionDigest
);
functionObject.versionLogicalId = versionLogicalId;
const newVersionObject = {
[versionLogicalId]: versionResource,
};

newVersionOutput.Value = { Ref: versionLogicalId };
Object.assign(cfTemplate.Resources, newVersionObject);

Object.assign(cfTemplate.Outputs, {
[functionVersionOutputLogicalId]: newVersionOutput,
});
// Add function versions to Outputs section
const functionVersionOutputLogicalId = this.provider.naming.getLambdaVersionOutputLogicalId(
functionName
);
const newVersionOutput = this.cfOutputLatestVersionTemplate();

if (!functionObject.provisionedConcurrency) return;
if (!shouldVersionFunction) delete versionResource.DeletionPolicy;
newVersionOutput.Value = { Ref: versionLogicalId };

const provisionedConcurrency = Number(functionObject.provisionedConcurrency);
const aliasLogicalId = this.provider.naming.getLambdaProvisionedConcurrencyAliasLogicalId(
functionName
);
const aliasName = this.provider.naming.getLambdaProvisionedConcurrencyAliasName();

functionObject.targetAlias = { name: aliasName, logicalId: aliasLogicalId };

const aliasResource = {
Type: 'AWS::Lambda::Alias',
Properties: {
FunctionName: { Ref: functionLogicalId },
FunctionVersion: { 'Fn::GetAtt': [versionLogicalId, 'Version'] },
Name: aliasName,
ProvisionedConcurrencyConfig: {
ProvisionedConcurrentExecutions: provisionedConcurrency,
Object.assign(cfTemplate.Outputs, {
[functionVersionOutputLogicalId]: newVersionOutput,
});

if (!functionObject.provisionedConcurrency) return;
if (!shouldVersionFunction) delete versionResource.DeletionPolicy;

const provisionedConcurrency = Number(functionObject.provisionedConcurrency);
const aliasLogicalId = this.provider.naming.getLambdaProvisionedConcurrencyAliasLogicalId(
functionName
);
const aliasName = this.provider.naming.getLambdaProvisionedConcurrencyAliasName();

functionObject.targetAlias = { name: aliasName, logicalId: aliasLogicalId };

const aliasResource = {
Type: 'AWS::Lambda::Alias',
Properties: {
FunctionName: { Ref: functionLogicalId },
FunctionVersion: { 'Fn::GetAtt': [versionLogicalId, 'Version'] },
Name: aliasName,
ProvisionedConcurrencyConfig: {
ProvisionedConcurrentExecutions: provisionedConcurrency,
},
},
},
DependsOn: functionLogicalId,
};
DependsOn: functionLogicalId,
};

cfTemplate.Resources[aliasLogicalId] = aliasResource;
});
cfTemplate.Resources[aliasLogicalId] = aliasResource;
});
} else {
versionCompilationPromise = BbPromise.resolve();
}
Expand Down Expand Up @@ -719,6 +746,48 @@ class AwsCompileFunctions {
Value: 'Value',
};
}

addFileContentsToHashes(filePath, hashes) {
pwithams marked this conversation as resolved.
Show resolved Hide resolved
// Read the file in chunks and add them to the hash (saves memory and performance)
return new BbPromise((cb, reject) => {
const readStream = fs.createReadStream(filePath);
readStream
.on('data', chunk => {
hashes.forEach(hash => {
hash.write(chunk);
});
})
.on('close', () => {
cb();
})
.on('error', error => {
reject(new this.serverless.classes.Error(`Could not file content to hash: ${error}`));
pwithams marked this conversation as resolved.
Show resolved Hide resolved
});
});
}

extractLayerPropertiesFromFunction(functionProperties, cfTemplate) {
pwithams marked this conversation as resolved.
Show resolved Hide resolved
const cloudformationLayerVersionType = 'AWS::Lambda::LayerVersion';
const layerProperties = [];
Object.keys(functionProperties).forEach(key => {
const value = functionProperties[key];
pwithams marked this conversation as resolved.
Show resolved Hide resolved
if (!Array.isArray(value)) {
return;
}
value.forEach(potentialLayerObject => {
if (!_.isObject(potentialLayerObject)) {
return;
}
if ('Ref' in potentialLayerObject) {
pwithams marked this conversation as resolved.
Show resolved Hide resolved
const properties = cfTemplate.Resources[potentialLayerObject.Ref];
if (properties.Type === cloudformationLayerVersionType && properties.Properties) {
layerProperties.push({ name: potentialLayerObject.Ref, properties });
pwithams marked this conversation as resolved.
Show resolved Hide resolved
}
}
});
});
return layerProperties;
}
}

module.exports = AwsCompileFunctions;