Lambda Version fails to publish when CloudFormation changes but Lambda config doesn't #8251
Comments
|
I believe this was fixed by #8212 |
|
@brent-jackson great thanks for report. It's an interesting problem. As @thewizarodofoz pointed we're solving it in #8212 for few obvious cases, but indeed, where we deal with values that are resolved on AWS side during deployment we have limited means to properly detect weather real change happened or not.
This looks as interesting idea. I take this could only by solved by changing the result zip file. We could either add some Other issue, is that for some deployments users prepare zip artifacts on their own, and in that case I think we can't apply any of the above. We could also try to resolve those env vars prior hash calculation (we already resolve them for local invocation), but that's limited, as we resolve just some CF functions, and it's not possible to resolve every CF function before deployment. |
|
@medikoo the proposal that @brent-jackson has to inject the versionDigest into the function environment is sufficient to ensure that Lambda creates a new version; it's not necessary to modify the function code / artifact. It seems like a safe workaround to me. |
The solution implemented in #8212 won't resolve this issue. Same error message, different underlying reason.
I don't think we are limited to changing the zip. The AWS docs state that Lambda "doesn't publish a version if the function's configuration and code haven't changed since the last version." So we should be able to change either of them. Looking at the Lambda API docs for UpdateFunctionConfiguration and UpdateFunctionCode should give us an idea of which fields can be changed to allow a new Lambda::Version to be successfully published. The tricky part is that we are blind to the logic CloudFormation uses to convert a Lambda::Function resource into Lambda UpdateFunctionCode and UpdateFunctionConfiguration API calls. I feel that the most benign change available on the Lambda::Function resource is to add an environment variable along the lines of Your idea of including a |
@brent-jackson that sounds great to me. Indeed that's probably cleanest and most reliable solution. PR that adds that is welcome! |
|
have the same issue after update to 2.3.0 |
|
@Enase that issue was not yet solved. We welcome a PR that addresses that! |
In some instances the version hash may change due to a change involving CloudFormation instrisic functions, but upon resolving the functions, the resulting lambda configuration is identical to what is currently deployed. This causes the Lambda::Version to fail to publish since AWS sees the lambdas as identical. This commit embeds an environment variable onto each lambda function with the value of the version config hash, ensuring that anytime the config hash changes, the resulting lambda configuration will be changed, allowing a Lambda::Version to be published. Resolves issue serverless#8251
|
Unfortunately we needed to revert #8310, as unconditionally adding an env var will break lambdas attached to CloudFront distributions (note: we remove env vars for lambda explicitly configured with Do we have any better ideas on how to address that? Ideally if it's done via Lambda configuration, as patching lambda code is not a solution when we're relying on prebuilt artifacts |
|
Just dropping a note that I never had this problem before, but since updating |
|
@Bersaelor The same for me and the latest version without the issue is: |
|
@Enase thank you for dropping that info, after going back to 2.2.0 too ( |
|
@Bersaelor @Enase can you open other bug report? As what you reports looks as regression introduced with #8066, while this issue describes different problem (happening already before 2.2.0) |
|
I tried documenting it in #8392 but it seems I can only consistently reproduce it in out production platform which isn't great for experimenting with it. |
|
@medikoo That's definitely regression, but it's hard to find the reason in production environment. Hope I will have a time to test and share some details. |
|
@Enase @Bersaelor thanks for details. Yes I can confirm it's a regression, please see my comment: #8392 (comment) Ideally would be to find a proper solution for this issue, this will also make regression gone |
|
We see this issue as well after upgrading from 1.x to 2.6/2.7 |
|
@coyoteecd check #8392 (comment) |
|
@medikoo I already read the related tickets; I don't like the workaround of modifying all the lambdas. When it becomes annoying I'll probably try disabling versions instead, or rolling back to 2.2.0. But I'd rather see this fixed. I saw this error several times, including when deploying locally to a dev stage. What I noticed till now is that when it fails, it fails repeatedly. That means once it happens, I could put additional logging to see what's going wrong. Can I help with debugging, in which case some pointers where to look would be much appreciated? |
|
@coyoteecd if you feel that you're dealing with other problems than described here and #8392, then ideally if you open another issue with good explanation and minimal test case that will allow us to introduce the problem. In other case, ideally let's agree on a proper solution here and deliver PR so matter is finally addressed. |
|
No, I think I just misunderstood that comment. This other comment just clarified it. Somehow I thought that I had to modify the lambdas on every deploy, when in fact it seems that once I successfully deploy all functions with serverless 2.8, the error would not reappear. |
|
@medikoo I got this error again, and this time it's not related to upgrades from 1.x to 2.x. Luckily it happened together with a very limited changeset, describing this in case it helps addressing the problem:
Reading through the initial description of the issue, I believe the cause is in this case Serverless Framework generates a new function version because it sees the ImportValue key has changed. However the resolved value did not change, therefore deployment fails because the actual configuration is the same. Would it be possible to resolve these imports at deployment time, to make the resolved values part of the hash calculations? Meanwhile, my workaround in this case was to add a "dummy" env variable to each offending function, as a way to force a configuration (somewhat similar to the dummy code change above, but in serverless.yml) |
|
@coyoteecd great thanks for feedback. I think in this specific case, best solution is to either (1) rely on Serverless Framework variables (check: https://www.serverless.com/framework/docs/providers/aws/guide/variables/#reference-cloudformation-outputs) - they are guaranteed to be resolved prior deployment, (2) apply some ineffective update (e.g. whitespace change) to lambda code, it'll allow you to deploy such configuration without issues) Problem with attempting to resolve CF functions before deploying, is that for some cases it's not possible to resolve them before deployment (e.g. they may refer to meta of resources to be deployed with the stack, not yet existing) and there's no ideal documentation on AWS side how to resolve them. Introducing an attempt for resolving them on our side will definitely introduce a significant maintenance baggage, and extra pool of bugs to address. Technically our approach is, that we treat CF intrinsic functions as something AWS-specific, not really meant to be resolved on Framework side. We support them only in values intended to be put it as-is into result CF template. There are some exception where we attempt to resolve it (as e.g. local invocation) but support for it is only partial, and only for second-case scenarios (where first case is to pass them as-is to CF template). |
|
We're leaning into direction where ideally CF intrinsic functions are not used in the config, and it's only Framework variables through which we access data. So just replacing CF intrinsic functions with variables will solve this issue, and for a meantime no better solution seems available |
|
Closing, as this issue was triggered by a change with which lambda hashing algorithm slightly changed, and that created scenarios where in some cases hash was changed for unchanged function configuration. We didn't revert then, but promised to ourselves to not repeat the same mistake again, and another (hopefully final) change of version hashing algorithm was introduced behind the flag. If you're still affected by it, you can easily upgrade with the small helper we've created. See: https://www.serverless.com/framework/docs/guides/upgrading-v3#lambda-hashing-algorithm |
Description
CloudFormation can be used to help define a Lambda's configuration. In cases where the CloudFormation changes in a way such that the generated Lambda config remains the same as a previously published Lambda, the
sls deploystep fails due to CloudFormation being unable to create the Lambda::Version resource with the following error:The AWS Lambda API PublishVersion docs state:
However, since the serverless framework calculates the logical ID of the Lambda::Version resource based on the hashed handler code and lambda config (prior to CloudFormation intrinsic function resolution) [source], a change to the a Lambda's config will always generate a new Lambda::Version logical ID, even if the resolved CloudFormation generates an identical Lambda config.
Disabling
provider.versionFunctionswould likely work-around this issue, but is not something we want to do, as we want the benefits of being able to use Lambda traffic shifting between versions.serverless.yml before changeserverless.yml after changesls deploy --stage brentjoutputInstalled version
Proposed Solution
In some ideal world, the serverless framework is able detect that the raw CloudFormation will resolve to the same Lambda config, and prevent creating an entirely new Lambda::Version resource. But since the serverless framework uses CloudFormation as a deployment mechanism, this is impossible.
Instead, the framework needs a way to ensure that the generated Lambda config changes if the hash of the config it sees has changed. My proposed solution is to append an environment variable onto each Lambda::Function resource with a value of the hashed Lambda config (as calculated here) (or just the Lambda::Function logical ID). This will ensure that anytime the hash changed, the generated Lambda config has to have changed, meaning the deployment of the Lambda::Version should succeed.
I hastily threw together an plugin locally to verify that this would work, and it seems to. But I felt this was more of a bug than a feature, so it is better to be fixed in the framework itself.
Related Issues
These issues also discuss the lambda versioning logic, but are not duplicates:
I believe this issue may be describing a different side effect of the same behaviour/bug
The text was updated successfully, but these errors were encountered: