"A version for this Lambda function exists ( 8 ). Modify the function to create a new version.." even when functions have changed

[Bersaelor]

I have a serverless.yml with 29 functions (194 resources, just under the limit).

As of a few days ago, every time I change a function and sls deploy -s prod, I get an error during the Cloudformation
stack deployment:

  Serverless Error ---------------------------------------
 
  An error occurred: CognitoDashverificationDashemailLambdaVersionNfb3OPkVsKcH9oNj9EXkK8zZS2ELzrjV7yI1dpp7cU - A version for this Lambda function exists ( 8 ). Modify the function to create a new version..

This happens regardless of function changed. The error always mentions a different function than the one I had changed, since I obviously don't change every function every time. I tried it in 2.6.0, 2.5.0, 2.4.0, 2.3.0, all with the same result.
Going back to 2.2.0 it works again as before.

I tried reducing my yml to a minimal version that reproduces the issue, by deploying reduced versions to my dev environment. Unfortunately deploying to dev seemed to always work, regardless of the amount of lambda functions.
And in production environment, I can't just remove functions and deploy the reduced production list. (One should note here that in the dev environment, I have to remove the CandidatesDynamoDbTable otherwise it'll complain that the table already exists, thats the only difference in the serverless.yml between posting to dev and prod).

Here's a redacted serverless.yml that reproduces the issue:

serverless.yml

service: ARCMS # NOTE: update this with your service name
#app: your-app-name
#tenant: your-tenant-name

custom:
  myStage: ${opt:stage, self:provider.stage}
  myEnvironment:
    MESSAGE:
      prod: "This is production environment"
      dev: "This is development environment"
  users-role: ***
  region: ${opt:region, self:provider.region}
  environment:
    NEWORDERTOPIC: ***
    REQUESTDXFTOPIC: ***
    MESSAGE: ***
    CANDIDATE_TABLE: ***
    IMAGE_BUCKET: ***
    MODEL_BUCKET: ***
    STAGE: ***
    ACCESS_STORE: ***
    ACCESS_MANAGER: ***
    ACCESS_ADMIN: ***
    
provider:
  name: aws
  stage: dev
  runtime: nodejs10.x
  region: eu-central-1

package:
  individually: true
  exclude:
    - "**/*"
    - node_modules/**

resources:
  Resources:
    CandidatesDynamoDbTable:
      Type: 'AWS::DynamoDB::Table'
      DeletionPolicy: Retain
      Properties:
        AttributeDefinitions:
          - AttributeName: "id"
            AttributeType: "S"   
          - AttributeName: "sk"
            AttributeType: "S"   
        KeySchema:
          - AttributeName: "id"
            KeyType: "HASH"
          - AttributeName: "sk"
            KeyType: "RANGE"
        ProvisionedThroughput:
          ReadCapacityUnits: 2
          WriteCapacityUnits: 2
        StreamSpecification:
          StreamViewType: "NEW_AND_OLD_IMAGES"
        TableName: ${self:provider.environment.CANDIDATE_TABLE}


functions:
  appData:
    handler: categories/categories.appData
    role: ${self:custom.users-role}
    memorySize: 512
    description: Cached, public endpoint with categories and models
    package:
      include:
        - categories/categories.js
        - shared/**
    events:
      - http:
          path: /{brand}/app-data
          method: get
          cors: true
          request:
            parameters:
              paths:
                brand: true

  refresh-app-data:
    handler: categories/categories.refreshAppData
    role: ${self:custom.users-role}
    memorySize: 512
    description: Refresh the cached appdata of categories and models manually before it's expired
    package:
      include:
        - categories/categories.js
        - shared/**
    events:
      - http:
          path: /{brand}/refresh-app-data
          method: post
          cors: true
          request:
            parameters:
              paths:
                brand: true
          authorizer:
            name: authorizer
            arn: ***

  categories:
    handler: categories/categories.all
    role: ${self:custom.users-role}
    memorySize: 512
    description: Cached, public collections endpoint
    package:
      include:
        - categories/categories.js
        - shared/**
    events:
      - http:
          path: /{brand}/categories
          method: get
          cors: true
          request:
            parameters:
              paths:
                brand: true
          authorizer:
            name: authorizer
            arn: ***

  create-category:
    handler: categories/categories.createNew
    role: ${self:custom.users-role}
    memorySize: 512
    description: Create new category with posted values should the current user have enough rights
    package:
      include:
        - categories/categories.js
        - shared/**
    events:
      - http:
          path: /{brand}/categories
          method: post
          cors: true
          request:
            parameters:
              paths:
                brand: true
          authorizer:
            name: authorizer
            arn: ***

  set-category-status:
    handler: categories/categories.setStatus
    role: ${self:custom.users-role}
    memorySize: 512
    description: Set status of existing category
    package:
      include:
        - categories/categories.js
        - shared/**
    events:
      - http:
          path: /{brand}/category/{category}/setStatus
          method: post
          cors: true
          request:
            parameters:
              paths:
                brand: true
          authorizer:
            name: authorizer
            arn: ***x

  delete-category:
    handler: categories/categories.delete
    role: ${self:custom.users-role}
    memorySize: 512
    description: Delete category should the current user have enough rights
    package:
      include:
        - categories/categories.js
        - shared/**
    events:
      - http:
          path: /{brand}/categories/{id}
          method: delete
          cors: true
          request:
            parameters:
              paths:
                brand: true
                id: true
          authorizer:
            name: authorizer
            arn: ***

  models:
    handler: models/models.all
    role: ${self:custom.users-role}
    memorySize: 512
    description: Cached, public model endpoint
    package:
      include:
        - models/models.js
        - shared/**
    events:
      - http:
          path: /{brand}/category/{category}/models
          method: get
          cors: true
          request:
            parameters:
              paths:
                brand: true
                category: true

  get-model:
    handler: models/models.get
    role: ${self:custom.users-role}
    memorySize: 512
    description: get a single model should the current user have enough rights
    package:
      include:
        - models/models.js
        - shared/**
    events:
      - http:
          path: /{brand}/category/{category}/models/{id}
          method: get
          cors: true
          request:
            parameters:
              paths:
                brand: true
                category: true
                id: true
          authorizer:
            name: authorizer
            arn: ***

  set-model-status:
    handler: models/models.setStatus
    role: ${self:custom.users-role}
    memorySize: 512
    description: set publish/unpublish status on models
    package:
      include:
        - models/models.js
        - shared/**
    events:
      - http:
          path: /{brand}/category/{category}/models/{id}/setStatus
          method: post
          cors: true
          request:
            parameters:
              paths:
                brand: true
                category: true
                id: true
          authorizer:
            name: authorizer
            arn: ***

  create-model:
    handler: models/models.createNew
    role: ${self:custom.users-role}
    memorySize: 512
    description: Create new model with posted values should the current user have enough rights
    package:
      include:
        - models/models.js
        - shared/**
    events:
      - http:
          path: /{brand}/category/{category}/models
          method: post
          cors: true
          request:
            parameters:
              paths:
                brand: true
                category: true
          authorizer:
            name: authorizer
            arn: ***

  copy-model:
    handler: models/models.copy
    role: ${self:custom.users-role}
    memorySize: 512
    description: Copy an existing model to a new brand and collection
    package:
      include:
        - models/models.js
        - shared/**
    events:
      - http:
          path: /{brand}/category/{category}/models/{id}
          method: post
          cors: true
          request:
            parameters:
              paths:
                brand: true
                category: true
                id: true
          authorizer:
            name: authorizer
            arn: ***

  delete-model:
    handler: models/models.delete
    role: ${self:custom.users-role}
    memorySize: 512
    description: Delete model should the current user have enough rights
    package:
      include:
        - models/models.js
        - shared/**
    events:
      - http:
          path: /{brand}/category/{category}/models/{id}
          method: delete
          cors: true
          request:
            parameters:
              paths:
                brand: true
                category: true
                id: true
          authorizer:
            name: authorizer
            arn: ***

  tags:
    handler: tags/tags.all
    role: ${self:custom.users-role}
    memorySize: 512
    description: all tags for one brand
    package:
      include:
        - tags/tags.js
        - shared/**
    events:
      - http:
          path: tags/{brand}/
          method: get
          cors: true
          request:
            parameters:
              paths:
                brand: true
          authorizer:
            name: authorizer
            arn: ***

  create-tag:
    handler: tags/tags.createNew
    role: ${self:custom.users-role}
    memorySize: 512
    description: create new tag with name for brand
    package:
      include:
        - tags/tags.js
        - shared/**
    events:
      - http:
          path: tags/{brand}/
          method: post
          cors: true
          request:
            parameters:
              paths:
                brand: true
          authorizer:
            name: authorizer
            arn: ***

  my-brands:
    handler: user/brands.get
    role: ${self:custom.users-role}
    memorySize: 512
    description: Get all brands the current uses is member in
    package:
      include:
        - user/brands.js
        - shared/access_methods.js
        - brand_settings.json
    events:
      - http:
          path: user
          method: get
          cors: true
          authorizer:
            name: authorizer
            arn: ***

  create-user:
    handler: user/user.createNew
    role: ${self:custom.users-role}
    memorySize: 512
    description: Create new user with posted values should the current user have enough rights
    package:
      include:
        - user/user.js
        - shared/access_methods.js
    events:
      - http:
          path: user
          method: post
          cors: true
          authorizer:
            name: authorizer
            arn: ***

  resend-user-invite:
    handler: user/user.resendInvite
    role: ${self:custom.users-role}
    memorySize: 512
    description: Resend invitation for existing user
    package:
      include:
        - user/user.js
        - shared/access_methods.js
    events:
      - http:
          path: user/resend/
          method: post
          cors: true
          authorizer:
            name: authorizer
            arn: ***

  delete-user:
    handler: user/user.delete
    role: ${self:custom.users-role}
    memorySize: 512
    description: Delete user should the current user have enough rights
    package:
      include:
        - user/user.js
        - shared/access_methods.js
    events:
      - http:
          path: user/{id}
          method: delete
          cors: true
          request:
            parameters:
              paths:
                id: true
          authorizer:
            name: authorizer
            arn: ***

  users-by-brand:
    handler: brand/users.all
    role: ${self:custom.users-role}
    timeout: 30
    memorySize: 512
    description: Get all users for a given brand
    package:
      include:
        - brand/users.js
        - shared/access_methods.js
        - shared/pagination.js
    events:
      - http:
          path: users
          method: get
          cors: true
          authorizer:
            name: authorizer
            arn: ***

  get-devices:
    handler: devices/devices.all
    role: ${self:custom.users-role}
    memorySize: 512
    description: Get all devices for the current user
    package:
      include:
        - devices/devices.js
    events:
      - http:
          path: devices
          method: get
          cors: true
          authorizer:
            name: authorizer
            arn: ***

  check-device:
    handler: devices/devices.check
    role: ${self:custom.users-role}
    memorySize: 512
    description: Check in with a device, add it if needed or reply that the user has used up his quota
    package:
      include:
        - devices/devices.js
    events:
      - http:
          path: device/check
          method: post
          cors: true
          authorizer:
            name: authorizer
            arn: ***

  delete-device:
    handler: devices/devices.delete
    role: ${self:custom.users-role}
    memorySize: 512
    description: Delete a device from the current user
    package:
      include:
        - devices/devices.js
    events:
      - http:
          path: device/{id}
          method: delete
          request:
            parameters:
              paths:
                id: true
          cors: true
          authorizer:
            name: authorizer
            arn: ***

  get-orders-v2:
    handler: orders/orders.allPaginated
    role: ${self:custom.users-role}
    memorySize: 512
    description: Get all orders for the current user or a specified third user depending on the accessLvl
    package:
      include:
        - orders/orders.js
        - shared/access_methods.js
        - brand_settings.json
        - shared/pagination.js
        
    events:
      - http:
          path: v2/orders
          method: get
          cors: true
          authorizer:
            name: authorizer
            arn: ***

  copy-orders:
    handler: orders/convert.copyFromTo
    role: ${self:custom.users-role}
    memorySize: 512
    description: Copy all orders by one user to a different user, typically to generate a larger data set
    package:
      include:
        - orders/convert.js
        - shared/access_methods.js

  get-order:
    handler: orders/orders.order
    role: ${self:custom.users-role}
    memorySize: 512
    description: Get a specific order based on brand, user and timeStamp
    package:
      include:
        - orders/orders.js
        - shared/access_methods.js
        - shared/pagination.js
        - brand_settings.json
    events:
      - http:
          path: order/{id}
          method: get
          cors: true
          authorizer:
            name: authorizer
            arn: ***

  create-order:
    handler: orders/orders.create
    role: ${self:custom.users-role}
    memorySize: 512
    description: Create and save a new order
    package:
      include:
        - orders/orders.js
        - shared/pagination.js
        - shared/access_methods.js
        - brand_settings.json
    environment:
      emailSNSArn: { "Fn::Join" : ["", ["arn:aws:sns:${self:custom.region}:", { "Ref" : "AWS::AccountId" }, ":${self:provider.environment.NEWORDERTOPIC}" ] ]  }
      dxfFileRequestArn: { "Fn::Join" : ["", ["arn:aws:sns:${self:custom.region}:", { "Ref" : "AWS::AccountId" }, ":${self:provider.environment.REQUESTDXFTOPIC}" ] ]  }
    events:
      - http:
          path: order/create
          method: post
          cors: true
          authorizer:
            name: authorizer
            arn: ***
      - http:
          path: order
          method: post
          cors: true
          authorizer:
            name: authorizer
            arn: ***

  send-new-order-mails:
    handler: email-notifications/emails.newOrder
    role: ${self:custom.users-role}
    memorySize: 512
    description: Send email notifications when new order is received
    package:
      include:
        - email-notifications/**
        - '!email-notifications/tests/**'
        - node_modules/mustache/mustache.min.js
        - brand_settings.json
    events:
      - sns: ${self:provider.environment.NEWORDERTOPIC}

  create-dxf-files:
    handler: email-notifications/dxffilemail.newRequest
    role: ${self:custom.users-role}
    memorySize: 2048
    timeout: 300
    description: Send email with a number of created dxf files for the ordered frames
    package:
      include:
        - email-notifications/dxffilemail.js
        - email-notifications/DXFCombiner.js
        - email-notifications/locales.js
        - brand_settings.json
        - node_modules/nodemailer/**
        - node_modules/makerjs/**
        - node_modules/bezier-js/**
        - node_modules/clone/**
        - node_modules/graham_scan/**
        - node_modules/kdbush/**
        - node_modules/xml2js/**
        - node_modules/sax/**
        - node_modules/xmlbuilder/**
    events:
      - sns: ${self:provider.environment.REQUESTDXFTOPIC}

  cognito-verification-email:
    handler: email-notifications/cognito.verification
    role: ${self:custom.users-role}
    memorySize: 512
    description: Send email with verification code for password reset
    package:
      include:
        - email-notifications/**
        - '!email-notifications/tests/**'
        - node_modules/mustache/mustache.min.js
        - brand_settings.json

sls deploy -s prod output

Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Excluding development dependencies...
Serverless: Excluding development dependencies...
Serverless: Excluding development dependencies...
Serverless: Excluding development dependencies...
Serverless: Excluding development dependencies...
Serverless: Excluding development dependencies...
Serverless: Excluding development dependencies...
Serverless: Excluding development dependencies...
Serverless: Excluding development dependencies...
Serverless: Excluding development dependencies...
Serverless: Excluding development dependencies...
Serverless: Excluding development dependencies...
Serverless: Excluding development dependencies...
Serverless: Excluding development dependencies...
Serverless: Excluding development dependencies...
Serverless: Excluding development dependencies...
Serverless: Excluding development dependencies...
Serverless: Excluding development dependencies...
Serverless: Excluding development dependencies...
Serverless: Excluding development dependencies...
Serverless: Excluding development dependencies...
Serverless: Excluding development dependencies...
Serverless: Excluding development dependencies...
Serverless: Excluding development dependencies...
Serverless: Excluding development dependencies...
Serverless: Excluding development dependencies...
Serverless: Excluding development dependencies...
Serverless: Excluding development dependencies...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service appData.zip file to S3 (6.54 KB)...
Serverless: Uploading service refresh-app-data.zip file to S3 (6.54 KB)...
Serverless: Uploading service categories.zip file to S3 (6.54 KB)...
Serverless: Uploading service cognito-verification-email.zip file to S3 (52.15 KB)...
Serverless: Uploading service create-dxf-files.zip file to S3 (628.95 KB)...
Serverless: Uploading service send-new-order-mails.zip file to S3 (52.15 KB)...
Serverless: Uploading service create-order.zip file to S3 (6.1 KB)...
Serverless: Uploading service get-order.zip file to S3 (6.1 KB)...
Serverless: Uploading service copy-orders.zip file to S3 (2.35 KB)...
Serverless: Uploading service get-orders-v2.zip file to S3 (6.1 KB)...
Serverless: Uploading service delete-device.zip file to S3 (2.83 KB)...
Serverless: Uploading service check-device.zip file to S3 (2.83 KB)...
Serverless: Uploading service get-devices.zip file to S3 (2.83 KB)...
Serverless: Uploading service users-by-brand.zip file to S3 (3.24 KB)...
Serverless: Uploading service delete-user.zip file to S3 (5.28 KB)...
Serverless: Uploading service resend-user-invite.zip file to S3 (5.28 KB)...
Serverless: Uploading service create-user.zip file to S3 (5.28 KB)...
Serverless: Uploading service my-brands.zip file to S3 (2.74 KB)...
Serverless: Uploading service create-tag.zip file to S3 (4.21 KB)...
Serverless: Uploading service tags.zip file to S3 (4.21 KB)...
Serverless: Uploading service delete-model.zip file to S3 (8.18 KB)...
Serverless: Uploading service copy-model.zip file to S3 (8.18 KB)...
Serverless: Uploading service create-model.zip file to S3 (8.18 KB)...
Serverless: Uploading service set-model-status.zip file to S3 (8.18 KB)...
Serverless: Uploading service get-model.zip file to S3 (8.18 KB)...
Serverless: Uploading service models.zip file to S3 (8.18 KB)...
Serverless: Uploading service delete-category.zip file to S3 (6.54 KB)...
Serverless: Uploading service set-category-status.zip file to S3 (6.54 KB)...
Serverless: Uploading service create-category.zip file to S3 (6.54 KB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
........................................................................................................
Serverless: Operation failed!
Serverless: View the full error output: ***

Serverless Error ---------------------------------------

An error occurred: CopyDashordersLambdaVersionvk88Tcgtu1LRggMQjJl6aOhSHdDZS5FGb62HpriD14 - A version for this Lambda function exists ( 5 ). Modify the function to create a new version..

Get Support --------------------------------------------
Docs: docs.serverless.com
Bugs: github.com/serverless/serverless/issues
Issues: forum.serverless.com

Your Environment Information ---------------------------
Operating System: darwin
Node Version: 10.16.0
Framework Version: 2.6.0
Plugin Version: 4.0.4
SDK Version: 2.3.2
Components Version: 3.2.1

Installed version

Framework Core: 2.6.0
Plugin: 4.0.4
SDK: 2.3.2
Components: 3.2.1

[Bersaelor]

PS: I tried deploying my service under a different name and reducing the functions there, but that also doesn't cause any errors.

[Enase]

#8251 (comment)

looks as regression introduced with #8066

[medikoo]

@Bersaelor thanks for report.

I believe the issue comes from a fact that with #8066 we fixed some discovered issues in lambda version hash generation, and it started to produce different hashes for lambda version than previously. That means, that for a same function code and configuration you'll get a different hash, and this is where AWS will complain.

I didn't envision this when we were merging it. Interestingly there's just two reports for that.

I believe right fix will come with solution for #8251, Otherwise a workaround on your side could be to simply modify in some way code for all the functions. Maybe you have some common util file used by all, you can tweak it's content in non-destructive manner and that'll ensure that redeployment will work, and issue should be gone.

[Bersaelor]

Thanks @medikoo , that makes a lot of sense and from reading that PR I could have come up with that by myself 😅

I changed a variable name in a file that's imported by practically all lambda code files and the error went away.

[Enase]

@medikoo the same for me, everything works fine after code modification. Thanks.

[jaska120]

I am experiencing the same issue using ^2.8.0.

EDIT: By downgrading to 2.2.0 I managed to redeploy without errors.

[medikoo]

@jaska120 have you tried suggestion mentioned in this comment: #8392 (comment) ?

[jaska120]

@medikoo for now 2.2.0 is sufficient for us. I didn't try the mentioned suggestion, since I wasn't sure if it would be a one time fix or if all the future deploys should be working after new hashes have been generated once for all functions. Could you confirm which should be the case? It the latter then I am up to try the suggestion.

[Bersaelor]

@jaska120 for me, changing one method that is imported into all my lambda functions was enough.
Once I had done this after updating to post-2.2.0 the error never appeared again. I guess I just had to make sure all method were hashed with the new hash function.

[eldos-dl]

Seeing the same issue, when the function code is not modified.
But, I am only getting this issue when I add the configuration lambdaHashingVersion: 20201221.

[pgrzesik]

@eldos-dl This is expected when upgrading to `lambdaHashingVersion: 20201221

[medikoo]

@eldos-dl when adding lambdaHashingVersion: 20201221 ensure that there are some changes to lambdas code (that's needed only for first deployment). Having that it'll go smoothly

[sjwoodr]

I ran into this same problem after upgrading to serverless 2.72.2

[medikoo]

@sjwoodr I'm not aware of any new changes that could trigger that. Still, we've prepared some solution and guide on how to tackle such errors. See: https://www.serverless.com/framework/docs/guides/upgrading-v3#lambda-hashing-algorithm

[medikoo]

I'm going to close this issue, as solution I've mentioned in the previous comment should help mitigate such issues, and we will definitely not introduce any changes which break version hash again

[sjwoodr]

@sjwoodr I'm not aware of any new changes that could trigger that. Still, we've prepared some solution and guide on how to tackle such errors. See: https://www.serverless.com/framework/docs/guides/upgrading-v3#lambda-hashing-algorithm

This guide will help, thanks!