No xxx.zip provided when trying to deploy package using CodePipeline/CodeBuild

[VincentHuysmans1]

I'm trying to deploy a serverless app with two functions using CodeBuild. In the serverless.yml I used settings for packaging individually with includes and excludes. When I remove these settings from the yaml file, everything runs perfectly. Because we are going to add a lot more functions in the future we really need this functionality to work with AWS CodeBuild.

I feel like this is somehow related to this bug which is closed recently: #4720

serverless.yml

service: xxx
app: xxx-app

provider:
  name: aws
  runtime: nodejs12.x
  timeout: 10
  memorySize: 1546
  stage: ${opt:stage}
  iamRoleStatements:
      - Effect: 'Allow'
        Action:
          - 'secretsmanager:GetSecretValue'
        Resource:
              - ${self:custom.secretArn.${self:provider.stage}}
  region: eu-west-1
  apiGateway:
    restApiId: ${ssm:/api-gateway/${opt:apiName}/${self:provider.stage}/api-id~true}
    restApiRootResourceId: ${ssm:/api-gateway/${opt:apiName}/${self:provider.stage}/root-resource-id~true}

custom:
  secretArn:
    dev: xxx
  apiGatewayThrottling:
    maxRequestsPerSecond: 1000
    maxConcurrentRequests: 500
  apiKeys:
    - name: xxx
      value:
        encrypted: ${ssm:/kms/${opt:apiName}/${self:provider.stage}/api-key~true}
        kmsKeyRegion: eu-west-1

package:
  individually: true
  exclude:
    - ./**

functions:
  function1:
    handler: src/main/function1.getData
    package:
      include:
        - src/main/function1.js
        - src/libs/helper/**
        - node_modules/**
    events:
        - http:
            path: /x/x/x
            method: GET
            private: true
  function2:
    handler: src/main/function2.getData
    package:
      include:
        - src/main/function2.js
        - node_modules/**
    events:
        - http:
            path: /y/y/y
            method: GET
            cors: true

plugins:
  - serverless-api-gateway-throttling
  - serverless-add-api-key
  - serverless-aws-documentation

serverless deploy --stage $ENV --package $CODEBUILD_SRC_DIR/artifacts/$ENV -v output

Serverless Error ---------------------------------------
--
  
 No function1.zip file found in the package path you provided.
 

Installed version

 Operating System:          linux
 Node Version:              12.16.1
 Framework Version:         2.27.1
 Plugin Version:            4.4.3
 SDK Version:               2.3.2
 Components Version:        3.7.1

[pgrzesik]

Hello @VincentHuysmans1, thanks for reporting. I've tried to reproduce your issue locally with a simplified version of your config but with no luck - in the approach you're using, you have to first package your functions with package command - could you share how are you doing that? Could you also list the contents of the directory $CODEBUILD_SRC_DIR/artifacts/$ENV? Are there function artifacts as well as files with CF templates?

[VincentHuysmans1]

The package command:

serverless package --package artifacts/${ENV} --stage ${ENV} -v

The content of artifacts folder:

 -rw-r--r-- 1 root root     2077 Feb 23 21:09 cloudformation-template-create-stack.json
 -rw-r--r-- 1 root root    18533 Feb 23 21:09 cloudformation-template-update-stack.json
 -rw-r--r-- 1 root root    33995 Feb 23 21:09 serverless-state.json
 -rw-r--r-- 1 root root 16918705 Feb 23 21:09 function1.zip
 -rw-r--r-- 1 root root 16916396 Feb 23 21:09 function2.zip

Packaging is done by another CodeBuild step inside CodePipeline. The full artifact folder is used as input for the deploy step.

[VincentHuysmans1]

@pgrzesik Do you want me to provide some additional information?

[pgrzesik]

Hello @VincentHuysmans1 - I was just looking into it and trying to reproduce your issue but didn't manage to as of yet.

The listing of artifacts folder is from the separate build step or it's listed right before the serverless deploy --stage $ENV --package $CODEBUILD_SRC_DIR/artifacts/$ENV -v command is executed? It seems like the artifacts are not available inside the $CODEBUILD_SRC_DIR/artifacts/$ENV at the time the command gets executed

[VincentHuysmans1]

It is listed right before the deploy statement. When I remove the individually packaging from the serverless.yml, everything works just fine.

[pgrzesik]

While searching for potential clues, I've found a similar issue that a few people experienced in the past as well - #6964 - unfortunately, it seems like the fix didn't make it. Could you try using one of the workarounds proposed that uses relative path instead of the absolute one with CODEBUILD_SRC_DIR?

@medikoo As I see that you've been involved with the original issue - do you happen to remember what was the cause of it?

[medikoo]

@medikoo As I see that you've been involved with the original issue - do you happen to remember what was the cause of it?

@pgrzesik in #6964 bug was incorrect handling of --package option. I remember then that I investigated it, and that handling of that setting was spread across many places, and I think it was doubled at some point (but I can't be sure now, what speficically I saw)

I started working then on refactor #7009, and while refactoring handling of this option was relatively straightforward. I observed tens of tests being broken in many files, which drive me to conclusion that first those tests need to be refactored to runServerless, and unfortunately it got stuck then.

On revisiting, I would first confirm whether same issue won't be fixed by solution proposed here: #7298 If it's the case, I would focus just on that.

Otherwise I would revisit failing tests, and first ensure we have them moved to runServerless variant, and then refactor handling of this option.

[pgrzesik]

Thank you for insights @medikoo 🙇 I will dive a bit deep on that issue and see what I can find

[pgrzesik]

Hello @VincentHuysmans1 - thanks for patience here - unfortunatelly due to other priorities I won't be able to focus more on researching this issue during this week for sure - sorry about that. Did you try the proposed workaround from linked issue? Would that be an acceptable solution in the meantime?

[VincentHuysmans1]

Hi @pgrzesik - you too thanks for the time you've already spend on investigating. I have not tested the workaround yet, but I will try it today and come back at it.

[VincentHuysmans1]

@pgrzesik So I've tried using the relative path as described in #6964 - unfortunately I get the same error.

The command I've tried:

serverless deploy --stage $ENV --package ./artifacts/$ENV -v

[pgrzesik]

Thanks @VincentHuysmans1 - too bad that the workaround didn't help in this case - I'll try to dive deeper into that issue hopefully next week and report back with findings.

[lpturmel]

Do we have any news about this issue?

[pgrzesik]

Hello @ghostjester - unfortunately, due to other priorities I wasn't yet able to investigate it further.

[pgrzesik]

Hello @VincentHuysmans1 and @ghostjester - I've managed to find some time to investigate further but when using your specific example @VincentHuysmans1 I didn't manage to reproduce this issue locally. Are you able to deploy if you do following:

serverless package --package path/to/package
serverless deploy --package path/to/package

?

I'm trying to exclude any issues related with how CODEBUILD extracts the cached artifacts, so that information would be really helpful.

Also, thanks a lot for patience, unfortunately a lot of other priorities came up which didn't let me to focus on this one.

[jplandry908]

Hi @pgrzesik. I am experiencing the same issue and came across this looking for a solution. I am using AWS CodePipeline to Build and then Deploy the solution. It is breaking up the packages and building correctly, but the Deploy process is failing - stating that the .zip file is not there for the individual packages, but I can confirm it is in that directory. I have traced it to an invalid path in the serverless-state.json file, and attempting to do a sed command to change the path to what it needs to be. I'll let you know if I can find a workaround, which may help implement a fix.

[jplandry908]

For anyone else experiencing this issue, I finally identified the root cause and a simple workaround.

In summary, the "artifact" incorrectly has "$CODEBUILD_SRC_DIR" in the value. Also, the srcxxxxxxxxxx number changes between deploy and build, so you can't just do a find/replace for the exact string in DEPLOY since the value is the $CODEBUILD_SRC_DIR that was in BUILD.

For each function in the serverless-state.json, you will see an artifact property that looks similar to:
"artifact": "/codebuild/output/src887072543/src/.serverless/function.zip"
while it should actually look like this:
"artifact": ".serverless/function.zip"

To work around this issue, I simply added the following line in my BUILD commands, immediately above the 'serverless deploy ...' statement.

sed -i 's/\/codebuild\/output\/.*\/src\///' artifacts/${!stage}/serverless-state.json

Note: I am using the '${!stage}' value in my code, but this may be different in your implementation - probably $ENV.

This will transform the "/codebuild/output/src887072543/src/.serverless/function.zip" statements to ".serverless/function.zip", allowing the deploy process to find the file without error.

This took me a while to figure out, so I hope it helps others that may be experiencing the same issue.

[pessrini]

Hi @pgrzesik @jplandry908 Im facing the same issue. I am using azure pipeline to package and deploy artifacts, packaging and deploying in different azure pipeline agents so its getting this issue.

For each function in the serverless-state.json, I can see the the artifact that look similar to:
"artifact": "/root/agent09/_work/12/s/.serverless/service.zip"
while deploy artifacts , this path will change depend on available pipeline agent like "artifact": "/root/agent109/_work/12/s/.serverless/service.zip"

Installed Version

Framework Core: 2.72.1 (local) Plugin: 5.5.4 SDK: 4.3.0 Components: 3.18.2

[jplandry908]

Hi @pessrini. If you are having the same issue that I was having, then you need to strip out the /root/agent* code from the artifact line.

So in your example, you have:
"artifact": "/root/agent09/_work/12/s/.serverless/service.zip"

And it should look like:
"artifact": ".serverless/service.zip"

Using the sed command, you can remove all text from two strings using the following command:
sed 's/{START}.*{END}//'

So instead of the command I specified above, for your situation try:
sed -i 's/\/root\/.*\/s\///' artifacts/${!stage}/serverless-state.json
This should remove everything between /root/ and /s/, including those strings.

The -i command says to change it in place, and then change artifacts/${!stage}/serverless-state.json to the path your serverless-state.json file is located. Hopefully this works for you.

[coyoteecd]

@pgrzesik and others, I ran into what I think is the same issue, and I can reproduce it consistently in my local dev (as well as in the CI/CD where it originally happened). I switched our local stack to use serverless-esbuild for building and started getting these errors randomly.

Our stack:

• uses package individually: true for lambda functions
• uses serverless-esbuild for packaging
• has two separate stages for build (serverless package) and deploy (serverless deploy --package name-of-package). These stages do not run necessarily in the same root directory, they depend on the CI as explained in the previous comments.

If you look at the state file of the built package, it contains the FULL PATH of the zip file under each function entry:

        "name": "my-lambda",
        "package": {
          "artifact": "/home/jenkins/build/workspace/branch-name@2/.serverless/my-lambda.zip"
        },

I compared this with the state file before we upgraded to serverless-esbuild (were using serverless-webpack) and there we had a relative path as artifact:

        "package": {
          "artifact": ".serverless/my-lambda.zip"
        },

The code that's processing the deploy extracts the artifactFilePath from the package.artifact property, therefore getting an absolute path there which in the case of CI/CD systems, may not be the same between package and deploy.

I will report an issue to the serverless-esbuild plugin, given that your intention is to use this plugin as default (#10700).