Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug: 'null' values are not allowed in templates since 2.5.0 for Condition/IpAddress #8370

Closed
DennisTruemperMd opened this issue Oct 8, 2020 · 4 comments · Fixed by #8374
Closed

Bug: 'null' values are not allowed in templates since 2.5.0 for Condition/IpAddress #8370

DennisTruemperMd opened this issue Oct 8, 2020 · 4 comments · Fixed by #8374
Assignees
@medikoo
Labels
bug cat/config-schema

Comments

@DennisTruemperMd
Copy link

With earlyer versions cloudwatch template looked like this (ApiGatewayRestApi only):

"ApiGatewayRestApi": {
      "Type": "AWS::ApiGateway::RestApi",
      "Properties": {
        "Name": "local-myservice",
        "EndpointConfiguration": {
          "Types": [
            "EDGE"
          ]
        },
        "Policy": {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Principal": "*",
              "Action": "execute-api:Invoke",
              "Resource": [
                "execute-api:/*/*/*"
              ],
              "Condition": {}
            }
          ]
        }
      }
    },

with 2.5.0 it looks like this:

   "ApiGatewayRestApi": {
      "Type": "AWS::ApiGateway::RestApi",
      "Properties": {
        "Name": "local-myservice",
        "EndpointConfiguration": {
          "Types": [
            "EDGE"
          ]
        },
        "Policy": {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Principal": "*",
              "Action": "execute-api:Invoke",
              "Resource": [
                "execute-api:/*/*/*"
              ],
              "Condition": {
                "IpAddress": null
              }
            }
          ]
        }
      }
    },

ipAddress: null leads to error:

Error: The CloudFormation template is invalid: [/Resources/ApiGatewayRestApi/Type/Policy/Statement/0/Condition/IpAddress] 'null' values are not allowed in templates
at /home/runner/work/my-service-name/my-service-name/node_modules/serverless/lib/plugins/aws/deploy/lib/validateTemplate.js:20:13
at tryCatcher (/home/runner/work/my-service-name/my-service-name/node_modules/bluebird/js/release/util.js:16:23)
at Promise._settlePromiseFromHandler (/home/runner/work/my-service-name/my-service-name/node_modules/bluebird/js/release/promise.js:547:31)
at Promise._settlePromise (/home/runner/work/my-service-name/my-service-name/node_modules/bluebird/js/release/promise.js:604:18)
at Promise._settlePromise0 (/home/runner/work/my-service-name/my-service-name/node_modules/bluebird/js/release/promise.js:649:10)
at Promise._settlePromises (/home/runner/work/my-service-name/my-service-name/node_modules/bluebird/js/release/promise.js:725:18)
at _drainQueueStep (/home/runner/work/my-service-name/my-service-name/node_modules/bluebird/js/release/async.js:93:12)
at _drainQueue (/home/runner/work/my-service-name/my-service-name/node_modules/bluebird/js/release/async.js:86:9)
at Async._drainQueues (/home/runner/work/my-service-name/my-service-name/node_modules/bluebird/js/release/async.js:102:5)
at Immediate.Async.drainQueues [as _onImmediate] (/home/runner/work/my-service-name/my-service-name/node_modules/bluebird/js/release/async.js:15:14)
at processImmediate (internal/timers.js:456:21)

serverless.yml 
Could not reproducs minimal yaml by now, but I think it is not needed for this case, will try later again if needed

This might be the important part:
We are using a ipFilter to allow just some IPs. But not for all stages. 

For stages with whiteList there is an file. For the ones without we are using a default value.
 custom.ipWhiteListDefault: {}
custom.ipWhiteList: ${file(./serverless.ip.${self:provider.stage}.yml):ips, self:custom.ipWhiteListDefault}

This is how it is used in ApiGateway
- Resources:
      ApiGatewayRestApi:
        Type: AWS::ApiGateway::RestApi
        Properties:
          Name: ${self:provider.stage}-${self:service.name}
          Policy:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Principal: '*'
                Action: execute-api:Invoke
                Resource:
                  - execute-api:/*/*/*
                Condition:
                  IpAddress: ${self:custom.ipWhiteList}
npx sls deploy -s dev output 
$ npx sls deploy -s dev
Serverless: To ensure safe major version upgrades ensure "frameworkVersion" setting in service configuration (recommended setup: "frameworkVersion: ^2.5.0")
Serverless: Load command interactiveCli
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command upgrade
Serverless: Load command uninstall
Serverless: Load command webpack
Serverless: Load command offline
Serverless: Load command offline:start
Serverless: Load command create_domain
Serverless: Load command delete_domain
Serverless: Load command login
Serverless: Load command logout
Serverless: Load command generate-event
Serverless: Load command test
Serverless: Load command dashboard
Serverless: Load command output
Serverless: Load command output:get
Serverless: Load command output:list
Serverless: Load command param
Serverless: Load command param:get
Serverless: Load command param:list
Serverless: Load command studio
 
 Serverless Warning --------------------------------------

  A valid file to satisfy the declaration 'file(./serverless.ip.dev.yml):ips,self:custom.ipWhiteListDefault' could not be found.


 Serverless Warning --------------------------------------

  A valid file to satisfy the declaration 'file(./serverless.ip.dev.yml):ips,self:custom.ipWhiteListDefault' could not be found.

Serverless: Invoke deploy
Serverless: Invoke package
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:common:cleanupTempDir
Serverless: Invoke webpack:validate
Serverless: Invoke webpack:compile
Serverless: Bundling with Webpack...
Time: 12927ms
Built at: 2020-10-08 6:02:37 PM
     /*WBPACK LOG*/ 
Serverless: Invoke webpack:package
Serverless: No external modules needed
Serverless: Packaging service...
Serverless: Invoke aws:package:finalize
Serverless: serverless-cloudformation-sub-variables: Converting AWS CloudFormation Sub variables...
Serverless: Invoke aws:common:moveArtifactsToPackage
Serverless: Invoke aws:common:validate
Serverless: [AWS s3 200 0.174s 0 retries] getBucketLocation({ Bucket: 'my.eu-central-1.serverless.deploys' })
Serverless: [AWS apigatewayv2 200 0.364s 0 retries] getDomainName({ DomainName: 'dev.myapi.product.services' })
Serverless: Invoke aws:deploy:deploy
Serverless: [AWS cloudformation 200 0.162s 0 retries] describeStacks({ StackName: 'amya-dev' })
Serverless: [AWS s3 200 0.116s 0 retries] listObjectsV2({ Bucket: 'my.eu-central-1.serverless.deploys',
  Prefix: 'serverless/amya/dev' })
Serverless: [AWS s3 200 0.09s 0 retries] headObject({ Bucket: 'my.eu-central-1.serverless.deploys',
  Key:
   'serverless/amya/dev/1602170053586-2020-10-08T15:14:13.586Z/compiled-cloudformation-template.json' })
Serverless: [AWS s3 200 0.096s 0 retries] headObject({ Bucket: 'my.eu-central-1.serverless.deploys',
  Key:
   'serverless/amya/dev/1602170053586-2020-10-08T15:14:13.586Z/amya.zip' }) 
Serverless: [AWS lambda 200 0.149s 0 retries] getFunction({ FunctionName: 'amya-dev-clientAPIPost' })
Serverless: [AWS sts 200 0.422s 0 retries] getCallerIdentity({})
Serverless: Uploading CloudFormation file to S3...
Serverless: [AWS s3 200 0.113s 0 retries] putObject({ Body:
   <Buffer 7b 22 41 57 53 54 65 6d 70 6c 61 74 65 46 6f 72 6d 61 74 56 65 72 73 69 6f 6e 22 3a 22 32 30 31 30 2d 30 39 2d 30 39 22 2c 22 44 65 73 63 72 69 70 74 ... >,
  Bucket: 'my.eu-central-1.serverless.deploys',
  Key:
   'serverless/amya/dev/1602172962540-2020-10-08T16:02:42.540Z/compiled-cloudformation-template.json',
  ContentType: 'application/json',
  Metadata:
   { filesha256: '2uwGhPDmGam5Kx2m4MV5mFkap/xItakFvTRgQ3Rph5A=' } })
Serverless: Uploading artifacts...
Serverless: Uploading service amya.zip file to S3 (2.23 MB)...
Serverless: [AWS s3 200 0.383s 0 retries] putObject({ Body:
   <Buffer 50 4b 03 04 14 00 08 00 08 00 00 00 21 00 00 00 00 00 00 00 00 00 00 00 00 00 14 00 00 00 73 72 63 2f 61 70 6f 6c 6c 6f 2d 6c 61 6d 62 64 61 2e 6a 73 ... >,
  Bucket: 'my.eu-central-1.serverless.deploys',
  Key:
   'serverless/amya/dev/1602172962540-2020-10-08T16:02:42.540Z/amya.zip',   
  ContentType: 'application/zip',
  Metadata:
   { filesha256: 'nmMOU0fcqjx4F3PHlmfP97MhWqhSFHg0T8Rxe5hkv9c=' } })
Serverless: Validating template...
Serverless: [AWS cloudformation 400 0.151s 0 retries] validateTemplate({ TemplateURL:
   'https://s3.amazonaws.com/my.eu-central-1.serverless.deploys/serverless/amya/dev/1602172962540-2020-10-08T16:02:42.540Z/compiled-cloudformation-template.json' })
 
  Error --------------------------------------------------

  Error: The CloudFormation template is invalid: [/Resources/ApiGatewayRestApi/Type/Policy/Statement/0/Condition/IpAddress] 'null' values are not allowed in templates
      at provider.request.catch.error (C:\git\my-service-name\node_modules\serverless\lib\plugins\aws\deploy\lib\validateTemplate.js:20:13)
  From previous event:
      at AwsDeploy.validateTemplate (C:\git\my-service-name\node_modules\serverless\lib\plugins\aws\deploy\lib\validateTemplate.js:16:85)
  From previous event:
      at AwsDeploy.BbPromise.bind.then (C:\git\my-service-name\node_modules\serverless\lib\plugins\aws\deploy\index.js:119:39)
  From previous event:
      at Object.aws:deploy:deploy:validateTemplate [as hook] (C:\git\my-service-name\node_modules\serverless\lib\plugins\aws\deploy\index.js:115:30)       
      at BbPromise.reduce (C:\git\my-service-name\node_modules\serverless\lib\classes\PluginManager.js:510:55)
  From previous event:
      at PluginManager.invoke (C:\git\my-service-name\node_modules\serverless\lib\classes\PluginManager.js:510:22)
      at PluginManager.spawn (C:\git\my-service-name\node_modules\serverless\lib\classes\PluginManager.js:530:17)
      at AwsDeploy.BbPromise.bind.then (C:\git\my-service-name\node_modules\serverless\lib\plugins\aws\deploy\index.js:93:48)
  From previous event:
      at Object.deploy:deploy [as hook] (C:\git\my-service-name\node_modules\serverless\lib\plugins\aws\deploy\index.js:89:30)
      at BbPromise.reduce (C:\git\my-service-name\node_modules\serverless\lib\classes\PluginManager.js:510:55)
  From previous event:
      at PluginManager.invoke (C:\git\my-service-name\node_modules\serverless\lib\classes\PluginManager.js:510:22)
      at getHooks.reduce.then (C:\git\my-service-name\node_modules\serverless\lib\classes\PluginManager.js:545:24)
  From previous event:
      at PluginManager.run (C:\git\my-service-name\node_modules\serverless\lib\classes\PluginManager.js:545:8)
      at variables.populateService.then (C:\git\my-service-name\node_modules\serverless\lib\Serverless.js:168:33)
  From previous event:
      at Serverless.run (C:\git\my-service-name\node_modules\serverless\lib\Serverless.js:155:74)
      at serverless.init.then.then (C:\git\my-service-name\node_modules\serverless\scripts\serverless.js:50:26)
      at runCallback (timers.js:705:18)
      at tryOnImmediate (timers.js:676:5)
      at processImmediate (timers.js:658:5)
      at process.topLevelDomainCallback (domain.js:126:23)
  From previous event:
      at Object.<anonymous> (C:\git\my-service-name\node_modules\serverless\scripts\serverless.js:50:4)
      at Module._compile (internal/modules/cjs/loader.js:778:30)
      at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)      at Module.load (internal/modules/cjs/loader.js:653:32)
      at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
      at Function.Module._load (internal/modules/cjs/loader.js:585:3)
      at Module.require (internal/modules/cjs/loader.js:692:17)
      at require (internal/modules/cjs/helpers.js:25:18)
      at Object.<anonymous> (C:\git\my-service-name\node_modules\serverless\bin\serverless.js:47:1)
      at Module._compile (internal/modules/cjs/loader.js:778:30)
      at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)      at Module.load (internal/modules/cjs/loader.js:653:32)
      at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
      at Function.Module._load (internal/modules/cjs/loader.js:585:3)
      at Function.Module.runMain (internal/modules/cjs/loader.js:831:12)      
      at findNodeScript.then.existing (C:\Users\dtruemper\AppData\Roaming\npm\node_modules\npm\node_modules\libnpx\index.js:268:14)

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com

  Your Environment Information ---------------------------
     Operating System:          win32
     Node Version:              10.17.0
     Framework Version:         2.5.0 (local)
     Plugin Version:            4.0.4
     SDK Version:               2.3.2
     Components Version:        3.2.1

Installed version

Framework Core: 2.5.0 (local)
Plugin: 4.0.4
SDK: 2.3.2
Components: 3.2.1
@dhrrgn
Copy link

dhrrgn commented Oct 8, 2020

We ran into this issue, and it ended up being a case sensitivity issue. This PR (#8319) in added type coercion, which ended up making things case sensitive

@dhrrgn
Copy link

dhrrgn commented Oct 8, 2020

Did some digging, because I was curious. This is actually because your ipWhiteListDefault has a value of {}, which gets caught in the null checking in the PR I mentioned (because it tries to traverse the object and it is empty, so custom.ipWhiteList ends up being null, hence the error.

This change should have been marked as a breaking change, because it broke a lot of things.

/cc @medikoo

@DennisTruemperMd
Copy link
Author

DennisTruemperMd commented Oct 9, 2020 •
edited

Hi @dhrrgn , thanks for the quick response. I could not see if you will fix this in a later version or not, so I searched for a workaround.
My current solution to the problem is duplication of the apigateway resource. Now I have one with condition and one without in diffrent files, referencing per stage. Instead of just setting ipAddress value for Condition per stage.

For someone having the same problem:

resources: 
    # add apigateway with whitelist for prod and apigateway without whitelist for other stages
    - ${file(serverless.apigateway.${self:provider.stage}.yml),file(serverless.apigateway.default.yml)}

@medikoo medikoo mentioned this issue Oct 9, 2020
Merged
@medikoo
Copy link
Contributor

medikoo commented Oct 9, 2020

@DennisTruemperMd thanks for report.

I believe it's caused by an issue introduced with #8319 where properties with undefined value explicitly assigned where converted to null.

It's being fixed here #8374

If it's not that I would need a proper reproduction case

@medikoo medikoo added the bug label Oct 9, 2020
@medikoo medikoo self-assigned this Oct 9, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@medikoo medikoo

Labels
bug cat/config-schema
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Ensure to preserve "undefined" values properties when normalizing for schema serverless/serverless
3 participants
@medikoo @dhrrgn @DennisTruemperMd