Request validator limit of 20

[brian-geoghegan]

I am running into an issue with request validation. When using event schemas, a model is created and a request validator. The request validator limit per api is 20. So when your reach 21 schema's, 21 validators exist. Thus throwing the error.
Maximum number of Request Validators for this API has been reached. (Service: AmazonApiGateway; Status Code: 429; Error Code: LimitExceededException; Request ID: ; Proxy: null). and failing a deploy.

Could there be a change to reuse validators instead of creating new ones? I contacted AWS and this was the recommended solution, I tried using serverless-request-validator plugin and it creates a reusable validator that you can attach, but the issue still arises with the schema keywork in an event, as it still tries to create a new validator before attaching the existing one.

- http:
          path: /api/right-answers/favorites
          method: post
          request:
            schema:
              application/json: ${file(./right-answers/schema.json)}
          cors: true
          private: true
          authorizer: ${self:custom.myAuthorizerAzure}

⚠️⚠️ REPLACE WITH FULL COMMAND NAME output

⚠️⚠️ REPLACE WITH FULL COMMAND OUTPUT

Installed version2.20.0

$ serverless --version
Serverless: Running "serverless" installed locally (in service node_modules)
Framework Core: 2.20.1 (standalone)
Plugin: 4.5.3
SDK: n/a
Components: 3.8.1

[pgrzesik]

Hello @brian-geoghegan, in 2.28.0 (https://github.com/serverless/serverless/blob/master/CHANGELOG.md#2280-2021-02-26) we've introduced the ability to reuse schema validators: #7619

Would that be helpful to resolve your issue here?

[brian-geoghegan]

Hi pgrzesik thanks for the quick reponse!

So if I understand correctly, that will create a schema model that is reusable? But what about the validator itself? I don't want to reuse schema models, I want to reuse validators for different schema models. According to AWS support we should only need 3 validators and reuse them for many schema models, a body, header and parameter validator should be all that is needed. But in serverless deployments we get

As you can see there are alot of validators where the should only be 3.
Alternatively here are the models which are separate to the validators.

Does this make sense or have misunderstood how they even work XD

Judging by the PR

validatorName = modelName ?${modelName}Validator: null;
It still create a new validator for each model

[brian-geoghegan]

A solution for this should be, creating request validators and them in the schema, instead of creating a new request validator each time a schema is created? ${modelName}Validator should just be a generic bodyRequestValidator or whatever one would like to name it?.

Here is the response I received from AWS support:

Thank you for contacting AWS Premium Support, my name is Sean and I'll be assisting you today. I understand from the case details you are hitting a limit for the Maximum Number of Request Validators for a given API.

The limit for this is 20 and cannot be increased- however, these validators are intended to be reused and shared across APIs. Because request validators have three options ( 1.Validate body, 2.Validate query string parameters and headers, 3. Validate body, query string parameters, and headers) more than three validators shouldn't be required.

The serverless Framework creates a new Request Validator for every endpoint resource method. In this case you can list existing request validator using get-request-validators [1] CLI and provide the name of existing request validator in the CloudFormation template, so that CloudFormation will not create new request validator every time with random name and this will prevent from exceeding limit.

Furthermore, you can use AWSCLI command to list "get-request-validators" [2] and remove "delete-request-validator command" [3] any validator you do not need. After that, you could reuse validators for your future deployments.

[pgrzesik]

Thanks for clarification @brian-geoghegan - I've checked an in fact, we're creating separate validators which definitely seems unnecessary. I believe we should reuse validators instead of creating new ones. I'll dive deeper into the code to figure out a potential way of reusing them in an efficient and backward-compatible manner.

[brian-geoghegan]

Thanks @pgrzesik, a solution I tried was to create the request validator separately and first and create models after it. Could specify the validator separately at a stack level or create generic ones automatically. If you need anything more from me let me know!

[pgrzesik]

I believe the most reasonable approach would be to create the generic ones automatically whenever they are needed. From what I inspected it seems like we're always creating validators with the following params:

Type: 'AWS::ApiGateway::RequestValidator',
Properties: {
  RestApiId: { Ref: this.apiGatewayRestApiLogicalId },
  ValidateRequestBody: true,
  ValidateRequestParameters: true,
}

so it suggests that we could have a single validator in the stack that is reused across all AWS::ApiGateway::Method resources. What do you think @brian-geoghegan ?

[brian-geoghegan]

If you are able to check if it exists already after being created would be ideal instead of recreating it per lambda. Just so that it is only created once and is reused, if you reference the one already created while creating the model schema would be ideal.

I think the easiest for you guys could even just create one when an api gateway is created and then use that one if the schema key word appears. That way you can just focus on creating schema models when a lambda needs it.

The request validator plugin worked that way. Create a validator on the stack and then attach it to a lambda if you need it(if you also wanted to add a schema)

Thanks again for the quick response on this, love the framework!

[gambit66]

I hit this issue this last week and have submitted PR #9291 which will just create 1 request validator for all request schemas instead of creating a request validator per request schema as suggested by @pgrzesik and AWS support.

[pgrzesik]

Hello @gambit66 - thanks for proposing a PR that will address this issue. I have one concern though - did you confirm that the proposed change is not breaking already existing stacks that will have to recreate the validators? We wouldn't want to introduce any disruptions for existing services.

[gambit66]

Hello @gambit66 - thanks for proposing a PR that will address this issue. I have one concern though - did you confirm that the proposed change is not breaking already existing stacks that will have to recreate the validators? We wouldn't want to introduce any disruptions for existing services.

Hi @pgrzesik, thank you for taking the time to look at my PR. I have tested this update with a test project to try test all the different scenarios I could think of, such as having 1 function use events.http.request.schemas and another using the old events.http.request.schema property.

I also did a deployment using this code for an API project that was getting the Maximum number of Request Validators for this API has been reached error when we deployed. The 20 request validators that were previously created by a serverless deploy were successfully replaced by the new single shared request validator which is the behaviour we are expecting.

Before

After

[brian-geoghegan]

Thanks for working on this @gambit66

[poldark101]

FYI @gambit66 : One of our stacks creates the apig reqValidator. I have 2 other stacks using it for their events. If I leave in the reqValidator while deleting references to it. I get the error last reported above. If I remove the reqValidator and all references to it, I have a dependancy error on deploy (below).

An error occurred: aws-help-services-edge-api-sandbox - Export aws-help-services-edge-api-sandbox-bodyRequestValidator cannot be deleted as it is in use by aws-help-services-api-services-sandbox and aws-help-services-vpc-services-sandbox

The following worked for me.

1. (With Serverless version pre your change) Remove references to the validator dependent stacks - Then Deploy.
2. (With Serverless version post your change) Remove configuration for AWS::ApiGateway::RequestValidator - Then Deploy

[Pwntus]

Oh - it took some time for me to understand the solution implemented (and why I was unable to get it to work for me). Right now, serverless framework create a shared Request Validator per stack. Well, I've got more than 20+ stacks so the hard limit of 20 Request Validators per API Gateway is still hit.

What I ended up doing, in case someone is struggling with similar issues, was to install these two plugins:

serverless-disable-request-validators
serverless-reqvalidator-plugin

The first serverless-disable-request-validators will remove any automatically created Request Validators by serverless framework. I then manually define (through CloudFormation in the resources section of serverless.yml) in a shared stack the only used Request Validator and export it:

resources:
  Resources:
    ApiGatewayRequestValidator:
      Type: AWS::ApiGateway::RequestValidator
      Properties:
        RestApiId: !Ref ApiGatewayRestApi # Definition omitted
        ValidateRequestBody: true
        ValidateRequestParameters: true
        Name: 'Validate request body and query string parameters'

  Outputs:
    ApiGatewayRequestValidatorId:
      Value: !Ref ApiGatewayRequestValidator
      Export:
        Name: ApiGatewayRequestValidatorId

The serverless-reqvalidator-plugin then allows me to "associate" the shared Request Validator to resources that need it, like this:

functions:
  foo:
    handler: foo.main
    events:
      - http:
          path: bar/{param}
          method: get
          cors: true
          request:
            parameters:
              paths:
                param: true
          reqValidatorName: !ImportValue ApiGatewayRequestValidatorId