JSON schema validation is called for application/octet-stream

[nflaig]

As per fasitfy docs schema validation should only be called if body is application/json

Validation will only be attempted if the content type is application-json, as described in the documentation for the content type parser.

My server is configured like this

const server = fastify({
    ajv: {customOptions: {coerceTypes: "array"}},
    querystringParser: (str) => parseQueryString(str, {comma: true, parseArrays: false}),
});

server.addContentTypeParser(
  "application/octet-stream",
  {parseAs: "buffer"},
  async (_request: FastifyRequest, payload: Buffer) => {
    // This is called
    console.log(payload) // <Buffer 00 00 00 00 00 00 00 00 00 00 00 00 ... >
    return payload;
  }
);

The schema is really simple

schema: {
   body: {type: "array"}
}

I can confirm that my content type parser is called correctly but I am still getting a schema validation error

Error: body must be array
    at defaultSchemaErrorFormatter (.../node_modules/fastify/lib/context.js:108:10)
    at wrapValidationError (.../node_modules/fastify/lib/validation.js:228:17)
    at validate (.../node_modules/fastify/lib/validation.js:146:16)
    at preValidationCallback (.../node_modules/fastify/lib/handleRequest.js:92:25)
    at handler (.../node_modules/fastify/lib/handleRequest.js:76:7)
    at .../node_modules/fastify/lib/contentTypeParser.js:192:9
    at AsyncResource.runInAsyncScope (node:async_hooks:206:9)
    at done (.../node_modules/fastify/lib/contentTypeParser.js:186:14)
    at /.../node_modules/fastify/lib/contentTypeParser.js:275:27
    at processTicksAndRejections (node:internal/process/task_queues:95:5) {
  statusCode: 400,
  code: 'FST_ERR_VALIDATION',
  validation: [
    {
      instancePath: '',
      schemaPath: '#/type',
      keyword: 'type',
      params: [Object],
      message: 'must be array'
    }
  ],
  validationContext: 'body'
}

As per docs I would expect that JSON schema validation is skipped for non JSON bodies.

Let me know if I am just misreading what the docs are saying and if anyone has an idea how to work around this any hint would be appreciated.

[climba03003]

I believe the document need to re-phase.
By default, fastify only supports application-json and it only runs the validation through application-json.

So, after you add a new content type parser. The validation will still happen since it can understand the content now.

[nflaig]

So, after you add a new content type parser. The validation will still happen since it can understand the content now.

Thanks, that explains the observed behavior. What's the recommended approach to use in this case, can it be disabled per parser (I haven't found that option) or do you have to extend the schema to allow a Buffer as valid body?

[climba03003]

Currently, there is no way to skip the validation for dedicated Content-Type when schema is applied.
So, you have to allows the Buffer in schema or specify a always true validator per context or route.

[nflaig]

So, you have to allows the Buffer in schema

This is what I use as a workaround right now as all other attempts failed for me, I updated my schema like this now to allow Buffer

schema: {
   body: {oneOf: [{type: "array"}, {type: "object", required: ["buffer"], properties: {buffer: {}}}]};
}

it's not nice but the only alternative would be to completely drop body validation.

or specify a always true validator per context or route.

Is there a way to override the validator at the time of the request and remove the body schema?

Some things I tried without success:

1. Use fastify.setValidatorCompiler to dynamically apply validator, this might have worked but unfortunately it does not have access to content type although the type seems to suggests it.

2. Another thing which kinda worked is to delete the body schema from the request but the issue is that internal like this cannot be accessed easily since Hide internals with symbols fastify#1128 and symbols are not exported.

3. Does the preValidation hook give me some possibilities? unfortunately I think it does not..

Would be great if there was an easy way to at least disable validation for a specific content type (i.e. by setting a bool on request body so similar) and ideally should be able to apply this in content type parser.

Ideal solution would of course be to specify different schema per content type as this is part of openapi spec as well and should be possible (similar as fastify/fastify#3902 for responses)

[mcollina]

I don't really understand what you are trying to achieve here. Why are you trying to validate a Buffer?

[climba03003]

Why are you trying to validate a Buffer?

This is the issue I faced when receiving form-data and using @fastify/swagger.
Since schema is used in generating swagger and validation, it is needed to be specify.

@nflaig
The below is the method I use before.
You should instead use onRoute hook to provide the validatorCompiler which can skip the validation.

import Fastify from 'fastify'

const fastify = Fastify()
fastify.addContentTypeParser('application/octet-stream', {
  parseAs: 'buffer'
}, async function(request, payload) {
  return payload
})

fastify.post('/', {
  schema: {
    body: { type: 'array' }
  },
  validatorCompiler() {
    return () => true
  }
}, async function(request, reply) {
  return reply.send(request.body)
})


const response = await fastify.inject({
  method: 'POST',
  path: '/',
  headers: {
    'Content-Type': 'application/octet-stream'
  },
  body: Buffer.from('AAAAAAAA')
})

console.log(response.statusCode)
console.log(response.body)

If you need to validate the body in all content-type, the output of content-type-parser must be already serialized the data.
That's mean you should always return the desire JSON or Array in that function.

[mcollina]

@climba03003 I think we should document this in fastify-swagger.

[nflaig]

I don't really understand what you are trying to achieve here.

@mcollina Thanks for taking a look, my use case is an API that accepts two different content types as body, for example see this openapi definition. I would like to validate the body via JSON schema but only if the content of body is application/json, otherwise validation should be skipped.

As per openapi spec and the example above, it should be possible to define a schema per content type. A possible solution to support this would be to allow something like this

schema: {
   body:{
    "application/json": {type: "array"},
    "application/octet-stream": {}, // no validation
    // ...
   } 
}

You should instead use onRoute hook to provide the validatorCompiler which can skip the validation.

Thanks for the suggestion @climba03003. This is kinda what I mentioned in #1012 (comment) (1) but the problem is that contentType is not passed to the validation function and in the example you provided I would assume it skips validation for all requests?

Also just a note, the example I posted here is vastly simplified, in practice I still want schema validation for requests with application/octet-stream bodies, I just don't want to validate the body, but query, headers, etc. should still be validated.

If you need to validate the body in all content-type, the output of content-type-parser must be already serialized the data.
That's mean you should always return the desire JSON or Array in that function.

That would be a possible solution, unfortunately this doesn't work in my case as the serialized data is different from the raw json sent in the request, and converting it to json-like format to pass schema validation would be costly to do and some cases just wanna to use the raw bytes (Buffer) downstream, and validation of data is done differently.

While the solution in #1012 (comment) adding a schema that passes Buffer validation works, I don't like it because it affects UX since we return a oneOf error to the client and the error message including something about Buffer is far from ideal.

What I settled with now is really hacky but does exactly what I want without impacting UX

export function addSszContentTypeParser(server) {
  // Cache body schema symbol, does not change per request
  let bodySchemaSymbol;

  server.addContentTypeParser(
    "application/octet-stream",
    {parseAs: "buffer"},
    async (request, payload) => {
      if (bodySchemaSymbol === undefined) {
        // Get body schema symbol to be able to access validation function
        // https://github.com/fastify/fastify/blob/af2ccb5ff681c1d0ac22eb7314c6fa803f73c873/lib/symbols.js#L25
        bodySchemaSymbol = Object.getOwnPropertySymbols(request.context).find((s) => s.description === "body-schema");
      }
      // JSON schema validation will be applied to `Buffer` object, it is required to override validation function
      // See https://github.com/fastify/help/issues/1012, it not possible right now to skip validation based on content type
      request.context[bodySchemaSymbol] = () => true;

      return payload;
    }
  );
}

Maybe others will find this useful, and please forgive me my sins :)