Lost request body in middleware.

[ka-ramba]

Describe the bug

Originally I posted the issue on stackoverflow.

I have a pretty standard Vapor http server, It has a middleware where it inspects the request body and does some authentication validation.

func respond(to request: Request, chainingTo next: Responder) -> EventLoopFuture<Response> {
     var myStuff: SomeCommonPattern? =  try request.content.decode(SomeCommonPattern?.self)

SomeCommonPattern is a base protocol for all other requests in the group. That works very well in debug on Mac. When I run a release build in docker on my Mac laptop it still works as expected. I put it through a rigorous tests with a hundred clients making simultaneous requests for hours. Not a single failure. Now I moved the same docker image to a production machine, which is Ubuntu with lower specs than my machine, and my code still works with exception when it does not. About 1 request out of 5 fails to decode the request. I put some additional logging and sure enough, sometimes request.body.string is empty when most other same requests are fine. Anyone knows what could be happening ? It looks almost like the body of the request is still in transit when the middleware is being called. It happens even when I make requests from a single client.

There are get and put requests and parameters are always passed through a body even for get requests. Any of the endpoints whether it is get or post or put can fail.

Edit: I just realized that on my laptop the service is running directly, while on the linux machine it is behind apache 2 reverse proxy. It is probably a significant detail but I still do not know the reason of the intermittent behavior.

Edit 2: I moved the failing code out of the middleware and put in every endpoint implementation. It is much less convenient and more error prone but now my application works 100% of the time. That may confirm that apache is off the hook and the problem is indeed in Vapor, or my misunderstanding how to use it.

It could be the key to reproduce the issue is to start the server on a slow hardware and in some middleware check if

request.body.string

is empty when it is supposed to have the body of the request.

Environment

• Vapor Framework version: 4.49.2
• Vapor Toolbox version: 18.3.3
• OS version: Ubuntu 20.04.2 , Docker version 20.10.10, build b485636

[t-ae]

request.body can be nil if the bodyStorage is still stream.

vapor/Sources/Vapor/Request/Request+Body.swift

Lines 9 to 22 in 6a5a3b5

	public var data: ByteBuffer? {
	switch self.request.bodyStorage {
	case .collected(let buffer): return buffer
	case .none, .stream: return nil
	}
	}
	public var string: String? {
	if var data = self.data {
	return data.readString(length: data.readableBytes)
	} else {
	return nil
	}
	}

You can call collect method in your Middleware.

[0xTim]

This is related to #2735 - the full body isn't available to middleware if it's over a certain size

[Joannis]

I haven't been able to reproduce this when since I assigned this to myself. I think it was fixed before 2 may, but I might be wrong.

[marius-se]

@ka-ramba can you still reproduce this? I'm currently investigating this issue, but couldn't reproduce it so far.

[kevinzhow]

Here is a unit test to reproduce this bug @0xTim @marius-se @Joannis
Vapor version is 4.65.2
It only affects when app.testable(method: .running) and production / development enviroment.

@testable import App
import XCTVapor

final class AppTests: XCTestCase {
   
    var app: Application!
    
    override func setUp() async throws {
        app = Application(.testing)
        
//        try configure(app)
        
        try await app.autoRevert()
        try await app.autoMigrate()
    }
    
    override func tearDown() async throws {
        app.shutdown()
    }
    
    func testBugs() async throws {
        
        struct MiddleTest: AsyncMiddleware {
           
            init() {}
            
            func respond(to request: Request, chainingTo next: AsyncResponder) async throws -> Response {
                guard let body = request.body.string, body.count > 0 else {
                    XCTFail("body is empty")
                    return try await next.respond(to: request)
                }
                
                print(body)
                
                return try await next.respond(to: request)
            }
        }
        
        app.grouped(MiddleTest()).on(.GET, "bug") { req -> HTTPStatus in
            return .ok
        }
        
        try app.testable(method: .running).test(.GET, "bug", beforeRequest: { request in
            try request.content.encode(ByteBuffer(repeating: 0x41, count: 1390 * 8).string)
        }, afterResponse: { _ in
            
        })
        
    }
}