BulkIndexer: lines in body sometimes out of order, leading to 400 response

[gkdr]

hi,

we run the BulkIndexer with default number of workers, which on the machines we use ends up being 8.

sometimes during a flush, we see an error response like this:

HTTP/1.1 400 Bad Request
Content-Type: application/json; charset=UTF-8

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Malformed action/metadata line [1], expected START_OBJECT or END_OBJECT but found [VALUE_NUMBER]"}],"type":"illegal_argument_exception","reason":"Malformed action/metadata line [1], expected START_OBJECT or END_OBJECT but found [VALUE_NUMBER]"},"status":400}

caused by a request looking like this:

POST /_bulk HTTP/1.1
Host: elasticsearch
User-Agent: go-elasticsearch/7.17.1 (linux amd64; Go 1.18.4)
Content-Length: 3408
Authorization: Basic asdf
Content-Type: application/json
X-Elastic-Client-Meta: es=7.17.1,go=1.18.4,t=7.17.1,hc=1.18.4,h=bp
Accept-Encoding: gzip

{doc 1}
{action and meta 2}
{doc 2}
[...]
{action and meta 7}
{doc 7}
{action and meta 1}

this, of course, does not look like expected by the bulk indexer API, as the action/meta line of the first document ends up at the very bottom instead of the very top. all other bulk indexer items, of which there aren't that many, seem fine.

inspecting the bulk indexer code, we did not find any obvious errors - the mutex protecting the buffer, both local to a worker, seems to be locked and unlocked properly before write and flush. do you see where this could be going wrong?

we have been using the bulk indexer basically since it was added to the client, and i believe this error is fairly new. as it happens only rarely, i can't tell for sure though, as it could be just caused by increased load on our side as well.