v0.20.23 break "indexer-service"

[matsuro-hadouken]

Network: arbitrum-one

After full js side of the indexer stack upgrade to v0.20.23 indexer-service trow errors and not able to process paid queries.

{
    "level": 50,
    "time": 1698878802819,
    "err": {
        "type": "IndexerError",
        "message": "Failed to process paid query",
        "stack": "IndexerError: Failed to process paid query\n    at indexerError (/home/some-user/indexer/packages/indexer-common/dist/errors.js:173:12)\n    at /home/some-user/indexer/packages/indexer-service/dist/server/index.js:227:67\n    at Generator.throw (<anonymous>)\n    at rejected (/home/some-user/indexer/packages/indexer-service/dist/server/index.js:6:65)",
        "code": "IE032",
        "explanation": "https://github.com/graphprotocol/indexer/blob/main/docs/errors.md#ie032",
        "cause": {
            "type": "Error",
            "message": "Unable to sign the query response attestation",
            "stack": "Error: Unable to sign the query response attestation\n    at QueryProcessor.<anonymous> (/home/some-user/indexer/packages/indexer-service/dist/queries.js:89:31)\n    at Generator.next (<anonymous>)\n    at fulfilled (/home/some-user/indexer/packages/indexer-service/dist/queries.js:5:58)",
            "status": 500
        }
    },
    "msg": "Failed to handle paid query"
}

additionally:

{
    "level": 40,
    "time": 1698878854211,
    "name": "IndexerService",
    "component": "AllocationMonitor",
    "allocations": [],
    "err": {
        "type": "IndexerError",
        "message": "Failed to query indexer allocations",
        "stack": "IndexerError: Failed to query indexer allocations\n    at indexerError (/home/some-user/indexer/packages/indexer-common/dist/errors.js:173:12)\n    at /home/some-user/indexer/packages/indexer-service/dist/allocations.js:111:56\n    at Generator.next (<anonymous>)\n    at /home/some-user/indexer/packages/indexer-service/dist/allocations.js:8:71\n    at new Promise (<anonymous>)\n    at __awaiter (/home/some-user/indexer/packages/indexer-service/dist/allocations.js:4:12)\n    at refreshAllocations (/home/some-user/indexer/packages/indexer-service/dist/allocations.js:24:56)\n    at /home/some-user/indexer/node_modules/@graphprotocol/common-ts/dist/eventual/eventual.js:242:33\n    at Generator.next (<anonymous>)\n    at /home/some-user/indexer/node_modules/@graphprotocol/common-ts/dist/eventual/eventual.js:8:71",
        "code": "IE010",
        "cause": {
            "type": "GraphQLError",
            "message": "Syntax Error: Unexpected \"}\".",
            "stack": "GraphQLError: Syntax Error: Unexpected \"}\".\n    at syntaxError (/home/some-user/indexer/node_modules/graphql/error/syntaxError.js:15:10)\n    at Parser.unexpected (/home/some-user/indexer/node_modules/graphql/language/parser.js:1458:41)\n    at Parser.parseDefinition (/home/some-user/indexer/node_modules/graphql/language/parser.js:212:16)\n    at Parser.many (/home/some-user/indexer/node_modules/graphql/language/parser.js:1511:26)\n    at Parser.parseDocument (/home/some-user/indexer/node_modules/graphql/language/parser.js:122:25)\n    at Object.parse (/home/some-user/indexer/node_modules/graphql/language/parser.js:32:17)\n    at parseDocument (/home/some-user/indexer/node_modules/graphql-tag/lib/graphql-tag.umd.js:69:34)\n    at gql (/home/some-user/indexer/node_modules/graphql-tag/lib/graphql-tag.umd.js:98:16)\n    at /home/some-user/indexer/packages/indexer-service/dist/allocations.js:27:102\n    at Generator.next (<anonymous>)",
            "locations": [
                {
                    "line": 7,
                    "column": 9
                }
            ],
            "extensions": {}
        }
    },
    "msg": "Failed to query indexer allocations, keeping existing"
}

ValidationError: The 'X-Forwarded-For' header is set but the Express 'trust proxy' setting is false (default).
This could indicate a misconfiguration which would prevent express-rate-limit from accurately identifying users.
See https://express-rate-limit.github.io/ERR_ERL_UNEXPECTED_X_FORWARDED_FOR/ for more information.
    at Object.xForwardedForHeader (/home/some-user/indexer/node_modules/express-rate-limit/dist/index.cjs:166:13)
    at Object.wrappedValidations.<computed> [as xForwardedForHeader] (/home/some-user/indexer/node_modules/express-rate-limit/dist/index.cjs:324:22)
    at Object.keyGenerator (/home/some-user/indexer/node_modules/express-rate-limit/dist/index.cjs:583:20)
    at /home/some-user/indexer/node_modules/express-rate-limit/dist/index.cjs:633:32
    at async /home/some-user/indexer/node_modules/express-rate-limit/dist/index.cjs:615:5 {
  code: 'ERR_ERL_UNEXPECTED_X_FORWARDED_FOR',
  help: 'https://express-rate-limit.github.io/ERR_ERL_UNEXPECTED_X_FORWARDED_FOR/'
}

After multiple attempts to debug and fix this problem, I decided to revert to v0.20.22 and all problems magically disappered.

There is no issues with agent or anything also, the only indexer-service is affected.

[alex-pakalniskis]

Sharing more comments from Discord:

https://discord.com/channels/438038660412342282/791444480628490290/1169434170804670555

Yes, all good with DNS on my side. Roll back to 22 and problems solved. Just will watch agent, he is still on 23 for the now. - Daniel | ART3MIS.CLOUD

https://discord.com/channels/438038660412342282/791444480628490290/1169509897130487858

Scenario 1 :
cli = 0.22
indexer-agent = 0.23
indexer-service = 0.23

Result : No output, no error message with graph indexer status --network arbitrum-one, just hangs there

Scenario 2 :
cli = 0.23
indexer-agent = 0.23
indexer-service = 0.23

Result : No output, no error message with graph indexer status --network arbitrum-one, just hangs there (previous was throwup error)

Scenario 3 :
cli = 0.22
indexer-agent = 0.22
indexer-service = 0.22

Result = works just fine

• Hau | Pinax

https://discord.com/channels/438038660412342282/791444480628490290/1169694148702523482

same queries started flowing after rolling back service to .22
lots of IE10's & 32's. before and after rolling back to .22 - Vince | Nodeify/Pinax