🐛 Bug Report: Random HTTP failures with Node20

[drodil]

📜 Description

Sometimes catalog (or other backend) calls fail due to socket hang up error. For example one of our processors tries to fetch entities from the catalog and it randomly fails as follows:

Catalog processing error occurred for component:default/my-component owned by group:default/developers: InputError: Processor MyProcessor threw an error while postprocessing; caused by FetchError: request to http://localhost:7007/api/catalog/entities?fields=kind,metadata.namespace,metadata.name&filter=kind%3DComponent failed, reason: socket hang up

Might have some relation to this or other issues linked to this: node-fetch/node-fetch#1735 / nodejs/node#47130

👍 Expected behavior

Sockets should not hang up

👎 Actual Behavior with Screenshots

Sockets hang up

👟 Reproduction steps

1. Use NodeJS 20
2. Add a processor that fetches catalog entities

📃 Provide the context for the Bug.

    const credentials = await this.auth.getOwnServiceCredentials();
    const { token } = await this.auth.getPluginRequestToken({
      onBehalfOf: credentials,
      targetPluginId: 'catalog',
    });

    const components = await this.client.getEntities(
      {
        filter: {
          kind: 'Component',
        },
        fields: ['kind', 'metadata.namespace', 'metadata.name'],
      },
      { token },
    );

🖥️ Your Environment

OS: Darwin 23.4.0 - darwin/arm64
node: v20.12.0
yarn: 1.22.19
cli: 0.26.4 (installed)
backstage: 1.26.4

Dependencies:
@backstage/app-defaults 1.5.4
@backstage/backend-app-api 0.6.2, 0.7.1
@backstage/backend-common 0.21.7
@backstage/backend-defaults 0.2.17
@backstage/backend-dev-utils 0.1.4
@backstage/backend-openapi-utils 0.1.10
@backstage/backend-plugin-api 0.6.17
@backstage/backend-tasks 0.5.22
@backstage/backend-test-utils 0.3.7
@backstage/catalog-client 1.6.4
@backstage/catalog-model 1.4.5
@backstage/cli-common 0.1.13
@backstage/cli-node 0.2.5
@backstage/cli 0.26.4
@backstage/config-loader 1.8.0
@backstage/config 1.2.0
@backstage/core-app-api 1.12.4
@backstage/core-compat-api 0.2.4
@backstage/core-components 0.14.5
@backstage/core-plugin-api 1.9.2
@backstage/dev-utils 1.0.31
@backstage/e2e-test-utils 0.1.1
@backstage/errors 1.2.4
@backstage/eslint-plugin 0.1.7
@backstage/frontend-plugin-api 0.6.4
@backstage/integration-aws-node 0.1.12
@backstage/integration-react 1.1.26
@backstage/integration 1.10.0
@backstage/plugin-api-docs 0.11.4
@backstage/plugin-app-backend 0.3.65
@backstage/plugin-app-node 0.1.17
@backstage/plugin-auth-backend-module-atlassian-provider 0.1.9
@backstage/plugin-auth-backend-module-aws-alb-provider 0.1.9
@backstage/plugin-auth-backend-module-azure-easyauth-provider 0.1.0
@backstage/plugin-auth-backend-module-bitbucket-provider 0.1.0
@backstage/plugin-auth-backend-module-cloudflare-access-provider 0.1.0
@backstage/plugin-auth-backend-module-gcp-iap-provider 0.2.12
@backstage/plugin-auth-backend-module-github-provider 0.1.14
@backstage/plugin-auth-backend-module-gitlab-provider 0.1.14
@backstage/plugin-auth-backend-module-google-provider 0.1.14
@backstage/plugin-auth-backend-module-guest-provider 0.1.3
@backstage/plugin-auth-backend-module-microsoft-provider 0.1.12
@backstage/plugin-auth-backend-module-oauth2-provider 0.1.14
@backstage/plugin-auth-backend-module-oauth2-proxy-provider 0.1.10
@backstage/plugin-auth-backend-module-oidc-provider 0.1.8
@backstage/plugin-auth-backend-module-okta-provider 0.0.10
@backstage/plugin-auth-backend 0.22.4
@backstage/plugin-auth-node 0.4.12
@backstage/plugin-auth-react 0.1.1
@backstage/plugin-catalog-backend-module-aws 0.3.12
@backstage/plugin-catalog-backend-module-msgraph 0.5.25
@backstage/plugin-catalog-backend-module-scaffolder-entity-model 0.1.15
@backstage/plugin-catalog-backend-module-unprocessed 0.4.4
@backstage/plugin-catalog-backend 1.21.1
@backstage/plugin-catalog-common 1.0.22
@backstage/plugin-catalog-graph 0.4.4
@backstage/plugin-catalog-node 1.11.1
@backstage/plugin-catalog-react 1.11.3
@backstage/plugin-catalog-unprocessed-entities-common 0.0.1
@backstage/plugin-catalog-unprocessed-entities 0.2.3
@backstage/plugin-catalog 1.19.0
@backstage/plugin-devtools-backend 0.3.3
@backstage/plugin-devtools-common 0.1.9
@backstage/plugin-devtools 0.1.13
@backstage/plugin-events-backend 0.3.4
@backstage/plugin-events-node 0.3.3
@backstage/plugin-home-react 0.1.12
@backstage/plugin-home 0.7.3
@backstage/plugin-kubernetes-common 0.7.5
@backstage/plugin-notifications-backend 0.2.0
@backstage/plugin-notifications-common 0.0.3
@backstage/plugin-notifications-node 0.1.3
@backstage/plugin-notifications 0.2.0
@backstage/plugin-org 0.6.24
@backstage/plugin-permission-backend 0.5.41
@backstage/plugin-permission-common 0.7.13
@backstage/plugin-permission-node 0.7.28
@backstage/plugin-permission-react 0.4.22
@backstage/plugin-proxy-backend 0.4.15
@backstage/plugin-scaffolder-backend-module-azure 0.1.9
@backstage/plugin-scaffolder-backend-module-bitbucket-cloud 0.1.7
@backstage/plugin-scaffolder-backend-module-bitbucket-server 0.1.7
@backstage/plugin-scaffolder-backend-module-bitbucket 0.2.7
@backstage/plugin-scaffolder-backend-module-confluence-to-markdown 0.2.18
@backstage/plugin-scaffolder-backend-module-cookiecutter 0.2.41
@backstage/plugin-scaffolder-backend-module-gerrit 0.1.9
@backstage/plugin-scaffolder-backend-module-gitea 0.1.7
@backstage/plugin-scaffolder-backend-module-github 0.2.7
@backstage/plugin-scaffolder-backend-module-gitlab 0.3.3
@backstage/plugin-scaffolder-backend 1.22.4
@backstage/plugin-scaffolder-common 1.5.1
@backstage/plugin-scaffolder-node-test-utils 0.1.3
@backstage/plugin-scaffolder-node 0.4.3
@backstage/plugin-scaffolder-react 1.8.4
@backstage/plugin-scaffolder 1.19.3
@backstage/plugin-search-backend-module-catalog 0.1.23
@backstage/plugin-search-backend-module-elasticsearch 1.4.0
@backstage/plugin-search-backend-module-techdocs 0.1.22
@backstage/plugin-search-backend-node 1.2.21
@backstage/plugin-search-backend 1.5.7
@backstage/plugin-search-common 1.2.11
@backstage/plugin-search-react 1.7.10
@backstage/plugin-search 1.4.10
@backstage/plugin-signals-backend 0.1.3
@backstage/plugin-signals-node 0.1.3
@backstage/plugin-signals-react 0.0.3
@backstage/plugin-signals 0.0.5
@backstage/plugin-techdocs-backend 1.10.4
@backstage/plugin-techdocs-node 1.12.3
@backstage/plugin-techdocs-react 1.2.3
@backstage/plugin-techdocs 1.10.4
@backstage/plugin-user-settings-backend 0.2.16
@backstage/plugin-user-settings 0.8.5
@backstage/release-manifests 0.0.11
@backstage/repo-tools 0.8.0
@backstage/test-utils 1.5.4
@backstage/theme 0.5.3
@backstage/types 1.1.1
@backstage/version-bridge 1.0.8

👀 Have you spent some time to check if this bug has been raised before?

• I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

• I have read the Code of Conduct

Are you willing to submit PR?

None

[zinizhu]

We are seeing the same error and we are also on backstage v1.26.4

[freben]

Also reported for the elasticsearch module

https://discord.com/channels/687207715902193673/770283289327566848/1236038730469347430

[aramissennyeydd]

👋 Looks like the fix for this on the node-fetch side was published as part of node-fetch 2.6.13, what version are you using? If lower than that version, can you try upgrading?

[drodil]

Seems to be resolved to 2.6.7 on our side. Upgrading might help with the node-fetch side, true, but not sure about the pure fetch yet. Have to try it out if upgrading is enough

[benjdlambert]

@drodil would be good to see if this issue also is fixed with the native fetch too 🙏

[dweber019]

My installation resolves to 2.7.0 for node-fetch but still have connection reset issues.

$ yarn list --pattern "node-fetch"
yarn list v1.22.17
├─ @asyncapi/parser@3.0.14
│  └─ node-fetch@2.6.7
├─ @lerna/create@7.4.2
│  └─ node-fetch@2.6.7
├─ @types/node-fetch@2.6.6
├─ @whatwg-node/node-fetch@0.5.3
├─ lerna@7.4.2
│  └─ node-fetch@2.6.7
├─ node-fetch-commonjs@3.3.2
└─ node-fetch@2.7.0
Done in 1.70s.