hs fetch "Error: ESOCKETTIMEDOUT" / UPDATE: [ERROR] A RequestError has occurred. Error: ESOCKETTIMEDOUT

[newton730ia]

Description and Context

I am syncing files to and from a private GIT-repository in a script and fetching about 1000 files. Since a few days ago I am getting regular "Error: ESOCKETTIMEDOUT" messages(worked for about 6 months, # of files fluctuates between 300 and 1500 files). 1-10 files get this error. Sometimes retryin 2 minutes later works, sometimes not. Runtime for the sync is 8-20 seconds

hs-cli is updated to 4.0.1 (prev. 3.0.6)

Steps to reproduce

Issued call is:

hs fetch --overwrite / ./hubspot/ --use-env

Expected behavior

hs fetch finisheing successful

Screenshots

...
[ERROR] A Error has occurred. ESOCKETTIMEDOUT
[ERROR] A Error has occurred. ESOCKETTIMEDOUT
[ERROR] Not all files in folder "/" were successfully fetched.  Re-run the last command to try again

Output with -d for debug:

Wrote file "/home/gitlab-runner/tmp/hubspot/XXX/templates/system/subscriptions-confirmation.html"
[ERROR] A Error has occurred. ESOCKETTIMEDOUT
[DEBUG] Error: Error: ESOCKETTIMEDOUT
    at ClientRequest.<anonymous> (/usr/local/lib/node_modules/@hubspot/cli/node_modules/request/request.js:816:19)
    at Object.onceWrapper (events.js:420:28)
    at ClientRequest.emit (events.js:314:20)
    at TLSSocket.emitRequestTimeout (_http_client.js:715:9)
    at Object.onceWrapper (events.js:420:28)
    at TLSSocket.emit (events.js:326:22)
    at TLSSocket.Socket._onTimeout (net.js:484:8)
    at listOnTimeout (internal/timers.js:554:17)
    at processTimers (internal/timers.js:497:7) {
  [stack]: 'Error: ESOCKETTIMEDOUT\n' +
    '    at ClientRequest.<anonymous> (/usr/local/lib/node_modules/@hubspot/cli/node_modules/request/request.js:816:19)\n' +
    '    at Object.onceWrapper (events.js:420:28)\n' +
    '    at ClientRequest.emit (events.js:314:20)\n' +
    '    at TLSSocket.emitRequestTimeout (_http_client.js:715:9)\n' +
    '    at Object.onceWrapper (events.js:420:28)\n' +
    '    at TLSSocket.emit (events.js:326:22)\n' +
    '    at TLSSocket.Socket._onTimeout (net.js:484:8)\n' +
    '    at listOnTimeout (internal/timers.js:554:17)\n' +
    '    at processTimers (internal/timers.js:497:7)',
  [message]: 'ESOCKETTIMEDOUT',
  code: 'ESOCKETTIMEDOUT',
  connect: false
}

Could this be a server sided rate-limit problem (~100/s)? If so, how can I change used client-rate limit in the call - I was not able to find anything in source regarding hs fetch rate limiting at all.

[newton730ia]

I am still having this issue, current iteration is

[ERROR] A RequestError has occurred. Error: ESOCKETTIMEDOUT

Detailed debug-output: (portalid+access-token has been replaced with *)

/usr/local/bin/hs --debug fetch --overwrite / ./hubspot/

Debugging info

CLI version: 4.1.6
node version: v12.22.12
platform: linux

[DEBUG] Loading config from undefined
[DEBUG] Reading config from /home/gitlab-runner/tmp/hubspot.config.yml
[DEBUG] Attempting to track usage of "fetch" command
[DEBUG] Sending usage event to authenticated endpoint
[DEBUG] Sent usage tracking command event: {
action: 'cli-command',
os: 'linux',
nodeVersion: 'v12.22.12',
nodeMajorVersion: 'v12',
version: '4.1.6',
command: 'fetch',
authType: 'personalaccesskey',
mode: 'publish'
}
[ERROR] A RequestError has occurred. Error: ESOCKETTIMEDOUT
[DEBUG] Error: RequestError: Error: ESOCKETTIMEDOUT
at new RequestError (/usr/local/lib/node_modules/@hubspot/cli/node_modules/request-promise-core/lib/errors.js:14:15)
at Request.plumbing.callback (/usr/local/lib/node_modules/@hubspot/cli/node_modules/request-promise-core/lib/plumbing.js:87:29)
at Request.RP$callback [as _callback] (/usr/local/lib/node_modules/@hubspot/cli/node_modules/request-promise-core/lib/plumbing.js:46:31)
at self.callback (/usr/local/lib/node_modules/@hubspot/cli/node_modules/request/request.js:185:22)
at Request.emit (events.js:314:20)
at ClientRequest. (/usr/local/lib/node_modules/@hubspot/cli/node_modules/request/request.js:819:16)
at Object.onceWrapper (events.js:420:28)
at ClientRequest.emit (events.js:314:20)
at TLSSocket.emitRequestTimeout (_http_client.js:715:9)
at Object.onceWrapper (events.js:420:28) {
name: 'RequestError',
message: 'Error: ESOCKETTIMEDOUT',
cause: Error: ESOCKETTIMEDOUT
at ClientRequest. (/usr/local/lib/node_modules/@hubspot/cli/node_modules/request/request.js:816:19)
at Object.onceWrapper (events.js:420:28)
at ClientRequest.emit (events.js:314:20)
at TLSSocket.emitRequestTimeout (_http_client.js:715:9)
at Object.onceWrapper (events.js:420:28)
at TLSSocket.emit (events.js:326:22)
at TLSSocket.Socket._onTimeout (net.js:484:8)
at listOnTimeout (internal/timers.js:554:17)
at processTimers (internal/timers.js:497:7) {
[stack]: 'Error: ESOCKETTIMEDOUT\n' +
' at ClientRequest. (/usr/local/lib/node_modules/@hubspot/cli/node_modules/request/request.js:816:19)\n' +
' at Object.onceWrapper (events.js:420:28)\n' +
' at ClientRequest.emit (events.js:314:20)\n' +
' at TLSSocket.emitRequestTimeout (_http_client.js:715:9)\n' +
' at Object.onceWrapper (events.js:420:28)\n' +
' at TLSSocket.emit (events.js:326:22)\n' +
' at TLSSocket.Socket._onTimeout (net.js:484:8)\n' +
' at listOnTimeout (internal/timers.js:554:17)\n' +
' at processTimers (internal/timers.js:497:7)',
[message]: 'ESOCKETTIMEDOUT',
code: 'ESOCKETTIMEDOUT',
connect: false
},
error: Error: ESOCKETTIMEDOUT
at ClientRequest. (/usr/local/lib/node_modules/@hubspot/cli/node_modules/request/request.js:816:19)
at Object.onceWrapper (events.js:420:28)
at ClientRequest.emit (events.js:314:20)
at TLSSocket.emitRequestTimeout (_http_client.js:715:9)
at Object.onceWrapper (events.js:420:28)
at TLSSocket.emit (events.js:326:22)
at TLSSocket.Socket._onTimeout (net.js:484:8)
at listOnTimeout (internal/timers.js:554:17)
at processTimers (internal/timers.js:497:7) {
[stack]: 'Error: ESOCKETTIMEDOUT\n' +
' at ClientRequest. (/usr/local/lib/node_modules/@hubspot/cli/node_modules/request/request.js:816:19)\n' +
' at Object.onceWrapper (events.js:420:28)\n' +
' at ClientRequest.emit (events.js:314:20)\n' +
' at TLSSocket.emitRequestTimeout (_http_client.js:715:9)\n' +
' at Object.onceWrapper (events.js:420:28)\n' +
' at TLSSocket.emit (events.js:326:22)\n' +
' at TLSSocket.Socket._onTimeout (net.js:484:8)\n' +
' at listOnTimeout (internal/timers.js:554:17)\n' +
' at processTimers (internal/timers.js:497:7)',
[message]: 'ESOCKETTIMEDOUT',
code: 'ESOCKETTIMEDOUT',
connect: false
},
options: {
baseUrl: 'https://api.hubapi.com',
headers: {
'User-Agent': 'HubSpot CLI/4.1.6',
Authorization: 'Bearer *'
},
json: true,
simple: true,
timeout: 15000,
uri: 'content/filemapper/v1/download/%40root',
qs: {
buffer: false,
environmentId: 1,
version: undefined,
portalId: *
},
method: 'GET',
callback: [Function: RP$callback] {
[length]: 3,
[name]: 'RP$callback',
[prototype]: RP$callback { [constructor]: [Circular] }
},
transform: undefined,
resolveWithFullResponse: false,
transform2xxOnly: false
},
response: undefined,
[stack]: 'RequestError: Error: ESOCKETTIMEDOUT\n' +
' at new RequestError (/usr/local/lib/node_modules/@hubspot/cli/node_modules/request-promise-core/lib/errors.js:14:15)\n' +
' at Request.plumbing.callback (/usr/local/lib/node_modules/@hubspot/cli/node_modules/request-promise-core/lib/plumbing.js:87:29)\n' +
' at Request.RP$callback [as _callback] (/usr/local/lib/node_modules/@hubspot/cli/node_modules/request-promise-core/lib/plumbing.js:46:31)\n' +
' at self.callback (/usr/local/lib/node_modules/@hubspot/cli/node_modules/request/request.js:185:22)\n' +
' at Request.emit (events.js:314:20)\n' +
' at ClientRequest. (/usr/local/lib/node_modules/@hubspot/cli/node_modules/request/request.js:819:16)\n' +
' at Object.onceWrapper (events.js:420:28)\n' +
' at ClientRequest.emit (events.js:314:20)\n' +
' at TLSSocket.emitRequestTimeout (_http_client.js:715:9)\n' +
' at Object.onceWrapper (events.js:420:28)'
}
[DEBUG] Context: ApiErrorContext { accountId: *, request: '/', payload: '' }

[newton730ia]

I have found a fix for my company:
I have increased the httpTimeout from 150000 to 40000 (15s -> 30s) - currently this cannot be done via environment variables (--use-env), only with hubspot.config.yml, which is a problem since we keep access tokens outside of the system with hs-cli. Therefore I have changed the default in
./node_modules/@hubspot/cli-lib/http/requestOptions.js line 23
from
timeout: httpTimeout || 15000,
to
timeout: httpTimeout || 40000,

[papagunit]

@newton730ia you're a genius, that works!! Should send in a pull request.

For reference on windows, the requestOptions.js file is under \userprofile\AppData\Roaming\npm\node_modules\\@hubspot\cli\node_modules\\@hubspot\cli-lib\http

[techieshark]

Thanks for the workaround @newton730ia!

Anyone else landing here and able to update your hubspot.config.yml, try setting httpTimeout in the yml file itself, like:

defaultPortal: ...
httpTimeout: 40000
portals: ...