Z_BUF_ERROR on GET of https://intljs.rmtag.com/115557.ct.js through proxy

[johnrichardrinehart]

Describe the bug

• Node.js version: 15.3.0
• OS & version: Ubuntu 20.04

Fetching with got using a proxy Agent (hpagent) results in a Z_BUF_ERROR (unexpected content length). But, fetching with curl -x <agent> https://intljs.rmtag.com/115557.ct.js --compressed works just fine.

Actual behavior

Stack trace:

RequestError: unexpected end of file
    at PassThrough.<anonymous> (/home/john/Downloads/repos/puppeteer_api/.yarn/unplugged/got-npm-11.8.1-7775de79cb/node_modules/got/dist/source/core/index.js:797:31)
    at Object.onceWrapper (node:events:483:26)
    at PassThrough.emit (node:events:388:22)
    at emitErrorNT (node:internal/streams/destroy:188:8)
    at emitErrorCloseNT (node:internal/streams/destroy:153:3)
    at processTicksAndRejections (node:internal/process/task_queues:80:21)
    at Zlib.zlibOnError [as onerror] (node:zlib:182:17) {
  name: 'ReadError',
  code: 'Z_BUF_ERROR',
  timings: {
    start: 1613893520933,
    socket: 1613893532290,
    lookup: 1613893532290,
    connect: 1613893532290,
    secureConnect: 1613893533013,
    upload: 1613893533014,
    response: 1613893534192,
    end: 1613893537248,
    error: undefined,
    abort: 1613893537257,
    phases: {
      wait: 11357,
      dns: 0,
      tcp: 0,
      tls: 723,
      request: 1,
      firstByte: 1178,
      download: 3056,
      total: 16324
    }
  }
}

Expected behavior

Fetching the asset wouldn't produce a Z_BUF_ERROR.

Code to reproduce

Here is a console.log of the arguments passed to got(url, options) (console.log({url: url, options: options})).

{
url: "https://intljs.rmtag.com/115557.ct.js",
options: {
  cookieJar: CookieJar {
    rejectPublicSuffixes: true,
    enableLooseMode: false,
    allowSpecialUseDomain: false,
    store: { idx: {} },
    prefixSecurity: 'silent',
    _cloneSync: [Function (anonymous)],
    _importCookiesSync: [Function (anonymous)],
    getCookiesSync: [Function (anonymous)],
    getCookieStringSync: [Function (anonymous)],
    getSetCookieStringsSync: [Function (anonymous)],
    removeAllCookiesSync: [Function (anonymous)],
    setCookieSync: [Function (anonymous)],
    serializeSync: [Function (anonymous)]
  },
  method: 'GET',
  body: undefined,
  headers: {
    'sec-ch-ua': '" Not A;Brand";v="99", "Chromium";v="90", "Google Chrome";v="90"',
    'sec-ch-ua-mobile': '?0',
    'upgrade-insecure-requests': '1',
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4403.0 Safari/537.36',
    accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
    'accept-encoding': 'gzip, deflate, br',
    host: 'intljs.rmtag.com',
    'sec-fetch-mode': 'navigate',
    'sec-fetch-site': 'none',
    'sec-fetch-user': '?1'
  },
  agent: {
    https: HttpsProxyAgent {
      _events: [Object: null prototype],
      _eventsCount: 2,
      _maxListeners: undefined,
      defaultPort: 443,
      protocol: 'https:',
      options: [Object],
      requests: {},
      sockets: {},
      freeSockets: {},
      keepAliveMsecs: 1000,
      keepAlive: true,
      maxSockets: 256,
      maxFreeSockets: 256,
      scheduling: 'lifo',
      maxTotalSockets: Infinity,
      totalSocketCount: 0,
      maxCachedSessions: 100,
      _sessionCache: [Object],
      proxy: [URL],
      [Symbol(kCapture)]: false
    }
  },
  responseType: 'buffer',
  maxRedirects: 15,
  throwHttpErrors: false,
  ignoreInvalidCookies: true,
  followRedirect: false,
  timeout: 30000,
  decompress: true
}
}

Checklist

• I have read the documentation. **I can "fix" this with decompress: false, but that's undesirable for obvious reasons.
• I have tried my code with the latest version of Node.js and Got. ** I'm using got v11.8.1. **

[johnrichardrinehart]

@szmarczak Why was my Issue closed? What's the plan for this discussion?

[johnrichardrinehart]

You mean the Content-Encoding header? I read the attached issue (again). I didn't see any mention of a "compression header". Is it possible to obtain the (potentially corrupt) response as obtained by curl/puppeteer/normal browser/Go net/http.DefaultClient/etc.?

[szmarczak]

https://stackoverflow.com/questions/9050260/what-does-a-zlib-header-look-like

[johnrichardrinehart]

I didn't see any discussion in that link about zlib headers, but maybe the zlib header goes by many names.

But, the response is still healthy. Is it possible to obtain the response?

[szmarczak]

Yes. You can use the Stream API and then decompress the response manually. Also see #388

[johnrichardrinehart]

Fair enough. Thanks @szmarczak!