Site hangs got

[vladtreny]

Describe the bug

• Node.js version: 15
• OS & version: Ubuntu 20 / Windows

Just run this code and you will never get a response. Timeout doesn't help.

const response = await got("http://www.1a2f.com");
console.log(response.body);

If you enable handler, you will see

{ percent: 0, transferred: 0, total: undefined }
{ percent: 1, transferred: 0, total: 0 }
{ percent: 0, transferred: 0, total: 1927 }
{ percent: 1, transferred: 4605, total: 4605 }

Actual behavior

stucks at const response = await got("http://www.1a2f.com");
P.S. Timeout doesn't help.
...

Expected behavior

get console.log(response.body);
...

Code to reproduce

const response = await got("http://www.1a2f.com");
console.log(response.body);

Checklist

• I have read the documentation.
• I have tried my code with the latest version of Node.js and Got.

[sindresorhus]

Try setting a user-agent header.

https://github.com/sindresorhus/got#user-agent

[vladtreny]

Try setting a user-agent header.

https://github.com/sindresorhus/got#user-agent

Hello, it doesn't help. It is just a buggy site I encountered in a search engine (one per 500k).
It returns 503 error and hangs got somewhere.

I have a solution for this. But I think, in general, it would be good if timeout work in this case. So I reported this issue :)

[szmarczak]

Indeed this is a bug. I need to check if this is fixed in Got 12.

[vladtreny]

Indeed this is a bug. I need to check if this is fixed in Got 12.

Another example http://aadeshjain.com/

It returns 503 error the same way.

It took me a lot of time to find out what is wrong in my setup, hope will help other people :)

[szmarczak]

It doesn't hang. The retry-after header is set to 86400, which is 1 day. So Got is going to make a new retry after 1 day. You need to set a timeout in order to prevent this.

[vladtreny]

Did you read my initial post?
Timeout doesn't help.

[vladtreny]

Seems like this site fixed the error http://www.1a2f.com

Here is another example to reproduce this issue
http://aadeshjain.com/

[szmarczak]

Did you read my initial post?

Yes, I have and I cannot reproduce this (even with the attached website). It works here as expected with timeout. Make sure you're using latest Got.

[vladtreny]

Did you read my initial post?

Yes, I have and I cannot reproduce this (even with the attached website). It works here as expected with timeout. Make sure you're using latest Got.

Ok, I've narrowed down the issue. It is extend

This hangs

    console.log("START")
    let custom = got.extend();
    const data = await custom("http://aadeshjain.com/", {
        timeout: 3 * 1000,
    });
    console.log("FINISH")
    console.log(data)

This works
let dt1 = await got("http://aadeshjain.com/", {timeout: 3*1000})

[szmarczak]

Hmm... Looks like a normalization bug. In the upcoming Got 12 normalization is done in a much different, better way.

[NeverBehave]

I have encounter similar situation with the site https://www.mckinsey.com.cn
Got will hang if an user-agent was supplied.

Step to reproduce: https://runkit.com/neverbehave/6152cbc94fd89300080351dc

I am not sure about the cause, and there is no retry-after header or something strange, based on curl result:

* Connected to www.mckinsey.com.cn (173.222.162.145) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [233 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [102 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2668 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [148 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=CN; ST=Beijing; L=Beijing; O=McKinsey & Consulting Company Inc., Shanghai Beijing Branch; CN=solutions.mckinsey.com.cn
*  start date: Mar 13 00:00:00 2021 GMT
*  expire date: Mar 23 23:59:59 2022 GMT
*  subjectAltName: host "www.mckinsey.com.cn" matched cert's "www.mckinsey.com.cn"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7f8a0b80a400)
> GET / HTTP/2
> Host: www.mckinsey.com.cn
> Accept: */*
> user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 200
< content-type: text/html; charset=UTF-8
< link: <https://www.mckinsey.com.cn/wp-json/>; rel="https://api.w.org/"
< link: <https://www.mckinsey.com.cn/wp-json/wp/v2/pages/13297>; rel="alternate"; type="application/json"
< link: <https://www.mckinsey.com.cn/>; rel=shortlink
< strict-transport-security: max-age=31536000; includeSubDomains
< x-akamai-transformed: 9 70242 0 pmb=mTOE,2mRUM,1
< expires: Tue, 28 Sep 2021 08:18:04 GMT
< cache-control: max-age=0, no-cache, no-store
< pragma: no-cache
< date: Tue, 28 Sep 2021 08:18:04 GMT
< server-timing: cdn-cache; desc=MISS
< server-timing: edge; dur=572
< server-timing: origin; dur=254
< set-cookie: bm_mi=83EE8AEE13CB8255A62C532C401CBD60~7sbiGqKe87aAVOrXeRjZRFujUakZLUDTRzZ7n8uh7fZLZg1uZMpOm7g97fg0ucHGS4Da7MadBDU1+fEdbPQS7D62E0IRmqPbs/aXimR4NFa7K4sy+6nIesedZMEWE0orb0J1lRVkcqoZzRy0BJjr89y+Rpdosp2Uxnmts0GhvFpUhb0H+hNh682P18cqV/n21jFjogGcbEsd8j3shFci6BS8iK484FwOErOdckAnCcs=; Domain=.mckinsey.com.cn; Path=/; Max-Age=0; HttpOnly
< set-cookie: ak_bmsc=A8FB9AA30DB48A23767D5F6B7E8328BF~000000000000000000000000000000~YAAQkSwtF4c2dOR7AQAAX3h7Kw0DXwrRl+RUrTxHM9NW9cKE3ZtBh+tn+vEHz6M3NlNVHjeYm4dWF5+Clg8e5nOYnH59rXyE012ZgsH49C+ZLJ5/NFHkyirMVZTR4XwRTnh81g4QJsAOWpT/WCT2KWL+Z5n5OgmSNt1ESlfQvaz7qzvi/VO4TC7cIOP6bFFwhvr7DfD68gmH1DTElgPf9uUQjoDMwD8mI5iaGIcI0631ZPd9pkwsmVgplnjLuwou3RYCSaZ5F5SYjXo/eGEqJH3XDjjSdWqBgchhrAu39g4Z7ubiwcYqipG1i18F9ajkjPzoLEz0aLFjLGPPQ5lKQnmKW29Jy7xamHjKElM9CrnysZ7xRp+zWAaU2NenWIRUIOLIoT7WGJ3mb8izNg==; Domain=.mckinsey.com.cn; Path=/; Expires=Tue, 28 Sep 2021 10:18:03 GMT; Max-Age=7199; HttpOnly
<

version: 11.8.2

[szmarczak]

Can you please open a new issue? I can confirm this with Got 12.

[NeverBehave]

Opening rn

[NeverBehave]

Moved to #1876