Curl to libnvidia-container/gpgkey Timing out unexpectedly

[yooav]

I'm attempting to curl for the nvidia container toolkit GPG key https://nvidia.github.io/libnvidia-container/gpgkey following instructions here.

I get the following error that the curl request for the GPG key times out connecting to the host:

curl: (28) Failed to connect to nvidia.github.io port 443 after 278724 ms: Connection timed out

Any idea why this is happening and how we can fix this? It doesn't seem to be happening consistently on all our machines, and particularly we see this in a GCP node. Did we get blacklisted for querying too many times?

Related to NVIDIA/nvidia-container-runtime#164

[elezar]

@yooav the site is hosted as GitHub pages and as such the following usage limits are applicable: https://docs.github.com/en/pages/getting-started-with-github-pages/about-github-pages#usage-limits

It may be that you're getting rate limited, but this should not time out and should issue a 429 if this is the case.

Could you check the verbose output of curl?

[yooav]

Here's the verbose output, doesn't seem like we get any response

curl https://nvidia.github.io/nvidia-docker/gpgkey -v
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 185.199.109.153:443...
*   Trying 2606:50c0:8000::153:443...
* Immediate connect fail for 2606:50c0:8000::153: Cannot assign requested address
*   Trying 2606:50c0:8001::153:443...
* Immediate connect fail for 2606:50c0:8001::153: Cannot assign requested address
*   Trying 2606:50c0:8002::153:443...
* Immediate connect fail for 2606:50c0:8002::153: Cannot assign requested address
*   Trying 2606:50c0:8003::153:443...
* Immediate connect fail for 2606:50c0:8003::153: Cannot assign requested address
  0     0    0     0    0     0      0      0 --:--:--  0:02:10 --:--:--     0* connect to 185.199.109.153 port 443 failed: Connection timed out
*   Trying 185.199.108.153:443...
  0     0    0     0    0     0      0      0 --:--:--  0:03:35 --:--:--     0* After 84510ms connect time, move on!
* connect to 185.199.108.153 port 443 failed: Connection timed out
*   Trying 185.199.110.153:443...
  0     0    0     0    0     0      0      0 --:--:--  0:04:17 --:--:--     0* After 42255ms connect time, move on!
* connect to 185.199.110.153 port 443 failed: Connection timed out
*   Trying 185.199.111.153:443...
  0     0    0     0    0     0      0      0 --:--:--  0:04:38 --:--:--     0* After 21128ms connect time, move on!
* connect to 185.199.111.153 port 443 failed: Connection timed out
* Failed to connect to nvidia.github.io port 443 after 278872 ms: Connection timed out
  0     0    0     0    0     0      0      0 --:--:--  0:04:38 --:--:--     0
* Closing connection 0
curl: (28) Failed to connect to nvidia.github.io port 443 after 278872 ms: Connection timed out

[yooav]

I guess is this issue more relevant for github rather than nvidia then?

[elezar]

This seems like an issue with your networking rather than the URL itself. Do you have some firewall or proxy configured on these nodes?

On my local machine:

curl https://nvidia.github.io/nvidia-docker/gpgkey -v
*   Trying 185.199.111.153:443...
* Connected to nvidia.github.io (185.199.111.153) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES128-GCM-SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=GitHub, Inc.; CN=*.github.io
*  start date: Mar 15 00:00:00 2024 GMT
*  expire date: Mar 14 23:59:59 2025 GMT
...
9QWY
-----END PGP PUBLIC KEY BLOCK-----
* Connection #0 to host nvidia.github.io left intact