bug: use-proxy not working

[ghpu]

Are you use the envd server?

• Yes, I am using the envd server.
• No, I am not using the envd server.

Describe the bug

envd up --use-proxy is recognized, but HTTP(S)_PROXY en variables don't seemt o be sent to the moby/buildkit images, and build fails when trying to access the internet :
ERRO[2023-11-30T15:19:07+01:00] Buildkit error: failed to solve: failed to load cache key: failed to do request: Head "https://ghcr.io/v2/flyteorg/flytekit/manifests/py3.11-1.10.1": dial tcp 140.82.121.33:443: connect: connection refused

To Reproduce

• Have no direct internet access.
• configure access by proxy and fill HTTP(S)_PROXY variables
• envd up

Expected behavior

proxy variables sent to buildkit container, and build working.

The docker info output

Client: Docker Engine - Community
Version: 24.0.7
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.11.2
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.21.0
Path: /usr/libexec/docker/cli-plugins/docker-compose

Server:
Containers: 2
Running: 2
Paused: 0
Stopped: 0
Images: 116
Server Version: 24.0.7
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
runc version: v1.1.10-0-g18a0cb0
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.5.0-13-generic
Operating System: Ubuntu 23.10
OSType: linux
Architecture: x86_64
CPUs: 20
Total Memory: 62.48GiB
Name: yd-4GVSJX3
ID: c4ff2738-d309-454f-96df-1a0eb58714a4
Docker Root Dir: /var/lib/docker
Debug Mode: false
HTTP Proxy: [redacted]
HTTPS Proxy: [redacted]
No Proxy: localhost,127.0.0.1
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Default Address Pools:
Base: 172.30.0.0/16, Size: 24

The envd version output

envd: v0.3.40
BuildDate: 2023-09-01T08:46:03Z
GitCommit: d99cfa4
GitTreeState: clean
GitTag: v0.3.40
GoVersion: go1.19.10
Compiler: gc
Platform: linux/amd64

Additional context

No response

[gaocegege]

Thanks for raising this!

[kemingy]

Can you run envd --debug up --use-proxy to get the debug messages?

[ghpu]

Here is the redacted version, X for project PATH, Z for proxy ip and port

DEBU[2023-12-01T10:19:35+01:00] /home/X/.config/envd/id_rsa_envd.pub already present 
DEBU[2023-12-01T10:19:35+01:00] /home/X/config/envd/id_rsa_envd already present 
DEBU[2023-12-01T10:19:35+01:00] home manager initialized                      cache-dir=/home/X/cache/envd cache-map="map[]" cache-status=/home/X/.cache/envd/cache.status config-file=/home/mclf0736/.config/envd/config.envd context="{default [{default docker-container envd_buildkitd docker <nil>}]}" context-file=/home/X/.config/envd/contexts
DEBU[2023-12-01T10:19:35+01:00] telemetry initialization                      UID=Y
DEBU[2023-12-01T10:19:35+01:00] sending telemetry                            
DEBU[2023-12-01T10:19:35+01:00] tag not specified, using default             
DEBU[2023-12-01T10:19:35+01:00] starting up command                           builder-options="{/home/X/projects/flyte/build.envd /home/X/.config/envd/config.envd plain flyte:dev /home/X/projects/flyte build /home/X/.config/envd/id_rsa_envd.pub    true }" container-name=flyte detach=false
DEBU[2023-12-01T10:19:35+01:00] getting exporter entry                        entry="[]"
DEBU[2023-12-01T10:19:35+01:00] commandconn: starting docker with [exec -i envd_buildkitd buildctl dial-stdio] 
DEBU[2023-12-01T10:19:35+01:00] starting buildkitd                            buildkit-config="&{[]}" container=envd_buildkitd tag="docker.io/moby/buildkit:v0.10.6"
DEBU[2023-12-01T10:19:35+01:00] commandconn (docker):Error response from daemon: No such container: envd_buildkitd 
DEBU[2023-12-01T10:19:35+01:00] waiting to start                              container=/envd_buildkitd
DEBU[2023-12-01T10:19:36+01:00] the container is running                      container=/envd_buildkitd
DEBU[2023-12-01T10:19:36+01:00] container is running, check if it's ready at docker-container://envd_buildkitd...  container=envd_buildkitd driver=docker-container image="docker.io/moby/buildkit:v0.10.6" socket=envd_buildkitd
DEBU[2023-12-01T10:19:36+01:00] waiting to connect to buildkitd               container=envd_buildkitd driver=docker-container image="docker.io/moby/buildkit:v0.10.6" socket=envd_buildkitd
DEBU[2023-12-01T10:19:37+01:00] commandconn: starting docker with [exec -i envd_buildkitd buildctl dial-stdio] 
DEBU[2023-12-01T10:19:38+01:00] connected to buildkitd                        container=envd_buildkitd driver=docker-container image="docker.io/moby/buildkit:v0.10.6" socket=envd_buildkitd
DEBU[2023-12-01T10:19:38+01:00] evaluating config file                        language-version=v1 tag="flyte:dev"
DEBU[2023-12-01T10:19:38+01:00] interpret the file                            filename=/home/X/.config/envd/config.envd
DEBU[2023-12-01T10:19:38+01:00] interpret the file                            filename=/home/X/projects/flyte/build.envd
DEBU[2023-12-01T10:19:38+01:00] Execute build func                           
DEBU[2023-12-01T10:19:38+01:00] rule `base` is invoked, image=ghcr.io/flyteorg/flytekit:py3.11-1.10.1, dev=true  frontend=starlark
DEBU[2023-12-01T10:19:38+01:00] rule `install.python_packages` is invoked, name=[transformers], requirements=, local_wheels=[]  frontend=starlark
DEBU[2023-12-01T10:19:38+01:00] rule `install.apt_packages` is invoked, name=[git]  frontend=starlark
DEBU[2023-12-01T10:19:38+01:00] rule `runtime.environ` is invoked, env: map[Debug:True PYTHONPATH:/root _F_IMG_ID:localhost:30000/flytekit:CHRFCcdhdRIt8Zy3BHJfzg.. http_proxy:http://Z https_proxy:http://Z], extra_path: []  frontend=starlark
DEBU[2023-12-01T10:19:38+01:00] rule `config.pip_index` is invoked, index=https://pypi.org/simple, extraIndex=, trust=false  frontend=starlark
DEBU[2023-12-01T10:19:38+01:00] rule `install.python` is invoked, version=3.11  frontend=starlark
DEBU[2023-12-01T10:19:38+01:00] failed to check manifest update: image with hash 7502135eecfb2e36c99aff8968531891 not found  language-version=v1 tag="flyte:dev"
DEBU[2023-12-01T10:19:38+01:00] compile LLB                                   gid=1000 uid=1000
DEBU[2023-12-01T10:19:38+01:00] compile base image                            image="ghcr.io/flyteorg/flytekit:py3.11-1.10.1" language="[{python 0xc000f8a140}]"
DEBU[2023-12-01T10:19:38+01:00] Using registries.d directory /etc/containers/registries.d 
DEBU[2023-12-01T10:19:38+01:00] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[2023-12-01T10:19:38+01:00] Trying to access "ghcr.io/flyteorg/flytekit:py3.11-1.10.1" 
DEBU[2023-12-01T10:19:38+01:00] No credentials matching ghcr.io/flyteorg/flytekit found in /run/user/1000/containers/auth.json 
DEBU[2023-12-01T10:19:38+01:00] No credentials matching ghcr.io/flyteorg/flytekit found in /home/X/.config/containers/auth.json 
DEBU[2023-12-01T10:19:38+01:00] No credentials matching ghcr.io/flyteorg/flytekit found in /home/X/.docker/config.json 
DEBU[2023-12-01T10:19:38+01:00] No credentials matching ghcr.io/flyteorg/flytekit found in /home/X/.dockercfg 
DEBU[2023-12-01T10:19:38+01:00] No credentials for ghcr.io/flyteorg/flytekit found 
DEBU[2023-12-01T10:19:38+01:00]  No signature storage configuration found for ghcr.io/flyteorg/flytekit:py3.11-1.10.1, using built-in default file:///home/X/.local/share/containers/sigstore 
DEBU[2023-12-01T10:19:38+01:00] Looking for TLS certificates and private keys in /etc/docker/certs.d/ghcr.io 
DEBU[2023-12-01T10:19:38+01:00] GET https://ghcr.io/v2/                      
DEBU[2023-12-01T10:19:39+01:00] Ping https://ghcr.io/v2/ status 401          
DEBU[2023-12-01T10:19:39+01:00] GET https://ghcr.io/token?scope=repository%3Aflyteorg%2Fflytekit%3Apull&service=ghcr.io 
DEBU[2023-12-01T10:19:39+01:00] Increasing token expiration to: 60 seconds   
DEBU[2023-12-01T10:19:39+01:00] GET https://ghcr.io/v2/flyteorg/flytekit/manifests/py3.11-1.10.1 
DEBU[2023-12-01T10:19:39+01:00] Content-Type from manifest GET is "application/vnd.oci.image.index.v1+json" 
DEBU[2023-12-01T10:19:39+01:00] Using registries.d directory /etc/containers/registries.d 
DEBU[2023-12-01T10:19:39+01:00] No credentials matching ghcr.io/flyteorg/flytekit found in /run/user/1000/containers/auth.json 
DEBU[2023-12-01T10:19:39+01:00] No credentials matching ghcr.io/flyteorg/flytekit found in /home/X/.config/containers/auth.json 
DEBU[2023-12-01T10:19:39+01:00] No credentials matching ghcr.io/flyteorg/flytekit found in /home/X/.docker/config.json 
DEBU[2023-12-01T10:19:39+01:00] No credentials matching ghcr.io/flyteorg/flytekit found in /home/X/.dockercfg 
DEBU[2023-12-01T10:19:39+01:00] No credentials for ghcr.io/flyteorg/flytekit found 
DEBU[2023-12-01T10:19:39+01:00]  No signature storage configuration found for ghcr.io/flyteorg/flytekit:py3.11-1.10.1, using built-in default file:///home/mclf0736/.local/share/containers/sigstore 
DEBU[2023-12-01T10:19:39+01:00] Looking for TLS certificates and private keys in /etc/docker/certs.d/ghcr.io 
DEBU[2023-12-01T10:19:39+01:00] GET https://ghcr.io/v2/                      
DEBU[2023-12-01T10:19:40+01:00] Ping https://ghcr.io/v2/ status 401          
DEBU[2023-12-01T10:19:40+01:00] GET https://ghcr.io/token?scope=repository%3Aflyteorg%2Fflytekit%3Apull&service=ghcr.io 
DEBU[2023-12-01T10:19:40+01:00] Increasing token expiration to: 60 seconds   
DEBU[2023-12-01T10:19:40+01:00] HEAD https://ghcr.io/v2/flyteorg/flytekit/manifests/py3.11-1.10.1 
DEBU[2023-12-01T10:19:40+01:00] GET https://ghcr.io/v2/flyteorg/flytekit/manifests/sha256:709d2f9d14b9a49cb3d319bbd75473ae6c8c50ee09cf6ab4698373fffb84c883 
DEBU[2023-12-01T10:19:40+01:00] Content-Type from manifest GET is "application/vnd.oci.image.index.v1+json" 
DEBU[2023-12-01T10:19:40+01:00] GET https://ghcr.io/v2/flyteorg/flytekit/manifests/sha256:1fbae966f0baa54da59da84f5db73f5ce051d8a23fd8992f48912a314878f04b 
DEBU[2023-12-01T10:19:40+01:00] Content-Type from manifest GET is "application/vnd.oci.image.manifest.v1+json" 
DEBU[2023-12-01T10:19:40+01:00] Downloading /v2/flyteorg/flytekit/blobs/sha256:bab837efcb3df968715ca678c859e552ce70ce7156ee66008ad0b9611bc9cce0 
DEBU[2023-12-01T10:19:40+01:00] GET https://ghcr.io/v2/flyteorg/flytekit/blobs/sha256:bab837efcb3df968715ca678c859e552ce70ce7156ee66008ad0b9611bc9cce0 
DEBU[2023-12-01T10:19:41+01:00] inherit envs from base image: [PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin LANG=C.UTF-8 GPG_KEY=A035C8C19219BA821ECEA86B64E628F8D684696D PYTHON_VERSION=3.11.4 PYTHON_PIP_VERSION=23.1.2 PYTHON_SETUPTOOLS_VERSION=65.5.1 PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/0d8570dc44796f4369b652222cf176b3db6ac70e/public/get-pip.py PYTHON_GET_PIP_SHA256=96461deced5c2a487ddc65207ec5a9cffeca0d34e7af7ea1afc470ff0d746207 PYTHONPATH=/root FLYTE_INTERNAL_IMAGE=ghcr.io/flyteorg/flytekit:py3.11-1.10.1] 
DEBU[2023-12-01T10:19:41+01:00] apt/pypi calculated cacheID: /var/cache/apt/flyte-cpu 
DEBU[2023-12-01T10:19:41+01:00] apt/pypi calculated cacheID: /var/lib/apt/flyte-cpu 
DEBU[2023-12-01T10:19:41+01:00] using custom PyPI index                       index="https://pypi.org/simple"
DEBU[2023-12-01T10:19:41+01:00] Configure pip install statements              command="python -m pip install transformers"
DEBU[2023-12-01T10:19:41+01:00] apt/pypi calculated cacheID: /root/.cache/pip/flyte-cpu 
DEBU[2023-12-01T10:19:41+01:00] compiled build.envd                           language-version=v1 tag="flyte:dev"
DEBU[2023-12-01T10:19:41+01:00] building envd image                           language-version=v1 tag="flyte:dev"
DEBU[2023-12-01T10:19:41+01:00] build image with buildkit                     language-version=v1 tag="flyte:dev" type=docker
DEBU[2023-12-01T10:19:41+01:00] loading image to docker host                  language-version=v1 tag="flyte:dev"
DEBU[2023-12-01T10:19:41+01:00] serving grpc connection                      
DEBU[2023-12-01T10:19:41+01:00] running BuildFunc for envd                    language-version=v1 tag="flyte:dev"
DEBU[2023-12-01T10:19:41+01:00] final entrypoint: {[horust]}                  language-version=v1 tag="flyte:dev"
DEBU[2023-12-01T10:19:41+01:00] setting image config: {"architecture":"amd64","os":"linux","config":{"User":"envd","ExposedPorts":{"2222/tcp":{}},"Env":["https_proxy=http://Z","http_proxy=http://Z","GPG_KEY=A035C8C19219BA821ECEA86B64E628F8D684696D","PYTHONPATH=/root","_F_IMG_ID=localhost:30000/flytekit:CHRFCcdhdRIt8Zy3BHJfzg..","PYTHON_VERSION=3.11.4","PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/0d8570dc44796f4369b652222cf176b3db6ac70e/public/get-pip.py","PYTHON_GET_PIP_SHA256=96461deced5c2a487ddc65207ec5a9cffeca0d34e7af7ea1afc470ff0d746207","FLYTE_INTERNAL_IMAGE=ghcr.io/flyteorg/flytekit:py3.11-1.10.1","PYTHON_PIP_VERSION=23.1.2","PYTHON_SETUPTOOLS_VERSION=65.5.1","USER=envd","SHELL=bash","ENVD_WORKDIR=/home/envd/flyte","Debug=True","LANG=C.UTF-8","PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/conda/envs/envd/bin:/opt/conda/bin:/home/envd/.local/bin","LC_ALL=en_US.UTF-8","LANG=C.UTF-8"],"Entrypoint":["horust"],"WorkingDir":"/home/envd/flyte","Labels":{"ai.tensorchord.envd.apt.packages":"[\"git\"]","ai.tensorchord.envd.build.context":"/home/X/projects/flyte","ai.tensorchord.envd.build.digest":"7502135eecfb2e36c99aff8968531891","ai.tensorchord.envd.container.name":"flyte","ai.tensorchord.envd.graph.general":"{\"Languages\":[{\"Name\":\"python\",\"Version\":\"3.11\"}],\"EnvdSyntaxVersion\":\"v1\",\"Image\":\"ghcr.io/flyteorg/flytekit:py3.11-1.10.1\",\"User\":\"envd\",\"Shell\":\"bash\",\"Dev\":true,\"CUDA\":null,\"CUDNN\":\"8\",\"NumGPUs\":0,\"ShmSize\":0,\"UbuntuAPTSource\":null,\"CRANMirrorURL\":null,\"JuliaPackageServer\":null,\"PyPIIndexURL\":\"https://pypi.org/simple\",\"PyPIExtraIndexURL\":null,\"PyPITrust\":false,\"PublicKeyPath\":\"/home/X/.config/envd/id_rsa_envd.pub\",\"PyPIPackages\":[[\"transformers\"]],\"RequirementsFile\":null,\"PythonWheels\":null,\"RPackages\":[],\"JuliaPackages\":[],\"SystemPackages\":[\"git\"],\"VSCodePlugins\":null,\"UserDirectories\":[],\"Exec\":[],\"Copy\":null,\"Mount\":null,\"HTTP\":null,\"Entrypoint\":null,\"Repo\":{},\"EnvironmentName\":\"flyte\",\"EnvironmentPath\":\"/home/X/projects/flyte\",\"WorkingDir\":\"/home/envd/flyte\",\"DisableMergeOp\":false,\"environ\":{\"Debug\":\"True\",\"FLYTE_INTERNAL_IMAGE\":\"ghcr.io/flyteorg/flytekit:py3.11-1.10.1\",\"GPG_KEY\":\"A035C8C19219BA821ECEA86B64E628F8D684696D\",\"LANG\":\"C.UTF-8\",\"PATH\":\"/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",\"PYTHONPATH\":\"/root\",\"PYTHON_GET_PIP_SHA256\":\"96461deced5c2a487ddc65207ec5a9cffeca0d34e7af7ea1afc470ff0d746207\",\"PYTHON_GET_PIP_URL\":\"https://github.com/pypa/get-pip/raw/0d8570dc44796f4369b652222cf176b3db6ac70e/public/get-pip.py\",\"PYTHON_PIP_VERSION\":\"23.1.2\",\"PYTHON_SETUPTOOLS_VERSION\":\"65.5.1\",\"PYTHON_VERSION\":\"3.11.4\",\"SHELL\":\"bash\",\"USER\":\"envd\",\"_F_IMG_ID\":\"localhost:30000/flytekit:CHRFCcdhdRIt8Zy3BHJfzg..\",\"http_proxy\":\"http://Z\",\"https_proxy\":\"http://Z\"},\"env_paths\":[\"/usr/local/sbin\",\"/usr/local/bin\",\"/usr/sbin\",\"/usr/bin\",\"/sbin\",\"/bin\",\"/opt/conda/envs/envd/bin:/opt/conda/bin:/home/envd/.local/bin\"],\"Platform\":{\"architecture\":\"amd64\",\"os\":\"linux\"}}","ai.tensorchord.envd.graph.runtime":"{\"environ\":{\"Debug\":\"True\",\"FLYTE_INTERNAL_IMAGE\":\"ghcr.io/flyteorg/flytekit:py3.11-1.10.1\",\"GPG_KEY\":\"A035C8C19219BA821ECEA86B64E628F8D684696D\",\"LANG\":\"C.UTF-8\",\"PATH\":\"/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",\"PYTHONPATH\":\"/root\",\"PYTHON_GET_PIP_SHA256\":\"96461deced5c2a487ddc65207ec5a9cffeca0d34e7af7ea1afc470ff0d746207\",\"PYTHON_GET_PIP_URL\":\"https://github.com/pypa/get-pip/raw/0d8570dc44796f4369b652222cf176b3db6ac70e/public/get-pip.py\",\"PYTHON_PIP_VERSION\":\"23.1.2\",\"PYTHON_SETUPTOOLS_VERSION\":\"65.5.1\",\"PYTHON_VERSION\":\"3.11.4\",\"SHELL\":\"bash\",\"USER\":\"envd\",\"_F_IMG_ID\":\"localhost:30000/flytekit:CHRFCcdhdRIt8Zy3BHJfzg..\",\"http_proxy\":\"http://Z\",\"https_proxy\":\"http://Z\"},\"env_paths\":[\"/usr/local/sbin\",\"/usr/local/bin\",\"/usr/sbin\",\"/usr/bin\",\"/sbin\",\"/bin\",\"/opt/conda/envs/envd/bin:/opt/conda/bin:/home/envd/.local/bin\"]}","ai.tensorchord.envd.ports":"[{\"name\":\"ssh\",\"port\":2222}]","ai.tensorchord.envd.pypi.commands":"[\"transformers\"]","ai.tensorchord.envd.r.packages":"[]","ai.tensorchord.envd.repo":"{}","ai.tensorchord.envd.syntax.version":"v1","ai.tensorchord.envd.vendor":"envd"}},"rootfs":{"type":"layers","diff_ids":null}}  language-version=v1 tag="flyte:dev"
DEBU[2023-12-01T10:19:41+01:00] stopping session                             
#1 [internal] setting pip cache mount permissions
ERRO[2023-12-01T10:19:41+01:00] Buildkit error: failed to solve: failed to load cache key: failed to do request: Head "https://ghcr.io/v2/flyteorg/flytekit/manifests/py3.11-1.10.1": dial tcp 140.82.121.34:443: connect: connection refused
(1) attached stack trace
  -- stack trace:
  | github.com/tensorchord/envd/pkg/builder.generalBuilder.build.func1
  |     /home/runner/work/envd/envd/pkg/builder/build.go:265
  | golang.org/x/sync/errgroup.(*Group).Go.func1
  |     /home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75
  | runtime.goexit
  |     /opt/hostedtoolcache/go/1.19.10/x64/src/runtime/asm_amd64.s:1594
Wraps: (2) Buildkit error
Wraps: (3) failed to solve: failed to load cache key: failed to do request: Head "https://ghcr.io/v2/flyteorg/flytekit/manifests/py3.11-1.10.1": dial tcp 140.82.121.34:443: connect: connection refused
  | (1) failed to solve: failed to load cache key: failed to do request: Head "https://ghcr.io/v2/flyteorg/flytekit/manifests/py3.11-1.10.1": dial tcp 140.82.121.34:443: connect: connection refused
  | Error types: (1) *builder.BuildkitdErr
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *builder.BuildkitdErr 
#1 DONE 0.0s

#2 docker-image://ghcr.io/flyteorg/flytekit:py3.11-1.10.1
#2 resolve ghcr.io/flyteorg/flytekit:py3.11-1.10.1 0.1s done
#2 ERROR: failed to do request: Head "https://ghcr.io/v2/flyteorg/flytekit/manifests/py3.11-1.10.1": dial tcp 140.82.121.34:443: connect: connection refused

#3 docker-image://docker.io/tensorchord/horust:v0.2.3
#3 resolve docker.io/tensorchord/horust:v0.2.3 0.1s done
#3 CANCELED

#4 docker-image://docker.io/mambaorg/micromamba:1.0.0
#4 resolve docker.io/mambaorg/micromamba:1.0.0 0.1s done
#4 CANCELED

#5 docker-image://docker.io/tensorchord/starship:v0.0.1
#5 resolve docker.io/tensorchord/starship:v0.0.1 0.1s done
#5 CANCELED

#6 docker-image://docker.io/tensorchord/envd-sshd-from-scratch:v0.3.40
#6 resolve docker.io/tensorchord/envd-sshd-from-scratch:v0.3.40 0.1s done
#6 CANCELED
------
 > docker-image://ghcr.io/flyteorg/flytekit:py3.11-1.10.1:
------
ERRO[2023-12-01T10:19:41+01:00] failed to load docker image: Post "http://%2Fvar%2Frun%2Fdocker.sock/v1.42/images/load?quiet=1": context canceled  language-version=v1 tag="flyte:dev"
FATA[2023-12-01T10:19:41+01:00] failed to build the image: failed to build: failed to wait error group: Buildkit error: failed to solve: failed to load cache key: failed to do request: Head "https://ghcr.io/v2/flyteorg/flytekit/manifests/py3.11-1.10.1": dial tcp 140.82.121.34:443: connect: connection refused 

[kemingy]

From the debug messages, it seems envd can get the base image metadata through the proxy (on your host), but the buildkitd in the container cannot access the Internet.

1. Is your proxy a localhost address?
2. Can you try envd v1 with moby? You can create the moby context with envd context create --name moby-test --builder moby-worker --use. v1

[ghpu]

Proxy is not a localhost address,
Using envd1 works to access the images through the proxy ( I got another error later about groupadd and GID 1000 already existing, but that's another issue :)- )