(podman) buildkitd is not able to pull private images

[johnhamelink]

Hi there,

I'm trying to use earthly with podman (rootless), but I'm struggling to get our private registry (ghcr.io) to authenticate. Using podman pull ghcr.io/username/repo works as expected after podman login, but earthly - when trying to pull the same image - returns failed to fetch anonymous token: unexpected status: 401 Unauthorized .

• I'm running earthly 0.6.6 (installed via asdf) with podman 3.4.4 on Arch Linux.
• I had to modify my earthly config to define iptables-nft to get buildkitd to work with podman on my machine (because I use nf_tables on my host)
• Running podman logs against buildkitd produces the following logs: https://termsend.pl/o/qq354

[johnhamelink]

After removing all containers & images, podman info looks like so:

podman info
host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.1.0-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: bdb4f6e56cd193d40b75ffc9725d4b74a18cb33c'
  cpus: 20
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  hostname: jupiter.local
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.16.7-arch1-1
  linkmode: dynamic
  logDriver: journald
  memFree: 26548953088
  memTotal: 33325604864
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.4.2-1
    path: /usr/bin/crun
    version: |-
      crun version 1.4.2
      commit: f6fbc8f840df1a414f31a60953ae514fa497c748
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.1.12-1
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 7m 55.27s
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /home/john/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 0
    stopped: 2
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/john/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 5
  runRoot: /run/user/1000/containers
  volumePath: /home/john/.local/share/containers/storage/volumes
version:
  APIVersion: 3.4.4
  Built: 1639074640
  BuiltTime: Thu Dec  9 18:30:40 2021
  GitCommit: f6526ada1025c2e3f88745ba83b8b461ca659933
  GoVersion: go1.17.4
  OsArch: linux/amd64
  Version: 3.4.4

[dchw]

I got to the bottom of this. Two things you may be able to do as a workaround until the needed code is merged:

• Make sure your auth.json contains the right host. docker.io will not work; you'll need to login to https://index.docker.io/v1/ instead.
• Link your podman config into docker's default location. Something like ln $XDG_RUNTIME_DIR/containers/auth.json ~/.docker/config.json. This should allow buildkit to pick up your credentials as it stands now.

Let me know how this works, if you want to try!

[johnhamelink]

Hey @dchw, I will try this out today. Thank you :)

[johnhamelink]

@dchw apologies, didn't get round to this yesterday, but today I was able to try this fix and yes - I can now pull down private images from ghcr.io and infact build our container!

However when earthly gets around to saving the image, I get the following error:

              output | --> exporting outputs
              output | [██████████] 100% exporting layers
              output | [██████████] 100% exporting manifest sha256:e35559475547fed579e32e0905f9c03fdc998f2129db1374a416ebbc69ebc39e
              output | [██████████] 100% exporting config sha256:6e412f742d57c285300c889b0f51b3a8da4bce38c4e3381777e4561c11e43663
              output | [          ]   0% transferring ghcr.io/<redacted>/ocr:latest
              output | WARN: (exporting outputs) pull ping error: pull ping response: rpc error: code = Unknown desc = image pull: 1 error occurred:
              output | 	* command failed: podman pull 127.0.0.1:8371/sess-xvi6c4ddcan3pi7x7hmdgatb6/sp:img1: exit status 125: Trying to pull 127.0.0.1:8371/sess-xvi6c4ddcan3pi7x7hmdgatb6/sp:img1...
              output | Error: initializing source docker://127.0.0.1:8371/sess-xvi6c4ddcan3pi7x7hmdgatb6/sp:img1: pinging container registry 127.0.0.1:8371: Get "https://127.0.0.1:8371/v2/": http: server gave HTTP response to HTTPS client: exit status 125
              output | 
              output | 
Error: build target: build main: bkClient.Build: failed to solve: image pull: 1 error occurred:
	* command failed: podman pull 127.0.0.1:8371/sess-xvi6c4ddcan3pi7x7hmdgatb6/sp:img1: exit status 125: Trying to pull 127.0.0.1:8371/sess-xvi6c4ddcan3pi7x7hmdgatb6/sp:img1...
Error: initializing source docker://127.0.0.1:8371/sess-xvi6c4ddcan3pi7x7hmdgatb6/sp:img1: pinging container registry 127.0.0.1:8371: Get "https://127.0.0.1:8371/v2/": http: server gave HTTP response to HTTPS client: exit status 125

Adding --insecure to the SAVE IMAGE command does not change this error. So close though!

Thank you for your help so far!

[dchw]

Yep! For now, as another workaround, you can add 127.0.0.1:8371 to your registries.conf as an insecure registry. Both of these should be fixed in the next release with the merge of this PR: #1675, and this PR: #1685. As in, we will honor the default location for podman credentials, and allow the local registry helper to function without tweaking registries.conf.

Sample for adding an insecure registry:

[[registry]]
location = "127.0.0.1:8371"
insecure = true