Failed to copy container. Digest did not match

[kvz]

Describe the bug
I'm trying to copy my container to AWS ECR from my Linux X64 CI server, but hitting the following issue. Failed to copy container. Digest did not match.

To reproduce

  # also tried with `--system x86_64-linux`, does not make a difference
  devenv container build processes

  # ecr login
  aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 123123123123.dkr.ecr.us-east-1.amazonaws.com
  # ecr create repository
  if ! aws ecr describe-repositories --repository-names processes --region us-east-1; then
    aws ecr create-repository --repository-name processes --region us-east-1
  fi
  # ecr push. also tried with `--copy-args="--preserve-digests"`. does not make a difference
  devenv container \
    --registry docker://123123123123.dkr.ecr.us-east-1.amazonaws.com/ \
  copy processes

Output:

• Building processes container ...
• Using Cachix: devenv

/nix/store/31g4rq1p6spmihc95qm3qc2iy8w7hm5b-image-processes.json
✔ Building processes container in 2.2s.
• Copying processes container ...• Running /nix/store/g96s72rwnk3vdkqnvd5wmkmnrmk0z9sk-copy-container /nix/store/31g4rq1p6spmihc95qm3qc2iy8w7hm5b-image-processes.json docker://123123123123.dkr.ecr.us-east-1.amazonaws.com/ --preserve-digests

Copying container /nix/store/31g4rq1p6spmihc95qm3qc2iy8w7hm5b-image-processes.json to docker://123123123123.dkr.ecr.us-east-1.amazonaws.com/processes:latest

Getting image source signatures
Copying blob sha256:67eb02cf19ceef2d35b6a7373359a5f7ede6f478e59ef55ef0688be70b03[97](https://github.com/transloadit/api2/actions/runs/8615325344/job/23610670898#step:6:98)d2
Copying blob sha256:2ac708344cd9024282848c53dbd92614463d4ce574312949cd43f09ff92e8e85
time="2024-04-09T12:10:47Z" level=fatal msg="writing blob: Patch \"https://123123123123.dkr.ecr.us-east-1.amazonaws.com/v2/processes/blobs/uploads/27a9a11e-ad6e-46cb-92fe-56617689e652\": happened during read: Digest did not match, expected sha256:67eb02cf19ceef2d35b6a7373359a5f7ede6f478e59ef55ef0688be70b0397d2, got sha256:8b155f8b3b4bca1f56a387a409238782eb8578fa86fe57df5a9d4961cb13a6d2"

✔ Copying processes container in 70.3s.
Error:   × Failed to copy container

Version

• While developing (locally): devenv 1.0.3 (aarch64-darwin)
• While building (CI): devenv 1.0.3 (x86_64-linux)

Not sure what to do next 🤔 any idea?

[kvz]

Extra datapoint, I just tried to push the container locally (on linux ARM for linux ARM, as opposed to X64 earlier), and pushing to the GitHub Container Registry (as opposed to ECR earlier), and the error is the same. So I guess that puts the suspicion on non-architecture/registry things.

[kvz]

It also happens just when running, not just copying:

vagrant at vbox@local-vagrantisp in /srv/current/crm on api2/vbox#00000 
$ devenv container run processes
• Building processes container ...
• Using Cachix: devenv
warning: Ignoring setting 'auto-allocate-uids' because experimental feature 'auto-allocate-uids' is not enabled
warning: Ignoring setting 'impure-env' because experimental feature 'configurable-impure-env' is not enabled
/nix/store/x2pavkjn8n0c3jqmdiizldyd399i9gdc-image-processes.json
✔ Building processes container in 35.4s.
• Running /nix/store/wr98nrk2xwqln5lgg5qln83zagk15006-copy-container /nix/store/x2pavkjn8n0c3jqmdiizldyd399i9gdc-image-processes.json docker-daemon: 

Copying container /nix/store/x2pavkjn8n0c3jqmdiizldyd399i9gdc-image-processes.json to docker-daemon:processes:latest

Getting image source signatures
Copying blob 3e07a34a1ca1 done   | 
Copying blob 0fb47a54c2cb [=====================================>] 1.6GiB / 1.6GiB | 643.9 MiB/s
FATA[0003] writing blob: writing to temporary on-disk layer: happened during read: Digest did not match, expected sha256:0fb47a54c2cb3ce900976003c42db74ebb24fbf9bfefd458c0a7d2cd5f15a1f3, got sha256:d2e783b2eeb88149765f06c6dff4dfc85b84db7b335484c7d73b519ff748cf2c 
✔ Copying processes container in 17.0s.
Error:   × Failed to copy container

My devenv.nix

# - https://shyim.me/blog/devenv-compose-developer-environment-for-php-with-nix/
# - https://github.com/cachix/devenv/blob/main/src/modules/languages/php.nix
{ pkgs, config, lib, ... }:

let
  domain = "localhost";
  app = "webapi";
  dataDir = "/srv/current/crm/webroot";

  crmEnv = {
    MYSQL_GLOBAL_DBNM = "transloadit";
    MYSQL_GLOBAL_HOST = "localhost";
    MYSQL_GLOBAL_PORT = "3306";
    MYSQL_GLOBAL_USER = "root";
    MYSQL_GLOBAL_PASSWORD = "root";
    APP_NAME="crm";
  };
in {
  # https://devenv.sh/basics/
  env.GREET = "devenv";

  # https://devenv.sh/packages/
  packages = [
    pkgs.htop
  ];

  # https://devenv.sh/scripts/
  scripts.hello.exec = "echo hello from $GREET";

  # https://devenv.sh/services/
  services.mysql.enable = true;
  services.mysql.package = pkgs.mysql80;
  services.mysql.initialDatabases = [{ name = crmEnv.MYSQL_GLOBAL_DBNM; }];
  services.mysql.ensureUsers = [
    {
      name = crmEnv.MYSQL_GLOBAL_USER;
      password = crmEnv.MYSQL_GLOBAL_PASSWORD;
      ensurePermissions = { "${crmEnv.MYSQL_GLOBAL_DBNM}.*" = "ALL PRIVILEGES"; };
    }
  ];

  # https://devenv.sh/languages/
  languages.typescript.enable = true;
  languages.php = {
    enable = true;
    version = "7.4";
    ini = ''
      memory_limit = 256M
    '';

    fpm.pools.${app} = {
      settings = {
        "pm" = "dynamic";
        "pm.max_children" = 5;
        "pm.start_servers" = 2;
        "pm.min_spare_servers" = 1;
        "pm.max_spare_servers" = 5;
        "php_admin_value[error_log]" = "stderr";
        "php_admin_flag[log_errors]" = true;
        "catch_workers_output" = true;
        "env[MYSQL_GLOBAL_DBNM]" = crmEnv.MYSQL_GLOBAL_DBNM;
        "env[MYSQL_GLOBAL_HOST]" = crmEnv.MYSQL_GLOBAL_HOST;
        "env[MYSQL_GLOBAL_PORT]" = crmEnv.MYSQL_GLOBAL_PORT;
        "env[MYSQL_GLOBAL_USER]" = crmEnv.MYSQL_GLOBAL_USER;
        "env[MYSQL_GLOBAL_PASSWORD]" = crmEnv.MYSQL_GLOBAL_PASSWORD;
        "env[APP_NAME]" = crmEnv.APP_NAME;
      };
    };
  };

  services.nginx = {
    enable = true;
    httpConfig = ''
      server {
        listen 1113;
        server_name ${domain};
        root ${dataDir};
        index index.php index.html index.htm;
        location / {
          try_files $uri $uri/ /index.php?$query_string;
        }
        location ~ \.php$ {
          include ${pkgs.nginx}/conf/fastcgi_params;
          fastcgi_pass unix:${config.languages.php.fpm.pools.${app}.socket};
          fastcgi_index index.php;
          fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        }
      }
    '';
  };
}

[ento]

There's a similar issue about mismatched digest in the context of devenv reported in nix2container's repo: nlewo/nix2container#127

[domenkozar]

The fix has hit Nix, now we're verifying if it fixes the issue in nlewo/nix2container#127 (comment)

[domenkozar]

Does this now work with Nix 2.22.1?

[sandydoo]

@domenkozar, bumped nix-daemon from ~2.20 to 2.22.1. No more digest errors on aarch64-linux.