S3 Storage Fails with storage: read exceeds limit

[mhkarimi1383]

Description

We are building an image based on php:8.2-cli-bookworm (using earthly)
and our registry storage is S3, and we are able to push image without problem, but when we want to pull image we are getting
500: Internal Server error, with unknown error in log and in details we are getting storage: read exceeds limit

Reproduce

Here is my Earthfile

VERSION 0.8
ARG --required PHP_VERSION
ARG --required DEBIAN_VERSION
FROM docker.io/library/php:${PHP_VERSION}-cli-${DEBIAN_VERSION}

WORKDIR /var/www/html/
ENV DEBIAN_FRONTEND=noninteractive
ENV LC_ALL=C.UTF-8
ENV TERM=xterm-color
ENV COMPOSER_ALLOW_SUPERUSER 1
ENV PATH /var/www/html/vendor/bin:/composer/vendor/bin:$PATH

RUN rm -f /etc/apt/apt.conf.d/docker-clean; echo 'Binary::apt::APT::Keep-Downloaded-Packages "true";' > /etc/apt/apt.conf.d/keep-cache

reqs:
    RUN set -eux; \
        apt-get update; \
        apt-get install -y apt-transport-https; \
        pecl channel-update pecl.php.net \
        && apt-get install -yqq --no-install-recommends \
                apt-utils \
                gnupg \
                gosu \
                git \
                curl \
                ca-certificates \
                supervisor \
                libmemcached-dev \
                libz-dev \
                libbrotli-dev \
                libc-ares-dev \
                libpq-dev \
                libjpeg-dev \
                libpng-dev \
                libfreetype6-dev \
                libssl-dev \
                libwebp-dev \
                libmcrypt-dev \
                libonig-dev \
                libzip-dev zip unzip \
                libargon2-1 \
                libidn2-0 \
                libpcre2-8-0 \
                libpcre3 \
                libxml2 \
                libzstd1 \
                procps \
                libcurl4-openssl-dev

    RUN apt-get install -y --no-install-recommends vim nano unzip net-tools \
        libmagickwand-dev libcurl4-gnutls-dev libgmp-dev libxslt1-dev zlib1g-dev libicu-dev g++
    
    RUN pecl install -f \
       igbinary

    RUN docker-php-ext-enable \
      igbinary

    RUN docker-php-ext-install \
        bz2 \
        calendar \
        curl \
        exif \
        gettext \
        gmp \
        iconv \
        pdo \
        pdo_pgsql \
        shmop \
        soap \
        sysvmsg \
        sysvsem \
        sysvshm \
        xml \
        xsl

    RUN php -m

    RUN set -xe; \
        docker-php-ext-configure zip \
                && docker-php-ext-install zip \
                && php -m | grep -q 'zip'; \
        docker-php-ext-install \
                mbstring \
                pdo_mysql; \
        docker-php-ext-configure gd \
                --prefix=/usr \
                --with-jpeg \
                --with-webp \
                --with-freetype \
        && docker-php-ext-install gd; \
        php -r 'var_dump(gd_info());'

    RUN docker-php-ext-install opcache
    RUN pecl install -o -f redis && \
        rm -rf /tmp/pear && \
        docker-php-ext-enable redis
    RUN docker-php-ext-install pcntl
    RUN docker-php-ext-install bcmath
    RUN pecl install -D 'enable-openssl="yes" enable-http2="yes" enable-swoole-curl="yes" enable-cares="yes"' swoole
    RUN docker-php-ext-enable swoole && \
        php -m | grep -q "swoole"
    RUN docker-php-ext-configure intl && \
        docker-php-ext-install intl
    
    ARG INSTALL_PG_CLIENT=true
    ARG INSTALL_POSTGIS=true
    IF [ ${INSTALL_PG_CLIENT} = true ]
        RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
            --mount=type=cache,target=/var/lib/apt,sharing=locked \
            apt-get install -yqq gnupg \
            && . /etc/os-release \
            && echo "deb http://apt.postgresql.org/pub/repos/apt $VERSION_CODENAME-pgdg main" > /etc/apt/sources.list.d/pgdg.list \
            && curl -sL https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add - \
            && apt-get update -yqq \
            && apt-get install -yqq --no-install-recommends --show-progress postgresql-client-16 postgis;
        IF [ ${INSTALL_POSTGIS} = true ]
            RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
            --mount=type=cache,target=/var/lib/apt,sharing=locked \
            apt-get install -yqq --no-install-recommends --show-progress postgis;
        END
    END
    RUN curl -s https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin/ --filename=composer
    RUN composer global require bamarni/symfony-console-autocomplete

    RUN curl -sL https://deb.nodesource.com/setup_20.x | bash - \
        && apt-get install -y nodejs \
        && npm install -g npm \
        && curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - \
        && echo "deb https://dl.yarnpkg.com/debian/ stable main" > /etc/apt/sources.list.d/yarn.list \
        && apt-get update \
        && apt-get install -y yarn

docker:
    FROM +reqs
    ARG --required CONTAINER_IMAGE_PREFIX
    ARG --required PHP_VERSION
    ARG --required DEBIAN_VERSION

    SAVE IMAGE --cache-hint --push ${CONTAINER_IMAGE_PREFIX}/php:${PHP_VERSION}-cli-${DEBIAN_VERSION}

Expected behavior

Pull image without problem

registry version

image tag: registry:2
version: registry github.com/docker/distribution 2.8.3

Additional Info

Image size is about 600 MBs

But there was nothing in directories related to this image/manifest in my S3 bucket

and

There is no problem while using Dockerfile (I think there are some problems with buildkit/earthly)

[milosgajdos]

There is a read limit on pulls to prevent memory allocations (

distribution/registry/storage/io.go

Line 13 in 5d5c60f

maxBlobGetSize = 4 << 20

), indeed. How are you pulling these images? Using docker pull?

[mhkarimi1383]

Both using docker pull and FROM keyword, and also K8s pull using ContainerD
Also I have a worker that keeps my images up to date based on policies it will check for updates every one minute per image, but why switching to Dockerfile fixed problem?

[mhkarimi1383]

I have searched for the error message (after storage: prefix but found nothing)
Thank you for sharing the code

[mhkarimi1383]

As I see on code I think I was trying to make the image to be a single layer So it was exceeding that limit

[milosgajdos]

I would also recommend you to switch to v3 release -- it has a lot of patches applied and a bunch of performance improvements. Consider v2.x release to be in maintenance mode at this point

[mhkarimi1383]

Thanks
Is that stable?
Also what are changes?
I saw nothing in docs about that

[milosgajdos]

It's not been marked as stable yet, but the changes have been accumulated over the past couple of years -- before we released v3 there had been no release for over 5 years. See the not-fully complete changelog here: https://github.com/distribution/distribution/releases/tag/v3.0.0-alpha.1

[mhkarimi1383]

Thank you again
I have some problems with garbage collection
That when I run that it will not remove things completely and the removed tag is listed in the tag list api

Also I have some integrations with HTTP V2 API and Debug API
And I need a lot of tests and more list of deprications on API, etc.