Push hangs at first layer

[mayorovp]

Description

I cannot push anything on private registry, like this:

root@gateway:/home/ubuntu# docker push registry.mcs.local/alpine
Using default tag: latest
The push refers to repository [registry.mcs.local/alpine]
5af4f8f59b76: Pushing [=======>                                           ]  1.065MB/7.377MB

This push is hang forever without any progress.
Tcpdump shows ACK packets with zero window from registry to docker (docker wants to send more bytes on HTTP connection, but registry never reads his tcp socket).

Reproduce

1. start registry with image distribution/distribution:edge using s3 from vk cloud
2. configure docker to use insecure private registry
3. try to push something to it
4. see nothing on docker side, and infinite s3/ListParts requests on registry side in logs (>10 requests per second)

Expected behavior

Push is successfull. Or returned useful error message.
At least write some useful error message to logs!

registry version

registry:2.8.3, distribution/distribution:edge

Additional Info

My registry configuration:

version: 0.1
log:
  level: debug
storage:
  s3:
    accesskey: [redacted]
    secretkey: [redacted]
    region: kz-ast
    regionendpoint: https://hb.kz-ast.vkcs.cloud
    bucket: registry-images
    rootdirectory: /own
    loglevel: debug
  redirect:
    disable: false
  delete:
    enabled: false
  cache:
    blobdescriptor: inmemory
  maintenance:
    uploadpurging:
      enabled: true
      age: 168h
      interval: 24h
      dryrun: false
    readonly:
      enabled: false
http:
  addr: 0.0.0.0:80
  headers:
    X-Content-Type-Options: [nosniff]
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3

Looks like storage redirect don't matter (i tried both configurations)

Logs are in attachment

Docker version:

root@gateway:/home/ubuntu# docker version
Client:
 Version:           24.0.5
 API version:       1.43
 Go version:        go1.20.3
 Git commit:        24.0.5-0ubuntu1~22.04.1
 Built:             Mon Aug 21 19:50:14 2023
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          24.0.5
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.3
  Git commit:       24.0.5-0ubuntu1~22.04.1
  Built:            Mon Aug 21 19:50:14 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.2
  GitCommit:
 runc:
  Version:          1.1.7-0ubuntu1~22.04.1
  GitCommit:
 docker-init:
  Version:          0.19.0
  GitCommit:

[milosgajdos]

I've tested this on AWS S3 and this works like charm. My suspicion is this is related to #3940

[mayorovp]

My suspiction is incorrect NextPartNumberMarker in s3 reply, but i still want useful error message.

[milosgajdos]

I haven't checked your logs and was responding from GH mobile app, hence my marking the comment I made as suspicion.

Feel free to open a PR with a fix. We welcome all contributions.