Istio is failing with upstream_reset_after_response_started

[kesavkolla]

Description

I have installed registry:2 in my kubernetes setup. I have created a virtualservice in istio to keep an external hostname to direct traffic to registry service.

When I tried to push images to my registry everything was fine. When I tried to pull images from the registry I am getting error:

Istio version: 1.20.1
registry version: registry github.com/docker/distribution 2.8.3

docker pull registry.k8s.kesavkolla.me/my-app
latest: Pulling from my-app
bc0734b949dc: Retrying in 1 second 
b5de22c0f5cd: Retrying in 1 second 
917ee5330e73: Retrying in 1 second 
b43bd898d5fb: Waiting 
7fad4bffde24: Waiting 
d685eb68699f: Waiting 
107007f161d0: Waiting 
02b85463d724: Waiting 
a6fb4b3b2b32: Waiting 
1955d7b2ed2e: Waiting 
d6b1b9086b58: Waiting 
1a015ef13f1f: Waiting 

When I look into the istio logs:

[2024-01-02T22:19:52.809Z] "GET /v2/jeeves-app/blobs/sha256:1955d7b2ed2e6d0d103e780b279b3e6578067b9107e772c37d3b80b1b15a79a5 HTTP/1.1" 200 UPE upstream_reset_after_response_started{protocol_error} - "-" 0 0 14 14 "10.0.0.224" "docker/24.0.7 go/go1.20.10 git-commit/311b9ff kernel/6.2.0-39-generic os/linux arch/amd64 UpstreamClient(Docker-Client/24.0.7 \(linux\))" "63999b34-27a2-4ee2-bf84-2c76e3cb1628" "registry.k8s.kesavkolla.me" "10.0.1.66:5000" outbound|5000||docker-registry.registry.svc.cluster.local 10.0.2.19:35200 10.0.2.19:8443 10.0.0.224:34454 registry.k8s.kesavkolla.me -

Registry logs are showing like this:

time="2024-01-02T22:19:52.823809885Z" level=info msg="response completed" go.version=go1.20.8 http.request.host=registry.k8s.kesavkolla.me http.request.id=a274fc60-1f5a-421f-8732-a5742413b963 http.request.method=GET http.request.remoteaddr=10.0.0.224 http.request.uri="/v2/jeeves-app/blobs/sha256:1955d7b2ed2e6d0d103e780b279b3e6578067b9107e772c37d3b80b1b15a79a5" http.request.useragent="docker/24.0.7 go/go1.20.10 git-commit/311b9ff kernel/6.2.0-39-generic os/linux arch/amd64 UpstreamClient(Docker-Client/24.0.7 \(linux\))" http.response.contenttype="application/octet-stream" http.response.duration=10.503928ms http.response.status=200 http.response.written=0

Reproduce

setup registry deployment
configure istio virtual service

docker pull image

Expected behavior

The pull images should successfully be retireve.

registry version

registry github.com/docker/distribution 2.8.3

Additional Info

No response