During image push and pull, error building image: stream error: stream ID 13; NO_ERROR; received from peer

[rashmigspai]

Actual behavior
As part of image build, using Kaniko and while during image build getting the error as below

INFO[2024-04-17T14:14:12Z] Building stage 'private.icr.io/namespace/tenant/master/liberty:23.0.0.12-506' [idx: '0', base-idx: '-1']
INFO[2024-04-17T14:14:12Z] Unpacking rootfs as cmd RUN mkdir -p $RTI  && chown -R 1001:0 $RTI && chown -R 1001:1001 /opt/ear && chmod -Rf 777 /opt/ear requires it.
WARN[2024-04-17T14:19:19Z] Retrying operation after 1s due to stream error: stream ID 17; NO_ERROR; received from peer
WARN[2024-04-17T14:25:21Z] Retrying operation after 2s due to stream error: stream ID 27; NO_ERROR; received from peer
WARN[2024-04-17T14:31:37Z] Retrying operation after 4s due to stream error: stream ID 41; NO_ERROR; received from peer

Can you please suggest why we getting this error ? Is it because of network issue ? Tried to use --use-new-run and --snapshot-mode=redo but none of them helping. Any suggestion on how to overcome or what we can do is highly appreciated as this is totally blocking us with failures.

And randomly seeing failure with push also

INFO[2024-04-17T08:23:32Z] Pushing image to private.icr.io/namespace/tenant/prod/apiserver:<tag>
WARN[2024-04-17T08:38:45Z] Retrying operation after 1s due to Patch "https://private.icr.io/v2/namespace/tenant/prod/apiserver/blobs/uploads/01baac8d-2e58-4568-8c1b-14d44037e403?_state=Io6SjctL0Ps8RAUbJ95G92W7yHowbtgqUxyLMz7_rcF7Ik5hbWUiOiJvcmRlci1tZ210LXByL2pscHVrL3FhLTEvYXBpc2VydmVyIiwiVVVJRCI6IjAxYmFhYzhkLTJlNTgtNDU2OC04YzFiLTE0ZDQ0MDM3ZTQwMyIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyNC0wNC0xN1QwODoyMzozMy43NjE5NjcxMTlaIn0%3D": stream error: stream ID 37; NO_ERROR; received from peer

We are unable to understand what does stream error: stream ID 37; NO_ERROR; received from peer mean in Kaniko.

This is totally blocking us and hence any immediate attention to answer the questions will be highly appreciated

Expected behavior
Expecting more clear and detail logging around why this occurred - is it related to failure to download/upload the image?

I have verified that this works with smaller images to rule out any other registry issues (proxy/auth/etc.)

To Reproduce
Steps to reproduce the behavior:

N/A at this point

Additional Information

• Dockerfile
  Please provide either the Dockerfile you're trying to build or one that can reproduce this error.

USER root
RUN mkdir -p $RTI  && chown -R 1001:0 $RTI && chown -R 1001:1001 /opt/ear && chmod -Rf 777 /opt/ear
USER 1001
COPY server.xml /opt/wlp/usr/servers/defaultServer/server.xml
COPY deploy.ear /opt/ear/deploy.ear

• Build Context
  Please provide or clearly describe any files needed to build the Dockerfile (ADD/COPY commands)
• Kaniko Image (fully qualified with digest)

gcr.io/kaniko-project/executor:v1.9.1

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko	- [Yes]
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile	- [ No]

[aaron-prindle]

Dupe of #3116

[rashmigspai]

@aaron-prindle would it b possible to explain the meaning of this error ? What all can cause it to issue this error.