[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1006)

[pavel-vhive]

Describe the bug

While downloading files ~16MB each, sometimes it fails with error:
[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1006)

Expected Behavior

download files without exceptions

Current Behavior

[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1006)

Reproduction Steps

import boto3
import time
import os
from boto3.s3.transfer import TransferConfig
from concurrent.futures import ThreadPoolExecutor

NUM_THREADS = os.cpu_count() or 1

if __name__ == "__main__":
    client = boto3.client('s3')
    config = TransferConfig(
        multipart_chunksize=1 * boto3.s3.transfer.MB,
        multipart_threshold=5 * boto3.s3.transfer.MB,
        max_concurrency=20
    )
    keys = [<list of files to download>]

    def download_file(key: str):
        file_name: str = os.path.basename(key)
        client.download_file(Bucket='<your bucket>',
                         Filename=file_name,
                         Key=key)

    with ThreadPoolExecutor(max_workers=NUM_THREADS) as executor:
        for key in keys:
            executor.submit(download_file, key)

Possible Solution

No response

Additional Information/Context

No response

SDK version used

1.28.74

Environment details (OS name and version, etc.)

ubuntu 22.04, python 3.11, botocore 1.31.74

[tim-finnigan]

Hi @pavel-vhive thanks for reaching out. This appears to be an SSL issue rather than directly a boto3/botocore issue. Which SSL version do you have installed? As suggested here this error may be due to OpenSSL 3 dependency issues.

But you mentioned that the issue only occurs sometimes. Could you provide more details on the conditions that cause this to occur and how often it occurs? Are there any network configurations that could interfere with your requests? (For example, firewalls, proxies...).

[pavel-vhive]

Hi @tim-finnigan thanks for your response. My SSL version is OpenSSL 3.0.2. As for networking I don't have any proxy or firewalls. I use docker image of ubuntu:22.04. It seems some files downloaded successfully but some failed with that error. More over, smaller files around 8MB seems have no issues, files around 16MB increase the issue rate. Could it be related to multipart configuration I used in my example? Or should I update my SSL dependancies as suggested in the link you added, but I am nor sure which dependencies must be updated.

[tim-finnigan]

Hi @pavel-vhive thanks for following up. You may want to update your OpenSSL version if possible since 3.0.2 is a bit behind and may be missing some important updates (here are the changelogs for reference).

We would expect to see more reports of this issue if there there is some kind of incompatibility. There may be a very specific set of circumstances where this could occur, so we'd need to narrow this down further. I found an issue that may be related in the urllib3 repository, but we don't have enough information to say for sure.

Can you share your debug logs (with any sensitive info redacted) by adding boto3.set_stream_logger('') to your script? That may help provide some more context. Also are you able to test in any different environments, and rule out if this issue occurs only with specific versions of Python/OpenSSL/urllib3?

[github-actions]

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.

[agentpietrucha]

@pavel-vhive have you solved your problem? If so, could you please share your solution? I am facing very similar one

[pavel-vhive]

@agentpietrucha the workaround was to freeze boto3 to v.1.28.50 and botocore to v1.31.50. Because boto3 uses botocore and botocore uses urllib3 which from v2.x.x has SSL issues. So basically urllib3 new version was the root cause.