You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Originally posted by all4one-max April 21, 2024
I'm in the process of migrating our Python package, used by various applications within our organization, from boto3 to aiobotocore. In our existing implementation with boto3, we ensure singleton initialization of the S3 client to optimize resource usage and avoid redundant client creation.
However, transitioning to aiobotocore poses a challenge as it requires using context managers for client creation, and the client doesn't exist outside the context. I aim to encapsulate this singleton logic within our package itself, rather than relying on configuration in the application lifecycle.
I've explored several sources, including the aiobotocore documentation and relevant discussions on GitHub and Stack Overflow, but haven't found a satisfactory solution yet.
I tried implementing using the below code but it did not work I was getting Runtime error, Event loop is closed. I am executing operations such as download, upload in my celery task.
classAsyncS3ClientSingleton(metaclass=AWSMeta):
_async_s3_client=None@classmethodasyncdefget_s3_client(cls) ->Any:
ifnotcls._async_s3_client:
# Lazy initialization of the S3 clientsession=aioboto3.Session()
ctx=session.client(
"s3",
aws_access_key_id=AWS_ACCESS_KEY,
aws_secret_access_key=AWS_ACCESS_KEY_SECRET,
region_name="us-east-1",
)
cls._async_s3_client=awaitctx.__aenter__()
returncls._async_s3_client
But when I create the session and client for every operation as shown below it works fine, though it defeats the purpose of reusing the client.
Please help me fix this. I feel it has something to do with session being tied to an event loop and everytime a new celery task is initiated, new event loop is created. Only speculation though
Discussed in #1110
Originally posted by all4one-max April 21, 2024
I'm in the process of migrating our Python package, used by various applications within our organization, from boto3 to aiobotocore. In our existing implementation with boto3, we ensure singleton initialization of the S3 client to optimize resource usage and avoid redundant client creation.
Existing Implementation with boto3:
However, transitioning to aiobotocore poses a challenge as it requires using context managers for client creation, and the client doesn't exist outside the context. I aim to encapsulate this singleton logic within our package itself, rather than relying on configuration in the application lifecycle.
I've explored several sources, including the aiobotocore documentation and relevant discussions on GitHub and Stack Overflow, but haven't found a satisfactory solution yet.
Sources referred:
The text was updated successfully, but these errors were encountered: