Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Minio/DataModel/Notification/ObjectMeta.cs class uses int for filesize causing problems when serializing event notifications from big files #977

Open
TheRealRolandDeschain opened this issue Jan 8, 2024 · 0 comments
Assignees
Labels

Comments

@TheRealRolandDeschain
Copy link

In my Minio wrapper class I am using a function to add callbacks for notifications as follows:

    public void AddObjectEventListener(string name, Func<MinioNotification, Task> callback, CancellationToken ct)
    {
        _logger.LogDebug("Listening for Object creation: {name}", name);
        var events = new List<EventType> { EventType.ObjectCreatedAll };

        ListenBucketNotificationsArgs args = new ListenBucketNotificationsArgs()
                                                 .WithBucket(_minioConfig.Bucket)
                                                 .WithEvents(events)
                                                 .WithPrefix(name);
        IObservable<MinioNotificationRaw> observable = _minio.ListenBucketNotificationsAsync(args, ct);
        IDisposable subscription = observable.Subscribe(
            notification =>
            {
                callback(JsonSerializer.Deserialize<MinioNotification>(notification.Json)!);
            },
            ex => _logger.LogError("Minio Error while listening to bucket event: {ex}", ex),
            () => _logger.LogDebug("Stopped listening for bucket notifications"));
    }

The callback function then might looks something like this:

    private async Task CompleteFileUpload(MinioNotification notification)
    {
        if (notification.Records is null)
        {
            logger.LogWarning("S3 notification did not contain any records!");
            return;
        }

        foreach (var record in notification.Records)
        {
            if (string.IsNullOrWhiteSpace(record?.S3?.ObjectMeta?.Key))
            {
                logger.LogWarning("Unable to exctract object data from notification record!");
                return;
            }

            string secureObjectName = record.S3.ObjectMeta.Key;
            (CancellationTokenSource cts, int fileID) = _fileUploadEventInfos[secureObjectName];
            var uploadedTime = DateTime.UtcNow;
            await fileAccess.UpdateUploadStatus(fileID, UploadState.Ready, uploadedTime);
            cts.Cancel();
        }
    }

Long story short, due to the int type for the object size in Minio/DataModel/Notification/ObjectMeta.cs serialization fails for bigger files. For the moment I fixed it on my side by using the raw json string directly and parse the needed information manually, but I though I'd report here, since it should be relatively easy to change this to long.

@ebozduman ebozduman self-assigned this Jan 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants