upload to minio suddenly goes very slow

[zhonghuafy]

I have a minio cluster server with 2 nodes on linux (minio version Release.2023-05-04T21-44-30Z). Minio runs with default config. The minio is used by a java (spring boot) application which manages file upload and download.

The application runs good and fast both upload and download, but very rarely uploading files suddenly goes very slow. Normally upload a file would be finised in a second, but when this problem happens it needs more than 1 minite (file size less than 100KB). And the problem will auto healed after about half a hour.

The application and minio seems good when problem happens. I tried restart the minio and java application but not worked. The network, cpu and memory is quite normal, I tried to upload files using minio console and it works fine. I just don't understand.

The java application is as below:

pom file:

<dependency>
    <groupId>io.minio</groupId>
        <artifactId>minio</artifactId>
     <version>8.5.2</version>
</dependency>

java code:

@Configuration
public class MinIoClientConfig {

    @Value("${minio.url}")
    private String url;

    @Value("${minio.username}")
    private String userName;

    @Value("${minio.password}")
    private String password;

    @Bean
    public MinioClient minioClient(){
        return MinioClient.builder().endpoint(url).credentials(userName, password).build();
    }
}

@Service
@Slf4j
public class MinIoServiceImpl implements IMinIoService {

    @Autowired
    private MinioClient minioClient;

    public FileBo upload(UploadFileTypeEnum uploadFileTypeEnum, UploadFileParam uploadFileParam) {
        //...
        try(InputStream inputStream = uploadFileParam.getFile().getInputStream()) {
            minioClient.putObject(PutObjectArgs.builder()
                    .bucket(uploadFileTypeEnum.getBucket().getName())
                    .object(savedPathName).userMetadata(generateMeta(uploadFileTypeEnum, uploadFileParam))
                    .stream(inputStream, uploadFileParam.getFile().getSize(), -1)
                    .build());
            return fileBo;
        }
    }
}

[jiuker]

@zhonghuafy Have you tried the mc cp command when this happens? Does it have this problem too?

[zhonghuafy]

No, I did't. This problem occors 3 times after minio producted 2 months ago. I tried minio console and worked fine. Does minio have something like connection pool need to configed? I traced the log and seems it stucked at minioClient.putObject().

[jiuker]

@zhonghuafy Post the logs plz.

[zhonghuafy]

minio doesn't have logs, only my java application logs.
When problem happens, minioClient.putObject() needs more than 1 minite:

2023-06-19 10:56:13:322 ┆ 13-exec-16 ┆ FILE ┆ INFO  ┆ c.p.base.file.service.minio.MinIoServiceImpl       :71    ┆ minIo上传文件 original name blob.jpeg saved name online-insurance/2023/202306/evidenceImg/c3c7bb9e-1d24-4e2c-933b-e6af1337b32d.jpeg content type image/jpeg
2023-06-19 10:57:29:140 ┆ 13-exec-16 ┆ FILE ┆ INFO  ┆ c.p.base.file.service.minio.MinIoServiceImpl       :78    ┆ minIo上传文件成功 {"fileExt":"jpeg","fileName":"blob.jpeg","fileSize":596831,"fileSrc":"online-insurance/2023/202306/evidenceImg/c3c7bb9e-1d24-4e2c-933b-e6af1337b32d.jpeg","fileSrcType":2,"fileUuid":"c3c7bb9e-1d24-4e2c-933b-e6af1337b32d","id":2503672,"serviceName":"B","uploadIp":"61.150.78.161","uploadType":"evidenceImg"}
2023-06-19 10:57:29:140 ┆ 13-exec-16 ┆ FILE ┆ INFO  ┆ c.p.base.file.service.minio.MinIoServiceImpl       :169   ┆ minIo 获取文件访问链接 blob.jpeg https://xxx.xxx/online-insurance/online-insurance/2023/202306/evidenceImg/c3c7bb9e-1d24-4e2c-933b-e6af1337b32d.jpeg

As comparision, underbelow is normal logs, minioClient.putObject() just used 60 miliseconds:

2023-06-20 10:08:55:320 ┆ 13-exec-42 ┆ FILE ┆ INFO  ┆ c.p.base.file.service.minio.MinIoServiceImpl       :71    ┆ minIo上传文件 original name AJINJ50Y2023B113110I_LS050320.pdf saved name offline-insurance/2023/202306/elePolicy/884e702d-4d12-442f-8b27-eed35406beda.pdf content type application/pdf
2023-06-20 10:08:55:382 ┆ 13-exec-42 ┆ FILE ┆ INFO  ┆ c.p.base.file.service.minio.MinIoServiceImpl       :78    ┆ minIo上传文件成功 {"fileExt":"pdf","fileName":"AJINJ50Y2023B113110I_LS050320.pdf","fileSize":409583,"fileSrc":"offline-insurance/2023/202306/elePolicy/884e702d-4d12-442f-8b27-eed35406beda.pdf","fileSrcType":2,"fileUuid":"884e702d-4d12-442f-8b27-eed35406beda","id":2520383,"serviceName":"B","uploadIp":"27.211.163.7","uploadType":"elePolicy"}
2023-06-20 10:08:55:382 ┆ 13-exec-42 ┆ FILE ┆ INFO  ┆ c.p.base.file.service.minio.MinIoServiceImpl       :169   ┆ minIo 获取文件访问链接 AJINJ50Y2023B113110I_LS050320.pdf https://xxx.xxx/offline-insurance/offline-insurance/2023/202306/elePolicy/884e702d-4d12-442f-8b27-eed35406beda.pdf

[harshavardhana]

@zhonghuafy please upgrade your setup always before reporting issues. Most issues get fixed fairly quickly so you never know if this was already fixed.

[harshavardhana]

You are running MinIO version that is 1 month older. Please upgrade to latest release and let us know.

Also provide details of your setup i.e how many disks, how many nodes. And also provide a client program to reproduce the problem.

[zhonghuafy]

That's not cool to update prod env. Currently we won't plan to do that.
minio runs at linux platform with 2 servers 2 drives.
Os info: centos 7.6.1810, cup 8 cores, 32G memory
node1: disk 900GB capacity
node2: disk 2.5TB capacity
minio starts with default config:
nohup /data/minio/minio server --address :9001 --console :9002 http://ip1/data/minio/data http://ip2/data/minio/data >/data/minio/minio.log 2>&1 &

[harshavardhana]

This is a terrible setup

• Wrong OS CentOS is deprecated move to Rocky Linux 8.x minimum
• Wrong disk capacity heterogenous deployment
• Only 2 node deployment with no ellipses used also wrong

You have basically not followed any of our documentation. This setup won't work - no wonder you have issues.

[zhonghuafy]

Does minio have any configs like max connections for one ip or else? I didn't find docs intruducing

[klauspost]

It does not.

[zhonghuafy]

I think I had solved this.
I create a thread pool with lots threads, and use those threads to upload files. Every thread declares a MinioClient object, and problem solved.
So, seems that a MinioClient have its max capicity.

[maolaoke]

its a single minioClient before you solved?

[Vitorio88]

Good day everyone. I have the following MINIO settings
2 balancers with 4 nodes each
A replica site is configured
Such a question, with the help of a script we load from the Oracle database into one of the buckets of the required data, for example, on 1 cluster on which a replica was created, there is less data and the volume shows less on the web, and on the second cluster the entire volume.
When executing the command mcli du kiev/reports
everything is true both on 1 cluster and on the second
Minio version 2024-02-17T01:15:57Z
The settings are all identical
1 balancer
4 nodes with 4 disks of 50 GB each
When executing the command on one of the balancers of the mcli du kiev/reports command, the calculation is very long, a long delay of more than an hour, on another cluster after 20 minutes, the data volume is 7.1 Gigabytes and the data amount is 746000
Why can there be a delay, I receive information about the cluster perfectly
Thanks for any information, help