Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

improve performance for sequence read and warmup on big file #4769

Open
zhoucheng361 opened this issue Apr 24, 2024 · 0 comments
Open

improve performance for sequence read and warmup on big file #4769

zhoucheng361 opened this issue Apr 24, 2024 · 0 comments
Labels

Comments

@zhoucheng361
Copy link
Contributor

zhoucheng361 commented Apr 24, 2024

What would you like to be added:
bandwidth of object storage is 400M, the bandwidth of read is only 140M.
we have set buffer-size to 4G, but the process only use 40M buffer when reading.

root@bench-01:~/juicefs# /root/juicefs/juicefs mount tikv://172.27.0.1:2379/test1 /jfs -d --buffer-size 4096
root@bench-01:~/juicefs# md5sum /jfs/bigfile
root@bench-01:~/juicefs# ./juicefs stats /jfs

---------usage--------- ----------fuse--------- -------------meta------------ -blockcache ---------------------object--------------------
 cpu   mem   buf  cache| ops   lat   read write| ops   lat   txn   lat  retry| read write| get  get_c  lat   put  put_c  lat  del_c  lat
35.5%  271M   36M    0 |1142  2.63   141M    0 |   5  0.71     0     0     0 |   0   144M| 144M   36   158     0     0     0     0     0
29.2%  271M   36M    0 | 990  3.14   121M    0 |   6  0.80     0     0     0 |   0   112M| 112M   28   166     0     0     0     0     0
40.8%    0    32M 8192K|1294  1.96   160M    0 |   5  0.77     0     0     0 |   0   156M| 164M   41   145     0     0     0     0     0
25.4%  271M   36M    0 | 766  4.85    94M    0 |   3  0.64     0     0     0 |   0   112M| 104M   26   171     0     0     0     0     0
37.0%  271M   36M    0 |1284  2.23   158M    0 |   5  0.75     0     0     0 |   0   152M| 152M   38   152     0     0     0     0     0
33.3%  265M   32M    0 | 922  3.47   113M    0 |   4  0.77     2  2.08     0 |   0   128M| 128M   32   187     0     0     0     0     0
29.5%  267M   40M    0 |1116  2.68   137M    0 |   6  0.68     0     0     0 |   0   112M| 112M   28   184     0     0     0     0     0
39.7%  267M   36M 4096K|1206  2.47   150M    0 |   4  0.76     0     0     0 |   0   160M| 164M   41   155     0     0     0     0     0


root@bench-01:~/juicefs# ./juicefs warmup /tmp/jfs/bigfile
warmup cache file: 0                             0.0/s                          ⠹
warmup cache size: 192.0 MiB (201326592 Bytes)   21.9 MiB/s                     ⠹

root@bench-01:~# dstat
You did not select any stats, using -cdngy by default.
--total-cpu-usage-- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai stl| read  writ| recv  send|  in   out | int   csw
  2   1  97   0   0| 325k 2303k|   0     0 |   0     0 |2457    36k
  2   1  96   2   0|4096B   82M|  25M  150k|   0     0 |  14k   36k
  2   1  97   0   0|   0    16k|  17M  120k|   0     0 |  12k   34k
  1   1  97   0   0|   0   264k|  13M   81k|   0     0 |  11k   32k
  2   1  97   0   0|   0  8192B|  17M  100k|   0     0 |  11k   33k
   ⠋

Why is this needed:

@zhoucheng361 zhoucheng361 added kind/feature New feature or request kind/enhancement and removed kind/feature New feature or request labels Apr 24, 2024
@zhoucheng361 zhoucheng361 changed the title improve performance for sequence read on big file improve performance for sequence read and warmup on big file Apr 26, 2024
@SandyXSD SandyXSD added the priority/normal The default priority label May 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants