Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

juicefs load cost too much memory and OOM. #4766

Open
zhoucheng361 opened this issue Apr 24, 2024 · 0 comments · May be fixed by #4777
Open

juicefs load cost too much memory and OOM. #4766

zhoucheng361 opened this issue Apr 24, 2024 · 0 comments · May be fixed by #4777
Labels
kind/enhancement priority/normal The default priority

Comments

@zhoucheng361
Copy link
Contributor

zhoucheng361 commented Apr 24, 2024 •
edited

What would you like to be added:

root@bench-01:~/juicefs# ps -ef | grep mount
root      632878       1  0 09:55 ?        00:00:02 ./juicefs mount tikv://172.27.0.1:2379/test1 /jfs -d
root      638901  632878  0 13:32 ?        00:00:01 /root/juicefs/juicefs mount tikv://172.27.0.1:2379/test1 /jfs -d
root@bench-01:~/juicefs# df -ih /jfs
Filesystem         Inodes IUsed IFree IUse% Mounted on
JuiceFS:tikv-test1    40M   30M   10M   75% /jfs
root@bench-01:~/juicefs# ./juicefs summary /jfs/
/jfs/: 32000006                        51508.4/s
/jfs/: 2.1 TiB (2330095271936 Bytes)   3.5 GiB/s
+----------------------+---------+----------+----------+
|         PATH         |   SIZE  |   DIRS   |   FILES  |
+----------------------+---------+----------+----------+
| /                    | 2.1 TiB | 15000003 | 17000003 |
| bigfile-clone        | 1.0 TiB |        0 |        1 |
| bigfile              | 1.0 TiB |        0 |        1 |
| big_dir/             |  42 GiB |  5000001 |  6000000 |
| big_dir/file_828878  | 4.0 KiB |        0 |        1 |
| big_dir/dir_3659923/ | 4.0 KiB |        1 |        0 |
| big_dir/file_1338817 | 4.0 KiB |        0 |        1 |
| big_dir/file_2391576 | 4.0 KiB |        0 |        1 |
| big_dir/file_2615668 | 4.0 KiB |        0 |        1 |
| big_dir/file_1144848 | 4.0 KiB |        0 |        1 |
| big_dir/file_1608507 | 4.0 KiB |        0 |        1 |
| big_dir/file_3993541 | 4.0 KiB |        0 |        1 |
| big_dir/file_54972   | 4.0 KiB |        0 |        1 |
| big_dir/dir_2413165/ | 4.0 KiB |        1 |        0 |
| big_dir/...          |  42 GiB |  4999998 |  5999992 |
| bigdir2/             |  38 GiB |  5000001 |  5000000 |
| bigdir2/dir_4/       | 7.7 GiB |  1010101 |  1010100 |
| bigdir2/dir_1/       | 7.7 GiB |  1010101 |  1010100 |
| bigdir2/dir_3/       | 7.7 GiB |  1010101 |  1010100 |
| bigdir2/dir_2/       | 7.7 GiB |  1010101 |  1010100 |
| bigdir2/dir_13/      |  79 MiB |    10101 |    10100 |
| bigdir2/dir_74/      |  79 MiB |    10101 |    10100 |
| bigdir2/dir_94/      |  79 MiB |    10101 |    10100 |
| bigdir2/dir_45/      |  79 MiB |    10101 |    10100 |
| bigdir2/dir_22/      |  79 MiB |    10101 |    10100 |
| bigdir2/dir_14/      |  79 MiB |    10101 |    10100 |
| bigdir2/...          | 6.9 GiB |   898990 |   899000 |
| file_2277285         | 4.0 KiB |        0 |        1 |
| file_667499          | 4.0 KiB |        0 |        1 |
| hardlink_206773      | 4.0 KiB |        0 |        1 |
| dir_4948909/         | 4.0 KiB |        1 |        0 |
| dir_2150462/         | 4.0 KiB |        1 |        0 |
| dir_2903805/         | 4.0 KiB |        1 |        0 |
| ...                  |  42 GiB |  4999997 |  5999998 |
+----------------------+---------+----------+----------+
root@bench-01:~/juicefs# ./juicefs dump tikv://172.27.0.1:2379/test1 dump.json.gz
root@bench-01:~/juicefs# ./juicefs load badger://test.badger dump.json.gz
2024/04/24 13:28:57.863957 juicefs[638835] <INFO>: Meta address: badger://test.badger [interface.go:504]
2024/04/24 13:28:57.870275 badger[638835] <INFO>: All 0 tables opened in 0s [logger.go:46]
2024/04/24 13:28:57.871206 badger[638835] <INFO>: Discard stats nextEmptySlot: 0 [logger.go:46]
2024/04/24 13:28:57.871257 badger[638835] <INFO>: Set nextTxnTs to 0 [logger.go:46]
2024/04/24 13:28:57.871567 juicefs[638835] <INFO>: Loading from file ... [dump.go:377]
2024/04/24 13:29:24.758210 badger[638835] <INFO>: [0] [E] LOG Compact 0->5 (5, 0 -> 40 tables with 1 splits). [00046 00047 00048 00049 00050 . .] -> [00051 00052 00053 00054 00055 00056 00057 00058 00059 00060 00061 00062 00063 00064 00065 00066 00067 00068 00069 00070 00071 00072 00073 00074 00075 00076 00077 00078 00079 00080 00081 00082 00083 00084 00085 00086 00087 00088 00089 00090 .], took 2.096s
, deleted -4061211 bytes [logger.go:46]
2024/04/24 13:30:17.343764 badger[638835] <INFO>: [0] [E] LOG Compact 5->6 (1, 2 -> 2 tables with 1 splits). [00355 . 00421 00189 .] -> [00447 00452 .], took 2.121s
, deleted -225422 bytes [logger.go:46]
2024/04/24 13:30:28.876531 badger[638835] <INFO>: [1] [E] LOG Compact 0->5 (5, 2 -> 22 tables with 1 splits). [00439 00457 00458 00459 00460 . 00373 00374 .] -> [00461 00462 00463 00464 00465 00466 00467 00468 00469 00470 00471 00472 00473 00474 00475 00476 00477 00478 00479 00480 00481 00482 .], took 2.198s
, deleted 62362 bytes [logger.go:46]
Loaded entries: 10500001/30999843 [=============>----------------------------]  107361.8/s ETA: 3m10s
Killed

root@bench-01:~/juicefs# dmesg
[9237353.615547] [ 603759]  3532 603759  2844772  2135945 20062208        0             0 tikv-server
[9237353.615548] [ 632878]     0 632878   439472     3678   425984        0         -1000 juicefs
[9237353.615549] [ 632890]     0 632890  1635554    45899  9248768        0         -1000 juicefs
[9237353.615550] [ 633483]   112 633483   613846    96037  1232896        0             0 mysqld
[9237353.615551] [ 637913]     0 637913     3453      364    69632        0             0 sshd
[9237353.615552] [ 637940]     0 637940     2756      444    57344        0             0 bash
[9237353.615554] [ 638577]     0 638577     3453      364    69632        0             0 sshd
[9237353.615554] [ 638607]     0 638607     2723      411    57344        0             0 bash
[9237353.615555] [ 638778]     0 638778     3451      362    65536        0             0 sshd
[9237353.615556] [ 638804]     0 638804     2723      397    53248        0             0 bash
[9237353.615557] [ 638815]     0 638815     2962      183    65536        0             0 top
[9237353.615558] [ 638835]     0 638835  5694490  4351015 37003264        0             0 juicefs
[9237353.615559] [ 638878]   117 638878    14479       59    90112        0             0 fdbserver
[9237353.615560] [ 638888]   114 638888    54705     1029   159744        0             0 postgres
[9237353.615561] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-0.slice/session-28768.scope,task=juicefs,pid=638835,uid=0
[9237353.615601] Out of memory: Killed process 638835 (juicefs) total-vm:22777960kB, anon-rss:17404060kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:36136kB oom_score_adj:0

Why is this needed:
@zhoucheng361 zhoucheng361 added kind/feature New feature or request kind/enhancement and removed kind/feature New feature or request labels Apr 24, 2024
@jiefenghuang jiefenghuang linked a pull request Apr 25, 2024 that will close this issue
Draft
@SandyXSD SandyXSD added the priority/normal The default priority label May 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/enhancement priority/normal The default priority
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

cmd/load: reduce memory usage jiefenghuang/juicefs
2 participants
@SandyXSD @zhoucheng361