cmd/load: reduce memory usage #4777
Conversation
Signed-off-by: jiefenghuang <jiefeng@juicedata.io>
| @@ -587,6 +596,11 @@ func decodeEntry(dec *json.Decoder, parent Ino, cs *DumpedCounters, parents map[ | |||
| return nil, fmt.Errorf("decode %v: %s", name, err) | |||
| } | |||
| } | |||
| if decodeCounter >= gcThreshold || len(e.CompressedEntries) > gcThreshold { | |||
| decodeCounter = 0 | |||
| runtime.GC() | |||
There was a problem hiding this comment.
it's better to use mem pool for DumpedEntry and compressedAttr
There was a problem hiding this comment.
yes, mem pool can improve performance and reduce GC pressure.
However, testing shows that in this scenario, with 11M files in a single folder, if GC is not triggered in time, memory will still continue to grow, leading to OOM.
There was a problem hiding this comment.
We don't need to over optimize for this single case.
btw, since we don't use JSON to serialize the whole tree directly, we can change type of Entries of DumpedEntry to slice of compressAttr.
There was a problem hiding this comment.
btw, we can optmize this after 1.2
There was a problem hiding this comment.
We don't need to over optimize for this single case.
btw, since we don't use JSON to serialize the whole tree directly, we can change type of Entries of DumpedEntry to slice of compressAttr.
Entries map[string]*DumpedEntry json:"entries,omitempty"``
This variable is still necessary during the dump and load processes.
close #4766
test: