Limiting Memory Usage When Loading Large Collections

[legion15q]

Hello Milvus Community!

I am encountering an issue with high memory consumption when using the collection.load function for large datasets. Despite applying approaches like mmap and using diskANN and HSNW indexes, which indeed reduce memory usage, they do not allow for a strict limitation on maximum memory usage.

Is there a way in Milvus to configure data loading such that the search process can be batched or not entirely in-memory? I am willing to sacrifice search speed to reduce memory usage.

I would appreciate any suggestions or recommendations!

[yhmo]

In my opinion, to limit memory, you should use k8s resources.limit.memory.

[legion15q]

Thank you for the response! But in this case, k8s will kill the pod when it exceeds the limit, and I need to shift the load to the disk, or limit it using batches, or in some other way, but not kill the pod.

[yhmo]

In milvus.yaml, change this value to be smaller:

queryCoord:
  overloadedMemoryThresholdPercentage: 90 # The threshold percentage that memory overload

The default value is 90.
When a query node is loading segments one by one, it checks the current memory usage. If the memory usage exceeds 90%(of the memory capacity of the node), the query node will deny loading segment, and always return an error for search() that says "load segment failed, OOM if load".

Change it to 80 or 70 to avoid potential OOM.

[legion15q]

Thank you, useful advice, but in that case I won't be able to upload the collection and search through it. The question is more about how to work with vector search when there is limited RAM available. What are the ways to load and search a collection when there is not enough RAM?

[yhmo]

Different index type requires different memory size.
Typically, the index size compare to the original vectors size:

• IVF_FLAT/HNSW, a bit larger than the original vectors size
• IVF_SQ8/IVF_PQ, 25% ~ 30% of the original vectors size, the recall rate of IVF_PQ is not good.
• DISKANN, 1/4 ~ 1/6 of the original vectors size, performance is not good, requires high-performance hard disk.

If you have no enough RAM to load IVF_FLAT/HNSW, you can choose IVF_SQ8/IVF_PQ.
If you have no enough RAM to load IVF_SQ8/IVF_PQ, you can choose DISKANN.

If you have no enough RAM to load DISKANN, you can enable MMAP: https://milvus.io/docs/mmap.md#MMap-enabled-Data-Storage
With insufficient memory, MMAP still can work for search, but the search performance is poor.