[SOS]querynode CrashLoopBackOff

[yongxiaodong]

Deployment method:

Use helm to deploy to k8s

Hardware configuration

3 32-core 32G servers

Collection data:

2800W, one partition, using IVF_FLAT index

Question

The query node restarts forever. It seems that too much memory is used and OOM occurs on the node. How to solve this problem? Does the query node have to load all data from storage when it starts? Can it be loaded in batches?

Does the node memory have to be larger than the storage capacity?

[xiaofan-luan]

2800

yet, querynode has to load all data in memory if use ivf_flat index.
28m takes a little bit smaller than 200GB memory.

Fix:

1. drop the index and change to ivf_sq index or diskann index(if you have high performance disk)
2. change the index load mode to mmap, see https://milvus.io/docs/mmap.md
3. add more nodes.

Or, try zilliz cloud capacity instnace or contact us for more help

[xiaofan-luan]

That's why query coordinator exist. Query coordinator balance the data to make sure all segment is distributed evenly on all querynodes.

[yongxiaodong]

I don't quite understand the loading principle of queryNode. OOM may occur when a single queryNode tries to load a huge collection. Can horizontally expanding low-configuration queryNode solve it?

[xiaofan-luan]

for sure you can do that.
we encourage querynode to have 8G memory or more. 32G querynode load similar amount data as 2 * 16G, but the overhead might be less.
The best practice is to use 32G or 64G querynode with large data set

[yongxiaodong]

Thanks for your previous reply，after your last suggestion, I have currently written 400 million vector data. The two queryNodes use about 350G of memory, but one of them uses 230G. The load is unbalanced. Do you have any suggestions?

[yongxiaodong]

The good news is that they all run fine, it's just that the memory load is unbalanced