Calculate the static data index outside the cluster, then upload it to minio, and register the metadata to etcd; in this way, you can speed up the import of data.

[ai-smalleryu]

Need ideas: Milvus index offline construction tool supports Spark-based offline index construction and loading
Requirements sorting: If you want to use computing resources outside the milvus cluster, first calculate the index data for the original data according to the algorithm that conforms to milvus hnsw or scan, then upload it together with the data to the minio storage, and register the metadata to etcd; to reduce import and index creation time for larger data sets

[xiaofan-luan]

milvus index building is much faster than outside service.
Our recommendation is to use spark to prepare data, bulk insert into milvus and build index with milvus index node

[TonyAnn]

hi xiaofan @xiaofan-luan
Currently, our data volume is large, and 90% of it is static data, which will not increase in the future;

Currently, bulk insert is used to load, but index construction takes a long time, so we want to calculate the index offline and then upload the index data to minio to simplify data import.

[TonyAnn]

In addition, our computing resource pool is very sufficient. If we can calculate the index offline, it will be of great help to us.

[xiaofan-luan]

increase the indexnode number will work. and you can also share indexnode resources with offline resource pool.

we don't accept outside index build tool because there is no ganrautee that index format will be compatible forever. any outside tool can not guarantee it can sync with milvue.

you can write you own tool to use knowhere and build index. However this is not recommended from our perspective and it is too heavy to maintain tens of different types.

[TonyAnn]

The computing resource pool is not in the same location as the milvus cluster on the production line, so resources cannot be added directly to the milvus cluster,

therefore we want to calculate the index offline based on knowhere. We currently only use hnsw index, so the amount of development will not be too large.

[xiaofan-luan]

this is actually doable.

1. bulknsert the collecton, don't create partition
2. disable compaction
3. use spark connector read all milvus file, then build index with knowhere.
4. hack the index meta(Not support for now), point it the file you already put on S3