What is milvus format file

[Mutezebra]

I saw a project requirement in the OSPP that calls for supporting CSV, Milvus, and HDF5 formats for Milvus bulk load.

I have already completed the implementation demo for the CSV format, although it still lacks thorough testing.

Now, I am starting to work on the bulk load for Milvus format files. I tried to search online to understand what the Milvus format is, but I couldn't find a suitable answer. I revisited the Milvus documentation and came across MilvusDM. After reviewing the MilvusDM documentation, it seems that it is used for importing data directly from one Milvus instance to another, rather than importing via a Milvus format file.

Subsequently, I attempted to find methods for exporting Milvus data to see if I could export a file and analyze its structure. However, I didn't find any related functionality in the official documentation.

Therefore, my question remains: What exactly is the Milvus format and what its file look like?

[yhmo]

The description of MilvusDM mentioned it is only for Milvus v1.x: https://milvus.io/docs/migrate_overview.md
Currently, MilvusDM is only supported in Milvus 1.x version.

So, it doesn't work for Milvus v2.x.
The file format of Milvus v2.x is based on Parquet, but not a pure parquet format. Each file is wrapped by an extra header and we call it binlog files.
Outside tools shouldn't directly touch the file format of Milvus.

To export Milvus data, use iterator to fetch data row by row: https://milvus.io/docs/with-iterators.md#Query-with-an-iterator

To import files into Milvus, use bulkinsert: https://milvus.io/docs/v2.3.x/bulk_insert.md. This interface accepts JSON/Parquet/Numpy files in a certain format to import data. The pymilvus provides a smart tool based on bulkinsert interface, helps user to create correct files by append_row() method: https://milvus.io/api-reference/pymilvus/v2.4.x/DataImport/BulkFileType.md

[Mutezebra]

First, I would like to express my gratitude for the information and assistance you've provided earlier. Your explanations have given me a deeper understanding of Milvus's operations, especially regarding iterators and data exportation.

I would like to clarify my actual needs further, as I may not have communicated them clearly before. Currently, I am working on extending the file format support for Milvus's bulk insert interface. I have already managed to implement basic support for CSV format bulk inserts, but now I find myself puzzled by the Milvus-specific file format.

As you mentioned, Milvus v2.x uses a file format based on Parquet with an additional header, known as the binlog file. I am seeking a way to comprehend and handle this special format to effectively add support for it in the bulk insert interface.

However, you indicated that external tools cannot directly handle Milvus's file format, which has left me somewhat perplexed. How should users go about obtaining Milvus format files from Milvus if they wish to import them? This is particularly relevant because the requirements from ospp explicitly state: "为 Milvus Bulk Load 添加以下文件格式支持：i）CSV（comma separated values），ii）Milvus 格式，iii）HDF5（可选）"

[yhmo]

Workflow of bulkinsert:

Users create JSON/Numpy/Parquet files, upload the files to MinIO/S3, then call bulkinsert interface to pass the file paths to Milvus. Milvus data node will get the files by the paths and parse the data from the files. The data is stored as binlog files by data node. Users no need to care what the binlog is.

If you want to extend the ability of "import data", you can convert the CSV/HDF5 to JSON/Numpy/Parquet files then call bulkinsert interface to import.

If let me do this work, I would like to use the bulkwriter tool to help my work:

import csv
from pymilvus import (
    connections,
    CollectionSchema,
    Collection,
    RemoteBulkWriter, BulkFileType,
    utility,
)

connections.connect(host='localhost', port='19530')

collection = Coillection(collection_name)

writer = RemoteBulkWriter(
    schema=collection.schema,
    remote_path="files_to_import_path",
    connect_param=RemoteBulkWriter.ConnectParam()
    segment_size=512*1024*1024,
    file_type=BulkFileType.JSON
)

with open(csv_file_path, 'r', encoding='utf-8') as csvfile:
    reader = csv.reader(csvfile)
    for row in reader:
        row_dict = convert_to_dict(row)  # append_row() requires dict object, convert one line to a dict like {"id": 122, "vector": [xxxx]}
        writer.append_row(row_dict)    # append_row() requires dict object

writer.commit()

file_paths = writer.batch_files

task_ids = []
for files in file_paths:
    task_id = utility.do_bulk_insert(collection_name=collection_name, files=files)
    task_ids.append(task_id)

while len(task_ids) > 0:
    print("Wait 1 second to check bulkinsert tasks state...")
    time.sleep(1)
    for id in task_ids:
        state = utility.get_bulk_insert_state(task_id=id)
        if state.state == BulkInsertState.ImportFailed or state.state == BulkInsertState.ImportFailedAndCleaned:
            print(f"The task {state.task_id} failed, reason: {state.failed_reason}")
            task_ids.remove(id)
        elif state.state == BulkInsertState.ImportCompleted:
            print(f"The task {state.task_id} completed")
            task_ids.remove(id)

Not a difficult work, right?

[Mutezebra]

Thank you for your detailed response. However, what I need to do is to support direct import of CSV files into Milvus, rather than converting CSV files into other formats. I will provide support for the CSV format for Milvus's bulk insert feature.

[yhmo]

I am afraid CSV format is not in the design scope of bulkinsert, as @xiaofan-luan mentioned.

[xiaofan-luan]

First, I would like to express my gratitude for the information and assistance you've provided earlier. Your explanations have given me a deeper understanding of Milvus's operations, especially regarding iterators and data exportation.

I would like to clarify my actual needs further, as I may not have communicated them clearly before. Currently, I am working on extending the file format support for Milvus's bulk insert interface. I have already managed to implement basic support for CSV format bulk inserts, but now I find myself puzzled by the Milvus-specific file format.

As you mentioned, Milvus v2.x uses a file format based on Parquet with an additional header, known as the binlog file. I am seeking a way to comprehend and handle this special format to effectively add support for it in the bulk insert interface.

However, you indicated that external tools cannot directly handle Milvus's file format, which has left me somewhat perplexed. How should users go about obtaining Milvus format files from Milvus if they wish to import them? This is particularly relevant because the requirements from ospp explicitly state: "为 Milvus Bulk Load 添加以下文件格式支持：i）CSV（comma separated values），ii）Milvus 格式，iii）HDF5（可选）"

Currently milvus bulkinsertion support parquet and json. the bulkloader itself convert these format to milvus binlog.
we don't support csv because csv is not flexible enough to all milvus data format, like array and sparse vector. please let us know if you have any solution on that

[xiaofan-luan]

hdf5 and numpy is not good at supporting scalar datas

[Mutezebra]

Regarding the implementation of the CSV format, I will improve the demo code as soon as possible and then show you the results of the import directly. Although the support for dynamic fields may not be as good as importing directly from JSON files due to the limitations of the CSV plain text format, I believe this might be acceptable.

[Mutezebra]

I built Milvus-Standalone locally and then uploaded the JSON file and CSV file to Minio a-bucket, respectively. Next, I sent a request to Milvus via PyMilvus to import the data. Their source data is almost identical, except that I randomly modified some data in the 'vec' section. It appears that there is no difference between the import results of JSON and CSV. As for the issue with arrays and sparse vectors you mentioned, hmm... perhaps the precision of CSV and JSON is the same, I guess.And for the text data coming from CSV, perhaps I could use the math/big package instead of strconv to meet the higher precision requirements.

ps: picture from attu
source data

"book_id","book_name",book_intro,book_count,book_meta,book_reader
2010,"19530","[10.20,3.2]",20,meta,"[cat,dog,123,""none""]"
2011,向量指南,"[10.10,20.20]",30,meta,"[cat,dog,123,""none""]"
2012,~,"[10.10,20.20]",30,meta,"[cat,dog,123,""none""]"
2013,wwww,"[10.10,20.20]",30,meta,"[cat,dog,123,""none""]"
2014,"""mutezebra""","[10.10,20.20]",30,meta,"[cat,dog,123,""none""]"