[Bug]: Flush func in DataCoord returns only SegmentLevel_L1 segments as flushed segments

[ghu613]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: v2.4.0
- Deployment mode(standalone or cluster): cluster
- MQ type(rocksmq, pulsar or kafka): kafka
- SDK version(e.g. pymilvus v2.0.0rc2): pymilvus v2.4.0
- OS(Ubuntu or CentOS): RockyLinux
- CPU/Memory: 4c16G 
- GPU: N/A
- Others:

Current Behavior

Hello,

After upgrading Milvus from v2.3.10 to v2.4.0 in the test environment, milvus-backup tool only backed up three out of five segments in a collection with ~6million entities. The three segments all have level: SegmentLevel_L1 and the other two segments have level: SegmentLevel_Legacy (the default value). All of the segments are in Flushed state.

It seems that there's a change between v2.3.10 and v2.4.0 to the Flush func in DataCoord service which filters only segments of level: SegmentLevel_L1. This could be the reason that milvus-backup didn't back up all segments since it called Flush to get the flushed segments before copying their log files.

[Please feel free to correct here] A quick read of the codebase seems to suggest that level will be updated to SegmentLevel_L1 for segments after going through compaction and this info is saved to meta.

Appreciate it if anyone can look into this issue. It has become a blocker to upgrading in prod until back-up is verified to cover all segments. Thank you.

Expected Behavior

All flushed segments except probably the ones of SegmentLevel_L0 should be returned from Flush in DataCoord.

Steps To Reproduce

1. Set up Milvus with version v2.3.10
2. Prepare a collection with multiple segments 
3. Back up collection using milvus-backup tool
4. Upsert some vectors
5. Upgrade to v2.4.0
6. Back up collection using milvus-backup tool
7. Compare the back-up file sizes

Milvus Log

N/A

Anything else?

No response

[yanliang567]

/assign @wayblink
/unassign

[ghu613]

A temporary work-around might be to manually compact the collection, enforcing the meta to be updated. Just did that, and now all segments are of level: SegmentLevel_L1. This is more a Milvus issue than a milvus-backup tool issue. Thanks.

[wayblink]

@ghu613 Hi, thanks a lot for reporting this issue

[ghu613]

@ghu613 Hi, thanks a lot for reporting this issue

No problem. Thanks for the fix #33061. It might be good to ask the team/person who made the initial change to filter for SegmentLevel_L1 segments to review #33061. Personally curious why only SegmentLevel_L1 segments were selected in the first place and maybe SegmentInfo needs to be updated in meta in other code logic.