Add e2e tests to verify snapshot data compatibility in a mixed version cluster

[siyuanfoundation]

What would you like to be added?

e2e tests that verify in a mixed version cluster (3.5+3.4, and 3.6+3.5), when a snapshot is sent from the higher version leader to the lower version follower, the snapshot file is properly migrated and compatible to the lower version.

To achieve this, the following tasks need to be done:

• [3.5->3.4] Add --experimental-snapshot-catchup-entries flag in 3.5 to make sure snapshot is sent in e2e test. [3.5] add experimental-snapshot-catchup-entries flag #17808
• [3.5->3.4] Add tests to verify schema of db file using bbolt after snapshot is sent and applied in the mixed version cluster.
• [3.6->3.5] Add tests to verify schema of db file using bbolt after snapshot is sent and applied in the mixed version cluster, provided the online migration is implemented.

Why is this needed?

During implementing 3.5->3.4 downgrade, we caught an issue that could be easily slipped in downgrade designs: when a lower version follower receives a snapshot from a higher version leader, the snapshot might not be conformant to the lower version schema. In addition to offline data migration during downgrade process, special attention needs to paid to cover online migration as well.

This is important to guarantee data integrity and compatibility in a mixed version cluster. It is important not just to add the online migration, also to add proper tests to make sure of that.

[siyuanfoundation]

cc @serathius