Questions about data distribution in minio

[Hello-Linux]

I have a local cluster of two minio nodes, each with a disk drive. My question is how the data blocks and check blocks are distributed, is there a data block and check block on the disk of each node? Or is it the other way around: only data blocks on one node and check blocks on the other?

[klauspost]

Data and parity blocks are distributed across servers. If you have 10 servers objects will be split across all 10, with a number of these blocks being erasure codes (parity).

There are many documents on this. Maybe start with https://min.io/docs/minio/kubernetes/upstream/operations/concepts/architecture.html

[Hello-Linux]

Thank you very much for your answer. After reading the official website document, I still have some questions
To my question above，how the data blocks and check blocks are distributed, is there a data block and check block on the disk of each node? Or is it the other way around: only data blocks on one node and check blocks on the other?

I found a part1 file in the disk directory mounted on each node. Is this file a data block or a parity block?

[klauspost]

Is this file a data block or a parity block?

It can be either. The xl.meta in the parent will have that information. here is a tool that allows you to decode these files.

Going into details would be too much here, since it is a too big of an issue to cover.

         "EcDist": [
            16,
            1,
            2,
            3,
            4,
            5,
            6,
            7,
            8,
            9,
            10,
            11,
            12,
            13,
            14,
            15
          ],
          "EcIndex": 16,
          "EcM": 12,
          "EcN": 4,

EcDist is the shard distribution on disks. EcIndex is the index of the current disk (this will be different for each disk). EcM is the number of data shards, EcN is the number of parity shards.

Drive 16 (the drive with this xl.meta) will have the first data shard, drive 1 the second, drive 2 the third, until you have 12. Then the 4 parity shards follow.

[Hello-Linux]

Your explanation is very clear. According to your meaning, there are only two minio nodes in my local area, and there is only one disk on each node. Shouldn't the distribution of data shards and parity shards be that the first disk is data shard and the second disk is parity shards?

[Hello-Linux]

I use the xl-meta see my xl.meta,this is the output for two minio node:

{
  "Versions": [
    {
      "Header": {
        "Flags": 2,
        "ModTime": "2024-04-10T16:56:05.091560827+08:00",
        "Signature": "8d9c2a56",
        "Type": 1,
        "VersionID": "00000000000000000000000000000000"
      },
      "Idx": 0,
      "Metadata": {
        "Type": 1,
        "V2Obj": {
          "CSumAlgo": 1,
          "DDir": "CBMjrKUuRCeBxZ0z4VUElg==",
          "EcAlgo": 1,
          "EcBSize": 1048576,
          "EcDist": [
            2,
            1
          ],
          "EcIndex": 2,
          "EcM": 1,
          "EcN": 1,
          "ID": "AAAAAAAAAAAAAAAAAAAAAA==",
          "MTime": 1712739365091560827,
          "MetaSys": {},
          "MetaUsr": {
            "content-type": "application/octet-stream",
            "etag": "9d34d6d5b9f93ac9212181d1bd46a0ab"
          },
          "PartASizes": [
            32835984
          ],
          "PartETags": null,
          "PartNums": [
            1
          ],
          "PartSizes": [
            32835984
          ],
          "Size": 32835984
        },
        "v": 1710464839
      }
    }
  ]
}

the two node

{
  "Versions": [
    {
      "Header": {
        "Flags": 2,
        "ModTime": "2024-04-10T16:56:05.091560827+08:00",
        "Signature": "8d9c2a56",
        "Type": 1,
        "VersionID": "00000000000000000000000000000000"
      },
      "Idx": 0,
      "Metadata": {
        "Type": 1,
        "V2Obj": {
          "CSumAlgo": 1,
          "DDir": "CBMjrKUuRCeBxZ0z4VUElg==",
          "EcAlgo": 1,
          "EcBSize": 1048576,
          "EcDist": [
            2,
            1
          ],
          "EcIndex": 1,
          "EcM": 1,
          "EcN": 1,
          "ID": "AAAAAAAAAAAAAAAAAAAAAA==",
          "MTime": 1712739365091560827,
          "MetaSys": {},
          "MetaUsr": {
            "content-type": "application/octet-stream",
            "etag": "9d34d6d5b9f93ac9212181d1bd46a0ab"
          },
          "PartASizes": [
            32835984
          ],
          "PartETags": null,
          "PartNums": [
            1
          ],
          "PartSizes": [
            32835984
          ],
          "Size": 32835984
        },
        "v": 1710464839
      }
    }
  ]
}

This result shows that my data shards and parity shards are evenly distributed across the two disks on the two nodes

[klauspost]

Shouldn't the distribution of data shards and parity shards be that the first disk is data shard and the second disk is parity shards?

The starting offset is randomized for each object, otherwise reads would give uneven load on the disks.

[Hello-Linux]

I don't quite understand that "The starting offset is randomized for each object, otherwise reads would give uneven load on the disks." For my output above,，EC:M=1 EC:N=1，I only have two local nodes, each node has a hard disk, which disk stores data shards and the other disk stores parity shards, right? I understand that correctly？

[Hello-Linux]

My ultimate goal is to figure out how the data block and check block are distributed on each disk of my two nodes, and where can I see the data block and check block file