mds: create EC pool as secondary pool (backport #8452)

Documentation/ceph-filesystem-crd.md

@@ -86,6 +86,8 @@ spec:
     replicated:
       size: 3
   dataPools:
+    - replicated:
+        size: 3
     - erasureCoded:
         dataChunks: 2
         codingChunks: 1
@@ -94,6 +96,8 @@ spec:
     activeStandby: true
 ```
 
+**IMPORTANT**: For erasure coded pools, we have to create a replicated pool as the default data pool and an erasure-coded pool as a secondary pool.
+
 (These definitions can also be found in the [`filesystem-ec.yaml`](https://github.com/rook/rook/blob/{{ branchName }}/deploy/examples/filesystem-ec.yaml) file.
 Also see an example in the [`storageclass-ec.yaml`](https://github.com/rook/rook/blob/{{ branchName }}/deploy/examples/csi/cephfs/storageclass-ec.yaml) for how to configure the volume.)
 

deploy/examples/csi/cephfs/storageclass-ec.yaml

@@ -14,7 +14,10 @@ parameters:
 
   # Ceph pool into which the volume shall be created
   # Required for provisionVolume: "true"
-  pool: myfs-ec-data0
+
+  # For erasure coded pools, we have to create a replicated pool as the default data pool and an erasure-coded
+  # pool as a secondary pool.
+  pool: myfs-ec-data1
 
   # The secrets contain Ceph admin credentials. These are generated automatically by the operator
   # in the same namespace as the cluster.

deploy/examples/filesystem-ec.yaml

@@ -17,6 +17,10 @@ spec:
       size: 3
   # The list of data pool specs
   dataPools:
+    # For erasure coded pools, we have to create a replicated pool as the default
+    # data pool and an erasure-coded pool as a secondary pool.
+    - replicated:
+        size: 3
     # You need at least three `bluestore` OSDs on different nodes for this config to work
     - erasureCoded:
         dataChunks: 2