mds: create EC pool as secondary pool

When creating EC fs, create replicated pool as primary
pool and ec pool as secondary pool, creating ec pool
as primary is not encouraged and it will lead to failure.

Also, changing the pool name in storageclass-ec file.

Closes: #8210
Signed-off-by: subhamkrai <srai@redhat.com>

Documentation/ceph-filesystem-crd.md

@@ -86,6 +86,8 @@ spec:
     replicated:
       size: 3
   dataPools:
+    - replicated:
+        size: 3
     - erasureCoded:
         dataChunks: 2
         codingChunks: 1
@@ -94,6 +96,8 @@ spec:
     activeStandby: true
 ```
 
+**IMPORTANT**: For erasure coded pools, we have to create a replicated pool as the default data pool and an erasure-coded pool as a secondary pool.
+
 (These definitions can also be found in the [`filesystem-ec.yaml`](https://github.com/rook/rook/blob/{{ branchName }}/deploy/examples/filesystem-ec.yaml) file.
 Also see an example in the [`storageclass-ec.yaml`](https://github.com/rook/rook/blob/{{ branchName }}/deploy/examples/csi/cephfs/storageclass-ec.yaml) for how to configure the volume.)
 

deploy/examples/csi/cephfs/storageclass-ec.yaml

@@ -14,7 +14,10 @@ parameters:
 
   # Ceph pool into which the volume shall be created
   # Required for provisionVolume: "true"
-  pool: myfs-ec-data0
+
+  # For erasure coded pools, we have to create a replicated pool as the default data pool and an erasure-coded
+  # pool as a secondary pool.
+  pool: myfs-ec-data1
 
   # The secrets contain Ceph admin credentials. These are generated automatically by the operator
   # in the same namespace as the cluster.

deploy/examples/filesystem-ec.yaml

@@ -17,6 +17,10 @@ spec:
       size: 3
   # The list of data pool specs
   dataPools:
+    # For erasure coded pools, we have to create a replicated pool as the default
+    # data pool and an erasure-coded pool as a secondary pool.
+    - replicated:
+        size: 3
     # You need at least three `bluestore` OSDs on different nodes for this config to work
     - erasureCoded:
         dataChunks: 2