ceph: add a directory layout for secondary ec data pool

when creating ec fs, create replicated pool as primary
pool and ec pool as secondary pool, creating ec pool
as primary is not encouraged and it will lead to failure.
Also after this, user need to add a directory layout
on root for a secondary ec data pool.

Closes: #8210
Signed-off-by: subhamkrai <srai@redhat.com>

Documentation/ceph-filesystem-crd.md

@@ -86,6 +86,8 @@ spec:
     replicated:
       size: 3
   dataPools:
+    - replicated:
+        size: 3
     - erasureCoded:
         dataChunks: 2
         codingChunks: 1
@@ -94,6 +96,16 @@ spec:
     activeStandby: true
 ```
 
+**IMPORTANT**: For erasure coded pools, we have to create a replicated pool as the primary pool and erasureCoded as the secondary pool.
+Separately, a directory layout property needs to be added to use the secondary EC data pool.
+
+```console
+mkdir /mnt/cephfs/myssddir
+$ setfattr -n ceph.dir.layout.pool -v <ec-pool-name> /mnt/cephfs/myssddir #example ec-pool-name, myfs-ec
+```
+
+See [CephFS file layouts](https://docs.ceph.com/en/latest/cephfs/file-layouts/#adding-a-data-pool-to-the-file-system) for more details.
+
 (These definitions can also be found in the [`filesystem-ec.yaml`](https://github.com/rook/rook/blob/{{ branchName }}/cluster/examples/kubernetes/ceph/filesystem-ec.yaml) file.
 Also see an example in the [`storageclass-ec.yaml`](https://github.com/rook/rook/blob/{{ branchName }}/cluster/examples/kubernetes/ceph/csi/cephfs/storageclass-ec.yaml) for how to configure the volume.)