ceph: support ephemeral volumes with Ceph CSI RBD and CephFS driver

This commit make required changes for ceph csi drivers to work with
ephemeral volume support. With ephemeral volume support a user can
specify ephemeral volumes in its pod spec and tie the lifecycle
of the PVC with the POD.

An example POD spec looks like this:

```
kind: Pod
apiVersion: v1
metadata:
  name: csi-rbd-demo-ephemeral-pod
spec:
  containers:
    - name: web-server
      image: docker.io/library/nginx:latest
      volumeMounts:
        - mountPath: "/myspace"
          name: mypvc
  volumes:
    - name: mypvc
      ephemeral:
        volumeClaimTemplate:
          spec:
            accessModes: ["ReadWriteOnce"]
            storageClassName: "rook-ceph-block"
            resources:
              requests:
                storage: 1Gi
```

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>

Documentation/ceph-csi-drivers.md

@@ -92,3 +92,49 @@ kubectl create -f https://raw.githubusercontent.com/csi-addons/volume-replicatio
 2. Enable the volume replication controller:
    - For Helm deployments see the [csi.volumeReplication.enabled setting](helm-operator.md#configuration).
    - For non-Helm deployments set `CSI_ENABLE_VOLUME_REPLICATION: "true"` in operator.yaml
+
+## generic ephemeral volume support
+
+The generic ephemeral volume feature adds support for specifying PVCs in the
+`volumes` field to indicate a user would like to create a Volume as part of the pod spec.
+This feature requires the GenericEphemeralVolume feature gate to be enabled.
+Because this is a beta feature, it is enabled by default in kubernetes versions >=1.21.
+
+For example:
+
+```yaml
+kind: Pod
+apiVersion: v1
+metadata:
+  name: csi-rbd-demo-ephemeral-pod
+spec:
+  containers:
+    - name: web-server
+      image: docker.io/library/nginx:latest
+      volumeMounts:
+        - mountPath: "/myspace"
+          name: mypvc
+  volumes:
+    - name: mypvc
+      ephemeral:
+        volumeClaimTemplate:
+          spec:
+            accessModes: ["ReadWriteOnce"]
+            storageClassName: "rook-ceph-block"
+            resources:
+              requests:
+                storage: 1Gi
+```
+
+A volume claim template is defined inside the pod spec which refers to a volume
+provisioned and used by the pod with its lifecycle. The volumes are provisioned
+when pod get spawned and destroyed at time of pod delete.
+
+Refer to [ephemeral-doc]( https://kubernetes.io/docs/concepts/storage/ephemeral-volumes/#generic-ephemeral-volumes )
+for more info.
+
+### Prerequisites
+1. Requires Kubernetes v1.21+ which supports ephemeral volume.
+2. In [pod-ephemeral](https://github.com/rook/rook/tree/{{ branchName }}/cluster/examples/kubernetes/ceph/csi/rbd/pod-ephemeral.yaml),
+`storageclass` should be the name of the `storageclass` which is already available for the RBD
+provisioning to work.

cluster/examples/kubernetes/ceph/csi/cephfs/pod-ephemeral.yaml

@@ -0,0 +1,21 @@
+kind: Pod
+apiVersion: v1
+metadata:
+  name: csi-cephfs-demo-ephemeral-pod
+spec:
+  containers:
+    - name: web-server
+      image: docker.io/library/nginx:latest
+      volumeMounts:
+        - mountPath: "/myspace"
+          name: mypvc
+  volumes:
+    - name: mypvc
+      ephemeral:
+        volumeClaimTemplate:
+          spec:
+            accessModes: ["ReadWriteMany"]
+            storageClassName: "rook-cephfs"
+            resources:
+              requests:
+                storage: 1Gi

cluster/examples/kubernetes/ceph/csi/rbd/pod-ephemeral.yaml

@@ -0,0 +1,21 @@
+kind: Pod
+apiVersion: v1
+metadata:
+  name: csi-rbd-demo-ephemeral-pod
+spec:
+  containers:
+    - name: web-server
+      image: docker.io/library/nginx:latest
+      volumeMounts:
+        - mountPath: "/myspace"
+          name: mypvc
+  volumes:
+    - name: mypvc
+      ephemeral:
+        volumeClaimTemplate:
+          spec:
+            accessModes: ["ReadWriteOnce"]
+            storageClassName: "rook-ceph-block"
+            resources:
+              requests:
+                storage: 1Gi