ceph: support ephemeral volumes with Ceph CSI RBD and CephFS driver

This commit make required changes for ceph csi drivers to work with
ephemeral volume support. With ephemeral volume support a user can
specify ephemeral volumes in its pod spec and tie the lifecycle
of the PVC with the POD.

An example POD spec looks like this:

```
kind: Pod
apiVersion: v1
metadata:
  name: csi-rbd-demo-ephemeral-pod
spec:
  containers:
    - name: web-server
      image: docker.io/library/nginx:latest
      volumeMounts:
        - mountPath: "/myspace"
          name: mypvc
  volumes:
    - name: mypvc
      ephemeral:
        volumeClaimTemplate:
          spec:
            accessModes: ["ReadWriteOnce"]
            storageClassName: "rook-ceph-block"
            resources:
              requests:
                storage: 1Gi
```

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>

Documentation/ceph-csi-ephemeralvolume.md

@@ -0,0 +1,126 @@
+---
+title: Generic ephemeral volume support
+weight: 3250
+indent: true
+---
+
+The generic ephemeral volume feature adds support for specifying PVCs in the
+`volumes` field to indicate a user would like to create a Volume as part of the pod spec.
+This feature requires the GenericEphemeralVolume feature gate to be enabled.
+Because this is a beta feature, it is enabled by default in kubernetes versions >=1.21.
+
+For example:
+
+```yaml
+kind: Pod
+apiVersion: v1
+metadata:
+  name: csi-rbd-demo-ephemeral-pod
+spec:
+  containers:
+    - name: web-server
+      image: docker.io/library/nginx:latest
+      volumeMounts:
+        - mountPath: "/myspace"
+          name: mypvc
+  volumes:
+    - name: mypvc
+      ephemeral:
+        volumeClaimTemplate:
+          spec:
+            accessModes: ["ReadWriteOnce"]
+            storageClassName: "rook-ceph-block"
+            resources:
+              requests:
+                storage: 1Gi
+```
+
+A volume claim template is defined inside the pod spec which refers to a volume
+provisioned and used by the pod with its lifecycle. The volumes are provisioned
+when pod get spawned and destroyed at time of pod delete.
+
+Refer to [ephemeral-doc]( https://kubernetes.io/docs/concepts/storage/ephemeral-volumes/#generic-ephemeral-volumes )
+for more info.
+
+### Prerequisites
+1. Requires Kubernetes v1.21+ which supports ephemeral volume.
+
+## RBD Ephemeral Volume
+In
+[pod-ephemeral](https://github.com/rook/rook/tree/{{ branchName }}/cluster/examples/kubernetes/ceph/csi/rbd/pod-ephemeral.yaml),
+`storageclass` should be the name of the `storageclass` which is already available for the RBD
+provisioning to work.
+
+Create a new POD from the yaml
+
+### Verify RBD PVC got created successfully upon POD creation
+
+```console
+kubectl create -f cluster/examples/kubernetes/ceph/csi/rbd/pod-ephemeral.yaml
+pod/csi-rbd-demo-ephemeral-pod created
+
+kubectl get pvc
+NAME                               STATUS        VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       AGE
+csi-rbd-demo-ephemeral-pod-mypvc   Bound         pvc-19960142-8218-41c9-94fd-eda2ee9c47e3   1Gi        RWO            rook-ceph-block    31m
+
+kubectl get pods
+NAME                         READY   STATUS    RESTARTS   AGE
+csi-rbd-demo-ephemeral-pod   1/1     Running   0          32m
+
+```
+### Verify RBD PVC got deleted upon POD deletion
+
+```console
+kubectl delete pod csi-rbd-demo-ephemeral-pod
+pod "csi-rbd-demo-ephemeral-pod" deleted
+
+kubectl get pod
+
+```
+verify that PVC is also deleted
+
+```console
+kubectl get pvc
+
+```
+
+## CephFS Ephemeral Volume
+In
+[pod-ephemeral](https://github.com/rook/rook/tree/{{ branchName }}/cluster/examples/kubernetes/ceph/csi/cephfs/pod-ephemeral.yaml),
+`storageclass` should be the name of the `storageclass` which is already available for the CephFS
+provisioning to work.
+
+### Verify CephFS PVC got created successfully upon POD creation
+
+Create a new POD from the yaml
+
+```console
+kubectl create -f cluster/examples/kubernetes/ceph/csi/cephfs/pod-ephemeral.yaml
+pod/csi-cephfs-demo-ephemeral-pod created
+
+kubectl get pvc
+NAME                                  STATUS        VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       AGE
+cephfs-pvc                            Bound         pvc-f6bf616e-54ac-4b22-82a7-82a135af8096   1Gi        RWO            rook-cephfs        15d
+csi-cephfs-demo-ephemeral-pod-mypvc   Bound         pvc-3af76629-b2ea-4784-be31-d4742a1d0147   1Gi        RWX            rook-cephfs        3s
+
+kubectl get pods
+NAME                            READY   STATUS    RESTARTS   AGE
+csi-cephfs-demo-ephemeral-pod   1/1     Running   0          17s
+
+```
+
+### Verify CephFS PVC got deleted upon POD deletion
+
+```console
+kubectl delete pod csi-cephfs-demo-ephemeral-pod
+pod "csi-cephfs-demo-ephemeral-pod" deleted
+
+kubectl get pod
+
+```
+verify that PVC is also deleted
+
+```console
+kubectl get pvc
+
+```

cluster/examples/kubernetes/ceph/common.yaml

@@ -871,9 +871,15 @@ rules:
   - apiGroups: [""]
     resources: ["configmaps"]
     verbs: ["get", "list", "create", "delete"]
+  - apiGroups: [""]
+    resources: ["persistentvolumeclaims"]
+    verbs: ["get", "list", "watch", "update"]
   - apiGroups: ["coordination.k8s.io"]
     resources: ["leases"]
     verbs: ["get", "watch", "list", "delete", "update", "create"]
+  - apiGroups: ["storage.k8s.io"]
+    resources: ["storageclasses"]
+    verbs: ["get", "list", "watch"]
 # OLM: END CSI CEPHFS ROLE
 # OLM: BEGIN CSI CEPHFS ROLEBINDING
 ---
@@ -1045,9 +1051,15 @@ rules:
   - apiGroups: [""]
     resources: ["configmaps"]
     verbs: ["get", "list", "watch", "create", "delete", "update"]
+  - apiGroups: [""]
+    resources: ["persistentvolumeclaims"]
+    verbs: ["get", "list", "watch", "update"]
   - apiGroups: ["coordination.k8s.io"]
     resources: ["leases"]
     verbs: ["get", "watch", "list", "delete", "update", "create"]
+  - apiGroups: ["storage.k8s.io"]
+    resources: ["storageclasses"]
+    verbs: ["get", "list", "watch"]
 # OLM: END CSI RBD ROLE
 # OLM: BEGIN CSI RBD ROLEBINDING
 ---

cluster/examples/kubernetes/ceph/csi/cephfs/pod-ephemeral.yaml

@@ -0,0 +1,21 @@
+kind: Pod
+apiVersion: v1
+metadata:
+  name: csi-cephfs-demo-ephemeral-pod
+spec:
+  containers:
+    - name: web-server
+      image: docker.io/library/nginx:latest
+      volumeMounts:
+        - mountPath: "/myspace"
+          name: mypvc
+  volumes:
+    - name: mypvc
+      ephemeral:
+        volumeClaimTemplate:
+          spec:
+            accessModes: ["ReadWriteMany"]
+            storageClassName: "rook-cephfs"
+            resources:
+              requests:
+                storage: 1Gi

cluster/examples/kubernetes/ceph/csi/rbd/pod-ephemeral.yaml

@@ -0,0 +1,21 @@
+kind: Pod
+apiVersion: v1
+metadata:
+  name: csi-rbd-demo-ephemeral-pod
+spec:
+  containers:
+    - name: web-server
+      image: docker.io/library/nginx:latest
+      volumeMounts:
+        - mountPath: "/myspace"
+          name: mypvc
+  volumes:
+    - name: mypvc
+      ephemeral:
+        volumeClaimTemplate:
+          spec:
+            accessModes: ["ReadWriteOnce"]
+            storageClassName: "rook-ceph-block"
+            resources:
+              requests:
+                storage: 1Gi