ceph: support ephemeral volumes with Ceph CSI RBD and CephFS driver

This commit make required changes for ceph csi drivers to work with
ephemeral volume support. With ephemeral volume support a user can
specify ephemeral volumes in its pod spec and tie the lifecycle
of the PVC with the POD.

An example POD spec looks like this:

```
kind: Pod
apiVersion: v1
metadata:
  name: my-app
   spec:
  containers:
    - name: csi-rbd-demo-pod
      image: busybox
      volumeMounts:
        - mountPath: "/scratch"
          name: mypvc
      command: [ "sleep", "1000000" ]
  volumes:
    - name: mypvc
      ephemeral:
        volumeClaimTemplate:
          spec:
            accessModes: [ "ReadWriteOnce" ]
            storageClassName: "csi-rbd-sc"
            resources:
              requests:
                storage: 1Gi
```

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>

Documentation/ceph-csi-ephemeralvolume.md

@@ -0,0 +1,128 @@
+---
+title: Volume ephemeral
+weight: 3250
+indent: true
+---
+
+The generic ephemeral volume feature adds support for specifying PVCs in the
+`volumes` field to indicate a user would like to create a Volume as part of the pod spec.
+
+For ex:
+
+```
+kind: Pod
+apiVersion: v1
+metadata:
+  name: my-app
+   spec:
+  containers:
+    - name: csi-rbd-demo-pod
+      image: busybox
+      volumeMounts:
+        - mountPath: "/scratch"
+          name: mypvc
+      command: [ "sleep", "1000000" ]
+  volumes:
+    - name: mypvc
+      ephemeral:
+        volumeClaimTemplate:
+          spec:
+            accessModes: [ "ReadWriteOnce" ]
+            storageClassName: "csi-rbd-sc"
+            resources:
+              requests:
+                storage: 1Gi
+```
+A volume claim template is defined inside the pod spec which refers to a volume
+provisioned and used by the pod with its lifecycle. The volumes are provisioned
+when pod get spawned and destroyed at time of pod delete.
+
+Refer to [ephemeral-doc](https://kubernetes.io/docs/concepts/storage/ephemeral-volumes/)
+for more info.
+
+## RBD Ephemeral Volume
+
+### Prerequisites
+
+ 1. Requires Kubernetes v1.21+ which supports ephemeral volume.
+### Ephemeral Volume
+
+In
+[pod-ephemeral](https://github.com/rook/rook/tree/{{ branchName }}/cluster/examples/kubernetes/ceph/csi/rbd/pod-ephemeral.yaml),
+`storageclass` should be the name of the `storageclass` which is already available for the RBD
+provisioning to work.
+
+Create a new POD from the yaml
+
+### Verify RBD PVC got created successfully upon POD creation
+
+```console
+# kubectl create -f cluster/examples/kubernetes/ceph/csi/rbd/pod-ephemeral.yaml
+pod/csi-rbd-demo-ephemeral-pod created
+# kubectl get pvc
+NAME                               STATUS        VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       AGE
+rbd-pvc           		         Bound         pvc-f4d1d0ef-9c16-440e-9e77-d4b35654ecbe   1Gi        RWO            rook-ceph-block    19d
+csi-rbd-demo-ephemeral-pod-mypvc   Bound         pvc-19960142-8218-41c9-94fd-eda2ee9c47e3   1Gi        RWO            rook-ceph-block    31m
+
+# kubectl get pods
+NAME                         READY   STATUS    RESTARTS   AGE
+csi-rbd-demo-ephemeral-pod   1/1     Running   0          32m
+
+```
+### Verify RBD PVC got deleted upon POD deletion
+
+```
+# kubectl delete pod csi-rbd-demo-ephemeral-pod
+pod "csi-rbd-demo-ephemeral-pod" deleted
+
+# kubectl get pvc
+NAME              STATUS        VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       AGE
+rbd-pvc           Bound         pvc-f4d1d0ef-9c16-440e-9e77-d4b35654ecbe   1Gi        RWO            rook-ceph-block    19d
+
+# kubectl get pod
+#
+```
+
+## CephFS Ephemeral Volume
+
+
+### Prerequisites
+
+1. Requires Kubernetes v1.21+ which supports ephemeral volume.
+
+### Ephemeral Volume
+
+In
+[pod-ephemeral](https://github.com/rook/rook/tree/{{ branchName }}/cluster/examples/kubernetes/ceph/csi/cephfs/pod-ephemeral.yaml),
+`storageclass` should be the name of the `storageclass` which is already available for the CephFS
+provisioning to work.
+
+
+### Verify CephFS PVC got created successfully upon POD creation
+
+Create a new POD from the yaml
+
+```console
+#kubectl create -f cluster/examples/kubernetes/ceph/csi/cephfs/pod-ephemeral.yaml
+pod/csi-cephfs-demo-ephemeral-pod created
+# kubectl get pvc
+NAME                                  STATUS        VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       AGE
+cephfs-pvc                            Bound         pvc-f6bf616e-54ac-4b22-82a7-82a135af8096   1Gi        RWO            rook-cephfs        15d
+csi-cephfs-demo-ephemeral-pod-mypvc   Bound         pvc-3af76629-b2ea-4784-be31-d4742a1d0147   1Gi        RWX            rook-cephfs        3s
+#
+# kubectl get pods
+NAME                            READY   STATUS    RESTARTS   AGE
+csi-cephfs-demo-ephemeral-pod   1/1     Running   0          17s
+```
+### Verify CephFS PVC got deleted upon POD deletion
+
+```
+# kubectl delete pod csi-cephfs-demo-ephemeral-pod
+pod "csi-cephfs-demo-ephemeral-pod" deleted
+
+# kubectl get pod
+# kubectl get pvc
+NAME              STATUS        VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       AGE
+cephfs-pvc        Bound         pvc-f6bf616e-54ac-4b22-82a7-82a135af8096   1Gi        RWO            rook-cephfs        15d
+#
+```

cluster/examples/kubernetes/ceph/common.yaml

@@ -871,9 +871,15 @@ rules:
   - apiGroups: [""]
     resources: ["configmaps"]
     verbs: ["get", "list", "create", "delete"]
+  - apiGroups: [ "" ]
+    resources: [ "persistentvolumeclaims" ]
+    verbs: [ "get", "list", "watch", "update" ]
   - apiGroups: ["coordination.k8s.io"]
     resources: ["leases"]
     verbs: ["get", "watch", "list", "delete", "update", "create"]
+  - apiGroups: ["storage.k8s.io"]
+    resources: ["storageclasses"]
+    verbs: ["get", "list", "watch"]
 # OLM: END CSI CEPHFS ROLE
 # OLM: BEGIN CSI CEPHFS ROLEBINDING
 ---
@@ -1045,9 +1051,15 @@ rules:
   - apiGroups: [""]
     resources: ["configmaps"]
     verbs: ["get", "list", "watch", "create", "delete", "update"]
+  - apiGroups: [ "" ]
+    resources: [ "persistentvolumeclaims" ]
+    verbs: [ "get", "list", "watch", "update" ]
   - apiGroups: ["coordination.k8s.io"]
     resources: ["leases"]
     verbs: ["get", "watch", "list", "delete", "update", "create"]
+  - apiGroups: [ "storage.k8s.io" ]
+    resources: [ "storageclasses" ]
+    verbs: [ "get", "list", "watch" ]
 # OLM: END CSI RBD ROLE
 # OLM: BEGIN CSI RBD ROLEBINDING
 ---

cluster/examples/kubernetes/ceph/csi/cephfs/pod-ephemeral.yaml

@@ -0,0 +1,21 @@
+kind: Pod
+apiVersion: v1
+metadata:
+  name: csi-rbd-demo-ephemeral-pod
+spec:
+  containers:
+    - name: web-server
+      image: docker.io/library/nginx:latest
+      volumeMounts:
+        - mountPath: "/myspace"
+          name: mypvc
+  volumes:
+    - name: mypvc
+      ephemeral:
+        volumeClaimTemplate:
+          spec:
+            accessModes: ["ReadWriteMany"]
+            storageClassName: "rook-cephfs"
+            resources:
+              requests:
+                storage: 1Gi

cluster/examples/kubernetes/ceph/csi/rbd/pod-ephemeral.yaml

@@ -0,0 +1,21 @@
+kind: Pod
+apiVersion: v1
+metadata:
+  name: csi-rbd-demo-ephemeral-pod
+spec:
+  containers:
+    - name: web-server
+      image: docker.io/library/nginx:latest
+      volumeMounts:
+        - mountPath: "/myspace"
+          name: mypvc
+  volumes:
+    - name: mypvc
+      ephemeral:
+        volumeClaimTemplate:
+          spec:
+            accessModes: ["ReadWriteOnce"]
+            storageClassName: "rook-ceph-block"
+            resources:
+              requests:
+                storage: 1Gi