Add a controller for Ceph storage

[medkbadri]

Hello,

We're currently using Rook ceph for Kubernetes storage.
I experimented with this controller and it works like magic! Except that for my case, ceph OSD PVCs have block volumeMode:

Thus, kubelet_volume_stats_used_bytes and kubelet_volume_stats_capacity_bytes metrics for OSD PVCs are unavailable in Prometheus.

The idea is to:

• add specific annotations for ceph PVCs or cluser definition manifest (CephCluster CRD)
• use ceph metrics instead: ceph_cluster_total_bytes and ceph_cluster_total_used_bytes
• To be decided: update the PVC or CephCluster CRD (safer)

I'd be more than happy to contribute to the implementation of the idea if it is worth experimenting with.

++

[raffaelespazzoli]

hi @medkbadri, I like your proposal. Before we do something storage provider specific, let me ask you this. I'd expect the CSI spec to define also a set of metrics to be exported which would supposedly cover also the block volume use case. Do you know where we stand on that? I'd prefer to use those metrics as they'd offer two advantages:

1. they'd be implementation agnostic
2. the kubelet metrics are available only when the volume is mounted, presumably instead CSI metrics would always be available.

[ltartarini]

The following Prometheus query works well for all PVCs except for Ceph OSDs PVCs:

    (
      sum without(instance, node) (kubelet_volume_stats_used_bytes{job="kubelet",metrics_path="/metrics",persistentvolumeclaim=~".*"})
      /
      sum without(instance, node) (kubelet_volume_stats_capacity_bytes{job="kubelet", metrics_path="/metrics", persistentvolumeclaim=~".*"})
    ) * 100

Here's the PVCs spec:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    ...
  labels:
    ceph.rook.io/DeviceSet: set1
    ceph.rook.io/DeviceSetPVCId: set1-data-0
    ceph.rook.io/setIndex: "0"
  name: set1-data-abcdef
  namespace: xywz
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
  storageClassName: gp2
  volumeMode: Block
  volumeName: pvc-1234567890

is there any way to solve this issue?