v1.2.6

Improvements

Rook v1.2.6 is a patch release limited in scope and focusing on bug fixes.

Ceph

• Update default Ceph version to v14.2.8 (#4960, @leseb)
• Fix for OSDs on PVCs that were crashing on Ceph v14.2.8 (#4960, @leseb)
• Mount /udev so the osds can discover device info (#5001, @leseb)
• Query for the topology.kubernetes.io labels in K8s 1.17 or newer for the CRUSH hierarchy (#4989, @travisn)
• Log a warning when useAllNodes is true, but nodes are defined in the cluster CR (commit, @travisn)

v1.2.5

Improvements

Rook v1.2.5 is a patch release limited in scope and focusing on bug fixes.

Ceph

• Set recommended Ceph version v14.2.7 (#4898, @travisn)
• Allow mons from external cluster in the toolbox (#4922, @travisn)
• Set successful EC pool creation CR status on the pool CR (#4885, @travisn)
• Populate CSI configmap for external cluster mons (#4816, @leseb)
• CSI settings are configurable in the operator via a ConfigMap (#3239, @umangachapagain)
• Enabling balancer module with older clients (#4842, @leseb)
• Helm chart fix for deploying the CSI 2.0 driver (#4839, @rwd5213)
• Make replication setting optional for EC pools (#4750, @travisn)
• Docs: Set Ceph version for the PVC based example (#4869, @galexrt)

v1.2.4

Improvements

Rook v1.2.4 is a patch release limited in scope and focusing on bug fixes.

Ceph

• Stop garbage collector from deleting the CSI driver unexpectedly (#4820, @travisn)
• Upgrade legacy OSDs created with partitions created by Rook (#4799, @leseb)
• Ability to set the pool target_size_ratio (#4803, @leseb)
• Improve detection of drain-canaries and log significant nodedrain scheduling events (#4679, @rohantmp)
• Sort flexvolume docs and update for kubespray (#4747, @ftab)
• Add OpenShift common issues documentation (#4764, @umangachapagain)
• Improved integration test when cleaning devices (#4796, @leseb)

v1.2.3

Improvements

Rook v1.2.3 is a patch release limited in scope and focusing on bug fixes.

Ceph

• Fix garbage collection for the disappearing CSI driver (#4590, @travisn)
• Allow enabling of the Ceph-CSI v2.0.0 driver, with the default still as CSI v1.2.2 (#4763, @Madhu-1)
• Allow for empty "all" annotations (#4351, @travisn)
• Ability to enable the balancer module (#4784, @leseb)
• Fix TuneSlowDeviceClass spec (#4777, @leseb)
• Skip the upgrade checks in integration tests (#4710, @travisn)
• Add an external cluster functional test (#4742, @leseb)
• Remove unnecessary mon canary pods after an aborted mon failover (#3764, @jmolmo)
• Ensure MDS pod placement adheres to zone topology if specified (#4641, @jmolmo)
• Add nil check for ceph version in crash controller (#4762, @Madhu-1)
• Add ceph version to cluster status (#4357, @egafford)
• Ceph external cluster repeatedly saving mon endpoints to config (#4717, @eliaswimmer)
• Fix resource limits for the OSD prepare job (#4713, @leseb)
• Ensure OSD location is consistent to avoid unnecessary osds restarts (#4729, @ashangit)
• Fix external cluster config override configmap creation (#4686, @leseb)
• Improved ceph-volume logging in the OSD prepare job (#3888, @leseb)
• Integration test fix for the client CR (#4509, @mateuszlos)

v1.2.2

Improvements

Rook v1.2.2 is a patch release limited in scope and focusing on bug fixes.

Ceph

• Allow multiple clusters to set useAllDevices (#4692, @leseb)
• Operator start all mons before checking quorum if they are all down (#4531, @ashishranjan738)
• Ability to disable the crash controller (#4533, @leseb)
• Document monitoring options for the cluster CR (#4698, @umangachapagain)
• Apply node topology labels to PV-backed OSDs in upgrade from v1.1 (#4616, @rohan47)
• Update examples to Ceph version v14.2.6 (#4653, @leseb)
• Allow integration tests in minimal config to run on multiple K8s versions (#4674, @travisn)
• Wrong pod name and hostname shown in alert CephMonHighNumberOfLeaderChanges (#4665, @anmolsachan)
• Set hostname properly in the CRUSH map for non-portable OSDs on PVCs (#4658, @travisn)
• Update OpenShift example manifest to watch all namespaces for clusters (#4668, @likid0)
• Use min_size defaults set by Ceph instead of overriding with Rook's defaults (#4638, @leseb)
• CSI driver handling of upgrade from OCP 4.2 to OCP 4.3 (#4650, @Madhu-1)
• Add support for the k8s 1.17 failure domain labels (#4626, @BlaineEXE)
• Add option to the cluster CR to continue upgrade even with unclean PGs (#4617, @leseb)
• Add K8s 1.11 back to the integration tests as the minimum version (#4673, @travisn)

YugabyteDB

• Fixed replication factor flag and the master addresses (#4625, @Arnav15)

v1.1.9

Improvements

Rook v1.1.9 is a patch release limited in scope and focusing on bug fixes.

Ceph

• CSI driver handling of upgrade from OCP 4.2 to OCP 4.3 (#4650, @Madhu-1)
• Fix object bucket provisioner when rgw not on port 80 (#4049, @bsperduto)
• Only perform upgrade checks when the Ceph image changes (#4379, @travisn)

v1.2.1

Improvements

Rook v1.2.1 is a patch release limited in scope and focusing on bug fixes.

Ceph

• Add missing env var ROOK_CEPH_MON_HOST for OSDs (#4589, @leseb)
• Avoid logging sensitive info when debug logging is enabled (#4568, @jmolmo)
• Add missing vol mount for encrypted osds (#4583, @leseb)
• Bumping ceph-operator memory limit to 256Mi (#4561, @billimek)
• Fix object bucket provisioner when rgw not on port 80 (#4508, @bsperduto)

v1.2.0

Major Themes

• Security audit completed by Trail of Bits found no major concerns
• Ceph: Added a new "crash collector" daemon to send crash telemetry to the Ceph dashboard, support for priority classes, and a new CephClient resource to create user credentials
• EdgeFS: Added more flexible key-value backends (e.g., Samsung KV-SSDs), "Instant eventual snapshots", and the ability to send large data chunks to AWS S3.

Action Required

If you are running a previous Rook version, please see the corresponding storage provider upgrade guide:

• Ceph
• EdgeFS

Notable Features

• The minimum version of Kubernetes supported by Rook changed from 1.11 to 1.12.
• Discover daemon started by Ceph and EdgeFS storage providers:
  • When the Storage Operator is deleted, the Discover Daemon will also be deleted, as well as its Config Map
  • Device filtering is now configurable for the user by adding an environment variable
    • A new environment variable DISCOVER_DAEMON_UDEV_BLACKLIST is added through which the user can blacklist the devices
    • If no device is specified, the default values will be used to blacklist the devices

Ceph

• The job for detecting the Ceph version can be started with node affinity or tolerations according to the same settings in the Cluster CR as the mons.
• A new CR property skipUpgradeChecks has been added, which allows you force an upgrade by skipping daemon checks. Use this at YOUR OWN RISK, only if you know what you're doing. To understand Rook's upgrade process of Ceph, read the upgrade doc.
• Mon Quorum Disaster Recovery guide has been updated to work with the latest Rook and Ceph release.
• A new CRD property PreservePoolsOnDelete has been added to Filesystem(fs) and Object Store(os) resources in order to increase protection against data loss. If it is set to true, associated pools won't be deleted when the main resource (fs/os) is deleted. Creating again the deleted fs/os with the same name will reuse the preserved pools.
• A new ceph-crashcollector controller has been added for Ceph v14+ that will collect crash telemetry and send it to the Ceph dashboard. These new deployments will run on any node where a Ceph pod is running. Read more about this in the doc
• PriorityClassNames can now be added to the Rook/Ceph components to influence the scheduler's pod preemption.
  • mgr/mon/osd/rbdmirror: priority class names configuration settings
  • filesystem: metadata server settings
  • rgw: gateway settings
  • nfs: samples
• Rook is now able to create and manage Ceph clients client crd.
• The Status.Phase property has been introduced for Rook-Ceph CRDs. The current possible values of status are Processing, Ready and Failed. If the operator is performing any task regarding a Ceph related CR, its status will be reflected as Processing. The Status will be changed to Failed if the operator fails at some task related to the CR and will change to Ready once the Rook-Ceph operator finishes all the tasks related to the CR.
• New CR property available in the Operator: ROOK_UNREACHABLE_NODE_TOLERATION_SECONDS (5 seconds by default). Represents the time to wait until the node controller will move Rook pods to other nodes after detecting an unreachable node.

OSDs

• After the upgrade to v1.2, when the operator is updated to a new release, the OSD pods won't be restarted unless they are running on PVCs.
• Add a new CRD property devicePathFilter to support device filtering with path names, e.g. /dev/disk/by-path/pci-.*-sas-.*.
• Ceph OSD's admin socket is now placed under Ceph's default system location /run/ceph.
• The on-host log directory for OSDs is updated to be <dataDirHostPath>/log/<namespace>, the same as other Ceph daemons.
• Do not generate a config (during pod init) for directory-based or legacy filestore OSDs
• Support PersistentVolume backed by LVM Logical Volume for "OSD on PVC".
• When running an OSD on a PVC and the device is on a slow device class, Rook can adapt to that by tuning the OSD. This can be enabled by the CR setting tuneSlowDeviceClass.

ObjectStore / RGWs

• Ceph Object Gateway are automatically configured to not run on the same host if hostNetwork is activated

EdgeFS

• Rook EdgeFS operator adds support for single node, single device deployments. This is to enable embedded and remote developer use cases.
• Support for new EdgeFS backend, rtkvs, enables ability to operate on top of any key-value capable interface. Initial integration adds support for Samsung KV-SSD devices.
• Enhanced support for running EdgeFS in the AWS cloud. It is now possible to store data payload chunks directly in AWS S3 buckets, thus greatly reducing storage billing cost. Metadata chunks still will be in AWS EBS, thus provide low-latency and high-performance.
• It is now possible to configure ISGW Full-Mesh functionality without the need to create multiple ISGW services. Please read more about ISGW Full-Mesh functionality here.
• EdgeFS now capable of creating instant snapshots of S3 buckets. It supports billion of objects per-bucket use cases. A snapshot's metadata gets distributed among all the connected EdgeFS segments, such that cloning or accessing of snapshotted objects can be done without the need of full-delta transferring, i.e. on-demand.

Breaking Changes

Ceph

• The topology setting has been removed from the CephCluster CR. To configure the OSD topology, node labels must be applied.
  See the OSD topology topic. This setting only affects OSDs when they are first created, thus OSDs will not be impacted during upgrade.
  The topology settings only apply to bluestore OSDs on raw devices. The topology labels are not applied to directory-based OSDs.

Deprecations

Ceph

• Creation of new Filestore OSDs on disks is now deprecated. Filestore is in sustaining mode in Ceph.
  • The storeType storage config setting is now ignored
    • New OSDs created in directories are always Filestore type
    • New OSDs created on disks are always Bluestore type
  • Preexisting disks provisioned as Filestore OSDs will remain as Filestore OSDs
• Rook will no longer automatically remove OSDs if nodes are removed from the cluster CR to avoid the risk of destroying OSDs unintentionally. To remove OSDs manually, see the new doc on OSD Management

v1.1.8

Improvements

Rook v1.1.8 is a patch release limited in scope and focusing on bug fixes.

Ceph

• Continue orchestration on osd update errors to avoid retrying forever (#4418, @travisn)
• Operator crashing on concurrent map read and map write (#4350, @rohantmp)
• Ensure filesystem and object store are upgraded with new Ceph version (#4403, @leseb)
• Clarify log message for Ceph upgrades (#4360, @travisn)
• Ability to disable snapshotter from ceph-csi rbd (#4401, @Madhu-1)
• Update kubernetes CSI sidecar images (#4335, @Madhu-1)
• Update ceph-csi from v1.2.1 to v1.2.2 (#4352, @Madhu-1)
• Add delay between drain switches for managed PDBs (#4346, @rohantmp)

v1.1.7

Improvements

Rook v1.1.7 is a patch release limited in scope and focusing on bug fixes.

Ceph

• Skip osd prepare job creation if osd daemon exists for the pvc (#4277, @sp98)
• Stop osd process more quickly during pod shutdown to reduce IO unresponsiveness (#4328, @travisn)
• Add osd anti-affinity to the example of OSDs on PVCs (#4326, @travisn)
• Properly set app name on the cmdreporter (#4323, @BlaineEXE)
• Ensure disruption draining state is set and checked correctly (#4319, @rohantmp)
• Update LVM filter for OSDs on PVCs (#4312, @leseb)
• Fix topology logic for disruption drains (#4221, @rohantmp)
• Skip restorecon during ceph-volume configuration (#4260, @leseb)
• Added a note around snapshot CRD cleanup (#4302, @mohahmed13)
• Storage utilization alert threshold and timing updated (#4286, @anmolsachan)
• Silence disruption errors if necessary and add missing errors (#4288, @leseb)
• Create csi keys and secrets for external cluster (#4276, @leseb)
• Add retry to ObjectUser creation (#4149, @umangachapagain)