Migration procedure from flexvolume to ceph-csi

[yanchicago]

Is this a bug report or feature request?

• Bug Report

Deviation from expected behavior:
With ceph-csi support, the field deployment with flexvolume needs to be upgraded/migrated to ceph-csi. There's no procedure available.

Expected behavior:
Procedure document or script to support the migration. A "hacky" procedure in the forum:
scale down replicas

1. create new SC using the same pool
2. point claims to new SC
3. let provisioner to new volumes
   4).delete the provisioner created volume (make sure new PVC are not binded to any apps)
4. rename old volumes to csi-vol-UUID naming
   scale up

How to reproduce it (minimal and precise):

File(s) to submit:

• Cluster CR (custom resource), typically called cluster.yaml, if necessary
• Operator's logs, if necessary
• Crashing pod(s) logs, if necessary

To get logs, use kubectl -n <namespace> logs <pod name>
When pasting logs, always surround them with backticks or use the insert code button from the Github UI.
Read Github documentation if you need help.

Environment:

• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Cloud provider or hardware configuration:
• Rook version (use rook version inside of a Rook Pod):
• Storage backend version (e.g. for ceph do ceph -v):
• Kubernetes version (use kubectl version):
• Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift):
• Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox):

[Place1]

@yanchicago are the steps you've listed the correct way to do the migration?

[hSATAC]

+1 to this. Is there a recommended way to migrate from flexvolume to ceph-csi?

[cann0nf0dder]

+1

[Madhu-1]

a hacky way to migrate flex to CSI (confirmed by user it's working)

* scale down application pods which are using PVC
* Change reclaimpolicy of PVC to retain (as we don't want to delete PV and backend image)
* Delete the PVC ( we still have PV and backend image)
* create new SC using the same pool
* Add a new storage-class name to the PVC
* Create the new PVC with the same name (let the CSI created pvc related metadata in rados omap and create new rbd image in the backend)
*rename old volumes to csi-vol-UUID naming
* As the volume is renamed now delete the old PV
* scale-up application and see things are working fine

more info at https://rook-io.slack.com/archives/C46Q5UC05/p1585819976142800?thread_ts=1585734601.115000&cid=C46Q5UC05

try at own risk :P

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

[travisn]

For the flex and in-tree volume migration to CSI, we need a tool that will help automate the process. Here are some thoughts about how to create the tool.

Goals

For some high level goals I would propose:

• The admin should retain full control of his application pods. The admin owns stopping and starting the application. I don’t believe we want to get in the business of messing directly with the applications.
• The same tool should handle both flex and in-tree driver migration. They are very close in implementation so there is no reason for a separate tool.
• We should keep the design simple with low-cost development effort. The tool must be robust, but let’s avoid other engineering costs like new CI or new images to manage.

Admin Scenario

With those goals in mind, what about the following approach?

1. The admin stops the application (We can provide examples of scaling down a deployment or statefulset, but there are various types of applications that the admin ultimately needs to own stopping the app)
2. The admin starts Rook and the CSI driver (if not already running)
3. The admin creates a storage class for CSI based on the same pool where the rbd volumes are being migrated from
4. The admin connects to the Rook operator pod and executes a command such as the following:

rook migrate-to-csi <pvcName> --namespace <pvcNamespace> --storage-class <csi-sc>

5. The admin starts the application again with the migrated volume

Tool implementation

Include the tool as a command inside the rook operator since:

• The operator already has awareness and configuration for all ceph clusters and can run any ceph command needed
• The operator has context (and RBAC) to call the K8s API
• No new CI or images would be necessary for the tool, we could just use what Rook already has.

If we find new RBAC is needed to access the application PVCs, let's consider instead starting a special migration pod that would have only the RBAC needed.

The implementation of the tool would include these steps based on this example command:

rook migrate-to-csi <pvcName> --namespace <pvcNamespace> --storage-class <csi-sc>

1. Change the reclaim policy of the PVC to retain
2. Delete the PVC
3. Create a new PVC based on the new CSI storage class
4. Detect the rbd image name of the new PV
5. Delete the new rbd image
6. Rename the original rbd image to the new image name
7. Delete the old PV

@subhamkrai @Yuggupta27 @BlaineEXE @leseb @Madhu-1 thoughts?

[Yuggupta27]

Thanks @travisn for capturing all the points :)

For the flex and in-tree volume migration to CSI, we need a tool that will help automate the process. Here are some thoughts about how to create the tool.

Goals

For some high level goals I would propose:

• The admin should retain full control of his application pods. The admin owns stopping and starting the application. I don’t believe we want to get in the business of messing directly with the applications.
• The same tool should handle both flex and in-tree driver migration. They are very close in implementation so there is no reason for a separate tool.
• We should keep the design simple with low-cost development effort. The tool must be robust, but let’s avoid other engineering costs like new CI or new images to manage.

Sure, sounds good. Scaling down operation should be done by admins before migration, and migration tool should relay the message to the admin about the completion of the migration process, so that they can scale the apps back up.

Admin Scenario

With those goals in mind, what about the following approach?

1. The admin stops the application (We can provide examples of scaling down a deployment or statefulset, but there are various types of applications that the admin ultimately needs to own stopping the app)

2. The admin starts Rook and the CSI driver (if not already running)
3. The admin creates a storage class for CSI based on the same pool where the rbd volumes are being migrated from
4. The admin connects to the Rook operator pod and executes a command such as the following:

rook migrate-to-csi <pvcName> --namespace <pvcNamespace> --storage-class <csi-sc>

1. The admin starts the application again with the migrated volume

Just to emphasize, this process comes with downtime involved, but the goal of the tool will be to minimize the downtime as much as possible in comparison to if the migration would have been done manually.

Tool implementation

Include the tool as a command inside the rook operator since:

• The operator already has awareness and configuration for all ceph clusters and can run any ceph command needed
• The operator has context (and RBAC) to call the K8s API
• No new CI or images would be necessary for the tool, we could just use what Rook already has.

If we find new RBAC is needed to access the application PVCs, let's consider instead starting a special migration pod that would have only the RBAC needed.

The implementation of the tool would include these steps based on this example command:

rook migrate-to-csi <pvcName> --namespace <pvcNamespace> --storage-class <csi-sc>

We can also migrate all the PVCs in a cluster by passing an '--all' flag; so that it saves much time for complete migration(can be a patch later)

1. Change the reclaim policy of the PVC to retain
2. Delete the PVC
3. Create a new PVC based on the new CSI storage class
4. Detect the rbd image name of the new PV
5. Delete the new rbd image
6. Rename the original rbd image to the new image name
7. Delete the old PV

Just a note, The PVC created in ceph-csi should be of the same name as PVC created by flex/in-tree, so that there is no change in the app spec; and we can just scale everything up again.

[Madhu-1]

For the flex and in-tree volume migration to CSI, we need a tool that will help automate the process. Here are some thoughts about how to create the tool.

Goals

For some high level goals I would propose:

• The admin should retain full control of his application pods. The admin owns stopping and starting the application. I don’t believe we want to get in the business of messing directly with the applications.
• The same tool should handle both flex and in-tree driver migration. They are very close in implementation so there is no reason for a separate tool.
• We should keep the design simple with low-cost development effort. The tool must be robust, but let’s avoid other engineering costs like new CI or new images to manage.

Admin Scenario

With those goals in mind, what about the following approach?

1. The admin stops the application (We can provide examples of scaling down a deployment or statefulset, but there are various types of applications that the admin ultimately needs to own stopping the app)
2. The admin starts Rook and the CSI driver (if not already running)
3. The admin creates a storage class for CSI based on the same pool where the rbd volumes are being migrated from
4. The admin connects to the Rook operator pod and executes a command such as the following:

rook migrate-to-csi <pvcName> --namespace <pvcNamespace> --storage-class <csi-sc>

Instead of individual PVC's how about source storageclass (intree/felx) to destination storageclass?

migrate-to-csi --src-storage-class <intree/flex sc> --dest-storage-class <csi-sc>

1. The admin starts the application again with the migrated volume

Tool implementation

Include the tool as a command inside the rook operator since:

• The operator already has awareness and configuration for all ceph clusters and can run any ceph command needed
• The operator has context (and RBAC) to call the K8s API
• No new CI or images would be necessary for the tool, we could just use what Rook already has.

If we find new RBAC is needed to access the application PVCs, let's consider instead starting a special migration pod that would have only the RBAC needed.

As we are doing for both intree and flex, this can be a generic tool that can be used outside of rook? can this go to ceph project?

The implementation of the tool would include these steps based on this example command:

rook migrate-to-csi <pvcName> --namespace <pvcNamespace> --storage-class <csi-sc>

We need to check if PVC is in use to avoid user mistakes by checking the volume attachment object to make sure PVC is not in use?

1. Change the reclaim policy of the PVC to retain
2. Delete the PVC
3. Create a new PVC based on the new CSI storage class
4. Detect the rbd image name of the new PV
5. Delete the new rbd image
6. Rename the original rbd image to the new image name
7. Delete the old PV

@subhamkrai @Yuggupta27 @BlaineEXE @leseb @Madhu-1 thoughts?

[travisn]

rook migrate-to-csi <pvcName> --namespace <pvcNamespace> --storage-class <csi-sc>

Instead of individual PVC's how about source storageclass (intree/felx) to destination storageclass?

migrate-to-csi --src-storage-class <intree/flex sc> --dest-storage-class <csi-sc>

This option means that the admin must stop all their application pods, run the conversion, then start all the applications again. I would think in a production cluster the admin would want to do it more fine-grained so all applications don't need to be stopped at the same time. But I could certainly see that we could have several different options here for flexibility, such as 1) migrate a specific pvc, 2) migrate all pvcs in a namespace, 3) migrate all pvcs from one storage class to another.

Tool implementation

Include the tool as a command inside the rook operator since:

• The operator already has awareness and configuration for all ceph clusters and can run any ceph command needed
• The operator has context (and RBAC) to call the K8s API
• No new CI or images would be necessary for the tool, we could just use what Rook already has.

If we find new RBAC is needed to access the application PVCs, let's consider instead starting a special migration pod that would have only the RBAC needed.

As we are doing for both intree and flex, this can be a generic tool that can be used outside of rook? can this go to ceph project?

My main goal with this tool is that we don't need new CI or new images to manage. So if we're writing go code, it probably either needs to be in rook or the csi driver. Rook is a natural place for all the K8s implementation related to Ceph, but can you expand on how you see it needed in the ceph repo?

The implementation of the tool would include these steps based on this example command:

rook migrate-to-csi <pvcName> --namespace <pvcNamespace> --storage-class <csi-sc>

We need to check if PVC is in use to avoid user mistakes by checking the volume attachment object to make sure PVC is not in use?

agreed

[BlaineEXE]

My main goal with this tool is that we don't need new CI or new images to manage. So if we're writing go code, it probably either needs to be in rook or the csi driver. Rook is a natural place for all the K8s implementation related to Ceph, but can you expand on how you see it needed in the ceph repo?

I think the concern about standing up new CI is a good one, but I don't see why we would need to include this tool in an image. This could easily be a standalone go binary users download and run on their machines. Our Rook images are already huge.

If the tool is created in a different repo, it would be easy for the tool to use github.com/rook/rook/pkg code as a library, so I don't think it's necessary to create this tool in the Rook repo. While standing up new CI is not ideal, I think it could be pretty easy to use the primitives Rook has established under github.com/rook/rook/tests as a library to help new CI get going more quickly.

I think I would prefer to see the tool in a separate repo (under github.com/ceph or under github.com/rook is acceptable to me), especially given that it is partly intended to migrate Kubernetes in-tree RBD/cephfs claims to CSI claims. This concern is purely a Ceph concern, and having CI for these cases in Rook is unnecessary and adds CI bloat.

For conversions that Rook cares about, I think it's also valid to have a test case in Rook's CI that downloads the latest version of the external tool and validates the Flex->CSI conversions that Rook cares about in a way that is integrated with the Rook versions we care about.

---

Separately...

Will we give users one Rook release to migrate their PVCs before deprecating the FlexVolume driver? To me it is acceptable that users have to do the migration while they are on Rook v1.8 (1.7?), and then in v1.9 (1.8?) FlexVolume is removed from Rook.

[travisn]

I think the concern about standing up new CI is a good one, but I don't see why we would need to include this tool in an image. This could easily be a standalone go binary users download and run on their machines. Our Rook images are already huge.

If the tool is created in a different repo, it would be easy for the tool to use github.com/rook/rook/pkg code as a library, so I don't think it's necessary to create this tool in the Rook repo. While standing up new CI is not ideal, I think it could be pretty easy to use the primitives Rook has established under github.com/rook/rook/tests as a library to help new CI get going more quickly.

The tool needs access to the K8s api so it needs to be run inside the cluster. If you're downloading a tool, are you curling it into a container such as the toolbox?

Adding another go package to Rook isn't going to change the size of the image, or at least it will be insignificant. I don't see size or bloat as an issue here. I'm looking for engineering efficiency for a short-term tool.

I think I would prefer to see the tool in a separate repo (under github.com/ceph or under github.com/rook is acceptable to me), especially given that it is partly intended to migrate Kubernetes in-tree RBD/cephfs claims to CSI claims. This concern is purely a Ceph concern, and having CI for these cases in Rook is unnecessary and adds CI bloat.

Rook is the place we want people to come for running the CSI driver in K8s. Since we're migrating K8s volumes and you need the CSI driver running, I disagree that the tool should be elsewhere.

Will we give users one Rook release to migrate their PVCs before deprecating the FlexVolume driver? To me it is acceptable that users have to do the migration while they are on Rook v1.8 (1.7?), and then in v1.9 (1.8?) FlexVolume is removed from Rook.

The proposal is that the migration would need to be done on Rook v1.7, and the flex driver would be removed from Rook in v1.8. Clarifications added to #8076 and #8084.

[BlaineEXE]

The tool needs access to the K8s api so it needs to be run inside the cluster. If you're downloading a tool, are you curling it into a container such as the toolbox?

I don't know where that assertion is coming from. It's not necessary to run the utility inside the k8s cluster. It just needs network access to the API server. Kubectl, for example, doesn't need to be run inside the k8s cluster. And the k8s golang APIs make it very easy to create a client based on a kubeconfig file and the like.

Rook is the place we want people to come for running the CSI driver in K8s. Since we're migrating K8s volumes and you need the CSI driver running, I disagree that the tool should be elsewhere.

The CSI driver is independent of Rook, and users can deploy it manually without Rook. We (in the Rook project) want this and want to think that this revolves around us, but that isn't the reality of how things are set up. The Ceph CSI driver is a Ceph project.

I still think that the longest-lived use of the tool will be to migrate from in-tree drivers to Ceph-CSI, which is totally independent of Rook. The use case to migrate from Rook FlexVolume to Ceph-CSI will only be applicable for Rook v1.7. Quite a while after we stop caring about the tool in Rook, users will likely still be using it to migrate in-tree to Ceph-CSI.

There are no technical limitations to having a CLI tool that is not run in a k8s pod, and at least half of the functionality and a larger length of the lifetime of this tool will be outside of Rook's domain, so I don't believe it should be part of the primary Rook repo.

[BlaineEXE]

To put it another way, if we make this a separate repo, we don't have to triage and manage the GH issues and pull requests for the tool in this repository, which will not create extra work on the Rook maintainers.

[travisn]

In addition to the K8s api (which you're right could be used externally with the kubeconfig), I failed to point out that the tool also needs to run ceph commands and have that ceph context setup to access the ceph daemons. So access to the ceph tools and ceph context is the core reason the tool would run inside a container in the cluster.

If the tool is written in the ceph-csi repo I could see it working equally as well as if it were in the rook repo. I'd like to hear from the ceph-csi owners where it should live between the two repos. But creating a new repo still doesn't seem like the right approach for this small tool.

[Madhu-1]

The tool needs to make both k8s API calls and ceph calls. we can ship only the binary/binary in the cephcsi container. we can document the pre-req to run the tool like (kubeconfig, rbd/ceph libraries, etc).

Am fine either having a new repo in Ceph or including it in the cephcsi repo itself.

[Madhu-1]

CC @nixpanic @humblec

[travisn]

At least for the flex driver conversion, I don't see that we can ask the user to install ceph tools or assume access to the cluster. A container with all the tools will just solve that issue for us.

[humblec]

Rook was/is shipping the flex driver and it want a solution/tool/utility which helps on migrating earlier rook flex driver PVC to new CSI based driver PVCs. Thats why this issue exist and discussions around it. So, Rook is the correct place to host the tool. in tree migration came in as an addon to this tool. There is no harm to extend the addon/capability of intree migration in the utility to support those too ( additionally kube binary itself may have its migration path from intree to csi) , even we can have a disclaimer about the other migration path ie in-tree to csi in the utility if Rook does not care much about it or want to avoid supporting users came across the issues on the same..etc. But the latter use case should not be a blocker to host the utility here in this repo considering we are aiming Flex driver users which is part of Rook and flex driver migration tied to rook release ...etc. With that, imo, its good to host the utility here in this repo as the primary intention of the utility is supporting flex driver migration which got shipped with Rook. Also most of the users (95%) are hosting CSI with Rook, so this repo looks to be the correct place. Having a seperate project under ceph org is also viable, but imo, its an overkill too.

[subhamkrai]

@travisn now I think we close this issue...I'll just add doc pr link to this.