-
Notifications
You must be signed in to change notification settings - Fork 596
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[cinder-csi-plugin] Multi region/clouds support for controllerServer #2551
base: master
Are you sure you want to change the base?
Conversation
Welcome @MatthieuFin! |
Hi @MatthieuFin. Thanks for your PR. I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
As mentioned here and after some local tests it seems to be the proper implementation way. I propose to avoid usage of storageClass |
/ok-to-test Sorry, I won't have time to take a closer look before my vacations, will be back in 2 weeks. @kayrus might be the proper person to check this PR out. |
Looks like the tests have to be fixed here. |
/retest |
/retest |
Hi, My issue concern topology keys, currently only available is topology.cinder.csi.openstack.org/zone and obviously my provider OVH use same zone name (nova) on each of their openstack cluster, so I need to add possibility to manage another key to differentiate them. |
70b257c
to
33aec86
Compare
we have exactly the same problem (several openstack with same availability zone name) I'm interesseted by this feature let me know if you need some test feedback regards |
…ubernetes#2035) * feat(cinder-csi-plugin-controller): add support --cloud-name arg which permit to manage multiple config Global sections * feat(cinder-csi-plugin-controller): add support of key `cloud` from secret specified in storageClass to reference specific config Global section * feat(cinder-csi-plugin-node): add support --cloud-name arg which permit to specify one of config Global section Signed-off-by: MatthieuFin <matthieu2717@gmail.com>
…ubernetes#2035) * feat(cinder-csi-plugin-node): Additionnal topology keys `--additionnal-topology` are announced on NodeGetInfo to create proper CSINode object and could used in storage class allowedTopologies field, in combination with csi-provisioner `--feature-gates Topology=true` created PV will have nodeAffinity set with topologies presents in storageClass allowedTopologies and in `--additionnal-topology` argument. Signed-off-by: MatthieuFin <matthieu2717@gmail.com>
Signed-off-by: MatthieuFin <matthieu2717@gmail.com>
e4e424e
to
5861f43
Compare
add documentation and examples about multi cloud support configuration Co-authored-by: sebastienmusso <smusso@lfdj.com> Co-authored-by: FlorentLaloup <f.laloup@qwant.com> Signed-off-by: MatthieuFin <matthieu2717@gmail.com>
Thank you for this PR, very interesting idea. 👍 So the complexity of this PR is that node-pluging needs openstack api credentials for ephemeral storage which is deprecated. Perhaps we should ask about the removal process for it? The killer feature is to use one name of storageClass in stetefulset deployment. Base on nodAffinity/podAntiAffinity kubernetes scheduler spread the pods across the regions/zones. I quit to using allowedTopologies in storageClass they are immutable. It is very hard to maintain. It's better to use nodeSelector/nodeAffinity instead; you can change these parameters during the lifetime. To help kubernetes scheduler to choose the best region/zone (if not set) we need to expose free disk capacity. |
I've added a capacity capabilities - #2597 |
I don't know how other cloud are handling this kind of cross region (e.g AWS cloud provider handle this?) maybe someone can help comment and I knew openstack itself handling multiple cloud (region) is that not good so overall I think we may easily focus on compute (VM) provisioning and claim others as gap/follow up? |
Hi @jichenjc , Technically our openstack implementation doesn't support We can implement support of Indeed, GCP or AWS could implement that because all regions are uniformly managed by cloud API, so when you ask for a volume_id in DeleteVolumeRequest for example (same for others calls) to your cloud API, you have got response never mind in which region is the volume. In our case, manage multi openstack region is the same approach than manage multiple openstack clusters. So I have many n clouds api to n regions, where in GCP you have n region for only 1 cloud API. The "GCP/AWS like" behavior is probably to manage "regions" with AZ zone notion in openstack. But that is not my use case, I need to be able to support multiple openstack clusters not multiple AZ in one openstack cluster. And I don't know public cloud provider based on openstack which properly implement openstack availability zones... Moreover if we wanna handle multi clouds (openstack regions) with the help of
Personally these 2 solution really to complex to implement and maintain. This PR permit to do it easier way to understand and maintain imo. I have a storageClass per regions/clouds. and if i wanna deploy 1 sts cross 3 regions I create 3 pvc one per regions (each with a different SC: spec.volumeClaimTemplates:
- apiVersion: v1
kind:
name: data In that way I'm able to dispatch pods from same sts cross different sc and different openstack cluster. If you wanna a "multi-region" implementation like GCP or AWS
From my limited experience as a cloud consumer, openstack providers offer openstacks with multiple regions and not with multiple AZs. It's sad I know, but it seems to be the public offerings. Concerning network cross region indeed it shouldn't have to be managed by neutron, personally i managed it by myself out of openstack with some BGP routing and physical interconnections L2 (dark fiber) cross my DCs and ipsec tunnels for dev/poc links cross different DCs not physically linked (I shared my firsts reflections about it here but that's out of this PR scope). I know that @sebastienmusso run k8s clusters spread on multiples openstack clusters too, I don't know how you handle this point ? Implement capacity capabilities as proposed by @sergelogvinov could probably help in multi AZ environment but in this PR context it should probably not have impact. |
+1 I think we do not break anything with So we need to decide what the label we need to use I prefer to use well-known label |
Affected binaries:
What this PR does / why we need it:
Enable multi region / multi clouds support on cinder-csi-plugin controllerServer.
Which issue this PR fixes(if applicable):
fixes #2035
Special notes for reviewers:
I have a kubernetes cluster stretch on multiples openstack clusters (3 OVH and 1 onPremise linked by a private dark fiber).
That why my approche is more "multi clouds" than only "multi region"
I don't touch nodeServer behavior (I simply deploy a Daemonset with a nodeSelector on a label whitch select nodes based on their hypervisor, and mount a dedicated secret with openstack creds associated)
The purpose is on controllerServer which handle all kube requests to manage pvc, he had be able to CRUD volumes on all managed openstack cluster.
I propose to use gcfg subsection feature to be abble to handle multiple "sections"
Global
in config files. This permit to be backaward compatible with configfiles syntax.I choose to use StorageClass
parameters
to deal with cloud selection (i add an optionnal fieldcloud
which containt config Global subsection name). In this way when a createVolumeRequest income controllerServer could identify OSInstance to used based on match between SC fieldparametes.cloud
and his config Global subsections.Issue
This PR is currently a MVP, which permit to create volumes and attach them to correct VM / OpenStack cluster, but i have a bit issue to troubleshoot, indeed all csi spec message don't contains
volume_context
orparameters
, especially for DeleteVolumeRequest which only contain volumeID.I have 2 ideas to troubleshoot this issue:
Release note: