Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pool: add rbd ec pool support in external cluster #9276

Merged
merged 2 commits into from Dec 14, 2021
Merged

pool: add rbd ec pool support in external cluster #9276

merged 2 commits into from Dec 14, 2021

Conversation

subhamkrai
Copy link
Contributor

Adding cli agrument --rbd-metadata-ec-pool-name to read
rbd ec pool name to support ec pool in external cluster
and also updating the json blob.

Signed-off-by: subhamkrai srai@redhat.com

Description of your changes:

Which issue is resolved by this Pull Request:
Resolves #

Checklist:

  • Commit Message Formatting: Commit titles and messages follow guidelines in the developer guide.
  • Skip Tests for Docs: Add the flag for skipping the build if this is only a documentation change. See here for the flag.
  • Skip Unrelated Tests: Add a flag to run tests for a specific storage provider. See test options.
  • Reviewed the developer guide on Submitting a Pull Request
  • Documentation has been updated, if necessary.
  • Unit tests have been added, if necessary.
  • Integration tests have been added, if necessary.
  • Pending release notes updated with breaking and/or notable changes, if necessary.
  • Upgrade from previous release is tested and upgrade user guide is updated, if necessary.
  • Code generation (make codegen) has been run to update object specifications, if necessary.

@subhamkrai subhamkrai marked this pull request as draft November 30, 2021 12:32
@subhamkrai subhamkrai force-pushed the external-rbd-ec branch 2 times, most recently from 9ba9303 to db6c060 Compare November 30, 2021 13:04
@subhamkrai subhamkrai changed the title rbd: add rbd ec pool support in external cluster pool: add rbd ec pool support in external cluster Nov 30, 2021
@subhamkrai subhamkrai force-pushed the external-rbd-ec branch 2 times, most recently from 1fdf2d5 to 9078119 Compare November 30, 2021 14:35
@mergify
Copy link

mergify bot commented Nov 30, 2021

This pull request has merge conflicts that must be resolved before it can be merged. @subhamkrai please rebase it. https://rook.io/docs/rook/latest/development-flow.html#updating-your-fork

@subhamkrai subhamkrai force-pushed the external-rbd-ec branch 3 times, most recently from 250d954 to 2aeba62 Compare November 30, 2021 15:32
@subhamkrai
Copy link
Contributor Author

good for the first round of review..looking at how I can test. I can't test this in our CI as it requires 3 osds.

@subhamkrai subhamkrai force-pushed the external-rbd-ec branch 7 times, most recently from 19c23e2 to 2213cc8 Compare December 7, 2021 07:21
@leseb
Copy link
Member

leseb commented Dec 7, 2021

@subhamkrai what's the latest status of this? Thanks

@subhamkrai
Copy link
Contributor Author

@subhamkrai what's the latest status of this? Thanks

I'm still getting that command not found an error __main__.ExecutionFailureException: 'osd pool ls detail' command failed. Error: command not known

@subhamkrai subhamkrai force-pushed the external-rbd-ec branch 7 times, most recently from a7f58ad to 03dc371 Compare December 10, 2021 10:48
@subhamkrai subhamkrai force-pushed the external-rbd-ec branch 2 times, most recently from f7932a2 to 42cc0c8 Compare December 13, 2021 04:40
@subhamkrai
Copy link
Contributor Author

subhamkrai commented Dec 13, 2021 •
edited

So, now on local cluster with 3 osd and 3 nodes it is working,

python3 create-external-cluster-resources.py --rbd-data-pool-name=ec-pool --rbd-metadata-ec-pool-name=replicated-metadata-pool
[{"name": "rook-ceph-mon-endpoints", "kind": "ConfigMap", "data": {"data": "a=10.99.195.136:6789", "maxMonId": "0", "mapping": "{}"}}, {"name": "rook-ceph-mon", "kind": "Secret", "data": {"admin-secret": "admin-secret", "fsid": "777755e4-a564-4a1f-979c-91f528fbe0e1", "mon-secret": "mon-secret"}}, {"name": "rook-ceph-operator-creds", "kind": "Secret", "data": {"userID": "client.healthchecker", "userKey": "AQAeF7NhtHj3OBAANk50obljVg+AK+I+BSMEow=="}}, {"name": "rook-csi-rbd-node", "kind": "Secret", "data": {"userID": "csi-rbd-node", "userKey": "AQC2D7NhszCfDBAA32cBP3GlWHPtxrdBkK+5VQ=="}}, {"name": "monitoring-endpoint", "kind": "CephCluster", "data": {"MonitoringEndpoint": "10.244.2.10", "MonitoringPort": "9283"}}, {"name": "ceph-rbd", "kind": "StorageClass", "data": {"dataPool": "ec-pool", "pool": "replicated-metadata-pool"}}, {"name": "rook-ceph-dashboard-link", "kind": "Secret", "data": {"userID": "ceph-dashboard-link", "userKey": "https://10.244.2.10:8443/"}}, {"name": "rook-csi-rbd-provisioner", "kind": "Secret", "data": {"userID": "csi-rbd-provisioner", "userKey": "AQC1D7NhVs5GOxAAXtrZKz5eVBlbxh4iiqGOLg=="}}]

@subhamkrai subhamkrai marked this pull request as ready for review December 13, 2021 04:47
@mergify
Copy link

mergify bot commented Dec 13, 2021

This pull request has merge conflicts that must be resolved before it can be merged. @subhamkrai please rebase it. https://rook.io/docs/rook/latest/development-flow.html#updating-your-fork

leseb
leseb requested changes Dec 13, 2021
View reviewed changes
deploy/examples/pool-ec.yaml Show resolved Hide resolved
.github/workflows/canary-integration-test.yml Show resolved Hide resolved
deploy/examples/create-external-cluster-resources.py Outdated Show resolved Hide resolved
deploy/examples/create-external-cluster-resources.py Outdated Show resolved Hide resolved
deploy/examples/create-external-cluster-resources.py Outdated Show resolved Hide resolved
deploy/examples/create-external-cluster-resources.py Outdated Show resolved Hide resolved
deploy/examples/create-external-cluster-resources.py Outdated Show resolved Hide resolved
deploy/examples/create-external-cluster-resources.py Outdated Show resolved Hide resolved
@subhamkrai subhamkrai force-pushed the external-rbd-ec branch 3 times, most recently from 9fb2eae to 834a065 Compare December 14, 2021 06:41
@subhamkrai
pool: add rbd ec pool support in external cluster
2286581 
Adding cli agrument `--rbd-metadata-ec-pool-name` to read
rbd ec pool name to support ec pool in external cluster
and also updating the json blob.

Signed-off-by: subhamkrai <srai@redhat.com>
@subhamkrai
pool: update the failure domain of pool-ec file to host
fc88e4f 
Updating the failure domain of pool-ec file from
osd to host.

Signed-off-by: subhamkrai <srai@redhat.com>
@leseb leseb merged commit ca7d7dd into rook:master Dec 14, 2021
@mergify mergify bot mentioned this pull request Dec 14, 2021
Merged
@subhamkrai subhamkrai deleted the external-rbd-ec branch December 14, 2021 09:04
mergify bot added a commit that referenced this pull request Dec 14, 2021
@mergify
Merge pull request #9421 from rook/mergify/bp/release-1.8/pr-9276
df6167d 
pool: add rbd ec pool support in external cluster (backport #9276)
travisn
travisn reviewed Dec 15, 2021
View reviewed changes
.github/workflows/canary-integration-test.yml
@@ -59,6 +59,11 @@ jobs:
kubectl -n rook-ceph cp deploy/examples/create-external-cluster-resources.py $toolbox:/etc/ceph
timeout 10 sh -c "until kubectl -n rook-ceph exec $toolbox -- python3 /etc/ceph/create-external-cluster-resources.py --rbd-data-pool-name replicapool; do echo 'waiting for script to succeed' && sleep 1; done"

- name: test external script for erasure coded rbd pool
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The CI only creates a single OSD, but for the EC pool we need 3 OSDs. Therefore, any operation that actually requires the pool to be functional will hang since ceph cannot read or write to the pool. With #9363 I will disable this test for now, and separately we will need to see if we can get 3 OSDs running in the CI so this test can be created again.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, but my question is how it is not failing always sometimes both ec and replicapool are created and sometimes not.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When the pool controller creates the pool, there are basically two steps:

  1. Create the pool
  2. Init the pool

The EC pool is always created, but then hangs on initializing the pool. So if the EC pool is the last pool created, everything will appear to work since the initialization isn't critical to the test. But if any other pool hasn't been reconciled yet, it will never get created.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh I undertand now... thanks @travisn

@subhamkrai subhamkrai mentioned this pull request Dec 16, 2021
Merged
10 tasks
subhamkrai added a commit to subhamkrai/ocs-operator that referenced this pull request Dec 16, 2021
@subhamkrai
External: configure sc properties for external cluster EC RBD pool
b0f71c6 
Adding extra field in storageClass "dataPool" when pool is
erasure coded pool in case of RBD.

Links:
1) storageClass https://github.com/rook/rook/blob/master/deploy/examples/csi/rbd/storageclass-ec.yaml#L45
2) Json Blob in case of EC RBD pool rook/rook#9276 (comment)

Signed-off-by subhamkrai <srai@iedhat.com>
@subhamkrai subhamkrai mentioned this pull request Dec 16, 2021
Merged
subhamkrai added a commit to subhamkrai/ocs-operator that referenced this pull request Dec 16, 2021
@subhamkrai
External: configure sc properties for external cluster EC RBD pool
40069ff 
Adding extra field in storageClass "dataPool" when pool is
erasure coded pool in case of RBD.

Links:
1) storageClass https://github.com/rook/rook/blob/master/deploy/examples/csi/rbd/storageclass-ec.yaml#L45
2) Json Blob in case of EC RBD pool rook/rook#9276 (comment)

Signed-off-by: subhamkrai <srai@redhat.com>
subhamkrai added a commit to subhamkrai/ocs-operator that referenced this pull request Dec 21, 2021
@subhamkrai
External: configure sc properties for external cluster EC RBD pool
d8409a3 
Adding extra field in storageClass "dataPool" when pool is
erasure coded pool in case of RBD.

Links:
1) storageClass https://github.com/rook/rook/blob/master/deploy/examples/csi/rbd/storageclass-ec.yaml#L45
2) Json Blob in case of EC RBD pool rook/rook#9276 (comment)

Signed-off-by: subhamkrai <srai@redhat.com>
subhamkrai added a commit to subhamkrai/ocs-operator that referenced this pull request Dec 21, 2021
@subhamkrai
External: configure sc properties for external cluster EC RBD pool
99357d3 
Adding extra field in storageClass "dataPool" when pool is
erasure coded pool in case of RBD.

Links:
1) storageClass https://github.com/rook/rook/blob/master/deploy/examples/csi/rbd/storageclass-ec.yaml#L45
2) Json Blob in case of EC RBD pool rook/rook#9276 (comment)

Signed-off-by: subhamkrai <srai@redhat.com>
subhamkrai added a commit to subhamkrai/ocs-operator that referenced this pull request Dec 21, 2021
@subhamkrai
External: configure sc properties for external cluster EC RBD pool
976eebf 
Adding extra field in storageClass "dataPool" when pool is
erasure coded pool in case of RBD.

Links:
1) storageClass https://github.com/rook/rook/blob/master/deploy/examples/csi/rbd/storageclass-ec.yaml#L45
2) Json Blob in case of EC RBD pool rook/rook#9276 (comment)

Signed-off-by: subhamkrai <srai@redhat.com>
subhamkrai added a commit to subhamkrai/ocs-operator that referenced this pull request Dec 22, 2021
@subhamkrai
External: adding unit test to verify erasure code in external cluster
951032a 
Adding unit test to verify erasure coded pool in case of RBD in external
cluster.

Links:
1) storageClass https://github.com/rook/rook/blob/master/deploy/examples/csi/rbd/storageclass-ec.yaml#L45
2) Json Blob in case of EC RBD pool rook/rook#9276 (comment)

Signed-off-by: subhamkrai <srai@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@travisn travisn travisn left review comments

@leseb leseb leseb approved these changes

@aruniiird aruniiird Awaiting requested review from aruniiird

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@subhamkrai @leseb @travisn