test: test rgw multisite nightly #8931

BlaineEXE · 2021-10-06T19:39:38Z

Description of your changes:

In the nightly job, run the test with the latest Ceph version so we can
detect if there are RGW changes in Ceph that might break multisite. Use
a reusable GitHub action workflow to duplicate as little code as
possible.

Which issue is resolved by this Pull Request:
Resolves #

Checklist:

subhamkrai · 2021-10-07T02:46:07Z

.github/actions/rgw-multisite-test/action.yml

+        minikube version: 'v1.18.1'
+        kubernetes version: 'v1.19.2'


let's use latest version

Suggested change

minikube version: 'v1.18.1'

kubernetes version: 'v1.19.2'

minikube version: 'v1.23.2'

kubernetes version: 'v1.22.2'

These are the versions as copied from the canary.yml test. Should these really be changed?

subhamkrai · 2021-10-07T02:47:27Z

.github/actions/rgw-multisite-test/action.yml

+        cd cluster/examples/kubernetes/ceph/
+        IP_ADDR=$(kubectl -n rook-ceph get svc rook-ceph-rgw-multisite-store -o jsonpath="{.spec.clusterIP}")
+        yq w -i -d1 object-multisite-pull-realm-test.yaml spec.pull.endpoint http://${IP_ADDR}:80
+        BASE64_ACCESS_KEY=$(kubectl -n rook-ceph get secrets realm-a-keys -o jsonpath="{.data.access-key}")
+        BASE64_SECRET_KEY=$(kubectl -n rook-ceph get secrets realm-a-keys -o jsonpath="{.data.secret-key}")
+        sed -i 's/VzFjNFltMVdWRTFJWWxZelZWQT0=/'"$BASE64_ACCESS_KEY"'/g' object-multisite-pull-realm-test.yaml
+        sed -i 's/WVY1MFIxeExkbG84U3pKdlRseEZXVGR3T3k1U1dUSS9KaTFoUVE9PQ==/'"$BASE64_SECRET_KEY"'/g' object-multisite-pull-realm-test.yaml


can we write a new helper function and use it here?

This code isn't reused anywhere else, so I think it's fine here.

subhamkrai · 2021-10-07T02:47:42Z

.github/actions/rgw-multisite-test/action.yml

+        ns_name_primary='"rook-ceph/multisite-store"' # double quotes intended
+        ns_name_secondary='"rook-ceph-secondary/zone-b-multisite-store"' # double quotes intended
+        committed_msg="committing changes to RGW configuration period for CephObjectStore"
+        tests/scripts/github-action-helper.sh verify_operator_log_message "${committed_msg} ${ns_name_primary}"
+        tests/scripts/github-action-helper.sh verify_operator_log_message "${committed_msg} ${ns_name_secondary}"
+        tests/scripts/github-action-helper.sh restart_operator
+        not_committed_msg="there are no changes to commit for RGW configuration period for CephObjectStore"
+        tests/scripts/github-action-helper.sh wait_for_operator_log_message "${not_committed_msg} ${ns_name_primary}" 120
+        tests/scripts/github-action-helper.sh wait_for_operator_log_message "${not_committed_msg} ${ns_name_secondary}" 90


can we write a new helper function and use it here?

Also not reused, so I think it's fine. The things that do get reused and would be useful outside of this (verify_operator_log_message and wait_for_operator_log_message) are already helpers.

.github/workflows/daily-nightly-canary.yml

leseb · 2021-10-07T07:42:02Z

.github/actions/rgw-multisite-test/action.yml

@@ -0,0 +1,101 @@
+name: RGW Multisite Test


Happy to see we start using composite :)

mergify · 2021-10-12T14:26:53Z

This pull request has merge conflicts that must be resolved before it can be merged. @BlaineEXE please rebase it. https://rook.io/docs/rook/latest/development-flow.html#updating-your-fork

leseb · 2021-11-08T17:25:19Z

Looks like #8911 has merged. @BlaineEXE any plan to resume? Thanks!

BlaineEXE · 2021-11-11T01:02:25Z

@subhamkrai do you want to take a look and make sure my usage of the new setup-cluster-resources is correct?

subhamkrai · 2021-11-19T10:16:08Z

.github/workflows/rgw-multisite-test/action.yml

+    - name: wait for ceph cluster 1 to be ready
+      shell: bash --noprofile --norc -eo pipefail -x {0}
+      run: |
+        mkdir test


why create this dir?

I think validate_cluster.sh used to write data into test/. That's probably a carryover from then that I can remove.

subhamkrai · 2021-11-19T10:16:28Z

.github/workflows/rgw-multisite-test/action.yml

+  using: "composite"
+  steps:
+    - name: setup cluster resources
+      uses: ./.github/workflows/setup-cluster-resources


this look good to me

In the nightly job, run the test with the latest Ceph version so we can detect if there are RGW changes in Ceph that might break multisite. Use a reusable GitHub action workflow to duplicate as little code as possible. Signed-off-by: Blaine Gardner <blaine.gardner@redhat.com>

subhamkrai

LGTM!

leseb

Approved with some nits if you want to address them...

leseb · 2021-11-24T10:49:35Z

tests/scripts/github-action-helper.sh

+function replace_ceph_image() {
+  local file="$1"  # parameter 1: the file in which to replace the ceph image
+  local ceph_image="${2:-}"  # parameter 2: the new ceph image to use
+  if [[ -z ${ceph_image} ]]; then


You don't need double brackets when calling -z which is an argument from the test command.

leseb · 2021-11-24T10:50:49Z

tests/scripts/github-action-helper.sh

+  local ceph_image="${2:-}"  # parameter 2: the new ceph image to use
+  if [[ -z ${ceph_image} ]]; then
+    echo "No Ceph image given. Not adjusting manifests."
+    return 0


I believe this is implied but it doesn't hurt to explicitly have it...

BlaineEXE changed the title ~~Test rgw multisite in nightly tests~~ WIP: test: test rgw multisite nightly Oct 6, 2021

BlaineEXE added the do-not-merge DO NOT MERGE :) label Oct 6, 2021

BlaineEXE force-pushed the test-rgw-multisite-in-nightly-tests branch 18 times, most recently from cced94f to 37855ed Compare October 6, 2021 22:47

BlaineEXE changed the title ~~WIP: test: test rgw multisite nightly~~ test: test rgw multisite nightly Oct 6, 2021

BlaineEXE force-pushed the test-rgw-multisite-in-nightly-tests branch 5 times, most recently from fb082d4 to cb6992c Compare October 6, 2021 23:45

BlaineEXE marked this pull request as ready for review October 6, 2021 23:45

BlaineEXE requested review from travisn and leseb and removed request for travisn October 6, 2021 23:45

BlaineEXE requested a review from travisn October 6, 2021 23:46

BlaineEXE added test unit or integration testing and removed do-not-merge DO NOT MERGE :) labels Oct 6, 2021

subhamkrai requested changes Oct 7, 2021

View reviewed changes

leseb reviewed Oct 7, 2021

View reviewed changes

BlaineEXE force-pushed the test-rgw-multisite-in-nightly-tests branch 2 times, most recently from 9b590c3 to a0c2c58 Compare October 7, 2021 18:00

BlaineEXE added the ceph-rgw label Oct 7, 2021

subhamkrai mentioned this pull request Nov 3, 2021

ci: use github composition to reduce duplication #9081

Merged

10 tasks

BlaineEXE force-pushed the test-rgw-multisite-in-nightly-tests branch 3 times, most recently from 6bb7e01 to c4b6007 Compare November 11, 2021 00:59

BlaineEXE requested review from leseb and subhamkrai November 11, 2021 01:00

BlaineEXE force-pushed the test-rgw-multisite-in-nightly-tests branch from c4b6007 to 490e66b Compare November 11, 2021 01:02

BlaineEXE force-pushed the test-rgw-multisite-in-nightly-tests branch 5 times, most recently from e75ab73 to 99cdb03 Compare November 15, 2021 18:37

subhamkrai reviewed Nov 19, 2021

View reviewed changes

BlaineEXE force-pushed the test-rgw-multisite-in-nightly-tests branch from 99cdb03 to e803686 Compare November 19, 2021 16:56

subhamkrai approved these changes Nov 24, 2021

View reviewed changes

leseb approved these changes Nov 24, 2021

View reviewed changes

leseb merged commit ad2c3c2 into rook:master Nov 25, 2021

BlaineEXE deleted the test-rgw-multisite-in-nightly-tests branch November 30, 2021 15:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: test rgw multisite nightly #8931

test: test rgw multisite nightly #8931

BlaineEXE commented Oct 6, 2021 •

edited

subhamkrai Oct 7, 2021

BlaineEXE Oct 7, 2021

subhamkrai Oct 7, 2021

BlaineEXE Oct 7, 2021

subhamkrai Oct 7, 2021

BlaineEXE Oct 7, 2021

leseb Oct 7, 2021

mergify bot commented Oct 12, 2021

leseb commented Nov 8, 2021

BlaineEXE commented Nov 11, 2021

subhamkrai Nov 19, 2021

BlaineEXE Nov 19, 2021

subhamkrai Nov 19, 2021

subhamkrai left a comment

leseb left a comment

leseb Nov 24, 2021

leseb Nov 24, 2021

test: test rgw multisite nightly #8931

test: test rgw multisite nightly #8931

Conversation

BlaineEXE commented Oct 6, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mergify bot commented Oct 12, 2021

leseb commented Nov 8, 2021

BlaineEXE commented Nov 11, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

subhamkrai left a comment

Choose a reason for hiding this comment

leseb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

BlaineEXE commented Oct 6, 2021 •

edited