[RGW MS] Baremetal multisite config and suite changes due to disk space

[anrao19]

Description

Cluster details as of now
Extensa has all hdd disks (4 hdd disk in all 4 nodes with 3.6TB size each) total of 58TB size
Clara has all ssd disks (3 ssd disk in all 4 nodes with 218GB size each ) total of 2.6 TB size
Decided to shuffle clara and extensa nodes on primary and secondary clusters evenly based on disk size.
Decided to add clara001 and extensa001 nodes to multisite configuration.

click to expand checklist

• Create a test case in Polarion reviewed and approved.
• Create a design/automation approach doc. Optional for tests with similar tests already automated.
• Review the automation design
• Implement the test script and perform test runs
• Submit PR for code review and approve
• Update Polarion Test with Automation script details and update automation fields
• If automation is part of Close loop, update BZ flag qe-test_coverage “+” and link Polarion test

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: anrao19

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mkasturi18]

Looks good

[github-actions]

This Pull request has been automatically marked as STALE due to inactivity for 15 days and will be CLOSED on further inactivity on the PR for another 15 days.