mon: Set stretch tiebreaker reliably during failover #9282
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
leseb
reviewed
Dec 2, 2021
pkg/operator/ceph/cluster/mon/mon.go
Outdated
| logger.Infof("stretch mode is already enabled") | ||
| // only support arbiter failover if at least v16.2.7 | ||
| if !c.ClusterInfo.CephVersion.IsAtLeast(arbiterFailoverSupportedCephVersion) { | ||
| logger.Infof("stretch mode is already enabled") |
There was a problem hiding this comment.
Suggested change
| logger.Infof("stretch mode is already enabled") | |
| logger.Info("stretch mode is already enabled") |
BlaineEXE
reviewed
Dec 2, 2021
Comment on lines
500
to
508
| c.ClusterInfo.CephVersion = cephver.CephVersion{Major: 16, Minor: 2, Extra: 6} | ||
| err := c.ConfigureArbiter() | ||
| assert.NoError(t, err) | ||
| assert.False(t, setNewTiebreaker) | ||
| }) | ||
| t.Run("stretch mode already configured - new", func(t *testing.T) { | ||
| c.ClusterInfo.CephVersion = cephver.CephVersion{Major: 16, Minor: 2, Extra: 7} |
There was a problem hiding this comment.
Is there a specific reason the Ceph version is 16.2.6? Is there a specific reason it changes from 16.2.6 to 16.2.7 between these tests?
There was a problem hiding this comment.
Checking for the current tiebreaker mon requires 16.2.7. "ceph mon dump" results will be different on the older versions. The test isn't very clear about this, i'll update it.
The failover of the arbiter mon in a stretch cluster was sometimes failing due to the new tiebreaker not being set in ceph. Rook would repeatedly try to remove the old tiebreaker mon and keep failing because the new tiebreaker had not been set. Now we make setting the tiebreaker idempotent in case the operator restarts in the middle of the operation or some other corner case causes the expected tiebreaker to be set. In that case, the next reconcile will also ensure the tiebreaker mon is set as expected. Signed-off-by: Travis Nielsen <tnielsen@redhat.com>
travisn
force-pushed
the
failover-arbiter
branch
from
6cd8ad1
to
d556692
Compare
leseb
approved these changes
Dec 3, 2021
This was referenced Dec 3, 2021
mergify bot
added a commit
that referenced
this pull request
Dec 3, 2021
mon: Set stretch tiebreaker reliably during failover (backport #9282)
mergify bot
added a commit
that referenced
this pull request
Dec 3, 2021
mon: Set stretch tiebreaker reliably during failover (backport #9282)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of your changes:
The failover of the arbiter mon in a stretch cluster was sometimes failing due to the new tiebreaker not being set in ceph. Rook would repeatedly try to remove the old tiebreaker mon and keep failing because the new tiebreaker had not been set. Now we make setting the tiebreaker idempotent in case the operator restarts in the middle of the operation or some other corner case causes the expected tiebreaker to be set. In that case, the next reconcile will also ensure the tiebreaker mon is set as expected.
Checklist:
make codegen) has been run to update object specifications, if necessary.