Skip to content

Commit

Permalink
Fixes Readiness Check for statefulsets using partitioned rolling upda…
Browse files Browse the repository at this point in the history
…te. (#11774)

* Fixes Readiness Check for statefulsets using partitioned rolling update.
Fixes #11773

This change updates readiness check in ready.go to correctly
account for statefulsets that are utilizing a partitioned upgrade.
These statefulsets only upgrade a subset of the managed pods with each call
to helm upgrade. This causes the upgrade to legitimately hit the condition where
sts.status.CurrentRevision != sts.Status.UpdateRevision which causes helm to mark
the upgrade has failed when in fact it is successful.

This change fixes that behavior to only check when partition is unspecified or 0.

Signed-off-by: Aman Nijhawan <anijhawan@yugabyte.com>

* Adding a unit test to verify that partitioned rolling upgrade for a statefulset works.

Signed-off-by: Aman Nijhawan <anijhawan@yugabyte.com>

---------

Signed-off-by: Aman Nijhawan <anijhawan@yugabyte.com>
Co-authored-by: Aman Nijhawan <anijhawan@yugabyte.com>
  • Loading branch information
amannijhawan and amannijhawan committed Mar 30, 2023
1 parent 952708b commit eea2f27
Show file tree
Hide file tree
Showing 2 changed files with 11 additions and 2 deletions.
6 changes: 4 additions & 2 deletions pkg/kube/ready.go
Expand Up @@ -393,8 +393,10 @@ func (c *ReadyChecker) statefulSetReady(sts *appsv1.StatefulSet) bool {
c.log("StatefulSet is not ready: %s/%s. %d out of %d expected pods are ready", sts.Namespace, sts.Name, sts.Status.ReadyReplicas, replicas)
return false
}

if sts.Status.CurrentRevision != sts.Status.UpdateRevision {
// This check only makes sense when all partitions are being upgraded otherwise during a
// partioned rolling upgrade, this condition will never evaluate to true, leading to
// error.
if partition == 0 && sts.Status.CurrentRevision != sts.Status.UpdateRevision {
c.log("StatefulSet is not ready: %s/%s. currentRevision %s does not yet match updateRevision %s", sts.Namespace, sts.Name, sts.Status.CurrentRevision, sts.Status.UpdateRevision)
return false
}
Expand Down
7 changes: 7 additions & 0 deletions pkg/kube/ready_test.go
Expand Up @@ -189,6 +189,13 @@ func Test_ReadyChecker_statefulSetReady(t *testing.T) {
},
want: false,
},
{
name: "statefulset is ready when current revision for current replicas does not match update revision for updated replicas when using partition !=0",
args: args{
sts: newStatefulSetWithUpdateRevision("foo", 3, 2, 3, 3, "foo-bbbbbbb"),
},
want: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
Expand Down

0 comments on commit eea2f27

Please sign in to comment.