OCPBUGS-33181: Fixed audit-logs sigterm failing to terminate gracefully #3972

Joseph-Goergen · 2024-05-01T21:42:02Z

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, use fixes #<issue_number>(, fixes #<issue_number>, ...) format, where issue_number might be a GitHub issue, or a Jira story:
Fixes #

Checklist

Subject and description added to both, commit and PR.
Relevant issues have been referenced.
This change includes docs.
This change includes unit tests.

openshift-ci-robot · 2024-05-01T21:42:07Z

@Joseph-Goergen: This pull request references Jira Issue OCPBUGS-33181, which is invalid:

expected the bug to target either version "4.16." or "openshift-4.16.", but it targets "4.15" instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, use fixes #<issue_number>(, fixes #<issue_number>, ...) format, where issue_number might be a GitHub issue, or a Jira story:
Fixes #

Checklist

Subject and description added to both, commit and PR.

Relevant issues have been referenced.

This change includes docs.

This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

rtheis · 2024-05-02T08:53:43Z

control-plane-operator/controllers/hostedcontrolplane/kas/deployment.go

@@ -967,3 +966,12 @@ func buildKonnectivityVolumeClusterCerts(v *corev1.Volume) {
 		DefaultMode: pointer.Int32(0640),
 	}
 }
+
+func renderAuditLogScript(auditLogFilePath string) string {
+	var script = `#!/bin/bash


Why are you starting a new script when the command calling this is /bin/bash? Do you really need #!/bin/bash?

Probably not, I was mainly just following what others do when calling /bin/bash -clike

hypershift/contrib/ci/hypershift-ci-1.yaml

Lines 111 to 119 in 85148d5

- args:

- -c

- |

#!/bin/bash

set -euo pipefail

/usr/bin/oc get istag -n hypershift hypershift-operator:latest -ojsonpath='{.image.dockerImageReference}' > /installer-data/image-ref

command:

- /bin/bash

. There's quite a few more examples throughout the hypershift operator, so I thought it was common place for this repo.

But in this case, the previous code doesn't do that. So unless there's a good reason, please continue the pattern of the previous audit container setup.

Previously, it was calling tail directly, so it couldn't have - if we're going to use bash it seems prudent to continue with euo pipefail - it will reduce the chance we introduce any of the common bugs those guard against.

jeffnowicki · 2024-05-02T17:54:44Z

/jira refresh

openshift-ci-robot · 2024-05-02T17:54:50Z

@jeffnowicki: This pull request references Jira Issue OCPBUGS-33181, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.16.0) matches configured target version for branch (4.16.0)
bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

stevekuznetsov · 2024-05-05T22:14:53Z

control-plane-operator/controllers/hostedcontrolplane/kas/deployment.go

+
+func renderAuditLogScript(auditLogFilePath string) string {
+	var script = `#!/bin/bash
+trap exit SIGTERM


Are you relying on implicit behavior of bash when you call exit? Could we instead explicitly forward SIGTERM to children on SIGINT?

@stevekuznetsov is this what you'd like Joe to do: https://unix.stackexchange.com/questions/146756/forward-sigterm-to-child-in-bash?

#!/usr/bin/env bash set -o errexit set -o nounset set -o pipefail function cleanup() { for child in $( jobs -p ); do kill "${child}" done wait } trap cleanup EXIT for (( i = 0; i < 999; i++ )); do child.sh $i wait $! done

Something like this would work

@Joseph-Goergen ptal

Updated and tested(using a 4.15 release branch)

stevekuznetsov · 2024-05-15T14:24:51Z

control-plane-operator/controllers/hostedcontrolplane/kas/deployment.go

@@ -967,3 +966,23 @@ func buildKonnectivityVolumeClusterCerts(v *corev1.Volume) {
 		DefaultMode: pointer.Int32(0640),
 	}
 }
+
+func renderAuditLogScript(auditLogFilePath string) string {


Could we please export the function and re-use it in the OAPI deployment?

stevekuznetsov

/lgtm

Joseph-Goergen · 2024-05-15T16:10:24Z

@stevekuznetsov
If/when this merges, the pr for #3994 may have to rebase (if this gets in before it) and use the RenderAuditLogScript function for the audit-logs container. Or if that PR merges, I'll have to reconfigure this.

Joseph-Goergen · 2024-05-15T18:14:11Z

/retest-required

rtheis · 2024-05-16T00:56:38Z

/retest-required

rtheis · 2024-05-16T09:48:57Z

/retest-required

rtheis · 2024-05-16T11:59:29Z

/test e2e-kubevirt-aws-ovn

rtheis · 2024-05-16T13:22:14Z

control-plane-operator/controllers/hostedcontrolplane/kas/deployment.go

+	done
+	wait
+}
+trap cleanup EXIT


I tested this and it doesn't work unless EXIT is SIGTERM.

@rtheis I tested this using the openshift-apiserver

# time kubectl delete pod -n master-coskjif10c6fhqphi0hg openshift-apiserver-55476ddfc-h4ldv pod "openshift-apiserver-55476ddfc-h4ldv" deleted real 1m47.992s user 0m0.273s sys 0m0.109s

@Joseph-Goergen my point exactly, it shouldn't take 1 minute 47 seconds to fully terminate the pod.

rtheis · 2024-05-20T10:23:37Z

/retest-required

Joseph-Goergen · 2024-05-20T14:34:16Z

/test e2e-kubevirt-aws-ovn

Joseph-Goergen · 2024-05-20T17:54:43Z

/test e2e-kubevirt-aws-ovn

rtheis

/lgtm

Joseph-Goergen · 2024-05-20T20:37:45Z

/retest-required

rtheis · 2024-05-21T10:29:34Z

/retest-required

rtheis · 2024-05-21T12:39:43Z

/test e2e-kubevirt-aws-ovn

Joseph-Goergen · 2024-05-21T19:06:38Z

@stevekuznetsov May I get a re-review? I had to rebase

sjenning · 2024-05-22T18:36:43Z

/approve

openshift-ci · 2024-05-22T18:37:06Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Joseph-Goergen, rtheis, sjenning, stevekuznetsov

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [sjenning]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot · 2024-05-22T20:16:25Z

/retest-required

Remaining retests: 0 against base HEAD 0d10c82 and 2 for PR HEAD 82df7b3 in total

openshift-ci-robot · 2024-05-23T01:21:39Z

/retest-required

Remaining retests: 0 against base HEAD b5576e4 and 1 for PR HEAD 82df7b3 in total

openshift-ci-robot · 2024-05-23T08:37:43Z

/retest-required

Remaining retests: 0 against base HEAD 4148e4d and 0 for PR HEAD 82df7b3 in total

openshift-ci · 2024-05-23T12:46:52Z

@Joseph-Goergen: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-azure	`82df7b3`	link	false	`/test e2e-azure`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

rtheis · 2024-05-23T12:52:09Z

/test e2e-kubevirt-aws-ovn

openshift-ci-robot · 2024-05-23T22:54:26Z

@Joseph-Goergen: Jira Issue OCPBUGS-33181: All pull requests linked via external trackers have merged:

openshift/hypershift#3972

Jira Issue OCPBUGS-33181 has been moved to the MODIFIED state.

In response to this:

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, use fixes #<issue_number>(, fixes #<issue_number>, ...) format, where issue_number might be a GitHub issue, or a Jira story:
Fixes #

Checklist

Subject and description added to both, commit and PR.

Relevant issues have been referenced.

This change includes docs.

This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-bot · 2024-05-24T06:44:47Z

[ART PR BUILD NOTIFIER]

This PR has been included in build ose-hypershift-container-v4.17.0-202405240410.p0.g239a333.assembly.stream.el9 for distgit hypershift.
All builds following this will include this PR.

Joseph-Goergen · 2024-05-24T13:23:50Z

/cherry-pick release-4.15

openshift-cherrypick-robot · 2024-05-24T13:24:34Z

@Joseph-Goergen: new pull request created: #4089

In response to this:

/cherry-pick release-4.15

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

rtheis · 2024-05-24T13:27:09Z

/cherry-pick release-4.16

openshift-cherrypick-robot · 2024-05-24T13:27:52Z

@rtheis: new pull request created: #4090

In response to this:

/cherry-pick release-4.16

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels May 1, 2024

openshift-ci bot added the do-not-merge/needs-area label May 1, 2024

openshift-ci bot requested review from csrwng and hasueki May 1, 2024 21:42

openshift-ci bot added area/control-plane-operator Indicates the PR includes changes for the control plane operator - in an OCP release and removed do-not-merge/needs-area labels May 1, 2024

rtheis reviewed May 2, 2024

View reviewed changes

openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels May 2, 2024

stevekuznetsov reviewed May 5, 2024

View reviewed changes

Joseph-Goergen force-pushed the fix-audit-logs-SIGTERM-master branch from 97c40ae to 3ce718a Compare May 15, 2024 13:41

stevekuznetsov reviewed May 15, 2024

View reviewed changes

Joseph-Goergen force-pushed the fix-audit-logs-SIGTERM-master branch 3 times, most recently from cafc72c to 5052adc Compare May 15, 2024 15:46

stevekuznetsov approved these changes May 15, 2024

View reviewed changes

openshift-ci bot assigned stevekuznetsov May 15, 2024

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label May 15, 2024

rtheis suggested changes May 16, 2024

View reviewed changes

openshift-ci bot assigned rtheis May 16, 2024

openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label May 16, 2024

Fixed audit-logs sigterm failing to kill

82df7b3

Joseph-Goergen force-pushed the fix-audit-logs-SIGTERM-master branch from c918f83 to 82df7b3 Compare May 20, 2024 18:31

openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label May 20, 2024

rtheis approved these changes May 20, 2024

View reviewed changes

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label May 20, 2024

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 22, 2024

openshift-merge-bot bot merged commit 239a333 into openshift:main May 23, 2024
12 of 13 checks passed

openshift-cherrypick-robot mentioned this pull request May 24, 2024

[release-4.15] OCPBUGS-34423: Fixed audit-logs sigterm failing to terminate gracefully #4089

Open

openshift-cherrypick-robot mentioned this pull request May 24, 2024

[release-4.16] OCPBUGS-34554: Fixed audit-logs sigterm failing to terminate gracefully #4090

Merged

Joseph-Goergen mentioned this pull request May 24, 2024

OCPBUGS-33060: enable audit log for oauth-openshift #3994

Open

3 tasks

	- args:
	- -c
	- \|
	#!/bin/bash
	set -euo pipefail

	/usr/bin/oc get istag -n hypershift hypershift-operator:latest -ojsonpath='{.image.dockerImageReference}' > /installer-data/image-ref
	command:
	- /bin/bash

OCPBUGS-33181: Fixed audit-logs sigterm failing to terminate gracefully #3972

OCPBUGS-33181: Fixed audit-logs sigterm failing to terminate gracefully #3972

Conversation

Joseph-Goergen commented May 1, 2024

openshift-ci-robot commented May 1, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jeffnowicki commented May 2, 2024

openshift-ci-robot commented May 2, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stevekuznetsov May 6, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stevekuznetsov left a comment

Choose a reason for hiding this comment

Joseph-Goergen commented May 15, 2024 • edited

Joseph-Goergen commented May 15, 2024

rtheis commented May 16, 2024

rtheis commented May 16, 2024

rtheis commented May 16, 2024

Choose a reason for hiding this comment

Joseph-Goergen May 16, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rtheis commented May 20, 2024

Joseph-Goergen commented May 20, 2024

Joseph-Goergen commented May 20, 2024

rtheis left a comment

Choose a reason for hiding this comment

Joseph-Goergen commented May 20, 2024

rtheis commented May 21, 2024

rtheis commented May 21, 2024

Joseph-Goergen commented May 21, 2024

sjenning commented May 22, 2024

openshift-ci bot commented May 22, 2024

openshift-ci-robot commented May 22, 2024

openshift-ci-robot commented May 23, 2024

openshift-ci-robot commented May 23, 2024

openshift-ci bot commented May 23, 2024 • edited

rtheis commented May 23, 2024

openshift-ci-robot commented May 23, 2024

openshift-bot commented May 24, 2024

Joseph-Goergen commented May 24, 2024

openshift-cherrypick-robot commented May 24, 2024

rtheis commented May 24, 2024

openshift-cherrypick-robot commented May 24, 2024

stevekuznetsov May 6, 2024 •

edited

Joseph-Goergen commented May 15, 2024 •

edited

Joseph-Goergen May 16, 2024 •

edited

openshift-ci bot commented May 23, 2024 •

edited