[release-4.12] OCPBUGS-33665: Run resolv-prepender entirely async #4364
Conversation
Currently the resolv-prepender dispatcher script starts the systemd service and then waits for it to complete. This can cause the dispatcher script to time out if the runtimecfg image pull is slow or if resolv.conf does not get populated in a timely fashion (it's not entirely clear to me why the latter happens, but it does). This can cause configure-ovs to time out if there are a large number of interfaces on the system triggering the dispatcher script, such as when there are many VLANs configured. To avoid this, we can stop waiting for the systemd service in the dispatcher script. In fact, there's an argument that we shouldn't wait since we need to be able to handle asynchronous execution anyway for the slow image pull case (which was the entire reason the script was split into a service the way it is). I have found a few possible issues with async execution however: * If we start the service with an empty $DHCP6_FQDN_FQDN value and then later get a new value for that, we may not correctly apply the new value if the service is still running because we only ever "systemd start" the service, which is a noop if the service is already running. * Similarly, if new IP4/6_DOMAINS values come in on a later connection that may not be reflected in the service either. Even though these may sound like the same problem, I mention them separately on purpose because the solutions are different: * For the DHCP6 case, we can move that logic back into the dispatcher script so we will always set the hostname no matter what happens with the prepender code. One could argue that this should be in its own script anyway since it's largely unrelated to resolv.conf. * For the domains case, we do need to restart the service since the domains are involved in resolv.conf generation. However, we do not want to restart the service every time since that may be unnecessary and if we restart in the middle of the image pull it could result in a corrupt image (the whole thing we were trying to avoid by running this as a service in the first place). To avoid problems with restarting the service when we don't want to, I've added logic that only restarts the service if there are changed env values AND the runtimecfg image has already been pulled. This should mean the worst case scenario is that we don't properly set the domains and resolv.conf is temporarily generated with and incorrect search line. This should be resolved the next time any event that triggers the dispatcher script happens.
|
@openshift-cherrypick-robot: Jira Issue OCPBUGS-32208 has been cloned as Jira Issue OCPBUGS-33665. Will retitle bug to link to clone. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci
bot
changed the title
openshift-ci-robot
added
jira/severity-important
|
@openshift-cherrypick-robot: This pull request references Jira Issue OCPBUGS-33665, which is invalid:
Comment The bug has been updated to refer to the pull request using the external bug tracker. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
build image via cluster-bot and do pre-merge test passed. reboot time after applying 70 vlans based on bond: sh-4.4# systemd-analyze Startup finished in 2.297s (kernel) + 4.146s (initrd) + 56.291s (userspace) = 1min 2.735s graphical.target reached after 56.274s in userspace and there is no error message for bring up connection br-ex: sh-4.4# journalctl -b | grep "NM resolv.conf still empty of nameserver" | wc -l 4 sh-4.4# sh-4.4# journalctl -b | grep "configure-ovs exited with error" sh-4.4# sh-4.4# journalctl -b | grep "Cannot bring up connection br-ex after 10 attempts" sh-4.4# |
openshift-ci
bot
added
the
qe-approved
|
@openshift-cherrypick-robot: This pull request references Jira Issue OCPBUGS-33665, which is invalid:
Comment In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/lgtm |
openshift-ci
bot
added
the
backport-risk-assessed
openshift-ci
bot
assigned anuragthehatter, eurijon, mburman5, mhanss, ptalgulk01, rbbratta, reihl, rioliu-rh and rlobillo
|
/label cherry-pick-approved |
|
@qiowang721: Can not set label cherry-pick-approved: Must be member in one of these teams: [] In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
@zhaozhanqi could you help to add label cherry-pick-approved? seems i'm not in the team... |
|
/label cherry-pick-approved |
openshift-ci
bot
added
the
cherry-pick-approved
openshift-ci-robot
added
the
jira/valid-bug
|
@zhaozhanqi: This pull request references Jira Issue OCPBUGS-33665, which is valid. The bug has been moved to the POST state. 7 validation(s) were run on this bug
Requesting review from QA contact: In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci-robot
removed
the
jira/invalid-bug
|
/assign @sinnykumari |
|
@cybertron: GitHub didn't allow me to assign the following users: sinnykumari. Note that only openshift members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/approve |
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: cheesesashimi, cybertron, djoshy, openshift-cherrypick-robot The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
bot
added
the
approved
openshift-merge-bot
bot
merged commit dd2a0b3
into
openshift:release-4.12
|
@openshift-cherrypick-robot: Jira Issue OCPBUGS-33665: All pull requests linked via external trackers have merged: Jira Issue OCPBUGS-33665 has been moved to the MODIFIED state. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
[ART PR BUILD NOTIFIER] This PR has been included in build ose-machine-config-operator-container-v4.12.0-202405171807.p0.gdd2a0b3.assembly.stream.el8 for distgit ose-machine-config-operator. |
This is an automated cherry-pick of #4314
/assign cybertron