virt-controller, hot plug: Avoid race between status update and annotation update

[oshoval]

What this PR does

Determining if an annotation update is required is done according to the
interfaces in the VMI status.
Currently, updating the VMI status with multus infoSource happens
after the first annotation update attempt.
This causes an unrequired attempt to update annotations,
because the status is completely missing at this stage.

Fix it by updating annotations only after the VMI is running,
eliminating the transient state false hotplug detection.

Beside that:
Move updateMultusAnnotation next to GeneratePodAnnotations.
Both are related to network, make them next to each other.

Fixes #

Why we need it and why it was done in this way

The following tradeoffs were made:

The following alternatives were considered:

Links to places where the discussion took place:

Special notes for your reviewer

Checklist

This checklist is not enforcing, but it's a reminder of items that could be relevant to every PR.
Approvers are expected to review this list.

• Design: A design document was considered and is present (link) or not required
• PR: The PR description is expressive enough and will help future contributors
• Code: Write code that humans can understand and Keep it simple
• Refactor: You have left the code cleaner than you found it (Boy Scout Rule)
• Upgrade: Impact of this change on upgrade flows was considered and addressed if required
• Testing: New code requires new unit tests. New features and bug fixes require at least on e2e test
• Documentation: A user-guide update was considered and is present (link) or not required. You want a user-guide update if it's a user facing feature / API change.
• Community: Announcement to kubevirt-dev was considered

Release note

None

[kubevirt-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign enp0s3 for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[oshoval]

/cc @AlonaKaplan

[kubevirt-bot]

@oshoval: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kubevirt-unit-test-arm64	651c0e6	link	false	/test pull-kubevirt-unit-test-arm64
pull-kubevirt-e2e-k8s-1.30-sig-compute-serial	651c0e6	link	true	/test pull-kubevirt-e2e-k8s-1.30-sig-compute-serial

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[oshoval]

@AlonaKaplan @EdDev
do we want this or to close it ?

[oshoval]

i am closing it, if someone will see it is important we can reopen

the 2nd commits is fine, but it is just a nit
and later we can refactor to merge the two network related functions that update the annotations on
a follow-up