ipsec, auth: Avoid out-of-order events that lead to inconsistent node IDs

[pchaigno]

We found out that the logic to allocate and deallocate node IDs had multiple bugs because it wasn't handling out-of-order events properly. In this context, out-of-order events are e.g. when pod objects are received before their corresponding node object. We discussed those issues at length in #26725 (see review) and concluded we should try to avoid having to deal with such out-of-order events altogether.

This pull request therefore removes all handling of node IDs on pod events. Node IDs are now entirely managed from node events. The node IDs are removed from the ipcache (ep->node_id). That simplifies the agent code a lot (all ipcache is uncoupled from node ID logic) and only slightly complicated the datapath logic (one more map lookup).

Summary of commits:

• Commits 1-4 are preparatory work in the datapath and the agent, without functional changes.
• Commit 5 moves the node ID allocation to be consistent with the deallocation.
• Commit 6 introduces a new DROP reason.
• Commits 7-8 remove the two consumers of ep->node_id in the datapath and replace them with an additional map lookup.
• Commits 9-10 covers the new logic of the previous commit in BPF unit tests.
• Commits 11-12 remove the node IDs from the ipcache and clean up related agent logic.
• Commit 13 fixes a leak of node IDs when the IP addresses of a node are updated.
• Commit 14 adds a runtime check for the bogus state fixed in the previous commit.
• Commits 15-16 cover the node ID lifecycle in Golang integration tests.

See commits for details.

This PR is an alternative to the solution tried at #26725.

Fix a bug that could cause packet drops of type XfrmOutPolBlock when IPsec is enabled and node are recycled.
Fix a bug that could cause IPsec-encrypted packets to be sent to the wrong destination node when node churn is high.

[jrfastab]

actually now that it is internal to datapath/node can we get rid of all the exported GetNodeIP, GetNodeID. But DumpNodeIDs is used from cmd so might still need that.

And one small question. On restart can we miss a NodeDelete? If that happens do we need some logic to correct or should the normal nodeDelete logic eventually get called and nodeID map will also be fixed up (I think this is true?).

[jrfastab]

This is amazing!

[jrfastab]

I agree with the overall assessment trying to save a map lookup in the datapath introduces too much complexity in user space to warrant the effort/complexity/risk and anyways this is going to an ipsec stack that is going to encrypt the entire thing anyways so a single map lookup is optimizing something with no/little gain.

[pchaigno]

actually now that it is internal to datapath/node can we get rid of all the exported GetNodeIP, GetNodeID. But DumpNodeIDs is used from cmd so might still need that.

The Mutual Auth feature is also still using GetNodeID & co. It's probably doable to replace that, but I'd rather avoid touching Mutual Auth too much in this PR. It's already quite big in terms of commits.

And one small question. On restart can we miss a NodeDelete? If that happens do we need some logic to correct or should the normal nodeDelete logic eventually get called and nodeID map will also be fixed up (I think this is true?).

AFAIK, yes, the NodeDelete logic gets called on restart in that case. We should manually test it though.