pkg/service: Backends leak follow ups with revised fixes, debugging improvements and unit tests

[aditighag]

This is a follow-up PR with revised fixes along with debugging improvements, and test coverage for fixes merged in - #24681.

pkg/service: Extend logic to handle leaked backends

This commit revises the logic to handle all the leaked backend scenarios in one place.
Consider the following leaked backend scenarios where (1) and
(2a) were previously handled:
(1) Backend entries leaked, no duplicates: previously, such
      backends would be deleted as orphan backends after sync with
      kubernetes api server.
(2) Backend entries leaked with duplicates:
    (a) backend with overlapping L3nL4Addr hash is associated with service(s)
          Sequence of events:
          Backends were leaked prior to agent restart, but there was at least
          one service that the backend by hash is associated with.
          s.backendByHash will have a non-zero reference count for the
          overlapping L3nL4Addr hash.
    (b) none of the backends are associated with services
          Sequence of events:
          All the services these backends were associated with were deleted
          prior to agent restart.
          s.backendByHash will not have an entry for the backends hash.

pkg/service: Fix premature release of backend ID

(See commit description)
Fixes: 51b62f5684

pkg/service: Log skipped backends count during restore
pkg/service: Log orphan backends count

Debugging improvements

pkg/service: Unit test for handling of leaked backends in various scenarios described above

docs: Document backends leak issue

Document the issue along with affected Cilium versions.

Relates: #23551

Signed-off-by: Aditi Ghag aditi@cilium.io

[aditighag]

ConformanceKind hit - #22217.

[aditighag]

I'll piggyback on the PR to add documentation changes.

[zacharysarah]

@aditighag 👋🏻 Thanks for this update. I left some edits for clarity, otherwise LGTM from a docs perspective.

[aditighag]

Thanks for the docs review @zacharysarah.
@brb Ping... You had mentioned that you would take a look earlier this week. You are the original author for most of the code, so you have the most context.

[brb]

You are the original author for most of the code, so you have the most context.

It's no longer the case, as I stopped tracking changes the package long time ago.