bpf: lxc: support Pod->Service->Pod hairpinning with endpoint routes #27798
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
maintainer-s-little-helper
bot
added
the
dont-merge/needs-release-note-label
ti-mo
force-pushed
the
tb/svc-hairpinning
branch
from
46a1503
to
6d651aa
Compare
ti-mo
added
the
release-note/bug
maintainer-s-little-helper
bot
removed
the
dont-merge/needs-release-note-label
ti-mo
added
the
sig/datapath
2 tasks
|
/test |
qmonnet
requested changes
Aug 29, 2023
sypakine
reviewed
Aug 29, 2023
2 tasks
ti-mo
force-pushed
the
tb/svc-hairpinning
branch
from
6d651aa
to
662ce92
Compare
julianwiedmann
added
sig/policy
ti-mo
force-pushed
the
tb/svc-hairpinning
branch
2 times, most recently
from
e07938e
to
b465e93
Compare
qmonnet
reviewed
Aug 30, 2023
ti-mo
force-pushed
the
tb/svc-hairpinning
branch
2 times, most recently
from
100ec9e
to
b08c03f
Compare
|
/test |
ti-mo
added
needs-backport/1.12
needs-backport/1.13
qmonnet
added
the
backport-pending/1.13
maintainer-s-little-helper
bot
moved this from Needs backport from main
to Backport pending to v1.13
in 1.13.7
julianwiedmann
added
backport-done/1.13
maintainer-s-little-helper
bot
moved this from Backport pending to v1.13
to Backport done to v1.13
in 1.13.7
qmonnet
added
backport-done/1.12
maintainer-s-little-helper
bot
moved this from Backport pending to v1.12
to Backport done to v1.12
in 1.12.14
This was referenced Sep 9, 2023
gandro
added
the
backport/author
|
OK thank you I will look into it |
1 task
julianwiedmann
added
backport-done/1.14
2 tasks
jrajahalme
moved this from Needs backport from main
to Backport done to v1.14
in 1.14.3
julianwiedmann
added a commit
to julianwiedmann/cilium
that referenced
this pull request
May 17, 2024
The usual flow for handling service traffic to a local backend is as
follows:
* requests are load-balanced in from-container. This entails selecting
a backend (and caching the selection in a CT_SERVICE entry), DNATing the
packet, creating a CT_EGRESS entry for the resulting `client -> backend`
flow, applying egress network policy, and local delivery to the backend
pod. As part of the local delivery, we also create a CT_INGRESS entry and
apply ingress network policy.
* replies bypass the backend's egress network policy (because the CT
lookup returns CT_REPLY), and pass to the client via local delivery. In
the client's ingress path they bypass ingress network policy (the packets
match as reply against the CT_EGRESS entry), and we apply RevDNAT based on
the `rev_nat_index` in the CT_EGRESS entry.
For a loopback connection (where the client pod is selected as backend for
the connection) this looks slightly more complicated:
* As we can't establish a `client -> client` connection, the requests are
also SNATed with IPV4_LOOPBACK. Network policy in forward direction is
explicitly skipped (as the matched CT entries have the `.loopback` flag
set).
* In reply direction, we can't deliver to IPV4_LOOPBACK (as that's not a
valid IP for an endpoint lookup). So a reply already gets fully RevNATed
by from-container, using the CT_INGRESS entry's `rev_nat_index`. But this
means that when passing into the client pod (either via to-container, or
via the ingress policy tail-call), the packet doesn't match as reply to the
CT_EGRESS entry - and so we don't benefit from automatic network policy
bypass. We ended up with two workarounds for this aspect:
(1) when to-container is installed, it contains custom logic to match the
packet as a loopback reply, and skip ingress policy
(see cilium#27798).
(2) otherwise we skip the ingress policy tailcall, and forward the packet
straight into the client pod.
The downside of these workarounds is that we bypass the *whole* ingress
program, not just the network policy part. So the CT_EGRESS entry doesn't
get updated (lifetime, statistics, observed packet flags, ...), and we
have the hidden risk that when we add more logic to the ingress program,
it doesn't get executed for loopback replies.
This patch aims to eliminate the need for such workarounds. At its core,
it detects loopback replies in from-container and overrides the packet's
destination IP. Instead of attempting an endpoint lookup for IPV4_LOOPBACK,
we can now look up the actual client endpoint - and deliver to the ingress
policy program, *without* needing to early-RevNAT the packet. Instead the
replies follow the usual packet flow, match the CT_EGRESS entry in the
ingress program, naturally bypass ingress network policy, and are *then*
RevNATed based on the CT_EGRESS entry's `rev_nat_index`.
Consequently we follow the standard datapath, without needing to skip over
policy programs. The CT_EGRESS entry is updated for every reply.
Thus we can also remove the manual policy bypass for loopback replies,
when using per-EP routing. It's no longer needed and in fact the
replies will no longer match the lookup logic, as they haven't
been RevNATed yet. This effectively reverts
e2829a0 ("bpf: lxc: support Pod->Service->Pod hairpinning with endpoint routes").
Signed-off-by: Julian Wiedmann <jwi@isovalent.com>
julianwiedmann
added a commit
to julianwiedmann/cilium
that referenced
this pull request
May 22, 2024
The usual flow for handling service traffic to a local backend is as
follows:
* requests are load-balanced in from-container. This entails selecting
a backend (and caching the selection in a CT_SERVICE entry), DNATing the
packet, creating a CT_EGRESS entry for the resulting `client -> backend`
flow, applying egress network policy, and local delivery to the backend
pod. As part of the local delivery, we also create a CT_INGRESS entry and
apply ingress network policy.
* replies bypass the backend's egress network policy (because the CT
lookup returns CT_REPLY), and pass to the client via local delivery. In
the client's ingress path they bypass ingress network policy (the packets
match as reply against the CT_EGRESS entry), and we apply RevDNAT based on
the `rev_nat_index` in the CT_EGRESS entry.
For a loopback connection (where the client pod is selected as backend for
the connection) this looks slightly more complicated:
* As we can't establish a `client -> client` connection, the requests are
also SNATed with IPV4_LOOPBACK. Network policy in forward direction is
explicitly skipped (as the matched CT entries have the `.loopback` flag
set).
* In reply direction, we can't deliver to IPV4_LOOPBACK (as that's not a
valid IP for an endpoint lookup). So a reply already gets fully RevNATed
by from-container, using the CT_INGRESS entry's `rev_nat_index`. But this
means that when passing into the client pod (either via to-container, or
via the ingress policy tail-call), the packet doesn't match as reply to the
CT_EGRESS entry - and so we don't benefit from automatic network policy
bypass. We ended up with two workarounds for this aspect:
(1) when to-container is installed, it contains custom logic to match the
packet as a loopback reply, and skip ingress policy
(see cilium#27798).
(2) otherwise we skip the ingress policy tailcall, and forward the packet
straight into the client pod.
The downside of these workarounds is that we bypass the *whole* ingress
program, not just the network policy part. So the CT_EGRESS entry doesn't
get updated (lifetime, statistics, observed packet flags, ...), and we
have the hidden risk that when we add more logic to the ingress program,
it doesn't get executed for loopback replies.
This patch aims to eliminate the need for such workarounds. At its core,
it detects loopback replies in from-container and overrides the packet's
destination IP. Instead of attempting an endpoint lookup for IPV4_LOOPBACK,
we can now look up the actual client endpoint - and deliver to the ingress
policy program, *without* needing to early-RevNAT the packet. Instead the
replies follow the usual packet flow, match the CT_EGRESS entry in the
ingress program, naturally bypass ingress network policy, and are *then*
RevNATed based on the CT_EGRESS entry's `rev_nat_index`.
Consequently we follow the standard datapath, without needing to skip over
policy programs. The CT_EGRESS entry is updated for every reply.
Thus we can also remove the manual policy bypass for loopback replies,
when using per-EP routing. It's no longer needed and in fact the
replies will no longer match the lookup logic, as they haven't
been RevNATed yet. This effectively reverts
e2829a0 ("bpf: lxc: support Pod->Service->Pod hairpinning with endpoint routes").
Signed-off-by: Julian Wiedmann <jwi@isovalent.com>
julianwiedmann
added a commit
to julianwiedmann/cilium
that referenced
this pull request
May 23, 2024
The usual flow for handling service traffic to a local backend is as
follows:
* requests are load-balanced in from-container. This entails selecting
a backend (and caching the selection in a CT_SERVICE entry), DNATing the
packet, creating a CT_EGRESS entry for the resulting `client -> backend`
flow, applying egress network policy, and local delivery to the backend
pod. As part of the local delivery, we also create a CT_INGRESS entry and
apply ingress network policy.
* replies bypass the backend's egress network policy (because the CT
lookup returns CT_REPLY), and pass to the client via local delivery. In
the client's ingress path they bypass ingress network policy (the packets
match as reply against the CT_EGRESS entry), and we apply RevDNAT based on
the `rev_nat_index` in the CT_EGRESS entry.
For a loopback connection (where the client pod is selected as backend for
the connection) this looks slightly more complicated:
* As we can't establish a `client -> client` connection, the requests are
also SNATed with IPV4_LOOPBACK. Network policy in forward direction is
explicitly skipped (as the matched CT entries have the `.loopback` flag
set).
* In reply direction, we can't deliver to IPV4_LOOPBACK (as that's not a
valid IP for an endpoint lookup). So a reply already gets fully RevNATed
by from-container, using the CT_INGRESS entry's `rev_nat_index`. But this
means that when passing into the client pod (either via to-container, or
via the ingress policy tail-call), the packet doesn't match as reply to the
CT_EGRESS entry - and so we don't benefit from automatic network policy
bypass. We ended up with two workarounds for this aspect:
(1) when to-container is installed, it contains custom logic to match the
packet as a loopback reply, and skip ingress policy
(see cilium#27798).
(2) otherwise we skip the ingress policy tailcall, and forward the packet
straight into the client pod.
The downside of these workarounds is that we bypass the *whole* ingress
program, not just the network policy part. So the CT_EGRESS entry doesn't
get updated (lifetime, statistics, observed packet flags, ...), and we
have the hidden risk that when we add more logic to the ingress program,
it doesn't get executed for loopback replies.
This patch aims to eliminate the need for such workarounds. At its core,
it detects loopback replies in from-container and overrides the packet's
destination IP. Instead of attempting an endpoint lookup for IPV4_LOOPBACK,
we can now look up the actual client endpoint - and deliver to the ingress
policy program, *without* needing to early-RevNAT the packet. Instead the
replies follow the usual packet flow, match the CT_EGRESS entry in the
ingress program, naturally bypass ingress network policy, and are *then*
RevNATed based on the CT_EGRESS entry's `rev_nat_index`.
Consequently we follow the standard datapath, without needing to skip over
policy programs. The CT_EGRESS entry is updated for every reply.
Thus we can also remove the manual policy bypass for loopback replies,
when using per-EP routing. It's no longer needed and in fact the
replies will no longer match the lookup logic, as they haven't
been RevNATed yet. This effectively reverts
e2829a0 ("bpf: lxc: support Pod->Service->Pod hairpinning with endpoint routes").
Signed-off-by: Julian Wiedmann <jwi@isovalent.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This patch adds a check to determine whether a Pod is sending traffic to itself through a Service. With endpoint routes enabled, this traffic is subject to policy, and we want to avoid the user to have to explicitly allow traffic to+from itself.
Fixes: #27709
Strictly depends on #27602 (for backports)