Should eBPF programs always be run in a non-preemptible mode?

[Alan-Jowett]

Linux documents that BPF programs always execute in a non-preemptible mode.

eBPF-for-Windows has three possible modes:

1. Execute at IRQL == Dispatch - Programs run to completion without pre-emption.
2. Execute at IRQL == Passive - Programs can be pre-empted.
3. User mode - Programs can be pre-empted.

Should we simplify this logic and always run programs in a non-preemptible mode?

This would simplify:

1. Epoch tracking
2. Per-CPU map logic.

[dthaler]

For the kernel as an execution context, I would say yes dispatch level only (non-preemptible).
For user mode execution contexts, dispatch level is not possible so requires more thought.

[Alan-Jowett]

For kernel-mode:

1. Raise to IRQL on epoch_enter.
2. Restore the previous IRQL on epoch exit.

For user-mode, we can do the following:

1. Affinitize thread to current CPU on epoch_enter.
2. Acquire a per-CPU lock on epoch_enter.
3. Release a per-CPU lock on epoch_exit.
4. Restore thread affinity on epoch_exit.

This will give non-preemption behavior for user mode, with respect to the execution context. Only 1 thread per CPU will execute within the execution context and threads won't migrate between CPUs during the eBPF program execution.

Threads can still be pre-empted by other non-EC threads, but this is no different than DPC's being pre-empted by hyper-v in the kernel.

Thoughts?

[dthaler]

@qmonnet @teknoraver can you provide more info on whether programs are pre-emptible on Linux? E.g., can a program be preempted by another program? By an interrupt? Nothing? If preempted, it always continues on the same CPU?

[teknoraver]

from https://elixir.bootlin.com/linux/latest/source/include/linux/filter.h#L777

	/* Caller needs to hold rcu_read_lock() (!), otherwise program
	 * can be released while still running, or map elements could be
	 * freed early while still having concurrent users. XDP fastpath
	 * already takes rcu_read_lock() when fetching the program, so
	 * it's not necessary here anymore.
	 */

rcu_read_lock() disables preemption, so I can say that at least for XDP eBPF programs, preemption is disabled

[Alan-Jowett]

Given that BPF map implementation depends on RCU for correctness, I posit that all BPF programs execute with preemption disabled.

[Alan-Jowett]

I have a rough prototype of switching all execution in eBPF to be non-preemptive with respect to other eBPF execution.

https://github.com/Alan-Jowett/ebpf-for-windows/tree/non-preemption

The results show significant simplification in the eBPF epoch code.

[qmonnet]

I don't have a perfect understanding on that level, but I think most programs are non-preemptible, as stated above.

However, there was a need for some programs to be “sleep-able”, so they could use, for example, helpers like bpf_copy_from_user() which is susceptible to sleep, and this was implemented last summer. This is the case for LSM programs in particular, if I remember correctly. Note that sleepable programs must be explicitly declared as such at load time (there is a BPF_F_SLEEPABLE flag that can be passed). See related commit log for details. Note that it might be extended in the future, the commit message states:

This is the first step of introducing sleepable progs. Eventually dynamically
allocated hash maps can be allowed and networking program types can become
sleepable too.

[Alan-Jowett]

Thanks. It sounds like Linux is definitely moving towards having preemptible BPF programs.

Given that, I think the answer to this thread should then be to have eBPF for Windows support both preemptible and non-preemptible BPF programs.

[Alan-Jowett]

As per @qmonnet comment above Linux supports both preemptible and non-preemptible eBPF programs. Given that, eBPF for Windows should not drop support for preemptible eBPF programs and instead, permit attach points to define the contract for programs attached to them.

[shankarseal]

This is from BPF helper documentation:
https://www.man7.org/linux/man-pages/man7/bpf-helpers.7.html

@qmonnet - maybe the below should be updated?

`
u32 bpf_get_smp_processor_id(void)

          Description
                 Get the SMP (symmetric multiprocessing) processor
                 id. **Note that all programs run with preemption
                 disabled, which means that the SMP processor id is
                 stable during all the execution of the program.**

          Return The SMP id of the processor running the program.

`

[qmonnet]

Yes please, the update should go to include/uapi/linux/bpf.h in the kernel repository, then the man page will pick it from there eventually.

[dthaler]

@teknoraver Can you make the update to include/uapi/linux/bpf.h ?

[teknoraver]

Something like that?

teknoraver/linux@d63fb7f

[teknoraver]

Keep in mind, that even if preemption is enabled, migration is disabled, and the CPU id can't change

[teknoraver]

Fixed in https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git/commit/?id=336562752acc1a723f9a24b5b8129ae22e0478c6

[rob-scheepens]

This topic was also discussed in https://www.youtube.com/watch?v=EViAho-6qoc&t=12697s, BPF re-entrancy, eBPF Summit 2023.