subsys: samples: doc: arch: Make perf subsystem #72890

KushnerovMikhail · 2024-05-16T14:36:40Z

Add profiling util based on periodic stack unwinding. Perf from Linux was taken as a reference.

Code is based on Sampling profiler (#29304) by @Jongy.

The operation of module is based on frame pointer usage and saving registers during interruption handling by zephyr.

General work description:
The unwinding function stay in timer as expiry function so is called during interruption handling. Thus the function have access to saved registers (program counter and frame pointer in particular) of the current thread and use it to unwind the thread stack.
Timer starting and results printing function are made as shell commands for conveniency.
Obtained samples can be translated into flamegraph using stackcollapse.py script.

Flame graph example, generated from echo_server sample:

Kconfig optioins:

CONFIG_PERF enables perf subsystem in compilation.
CONFIG_PERF_BUFFER_SIZE specifies size of buffer, where samples are stored.

Kconfig requirements:

CONFIG_THREAD_STACK_INFO=y provides start and size thread stack values.
CONFIG_SMP=n SMP support is not implemented yet.
CONFIG_OMIT_FRAME_POINTER=n && CONFIG_OVERRIDE_FRAME_POINTER_DEFAULT=y provide frame pointer.
CONFIG_SHELL=y needs because subsystem provides shell commands.
CONFIG_RISCV=y || CONFIG_X86_64=y - subsystem implemented only for x86_64 and riscv yet.

github-actions · 2024-05-16T14:37:15Z

Hello @KushnerovMikhail, and thank you very much for your first pull request to the Zephyr project!
Our Continuous Integration pipeline will execute a series of checks on your Pull Request commit messages and code, and you are expected to address any failures by updating the PR. Please take a look at our commit message guidelines to find out how to format your commit messages, and at our contribution workflow to understand how to update your Pull Request. If you haven't already, please make sure to review the project's Contributor Expectations and update (by amending and force-pushing the commits) your pull request if necessary.
If you are stuck or need help please join us on Discord and ask your question there. Additionally, you can escalate the review when applicable. 😊

cfriedt

✨.Beautiful ✨.

cfriedt · 2024-05-17T01:19:38Z

doc/services/debugging/perf.rst

+
+   .. code-block:: bash
+
+      ./FlameGraph/flamegraph.pl perf_buf.foolded > graph.svg


Suggested change

./FlameGraph/flamegraph.pl perf_buf.foolded > graph.svg

./FlameGraph/flamegraph.pl perf_buf.folded > graph.svg

Thanks for the comment. Corrected it.

cfriedt · 2024-05-17T01:20:03Z

doc/services/debugging/perf.rst

+
+   .. code-block:: bash
+
+      python zephyr/scripts/perf/stackcollapse.py perf_buf <build_dir>/zephyr/zephyr.elf > perf_buf.foolded


Suggested change

python zephyr/scripts/perf/stackcollapse.py perf_buf <build_dir>/zephyr/zephyr.elf > perf_buf.foolded

python zephyr/scripts/perf/stackcollapse.py perf_buf <build_dir>/zephyr/zephyr.elf > perf_buf.folded

Thanks for the comment. Corrected it.

nashif

Please split the commit into multiple commits, i.e. 1 commit per arch, docs, adding the subsystem code and finally the sample.
Also, perf itself does not qualify as standalone subsystem. I suggest putting this under subsys/profiling/

ycsin

ARM64 supports stack unwinding as well. The docs can use some review from @kartben

ycsin · 2024-05-24T07:05:31Z

subsys/profiling/Kconfig

@@ -0,0 +1,14 @@
+# cOPYRIGHT (C) 2023 kns gROUP llc (yadro)


This and other similar ones, be consistent

Suggested change

# cOPYRIGHT (C) 2023 kns gROUP llc (yadro)

# Copyright (c) 2023 KNS Group LLC (YADRO)

I don't know how I let this happen. Thanks for the comment. Corrected it.

ycsin · 2024-05-24T07:07:45Z

subsys/profiling/Kconfig

+menuconfig PROFILING
+	bool "Profiling tools"
+	help
+		Enable profiling tools, such as perf


This and other Kconfig files, fix indent

Suggested change

Enable profiling tools, such as perf

Enable profiling tools, such as perf

Thanks for the comment. Corrected it.

ycsin · 2024-05-24T07:09:06Z

subsys/profiling/perf/Kconfig

+
+config PROFILING_PERF
+	bool "Perf support"
+	default n


n is already default

Suggested change

default n

Corrected it.

ycsin · 2024-05-24T07:11:05Z

subsys/profiling/perf/Kconfig

+	depends on !OMIT_FRAME_POINTER
+	depends on OVERRIDE_FRAME_POINTER_DEFAULT


Can use CONFIG_FRAME_POINTER after #72646 is merged

Сhanged code to use a CONFIG_FRAME_POINTER.

ycsin · 2024-05-24T07:14:45Z

subsys/profiling/perf/perf.c

+		perf_data.idx += trace_length;
+	} else {
+		--perf_data.idx;
+		printf("perf buf override!\n");


should print to the shell, have a look at k_timer_user_data_set

I had to create an additional handler thread because the shell_print does not work in interrupts.
I hope I understood correctly how you wanted me to use k_timer_user_data_set.

ycsin · 2024-05-24T07:16:47Z

subsys/profiling/perf/perf.c

+
+	k_timer_start(&perf_data.timer, K_NO_WAIT, K_NSEC(period));
+
+	printf("Enabled perf\n");


should print to the shell

Corrected it. It prints to shell now.

ycsin · 2024-05-24T07:19:10Z

subsys/profiling/perf/perf.c

+
+int cmd_perf_print(const struct shell *sh, size_t argc, char **argv)
+{
+	printf("Perf buf length %zu\n", perf_data.idx);


Please print to shell

Corrected it. It prints to shell now.

ycsin · 2024-05-24T07:22:57Z

subsys/profiling/perf/perf.c

+}
+
+SHELL_STATIC_SUBCMD_SET_CREATE(m_sub_perf,
+	SHELL_CMD_ARG(record, NULL, "Start recording for X ms on Y Hz", cmd_perf_record, 3, 0),


Maybe something like

#define CMD_HELP_RECORD \ "Start recording for <duration> ms on <frequency> Hz\n" \ "Usage: record <duration> <frequency>\n"

Suggested change

SHELL_CMD_ARG(record, NULL, "Start recording for X ms on Y Hz", cmd_perf_record, 3, 0),

SHELL_CMD_ARG(record, NULL, CMD_HELP_RECORD, cmd_perf_record, 3, 0),

Your help string is really better then mine. Thanks for help

ycsin · 2024-05-24T07:25:31Z

arch/riscv/core/perf.c

+}
+
+/*
+ * This function use frame poiters to unwind stack and get trace of return addresses.


Suggested change

* This function use frame poiters to unwind stack and get trace of return addresses.

* This function use frame pointers to unwind stack and get trace of return addresses.

Corrected it.

ycsin · 2024-05-24T07:26:40Z

arch/riscv/core/perf.c

+ * Return addresses are translated in corresponding function's names using .elf file.
+ * So we get function call trace
+ */
+size_t arch_perf_current_stack_trace(uintptr_t *buf, size_t size)


Any chance that this and https://github.com/zephyrproject-rtos/zephyr/blob/main/arch/riscv/core/stacktrace.c can share some of the unwinding logics?

Add profiling subsystem. Add perf util based on periodic stack unwinding. Perf from Linux was taken as a reference. The operation of module is based on frame pointer usage and saving registers during interruption handling. The unwinding function stay in timer as expiry functioin so is called during interruption handling. Thus the function have access to saved registers (program counter and frame pointer in particular) of the current thread and use it to unwind the thread stack. Timer starting and results printing function are made as shell commands for conveniency. Originally-by: Yonatan Goldschmidt <yon.goldschmidt@gmail.com> Signed-off-by: Mikhail Kushnerov <m.kushnerov@yadro.com>

Implement stack trace function for riscv arch, that get required thread register values and unwind stack with it. Signed-off-by: Mikhail Kushnerov <m.kushnerov@yadro.com>

Implement stack trace function for x86_64 arch, that get required thread register values and unwind stack with it. Originally-by: Yonatan Goldschmidt <yon.goldschmidt@gmail.com> Signed-off-by: Mikhail Kushnerov <m.kushnerov@yadro.com>

Samples, that were obtained by profiling perf tool, can be be translated into flamegraph using stackcollapse.py script. Originally-by: Yonatan Goldschmidt <yon.goldschmidt@gmail.com> Signed-off-by: Mikhail Kushnerov <m.kushnerov@yadro.com>

Add doc for profiling perf tool Signed-off-by: Mikhail Kushnerov <m.kushnerov@yadro.com>

The operation of perf tool can be checked using this sample. Signed-off-by: Mikhail Kushnerov <m.kushnerov@yadro.com>

zephyrbot added area: X86 x86 Architecture (32-bit) area: Debugging area: RISCV RISCV Architecture (32-bit & 64-bit) area: Samples Samples labels May 16, 2024

zephyrbot requested review from andyross, carlocaione, dcpleung, edersondisouza, fkokosinski, jhedberg, kartben, katsuster, kgugala, mgielda, nashif, npitre, tbursztyka, tgorochowik and ycsin May 16, 2024 14:37

zephyrbot assigned kartben May 16, 2024

kartben assigned nashif and unassigned kartben May 16, 2024

ycsin requested review from cfriedt and removed request for ycsin May 16, 2024 17:00

cfriedt previously approved these changes May 17, 2024

View reviewed changes

nashif requested changes May 18, 2024

View reviewed changes

KushnerovMikhail dismissed cfriedt’s stale review via 830c289 May 23, 2024 13:29

KushnerovMikhail force-pushed the add_perf_x86_riscv branch 2 times, most recently from 830c289 to 20d23f2 Compare May 23, 2024 15:45

ycsin reviewed May 24, 2024

View reviewed changes

ycsin mentioned this pull request May 31, 2024

arch: introduce arch_stack_walk() #73587

Open

Mikhail Kushnerov added 6 commits May 31, 2024 18:55

arch: riscv: Implement perf stack thrace func

a5d86b8

Implement stack trace function for riscv arch, that get required thread register values and unwind stack with it. Signed-off-by: Mikhail Kushnerov <m.kushnerov@yadro.com>

arch: x86: Implement perf stack thrace func

d41b43f

Implement stack trace function for x86_64 arch, that get required thread register values and unwind stack with it. Originally-by: Yonatan Goldschmidt <yon.goldschmidt@gmail.com> Signed-off-by: Mikhail Kushnerov <m.kushnerov@yadro.com>

scripts: profiling: Add stackcollapse script

8661898

Samples, that were obtained by profiling perf tool, can be be translated into flamegraph using stackcollapse.py script. Originally-by: Yonatan Goldschmidt <yon.goldschmidt@gmail.com> Signed-off-by: Mikhail Kushnerov <m.kushnerov@yadro.com>

doc: profiling: Add doc for perf tool

4a3ea1f

Add doc for profiling perf tool Signed-off-by: Mikhail Kushnerov <m.kushnerov@yadro.com>

samples: profiling: Add sample for perf tool

4fe8184

The operation of perf tool can be checked using this sample. Signed-off-by: Mikhail Kushnerov <m.kushnerov@yadro.com>

KushnerovMikhail force-pushed the add_perf_x86_riscv branch from 20d23f2 to 4fe8184 Compare May 31, 2024 16:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

subsys: samples: doc: arch: Make perf subsystem #72890

subsys: samples: doc: arch: Make perf subsystem #72890

KushnerovMikhail commented May 16, 2024 •

edited

github-actions bot commented May 16, 2024

cfriedt left a comment

cfriedt May 17, 2024

KushnerovMikhail May 23, 2024

cfriedt May 17, 2024

KushnerovMikhail May 23, 2024

nashif left a comment

ycsin left a comment •

edited

ycsin May 24, 2024 •

edited

KushnerovMikhail May 31, 2024

ycsin May 24, 2024

KushnerovMikhail May 31, 2024

ycsin May 24, 2024

KushnerovMikhail May 31, 2024

ycsin May 24, 2024

KushnerovMikhail May 31, 2024

ycsin May 24, 2024

KushnerovMikhail May 31, 2024

ycsin May 24, 2024

KushnerovMikhail May 31, 2024 •

edited

ycsin May 24, 2024

KushnerovMikhail May 31, 2024 •

edited

ycsin May 24, 2024

KushnerovMikhail May 31, 2024

ycsin May 24, 2024

KushnerovMikhail May 31, 2024

ycsin May 24, 2024

ycsin May 31, 2024


		.. code-block:: bash

		./FlameGraph/flamegraph.pl perf_buf.foolded > graph.svg

	./FlameGraph/flamegraph.pl perf_buf.foolded > graph.svg
	./FlameGraph/flamegraph.pl perf_buf.folded > graph.svg


		.. code-block:: bash

		python zephyr/scripts/perf/stackcollapse.py perf_buf <build_dir>/zephyr/zephyr.elf > perf_buf.foolded

	# cOPYRIGHT (C) 2023 kns gROUP llc (yadro)
	# Copyright (c) 2023 KNS Group LLC (YADRO)

	Enable profiling tools, such as perf
	Enable profiling tools, such as perf

		depends on !OMIT_FRAME_POINTER
		depends on OVERRIDE_FRAME_POINTER_DEFAULT


		k_timer_start(&perf_data.timer, K_NO_WAIT, K_NSEC(period));

		printf("Enabled perf\n");

	SHELL_CMD_ARG(record, NULL, "Start recording for X ms on Y Hz", cmd_perf_record, 3, 0),
	SHELL_CMD_ARG(record, NULL, CMD_HELP_RECORD, cmd_perf_record, 3, 0),

	* This function use frame poiters to unwind stack and get trace of return addresses.
	* This function use frame pointers to unwind stack and get trace of return addresses.

subsys: samples: doc: arch: Make perf subsystem #72890

Are you sure you want to change the base?

subsys: samples: doc: arch: Make perf subsystem #72890

Conversation

KushnerovMikhail commented May 16, 2024 • edited

github-actions bot commented May 16, 2024

cfriedt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nashif left a comment

Choose a reason for hiding this comment

ycsin left a comment • edited

Choose a reason for hiding this comment

ycsin May 24, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KushnerovMikhail May 31, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KushnerovMikhail May 31, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KushnerovMikhail commented May 16, 2024 •

edited

ycsin left a comment •

edited

ycsin May 24, 2024 •

edited

KushnerovMikhail May 31, 2024 •

edited

KushnerovMikhail May 31, 2024 •

edited