Red Hat SummitChapter 15. Creating uprobes with perf
Uprobes (user-space probes) provide dynamic instrumentation for user-space applications at runtime. They require no source code changes or recompilation.
There are two primary use cases where uprobes are useful:
- Debugging and performance analysis
-
Uprobesfunction similarly to watchpoints. You can insert them at specific locations in your application to count how often those code paths are run. Additionally, they can capture rich context such as call stacks or variable values. Use them to identify performance bottlenecks or track bugs. - Event-based data collection
-
Uprobesact as switching events for mechanisms such as circular buffers. They can control when data is recorded or flushed based on execution triggers in user space.
Uprobes integrate seamlessly with perf, which can both consume existing uprobes and create new ones. This flexibility enables powerful, non-intrusive observability and profiling of user-space behavior alongside kernel-space instrumentation (by using kprobes).
15.1. Creating uprobes at the function level with perf
Use the perf tool to create dynamic tracepoints at arbitrary points in a process or application. Use these tracepoints with perf stat or perf record to analyze process and application behavior.
Prerequisites
-
You have the
perfuser space tool installed. For more information, see Installing perf.
Procedure
Create the uprobe in the process or application you are interested in monitoring at a location of interest within the process or application:
# perf probe -x /path/to/executable -a functionAdded new event: probe_executable:function (on function in /path/to/executable) You can now use it in all perf tools, such as: perf record -e probe_executable:function -aR sleep 1
15.2. Creating uprobes on lines within a function with perf
You can use the tracepoints conjunction with other perf tools such as perf stat and perf record to better understand the process or application behavior.
Prerequisites
-
You have the
perfuser space tool installed. For more information, see Installing perf. You have received the debugging symbols for your executable:
# objdump -t ./your_executable | headNoteTo do this, install the
debuginfopackage of the executable. If the executable is a locally developed application, compile the application with debugging information by using the-goption in GCC.
Procedure
View the function lines where you can place a
uprobe:$ perf probe -x ./your_executable -L main<main@/home/user/my_executable:0> 0 int main(int argc, const char *argv) 1 { int err; const char *cmd; char sbuf[STRERR_BUFSIZE]; / libsubcmd init */ 7 exec_cmd_init("perf", PREFIX, PERF_EXEC_PATH, EXEC_PATH_ENVIRONMENT); 8 pager_init(PERF_PAGER_ENVIRONMENT);Create the
uprobefor the function line that you want to monitor:# perf probe -x ./my_executable main:8Added new event: probe_my_executable:main_L8 (on main:8 in /home/user/my_executable) you can now use it in all perf tools, such as: Perf record -e probe_my_executable:main_L8 -aR sleep 1
15.3. Perf script output of data recorded over uprobes
A common method to analyze data collected with uprobes is to run the perf script command. It which reads the perf.data file and displays a detailed trace of the recorded workload.
- In the perf script example output
-
A
uprobeis added to the functionisprime()in a program calledmy_prog ais a function argument added to theuprobe. Alternatively,acould be an arbitrary variable visible in the code scope of where you add youruprobe:# perf scriptmy_prog 1367 [007] 10802159.906593: probe_my_prog:isprime: (400551) a=2 my_prog 1367 [007] 10802159.906623: probe_my_prog:isprime: (400551) a=3 my_prog 1367 [007] 10802159.906625: probe_my_prog:isprime: (400551) a=4 my_prog 1367 [007] 10802159.906627: probe_my_prog:isprime: (400551) a=5 my_prog 1367 [007] 10802159.906629: probe_my_prog:isprime: (400551) a=6 my_prog 1367 [007] 10802159.906631: probe_my_prog:isprime: (400551) a=7 my_prog 1367 [007] 10802159.906633: probe_my_prog:isprime: (400551) a=13 my_prog 1367 [007] 10802159.906635: probe_my_prog:isprime: (400551) a=17 my_prog 1367 [007] 10802159.906637: probe_my_prog:isprime: (400551) a=19
-
A
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}