4.3. Profiling
The following sections showcase scripts that profile kernel activity by monitoring function calls.
4.3.1. Counting Function Calls Made 复制链接链接已复制到粘贴板!
复制链接链接已复制到粘贴板!
This section describes how to identify how many times the system called a specific kernel function in a 30-second sample. Depending on your use of wildcards, you can also use this script to target multiple kernel functions.
例 4.21. functioncallcount.stp
例 4.21 “functioncallcount.stp” takes the targeted kernel function as an argument. The argument supports wildcards, which enables you to target multiple kernel functions up to a certain extent.
The output of 例 4.21 “functioncallcount.stp” contains the name of the function called and how many times it was called during the sample time (in alphabetical order). 例 4.22 “例 4.21 “functioncallcount.stp” Sample Output” contains an excerpt from the output of stap functioncallcount.stp "*@mm/*.c":
例 4.22. 例 4.21 “functioncallcount.stp” Sample Output
4.3.2. Call Graph Tracing 复制链接链接已复制到粘贴板!
复制链接链接已复制到粘贴板!
This section describes how to trace incoming and outgoing function calls.
例 4.23. para-callgraph.stp
例 4.23 “para-callgraph.stp” takes two command-line arguments:
- The function(s) whose entry/exit you'd like to trace ($1).
- A second optional trigger function (
$2
), which enables or disables tracing on a per-thread basis. Tracing in each thread will continue as long as the trigger function has not exited yet.
例 4.23 “para-callgraph.stp” uses
thread_indent()
; as such, its output contains the timestamp, process name, and thread ID of $1
(the probe function you are tracing). For more information about thread_indent()
, see its entry in SystemTap Functions.
The following example contains an excerpt from the output for stap para-callgraph.stp 'kernel.function("*@fs/*.c")' 'kernel.function("sys_read")':
例 4.24. 例 4.23 “para-callgraph.stp” Sample Output
4.3.3. Determining Time Spent in Kernel and User Space 复制链接链接已复制到粘贴板!
复制链接链接已复制到粘贴板!
This section illustrates how to determine the amount of time any given thread is spending in either kernel or user-space.
例 4.25. thread-times.stp
例 4.25 “thread-times.stp” lists the top 20 processes currently taking up CPU time within a 5-second sample, along with the total number of CPU ticks made during the sample. The output of this script also notes the percentage of CPU time each process used, as well as whether that time was spent in kernel space or user space.
例 4.26 “例 4.25 “thread-times.stp” Sample Output” contains a 5-second sample of the output for 例 4.25 “thread-times.stp”:
例 4.26. 例 4.25 “thread-times.stp” Sample Output
4.3.4. Monitoring Polling Applications 复制链接链接已复制到粘贴板!
复制链接链接已复制到粘贴板!
This section describes how to identify and monitor which applications are polling. Doing so allows you to track unnecessary or excessive polling, which can help you pinpoint areas for improvement in terms of CPU usage and power savings.
例 4.27. timeout.stp
例 4.27 “timeout.stp” tracks how many times each application used the following system calls over time:
poll
select
epoll
itimer
futex
nanosleep
signal
In some applications, these system calls are used excessively. As such, they are normally identified as "likely culprits" for polling applications. Note, however, that an application may be using a different system call to poll excessively; sometimes, it is useful to find out the top system calls used by the system (see 第 4.3.5 节 “Tracking Most Frequently Used System Calls” for instructions). Doing so can help you identify any additional suspects, which you can add to 例 4.27 “timeout.stp” for tracking.
例 4.28. 例 4.27 “timeout.stp” Sample Output
You can increase the sample time by editing the timer in the second probe (
timer.s()
). The output of 例 4.21 “functioncallcount.stp” contains the name and UID of the top 20 polling applications, along with how many times each application performed each polling system call (over time). 例 4.28 “例 4.27 “timeout.stp” Sample Output” contains an excerpt of the script:
4.3.5. Tracking Most Frequently Used System Calls 复制链接链接已复制到粘贴板!
复制链接链接已复制到粘贴板!
例 4.27 “timeout.stp” from 第 4.3.4 节 “Monitoring Polling Applications” helps you identify which applications are polling by pointing out which ones used the following system calls most frequently:
poll
select
epoll
itimer
futex
nanosleep
signal
However, in some systems, a different system call might be responsible for excessive polling. If you suspect that a polling application is using a different system call to poll, you need to identify first the top system calls used by the system. To do this, use 例 4.29 “topsys.stp”.
例 4.29. topsys.stp
例 4.29 “topsys.stp” lists the top 20 system calls used by the system per 5-second interval. It also lists how many times each system call was used during that period. Refer to 例 4.30 “例 4.29 “topsys.stp” Sample Output” for a sample output.
例 4.30. 例 4.29 “topsys.stp” Sample Output
4.3.6. Tracking System Call Volume Per Process 复制链接链接已复制到粘贴板!
复制链接链接已复制到粘贴板!
This section illustrates how to determine which processes are performing the highest volume of system calls. In previous sections, we've described how to monitor the top system calls used by the system over time (第 4.3.5 节 “Tracking Most Frequently Used System Calls”). We've also described how to identify which applications use a specific set of "polling suspect" system calls the most (第 4.3.4 节 “Monitoring Polling Applications”). Monitoring the volume of system calls made by each process provides more data in investigating your system for polling processes and other resource hogs.
例 4.31. syscalls_by_proc.stp
例 4.31 “syscalls_by_proc.stp” lists the top 20 processes performing the highest number of system calls. It also lists how many system calls each process performed during the time period. See 例 4.32 “例 4.29 “topsys.stp” Sample Output” for a sample output.
例 4.32. 例 4.29 “topsys.stp” Sample Output
If you prefer the output to display the process IDs instead of the process names, use the following script instead.
例 4.33. syscalls_by_pid.stp
As indicated in the output, you need to manually exit the script in order to display the results. You can add a timed expiration to either script by simply adding a timer.s() probe; for example, to instruct the script to expire after 5 seconds, add the following probe to the script:
probe timer.s(5) { exit() }
probe timer.s(5)
{
exit()
}