Chapter 32. Stress testing real-time systems with stress-ng
The stress-ng tool measures the system’s capability to maintain a good level of efficiency under unfavorable conditions. The stress-ng tool is a stress workload generator to load and stress all kernel interfaces. It includes a wide range of stress mechanisms known as stressors. Stress testing makes a machine work hard and trip hardware issues such as thermal overruns and operating system bugs that occur when a system is being overworked.
There are over 270 different tests. These include CPU specific tests that exercise floating point, integer, bit manipulation, control flow, and virtual memory tests.
Use the stress-ng tool with caution as some of the tests can impact the system’s thermal zone trip points on a poorly designed hardware. This can impact system performance and cause excessive system thrashing which can be difficult to stop.
32.1. Testing CPU floating point units and processor data cache
A floating point unit is the functional part of the processor that performs floating point arithmetic operations. Floating point units handle mathematical operations and make floating numbers or decimal calculations simpler.
Using the --matrix-method option, you can stress test the CPU floating point operations and processor data cache.
Prerequisites
- You have root permissions on the systems
Procedure
To test the floating point on one CPU for 60 seconds, use the
--matrixoption:# stress-ng --matrix 1 -t 1mTo run multiple stressors on more than one CPUs for 60 seconds, use the
--timesor-toption:# stress-ng --matrix 0 -t 1m stress-ng --matrix 0 -t 1m --times stress-ng: info: [16783] dispatching hogs: 4 matrix stress-ng: info: [16783] successful run completed in 60.00s (1 min, 0.00 secs) stress-ng: info: [16783] for a 60.00s run time: stress-ng: info: [16783] 240.00s available CPU time stress-ng: info: [16783] 205.21s user time ( 85.50%) stress-ng: info: [16783] 0.32s system time ( 0.13%) stress-ng: info: [16783] 205.53s total time ( 85.64%) stress-ng: info: [16783] load average: 3.20 1.25 1.40The special mode with 0 stressors, query the available CPUs to run, removing the need to specify the CPU number.
The total CPU time required is 4 x 60 seconds (240 seconds), of which 0.13% is in the kernel, 85.50% is in user time, and
stress-ngruns 85.64% of all the CPUs.To test message passing between processes using a POSIX message queue, use the
-mqoption:# stress-ng --mq 0 -t 30s --times --perfThe
mqoption configures a specific number of processes to force context switches using the POSIX message queue. This stress test aims for low data cache misses.
32.2. Testing CPU with multiple stress mechanisms
The stress-ng tool runs multiple stress tests. In the default mode, it runs the specified stressor mechanisms in parallel.
Prerequisites
- You have root privileges on the systems
Procedure
Run multiple instances of CPU stressors as follows:
# stress-ng --cpu 2 --matrix 1 --mq 3 -t 5mIn the example,
stress-ngruns two instances of the CPU stressors, one instance of the matrix stressor and three instances of the message queue stressor to test for five minutes.To run all stress tests in parallel, use the
–alloption:# stress-ng --all 2In this example,
stress-ngruns two instances of all stress tests in parallel.To run each different stressor in a specific sequence, use the
--seqoption.# stress-ng --seq 4 -t 20In this example,
stress-ngruns all the stressors one by one for 20 minutes, with the number of instances of each stressor matching the number of online CPUs.To exclude specific stressors from a test run, use the
-xoption:# stress-ng --seq 1 -x numa,matrix,hddIn this example,
stress-ngruns all stressors, one instance of each, excludingnuma,hddandkeystressors mechanisms.
32.3. Measuring CPU heat generation
To measure the CPU heat generation, the specified stressors generate high temperatures for a short time duration to test the system’s cooling reliability and stability under maximum heat generation. Using the --matrix-size option, you can measure CPU temperatures in degrees Celsius over a short time duration.
Prerequisites
- You have root privileges on the system.
Procedure
To test the CPU behavior at high temperatures for a specified time duration, run the following command:
# stress-ng --matrix 0 --matrix-size 64 --tz -t 60 stress-ng: info: [18351] dispatching hogs: 4 matrix stress-ng: info: [18351] successful run completed in 60.00s (1 min, 0.00 secs) stress-ng: info: [18351] matrix: stress-ng: info: [18351] x86_pkg_temp 88.00 °C stress-ng: info: [18351] acpitz 87.00 °CIn this example, the
stress-ngconfigures the processor package thermal zone to reach 88 degrees Celsius over the duration of 60 seconds.Optional: To print a report at the end of a run, use the
--tzoption:# stress-ng --cpu 0 --tz -t 60 stress-ng: info: [18065] dispatching hogs: 4 cpu stress-ng: info: [18065] successful run completed in 60.07s (1 min, 0.07 secs) stress-ng: info: [18065] cpu: stress-ng: info: [18065] x86_pkg_temp 88.75 °C stress-ng: info: [18065] acpitz 88.38 °C
32.4. Measuring test outcomes with bogo operations
The stress-ng tool can measure a stress test throughput by measuring the bogo operations per second. The size of a bogo operation depends on the stressor being run. The test outcomes are not precise, but they provide a rough estimate of the performance.
You must not use this measurement as an accurate benchmark metric. These estimates help to understand the system performance changes on different kernel versions or different compiler versions used to build stress-ng. Use the --metrics-brief option to display the total available bogo operations and the matrix stressor performance on your machine.
Prerequisites
- You have root privileges on the system.
Procedure
To measure test outcomes with bogo operations, use with the
--metrics-briefoption:# stress-ng --matrix 0 -t 60s --metrics-brief stress-ng: info: [17579] dispatching hogs: 4 matrix stress-ng: info: [17579] successful run completed in 60.01s (1 min, 0.01 secs) stress-ng: info: [17579] stressor bogo ops real time usr time sys time bogo ops/s bogo ops/s stress-ng: info: [17579] (secs) (secs) (secs) (real time) (usr+sys time) stress-ng: info: [17579] matrix 349322 60.00 203.23 0.19 5822.03 1717.25The
--metrics-briefoption displays the test outcomes and the total real-time bogo operations run by thematrixstressor for 60 seconds.
32.5. Generating a virtual memory pressure
When under memory pressure, the kernel starts writing pages out to swap. You can stress the virtual memory by using the --page-in option to force non-resident pages to swap back into the virtual memory. This causes the virtual machine to be heavily exercised. Using the --page-in option, you can enable this mode for the bigheap, mmap and virtual machine (vm) stressors. The --page-in option, touch allocated pages that are not in core, forcing them to page in.
Prerequisites
- You have root privileges on the system.
Procedure
To stress test a virtual memory, use the
--page-inoption:# stress-ng --vm 2 --vm-bytes 2G --mmap 2 --mmap-bytes 2G --page-inIn this example,
stress-ngtests memory pressure on a system with 4GB of memory, which is less than the allocated buffer sizes, 2 x 2GB ofvmstressor and 2 x 2GB ofmmapstressor with--page-inenabled.
32.6. Testing large interrupts loads on a device
Running timers at high frequency can generate a large interrupt load. The –timer stressor with an appropriately selected timer frequency can force many interrupts per second.
Prerequisites
- You have root permissions on the system.
Procedure
To generate an interrupt load, use the
--timeroption:# stress-ng --timer 32 --timer-freq 1000000In this example,
stress-ngtests 32 instances at 1MHz.
32.7. Generating major page faults in a program
With stress-ng, you can test and analyze the page fault rate by generating major page faults in a page that are not loaded in the memory. On new kernel versions, the userfaultfd mechanism notifies the fault finding threads about the page faults in the virtual memory layout of a process.
Prerequisites
- You have root permissions on the system.
Procedure
To generate major page faults on early kernel versions, use:
# stress-ng --fault 0 --perf -t 1mTo generate major page faults on new kernel versions, use:
# stress-ng --userfaultfd 0 --perf -t 1m
32.8. Viewing CPU stress test mechanisms
The CPU stress test contains methods to exercise a CPU. You can print an output to view all methods using the which option.
If you do not specify the test method, by default, the stressor checks all the stressors in a round-robin fashion to test the CPU with each stressor.
Prerequisites
- You have root permissions on the system.
Procedure
Print all available stressor mechanisms, use the
whichoption:# stress-ng --cpu-method which cpu-method must be one of: all ackermann bitops callfunc cdouble cfloat clongdouble correlate crc16 decimal32 decimal64 decimal128 dither djb2a double euler explog fft fibonacci float fnv1a gamma gcd gray hamming hanoi hyperbolic idct int128 int64 int32Specify a specific CPU stress method using the
--cpu-methodoption:# stress-ng --cpu 1 --cpu-method fft -t 1m
32.9. Using the verify mode
The verify mode validates the results when a test is active. It sanity checks the memory contents from a test run and reports any unexpected failures.
All stressors do not have the verify mode and enabling one will reduce the bogo operation statistics because of the extra verification step being run in this mode.
Prerequisites
- You have root permissions on the system.
Procedure
To validate a stress test results, use the
--verifyoption:# stress-ng --vm 1 --vm-bytes 2G --verify -vIn this example,
stress-ngprints the output for an exhaustive memory check on a virtually mapped memory using thevmstressor configured with--verifymode. It sanity checks the read and write results on the memory.
