Chapter 16. Creating a performance profile
Learn about the Performance Profile Creator (PPC) and how you can use it to create a performance profile.
16.1. About the Performance Profile Creator
The Performance Profile Creator (PPC) is a command-line tool, delivered with the Performance Addon Operator, used to create the performance profile. The tool consumes must-gather
data from the cluster and several user-supplied profile arguments. The PPC generates a performance profile that is appropriate for your hardware and topology.
The tool is run by one of the following methods:
-
Invoking
podman
- Calling a wrapper script
16.1.1. Gathering data about your cluster using the must-gather command
The Performance Profile Creator (PPC) tool requires must-gather
data. As a cluster administrator, run the must-gather
command to capture information about your cluster.
Prerequisites
-
Access to the cluster as a user with the
cluster-admin
role. - Access to the Performance Addon Operator image.
-
The OpenShift CLI (
oc
) installed.
Procedure
-
Navigate to the directory where you want to store the
must-gather
data. Run
must-gather
on your cluster:$ oc adm must-gather --image=<PAO_image> --dest-dir=<dir>
NoteThe
must-gather
command must be run with theperformance-addon-operator-must-gather
image. The output can optionally be compressed. Compressed output is required if you are running the Performance Profile Creator wrapper script.Example
$ oc adm must-gather --image=registry.redhat.io/openshift4/performance-addon-operator-must-gather-rhel8:v4.9 --dest-dir=must-gather
Create a compressed file from the
must-gather
directory:$ tar cvaf must-gather.tar.gz must-gather/
16.1.2. Running the Performance Profile Creator using podman
As a cluster administrator, you can run podman
and the Performance Profile Creator to create a performance profile.
Prerequisites
-
Access to the cluster as a user with the
cluster-admin
role. - A cluster installed on bare metal hardware.
-
A node with
podman
and OpenShift CLI (oc
) installed.
Procedure
Check the machine config pool:
$ oc get mcp
Example output
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-acd1358917e9f98cbdb599aea622d78b True False False 3 3 3 0 22h worker-cnf rendered-worker-cnf-1d871ac76e1951d32b2fe92369879826 False True False 2 1 1 0 22h
Use Podman to authenticate to
registry.redhat.io
:$ podman login registry.redhat.io
Username: myrhusername Password: ************
Optional: Display help for the PPC tool:
$ podman run --entrypoint performance-profile-creator registry.redhat.io/openshift4/performance-addon-rhel8-operator:v4.9 -h
Example output
A tool that automates creation of Performance Profiles Usage: performance-profile-creator [flags] Flags: --disable-ht Disable Hyperthreading -h, --help help for performance-profile-creator --info string Show cluster information; requires --must-gather-dir-path, ignore the other arguments. [Valid values: log, json] (default "log") --mcp-name string MCP name corresponding to the target machines (required) --must-gather-dir-path string Must gather directory path (default "must-gather") --power-consumption-mode string The power consumption mode. [Valid values: default, low-latency, ultra-low-latency] (default "default") --profile-name string Name of the performance profile to be created (default "performance") --reserved-cpu-count int Number of reserved CPUs (required) --rt-kernel Enable Real Time Kernel (required) --split-reserved-cpus-across-numa Split the Reserved CPUs across NUMA nodes --topology-manager-policy string Kubelet Topology Manager Policy of the performance profile to be created. [Valid values: single-numa-node, best-effort, restricted] (default "restricted") --user-level-networking Run with User level Networking(DPDK) enabled
Run the Performance Profile Creator tool in discovery mode:
NoteDiscovery mode inspects your cluster using the output from
must-gather
. The output produced includes information on:- The NUMA cell partitioning with the allocated CPU ids
- Whether hyperthreading is enabled
Using this information you can set appropriate values for some of the arguments supplied to the Performance Profile Creator tool.
$ podman run --entrypoint performance-profile-creator -v /must-gather:/must-gather:z registry.redhat.io/openshift4/performance-addon-rhel8-operator:v4.9 --info log --must-gather-dir-path /must-gather
NoteThis command uses the performance profile creator as a new entry point to
podman
. It maps themust-gather
data for the host into the container image and invokes the required user-supplied profile arguments to produce themy-performance-profile.yaml
file.The
-v
option can be the path to either:-
The
must-gather
output directory -
An existing directory containing the
must-gather
decompressed tarball
The
info
option requires a value which specifies the output format. Possible values are log and JSON. The JSON format is reserved for debugging.Run
podman
:$ podman run --entrypoint performance-profile-creator -v /must-gather:/must-gather:z registry.redhat.io/openshift4/performance-addon-rhel8-operator:v4.9 --mcp-name=worker-cnf --reserved-cpu-count=20 --rt-kernel=true --split-reserved-cpus-across-numa=false --topology-manager-policy=single-numa-node --must-gather-dir-path /must-gather --power-consumption-mode=ultra-low-latency > my-performance-profile.yaml
NoteThe Performance Profile Creator arguments are shown in the Performance Profile Creator arguments table. The following arguments are required:
-
reserved-cpu-count
-
mcp-name
-
rt-kernel
The
mcp-name
argument in this example is set toworker-cnf
based on the output of the commandoc get mcp
. For single-node OpenShift use--mcp-name=master
.-
Review the created YAML file:
$ cat my-performance-profile.yaml
Example output
apiVersion: performance.openshift.io/v2 kind: PerformanceProfile metadata: name: performance spec: additionalKernelArgs: - nmi_watchdog=0 - audit=0 - mce=off - processor.max_cstate=1 - intel_idle.max_cstate=0 - idle=poll cpu: isolated: 1,3,5,7,9,11,13,15,17,19-39,41,43,45,47,49,51,53,55,57,59-79 reserved: 0,2,4,6,8,10,12,14,16,18,40,42,44,46,48,50,52,54,56,58 nodeSelector: node-role.kubernetes.io/worker-cnf: "" numa: topologyPolicy: single-numa-node realTimeKernel: enabled: true
Apply the generated profile:
NoteInstall the Performance Addon Operator before applying the profile.
$ oc apply -f my-performance-profile.yaml
16.1.2.1. How to run podman
to create a performance profile
The following example illustrates how to run podman
to create a performance profile with 20 reserved CPUs that are to be split across the NUMA nodes.
Node hardware configuration:
- 80 CPUs
- Hyperthreading enabled
- Two NUMA nodes
- Even numbered CPUs run on NUMA node 0 and odd numbered CPUs run on NUMA node 1
Run podman
to create the performance profile:
$ podman run --entrypoint performance-profile-creator -v /must-gather:/must-gather:z registry.redhat.io/openshift4/performance-addon-rhel8-operator:v4.9 --mcp-name=worker-cnf --reserved-cpu-count=20 --rt-kernel=true --split-reserved-cpus-across-numa=true --must-gather-dir-path /must-gather > my-performance-profile.yaml
The created profile is described in the following YAML:
apiVersion: performance.openshift.io/v2 kind: PerformanceProfile metadata: name: performance spec: cpu: isolated: 10-39,50-79 reserved: 0-9,40-49 nodeSelector: node-role.kubernetes.io/worker-cnf: "" numa: topologyPolicy: restricted realTimeKernel: enabled: true
In this case, 10 CPUs are reserved on NUMA node 0 and 10 are reserved on NUMA node 1.
16.1.3. Running the Performance Profile Creator wrapper script
The performance profile wrapper script simplifies the running of the Performance Profile Creator (PPC) tool. It hides the complexities associated with running podman
and specifying the mapping directories and it enables the creation of the performance profile.
Prerequisites
- Access to the Performance Addon Operator image.
-
Access to the
must-gather
tarball.
Procedure
Create a file on your local machine named, for example,
run-perf-profile-creator.sh
:$ vi run-perf-profile-creator.sh
Paste the following code into the file:
#!/bin/bash readonly CONTAINER_RUNTIME=${CONTAINER_RUNTIME:-podman} readonly CURRENT_SCRIPT=$(basename "$0") readonly CMD="${CONTAINER_RUNTIME} run --entrypoint performance-profile-creator" readonly IMG_EXISTS_CMD="${CONTAINER_RUNTIME} image exists" readonly IMG_PULL_CMD="${CONTAINER_RUNTIME} image pull" readonly MUST_GATHER_VOL="/must-gather" PAO_IMG="registry.redhat.io/openshift4/performance-addon-rhel8-operator:v4.9" MG_TARBALL="" DATA_DIR="" usage() { print "Wrapper usage:" print " ${CURRENT_SCRIPT} [-h] [-p image][-t path] -- [performance-profile-creator flags]" print "" print "Options:" print " -h help for ${CURRENT_SCRIPT}" print " -p Performance Addon Operator image" print " -t path to a must-gather tarball" ${IMG_EXISTS_CMD} "${PAO_IMG}" && ${CMD} "${PAO_IMG}" -h } function cleanup { [ -d "${DATA_DIR}" ] && rm -rf "${DATA_DIR}" } trap cleanup EXIT exit_error() { print "error: $*" usage exit 1 } print() { echo "$*" >&2 } check_requirements() { ${IMG_EXISTS_CMD} "${PAO_IMG}" || ${IMG_PULL_CMD} "${PAO_IMG}" || \ exit_error "Performance Addon Operator image not found" [ -n "${MG_TARBALL}" ] || exit_error "Must-gather tarball file path is mandatory" [ -f "${MG_TARBALL}" ] || exit_error "Must-gather tarball file not found" DATA_DIR=$(mktemp -d -t "${CURRENT_SCRIPT}XXXX") || exit_error "Cannot create the data directory" tar -zxf "${MG_TARBALL}" --directory "${DATA_DIR}" || exit_error "Cannot decompress the must-gather tarball" chmod a+rx "${DATA_DIR}" return 0 } main() { while getopts ':hp:t:' OPT; do case "${OPT}" in h) usage exit 0 ;; p) PAO_IMG="${OPTARG}" ;; t) MG_TARBALL="${OPTARG}" ;; ?) exit_error "invalid argument: ${OPTARG}" ;; esac done shift $((OPTIND - 1)) check_requirements || exit 1 ${CMD} -v "${DATA_DIR}:${MUST_GATHER_VOL}:z" "${PAO_IMG}" "$@" --must-gather-dir-path "${MUST_GATHER_VOL}" echo "" 1>&2 } main "$@"
Add execute permissions for everyone on this script:
$ chmod a+x run-perf-profile-creator.sh
Optional: Display the
run-perf-profile-creator.sh
command usage:$ ./run-perf-profile-creator.sh -h
Expected output
Wrapper usage: run-perf-profile-creator.sh [-h] [-p image][-t path] -- [performance-profile-creator flags] Options: -h help for run-perf-profile-creator.sh -p Performance Addon Operator image 1 -t path to a must-gather tarball 2 A tool that automates creation of Performance Profiles Usage: performance-profile-creator [flags] Flags: --disable-ht Disable Hyperthreading -h, --help help for performance-profile-creator --info string Show cluster information; requires --must-gather-dir-path, ignore the other arguments. [Valid values: log, json] (default "log") --mcp-name string MCP name corresponding to the target machines (required) --must-gather-dir-path string Must gather directory path (default "must-gather") --power-consumption-mode string The power consumption mode. [Valid values: default, low-latency, ultra-low-latency] (default "default") --profile-name string Name of the performance profile to be created (default "performance") --reserved-cpu-count int Number of reserved CPUs (required) --rt-kernel Enable Real Time Kernel (required) --split-reserved-cpus-across-numa Split the Reserved CPUs across NUMA nodes --topology-manager-policy string Kubelet Topology Manager Policy of the performance profile to be created. [Valid values: single-numa-node, best-effort, restricted] (default "restricted") --user-level-networking Run with User level Networking(DPDK) enabled
NoteThere two types of arguments:
-
Wrapper arguments namely
-h
,-p
and-t
- PPC arguments
-
Wrapper arguments namely
Run the performance profile creator tool in discovery mode:
NoteDiscovery mode inspects your cluster using the output from
must-gather
. The output produced includes information on:- The NUMA cell partitioning with the allocated CPU IDs
- Whether hyperthreading is enabled
Using this information you can set appropriate values for some of the arguments supplied to the Performance Profile Creator tool.
$ ./run-perf-profile-creator.sh -t /must-gather/must-gather.tar.gz -- --info=log
NoteThe
info
option requires a value which specifies the output format. Possible values are log and JSON. The JSON format is reserved for debugging.Check the machine config pool:
$ oc get mcp
Example output
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-acd1358917e9f98cbdb599aea622d78b True False False 3 3 3 0 22h worker-cnf rendered-worker-cnf-1d871ac76e1951d32b2fe92369879826 False True False 2 1 1 0 22h
Create a performance profile:
$ ./run-perf-profile-creator.sh -t /must-gather/must-gather.tar.gz -- --mcp-name=worker-cnf --reserved-cpu-count=2 --rt-kernel=true > my-performance-profile.yaml
NoteThe Performance Profile Creator arguments are shown in the Performance Profile Creator arguments table. The following arguments are required:
-
reserved-cpu-count
-
mcp-name
-
rt-kernel
The
mcp-name
argument in this example is set toworker-cnf
based on the output of the commandoc get mcp
. For single-node OpenShift use--mcp-name=master
.-
Review the created YAML file:
$ cat my-performance-profile.yaml
Example output
apiVersion: performance.openshift.io/v2 kind: PerformanceProfile metadata: name: performance spec: cpu: isolated: 1-39,41-79 reserved: 0,40 nodeSelector: node-role.kubernetes.io/worker-cnf: "" numa: topologyPolicy: restricted realTimeKernel: enabled: false
Apply the generated profile:
NoteInstall the Performance Addon Operator before applying the profile.
$ oc apply -f my-performance-profile.yaml
16.1.4. Performance Profile Creator arguments
Argument | Description |
---|---|
| Disable hyperthreading.
Possible values:
Default: Warning
If this argument is set to |
|
This captures cluster information and is used in discovery mode only. Discovery mode also requires the Possible values:
Default: |
|
MCP name for example |
| Must gather directory path. This parameter is required.
When the user runs the tool with the wrapper script |
| The power consumption mode. Possible values:
Default: |
|
Name of the performance profile to create. Default: |
| Number of reserved CPUs. This parameter is required. Note This must be a natural number. A value of 0 is not allowed. |
| Enable real-time kernel. This parameter is required.
Possible values: |
| Split the reserved CPUs across NUMA nodes.
Possible values:
Default: |
| Kubelet Topology Manager policy of the performance profile to be created. Possible values:
Default: |
| Run with user level networking (DPDK) enabled.
Possible values:
Default: |
16.2. Additional resources
-
For more information about the
must-gather
tool, see Gathering data about your cluster.