Chapter 14. Low latency tuning

Procedure

1. Label the target nodes for configuration by running the following command:

   $ oc label node <node_name> node-role.kubernetes.io/worker-cnf="" 1

   1

   Replace <node_name> with the name of your node. This example applies the worker-cnf label.

2. Create a MachineConfigPool resource containing the target nodes:

   1. Create a YAML file that defines the MachineConfigPool resource:

      Example mcp-worker-cnf.yaml file

      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfigPool
      metadata:
        name: worker-cnf 1
        labels:
          machineconfiguration.openshift.io/role: worker-cnf 2
      spec:
        machineConfigSelector:
          matchExpressions:
            - {
                 key: machineconfiguration.openshift.io/role,
                 operator: In,
                 values: [worker, worker-cnf],
              }
        paused: false
        nodeSelector:
          matchLabels:
            node-role.kubernetes.io/worker-cnf: "" 3

      1

      Specify a name for the MachineConfigPool resource.

      2

      Specify a unique label for the machine config pool.

      3

      Specify the nodes with the target label that you defined.

   2. Apply the MachineConfigPool resource by running the following command:

      $ oc apply -f mcp-worker-cnf.yaml

      Example output

      machineconfigpool.machineconfiguration.openshift.io/worker-cnf created

Procedure

1. Navigate to the directory where you want to store the must-gather data.

2. Collect cluster information by running the following command:

   $ oc adm must-gather

   The command creates a folder with the must-gather data in your local directory with a naming format similar to the following: must-gather.local.1971646453781853027.

3. Optional: Create a compressed file from the must-gather directory:

   $ tar cvaf must-gather.tar.gz <must_gather_folder> 1

   1

   Replace with the name of the must-gather data folder.

   Note

   Compressed output is required if you are running the Performance Profile Creator wrapper script.

Procedure

1. Check the machine config pool by running the following command:

   $ oc get mcp

   Example output

   NAME         CONFIG                                                 UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
   master       rendered-master-58433c8c3c0b4ed5feef95434d455490       True      False      False      3              3                   3                     0                      8h
   worker       rendered-worker-668f56a164f151e4a853229729b6adc4       True      False      False      2              2                   2                     0                      8h
   worker-cnf   rendered-worker-cnf-668f56a164f151e4a853229729b6adc4   True      False      False      1              1                   1                     0                      79m

2. Use Podman to authenticate to registry.redhat.io by running the following command:

   $ podman login registry.redhat.io

   Username: <user_name>
   Password: <password>

3. Optional: Display help for the PPC tool by running the following command:

   $ podman run --rm --entrypoint performance-profile-creator registry.redhat.io/openshift4/ose-cluster-node-tuning-rhel9-operator:v4.16 -h

   Example output

   A tool that automates creation of Performance Profiles
   
   Usage:
     performance-profile-creator [flags]
   
   Flags:
         --disable-ht                        Disable Hyperthreading
     -h, --help                              help for performance-profile-creator
         --info string                       Show cluster information; requires --must-gather-dir-path, ignore the other arguments. [Valid values: log, json] (default "log")
         --mcp-name string                   MCP name corresponding to the target machines (required)
         --must-gather-dir-path string       Must gather directory path (default "must-gather")
         --offlined-cpu-count int            Number of offlined CPUs
         --per-pod-power-management          Enable Per Pod Power Management
         --power-consumption-mode string     The power consumption mode.  [Valid values: default, low-latency, ultra-low-latency] (default "default")
         --profile-name string               Name of the performance profile to be created (default "performance")
         --reserved-cpu-count int            Number of reserved CPUs (required)
         --rt-kernel                         Enable Real Time Kernel (required)
         --split-reserved-cpus-across-numa   Split the Reserved CPUs across NUMA nodes
         --topology-manager-policy string    Kubelet Topology Manager Policy of the performance profile to be created. [Valid values: single-numa-node, best-effort, restricted] (default "restricted")
         --user-level-networking             Run with User level Networking(DPDK) enabled

4. To display information about the cluster, run the PPC tool with the log argument by running the following command:

   $ podman run --entrypoint performance-profile-creator -v <path_to_must_gather>:/must-gather:z registry.redhat.io/openshift4/ose-cluster-node-tuning-rhel9-operator:v4.16 --info log --must-gather-dir-path /must-gather

   • --entrypoint performance-profile-creator defines the performance profile creator as a new entry point to podman.

   • -v <path_to_must_gather> specifies the path to either of the following components:

     • The directory containing the must-gather data.
     • An existing directory containing the must-gather decompressed .tar file.

   • --info log specifies a value for the output format.

     Example output

     level=info msg="Cluster info:"
     level=info msg="MCP 'master' nodes:"
     level=info msg=---
     level=info msg="MCP 'worker' nodes:"
     level=info msg="Node: host.example.com (NUMA cells: 1, HT: true)"
     level=info msg="NUMA cell 0 : [0 1 2 3]"
     level=info msg="CPU(s): 4"
     level=info msg="Node: host1.example.com (NUMA cells: 1, HT: true)"
     level=info msg="NUMA cell 0 : [0 1 2 3]"
     level=info msg="CPU(s): 4"
     level=info msg=---
     level=info msg="MCP 'worker-cnf' nodes:"
     level=info msg="Node: host2.example.com (NUMA cells: 1, HT: true)"
     level=info msg="NUMA cell 0 : [0 1 2 3]"
     level=info msg="CPU(s): 4"
     level=info msg=---

5. Create a performance profile by running the following command. The example uses sample PPC arguments and values:

   $ podman run --entrypoint performance-profile-creator -v <path_to_must_gather>:/must-gather:z registry.redhat.io/openshift4/ose-cluster-node-tuning-rhel9-operator:v4.16 --mcp-name=worker-cnf --reserved-cpu-count=1 --rt-kernel=true --split-reserved-cpus-across-numa=false --must-gather-dir-path /must-gather --power-consumption-mode=ultra-low-latency --offlined-cpu-count=1 > my-performance-profile.yaml

   • -v <path_to_must_gather> specifies the path to either of the following components:

     • The directory containing the must-gather data.
     • The directory containing the must-gather decompressed .tar file.
   • --mcp-name=worker-cnf specifies the worker-=cnf machine config pool.
   • --reserved-cpu-count=1 specifies one reserved CPU.
   • --rt-kernel=true enables the real-time kernel.
   • --split-reserved-cpus-across-numa=false disables reserved CPUs splitting across NUMA nodes.
   • --power-consumption-mode=ultra-low-latency specifies minimal latency at the cost of increased power consumption.

   • --offlined-cpu-count=1 specifies one offlined CPU.

     Note

     The mcp-name argument in this example is set to worker-cnf based on the output of the command oc get mcp. For single-node OpenShift use --mcp-name=master.

     Example output

     level=info msg="Nodes targeted by worker-cnf MCP are: [worker-2]"
     level=info msg="NUMA cell(s): 1"
     level=info msg="NUMA cell 0 : [0 1 2 3]"
     level=info msg="CPU(s): 4"
     level=info msg="1 reserved CPUs allocated: 0 "
     level=info msg="2 isolated CPUs allocated: 2-3"
     level=info msg="Additional Kernel Args based on configuration: []"

6. Review the created YAML file by running the following command:

   $ cat my-performance-profile.yaml

   Example output

   ---
   apiVersion: performance.openshift.io/v2
   kind: PerformanceProfile
   metadata:
     name: performance
   spec:
     cpu:
       isolated: 2-3
       offlined: "1"
       reserved: "0"
     machineConfigPoolSelector:
       machineconfiguration.openshift.io/role: worker-cnf
     nodeSelector:
       node-role.kubernetes.io/worker-cnf: ""
     numa:
       topologyPolicy: restricted
     realTimeKernel:
       enabled: true
     workloadHints:
       highPowerConsumption: true
       perPodPowerManagement: false
       realTime: true

7. Apply the generated profile:

   $ oc apply -f my-performance-profile.yaml

   Example output

   performanceprofile.performance.openshift.io/performance created

Procedure

1. Create a file on your local machine named, for example, run-perf-profile-creator.sh:

   $ vi run-perf-profile-creator.sh

2. Paste the following code into the file:

   #!/bin/bash
   
   readonly CONTAINER_RUNTIME=${CONTAINER_RUNTIME:-podman}
   readonly CURRENT_SCRIPT=$(basename "$0")
   readonly CMD="${CONTAINER_RUNTIME} run --entrypoint performance-profile-creator"
   readonly IMG_EXISTS_CMD="${CONTAINER_RUNTIME} image exists"
   readonly IMG_PULL_CMD="${CONTAINER_RUNTIME} image pull"
   readonly MUST_GATHER_VOL="/must-gather"
   
   NTO_IMG="registry.redhat.io/openshift4/ose-cluster-node-tuning-rhel9-operator:v4.16"
   MG_TARBALL=""
   DATA_DIR=""
   
   usage() {
     print "Wrapper usage:"
     print "  ${CURRENT_SCRIPT} [-h] [-p image][-t path] -- [performance-profile-creator flags]"
     print ""
     print "Options:"
     print "   -h                 help for ${CURRENT_SCRIPT}"
     print "   -p                 Node Tuning Operator image"
     print "   -t                 path to a must-gather tarball"
   
     ${IMG_EXISTS_CMD} "${NTO_IMG}" && ${CMD} "${NTO_IMG}" -h
   }
   
   function cleanup {
     [ -d "${DATA_DIR}" ] && rm -rf "${DATA_DIR}"
   }
   trap cleanup EXIT
   
   exit_error() {
     print "error: $*"
     usage
     exit 1
   }
   
   print() {
     echo  "$*" >&2
   }
   
   check_requirements() {
     ${IMG_EXISTS_CMD} "${NTO_IMG}" || ${IMG_PULL_CMD} "${NTO_IMG}" || \
         exit_error "Node Tuning Operator image not found"
   
     [ -n "${MG_TARBALL}" ] || exit_error "Must-gather tarball file path is mandatory"
     [ -f "${MG_TARBALL}" ] || exit_error "Must-gather tarball file not found"
   
     DATA_DIR=$(mktemp -d -t "${CURRENT_SCRIPT}XXXX") || exit_error "Cannot create the data directory"
     tar -zxf "${MG_TARBALL}" --directory "${DATA_DIR}" || exit_error "Cannot decompress the must-gather tarball"
     chmod a+rx "${DATA_DIR}"
   
     return 0
   }
   
   main() {
     while getopts ':hp:t:' OPT; do
       case "${OPT}" in
         h)
           usage
           exit 0
           ;;
         p)
           NTO_IMG="${OPTARG}"
           ;;
         t)
           MG_TARBALL="${OPTARG}"
           ;;
         ?)
           exit_error "invalid argument: ${OPTARG}"
           ;;
       esac
     done
     shift $((OPTIND - 1))
   
     check_requirements || exit 1
   
     ${CMD} -v "${DATA_DIR}:${MUST_GATHER_VOL}:z" "${NTO_IMG}" "$@" --must-gather-dir-path "${MUST_GATHER_VOL}"
     echo "" 1>&2
   }
   
   main "$@"

3. Add execute permissions for everyone on this script:

   $ chmod a+x run-perf-profile-creator.sh

4. Use Podman to authenticate to registry.redhat.io by running the following command:

   $ podman login registry.redhat.io

   Username: <user_name>
   Password: <password>

5. Optional: Display help for the PPC tool by running the following command:

   $ ./run-perf-profile-creator.sh -h

   Example output

   Wrapper usage:
     run-perf-profile-creator.sh [-h] [-p image][-t path] -- [performance-profile-creator flags]
   
   Options:
      -h                 help for run-perf-profile-creator.sh
      -p                 Node Tuning Operator image
      -t                 path to a must-gather tarball
   A tool that automates creation of Performance Profiles
   
   Usage:
     performance-profile-creator [flags]
   
   Flags:
         --disable-ht                        Disable Hyperthreading
     -h, --help                              help for performance-profile-creator
         --info string                       Show cluster information; requires --must-gather-dir-path, ignore the other arguments. [Valid values: log, json] (default "log")
         --mcp-name string                   MCP name corresponding to the target machines (required)
         --must-gather-dir-path string       Must gather directory path (default "must-gather")
         --offlined-cpu-count int            Number of offlined CPUs
         --per-pod-power-management          Enable Per Pod Power Management
         --power-consumption-mode string     The power consumption mode.  [Valid values: default, low-latency, ultra-low-latency] (default "default")
         --profile-name string               Name of the performance profile to be created (default "performance")
         --reserved-cpu-count int            Number of reserved CPUs (required)
         --rt-kernel                         Enable Real Time Kernel (required)
         --split-reserved-cpus-across-numa   Split the Reserved CPUs across NUMA nodes
         --topology-manager-policy string    Kubelet Topology Manager Policy of the performance profile to be created. [Valid values: single-numa-node, best-effort, restricted] (default "restricted")
         --user-level-networking             Run with User level Networking(DPDK) enabled
         --enable-hardware-tuning            Enable setting maximum CPU frequencies

   Note

   You can optionally set a path for the Node Tuning Operator image using the -p option. If you do not set a path, the wrapper script uses the default image: registry.redhat.io/openshift4/ose-cluster-node-tuning-rhel9-operator:v4.16.

6. To display information about the cluster, run the PPC tool with the log argument by running the following command:

   $ ./run-perf-profile-creator.sh -t /<path_to_must_gather_dir>/must-gather.tar.gz -- --info=log

   • -t /<path_to_must_gather_dir>/must-gather.tar.gz specifies the path to directory containing the must-gather tarball. This is a required argument for the wrapper script.

     Example output

     level=info msg="Cluster info:"
     level=info msg="MCP 'master' nodes:"
     level=info msg=---
     level=info msg="MCP 'worker' nodes:"
     level=info msg="Node: host.example.com (NUMA cells: 1, HT: true)"
     level=info msg="NUMA cell 0 : [0 1 2 3]"
     level=info msg="CPU(s): 4"
     level=info msg="Node: host1.example.com (NUMA cells: 1, HT: true)"
     level=info msg="NUMA cell 0 : [0 1 2 3]"
     level=info msg="CPU(s): 4"
     level=info msg=---
     level=info msg="MCP 'worker-cnf' nodes:"
     level=info msg="Node: host2.example.com (NUMA cells: 1, HT: true)"
     level=info msg="NUMA cell 0 : [0 1 2 3]"
     level=info msg="CPU(s): 4"
     level=info msg=---

7. Create a performance profile by running the following command.

   $ ./run-perf-profile-creator.sh -t /path-to-must-gather/must-gather.tar.gz -- --mcp-name=worker-cnf --reserved-cpu-count=1 --rt-kernel=true --split-reserved-cpus-across-numa=false --power-consumption-mode=ultra-low-latency --offlined-cpu-count=1 > my-performance-profile.yaml

   This example uses sample PPC arguments and values.

   • --mcp-name=worker-cnf specifies the worker-=cnf machine config pool.
   • --reserved-cpu-count=1 specifies one reserved CPU.
   • --rt-kernel=true enables the real-time kernel.
   • --split-reserved-cpus-across-numa=false disables reserved CPUs splitting across NUMA nodes.
   • --power-consumption-mode=ultra-low-latency specifies minimal latency at the cost of increased power consumption.

   • --offlined-cpu-count=1 specifies one offlined CPUs.

     Note

     The mcp-name argument in this example is set to worker-cnf based on the output of the command oc get mcp. For single-node OpenShift use --mcp-name=master.

8. Review the created YAML file by running the following command:

   $ cat my-performance-profile.yaml

   Example output

   ---
   apiVersion: performance.openshift.io/v2
   kind: PerformanceProfile
   metadata:
     name: performance
   spec:
     cpu:
       isolated: 2-3
       offlined: "1"
       reserved: "0"
     machineConfigPoolSelector:
       machineconfiguration.openshift.io/role: worker-cnf
     nodeSelector:
       node-role.kubernetes.io/worker-cnf: ""
     numa:
       topologyPolicy: restricted
     realTimeKernel:
       enabled: true
     workloadHints:
       highPowerConsumption: true
       perPodPowerManagement: false
       realTime: true

9. Apply the generated profile:

   $ oc apply -f my-performance-profile.yaml

   Example output

   performanceprofile.performance.openshift.io/performance created

1. Create a performance profile that is appropriate for your hardware and topology.

2. Set nosmt as an additional kernel argument. The following example performance profile illustrates this setting:

   apiVersion: performance.openshift.io/v2
   kind: PerformanceProfile
   metadata:
     name: example-performanceprofile
   spec:
     additionalKernelArgs:
       - nmi_watchdog=0
       - audit=0
       - mce=off
       - processor.max_cstate=1
       - idle=poll
       - intel_idle.max_cstate=0
       - nosmt
     cpu:
       isolated: 2-3
       reserved: 0-1
     hugepages:
       defaultHugepagesSize: 1G
       pages:
         - count: 2
           node: 0
           size: 1G
     nodeSelector:
       node-role.kubernetes.io/performance: ''
     realTimeKernel:
       enabled: true

   Note

   When you configure reserved and isolated CPUs, the infra containers in pods use the reserved CPUs and the application containers use the isolated CPUs.

Procedure

1. Log in to the OpenShift Container Platform cluster as a user with cluster-admin privileges.
2. Set the performance profile apiVersion to use performance.openshift.io/v2.
3. Remove the globallyDisableIrqLoadBalancing field or set it to false.

4. Set the appropriate isolated and reserved CPUs. The following snippet illustrates a profile that reserves 2 CPUs. IRQ load-balancing is enabled for pods running on the isolated CPU set:

   apiVersion: performance.openshift.io/v2
   kind: PerformanceProfile
   metadata:
     name: dynamic-irq-profile
   spec:
     cpu:
       isolated: 2-5
       reserved: 0-1
   ...

   Note

   When you configure reserved and isolated CPUs, operating system processes, kernel processes, and systemd services run on reserved CPUs. Infrastructure pods run on any CPU except where the low latency workload is running. Low latency workload pods run on exclusive CPUs from the isolated pool. For more information, see "Restricting CPUs for infra and application containers".

Procedure

1. Log in to the OpenShift Container Platform cluster running the Node Tuning Operator as a user with cluster-admin privileges.
2. Create and apply a performance profile appropriate for your hardware and topology. For guidance on creating a profile, see the "Creating a performance profile" section.

3. Edit this created performance profile:

   $ oc edit -f <your_profile_name>.yaml

4. Populate the spec field with the net object. The object list can contain two fields:

   • userLevelNetworking is a required field specified as a boolean flag. If userLevelNetworking is true, the queue count is set to the reserved CPU count for all supported devices. The default is false.

   • devices is an optional field specifying a list of devices that will have the queues set to the reserved CPU count. If the device list is empty, the configuration applies to all network devices. The configuration is as follows:

     • interfaceName: This field specifies the interface name, and it supports shell-style wildcards, which can be positive or negative.

       • Example wildcard syntax is as follows: <string> .*
       • Negative rules are prefixed with an exclamation mark. To apply the net queue changes to all devices other than the excluded list, use !<device>, for example, !eno1.
     • vendorID: The network device vendor ID represented as a 16-bit hexadecimal number with a 0x prefix.

     • deviceID: The network device ID (model) represented as a 16-bit hexadecimal number with a 0x prefix.

       Note

       When a deviceID is specified, the vendorID must also be defined. A device that matches all of the device identifiers specified in a device entry interfaceName, vendorID, or a pair of vendorID plus deviceID qualifies as a network device. This network device then has its net queues count set to the reserved CPU count.

       When two or more devices are specified, the net queues count is set to any net device that matches one of them.

5. Set the queue count to the reserved CPU count for all devices by using this example performance profile:

   apiVersion: performance.openshift.io/v2
   kind: PerformanceProfile
   metadata:
     name: manual
   spec:
     cpu:
       isolated: 3-51,55-103
       reserved: 0-2,52-54
     net:
       userLevelNetworking: true
     nodeSelector:
       node-role.kubernetes.io/worker-cnf: ""

6. Set the queue count to the reserved CPU count for all devices matching any of the defined device identifiers by using this example performance profile:

   apiVersion: performance.openshift.io/v2
   kind: PerformanceProfile
   metadata:
     name: manual
   spec:
     cpu:
       isolated: 3-51,55-103
       reserved: 0-2,52-54
     net:
       userLevelNetworking: true
       devices:
       - interfaceName: "eth0"
       - interfaceName: "eth1"
       - vendorID: "0x1af4"
         deviceID: "0x1000"
     nodeSelector:
       node-role.kubernetes.io/worker-cnf: ""

7. Set the queue count to the reserved CPU count for all devices starting with the interface name eth by using this example performance profile:

   apiVersion: performance.openshift.io/v2
   kind: PerformanceProfile
   metadata:
     name: manual
   spec:
     cpu:
       isolated: 3-51,55-103
       reserved: 0-2,52-54
     net:
       userLevelNetworking: true
       devices:
       - interfaceName: "eth*"
     nodeSelector:
       node-role.kubernetes.io/worker-cnf: ""

8. Set the queue count to the reserved CPU count for all devices with an interface named anything other than eno1 by using this example performance profile:

   apiVersion: performance.openshift.io/v2
   kind: PerformanceProfile
   metadata:
     name: manual
   spec:
     cpu:
       isolated: 3-51,55-103
       reserved: 0-2,52-54
     net:
       userLevelNetworking: true
       devices:
       - interfaceName: "!eno1"
     nodeSelector:
       node-role.kubernetes.io/worker-cnf: ""

9. Set the queue count to the reserved CPU count for all devices that have an interface name eth0, vendorID of 0x1af4, and deviceID of 0x1000 by using this example performance profile:

   apiVersion: performance.openshift.io/v2
   kind: PerformanceProfile
   metadata:
     name: manual
   spec:
     cpu:
       isolated: 3-51,55-103
       reserved: 0-2,52-54
     net:
       userLevelNetworking: true
       devices:
       - interfaceName: "eth0"
       - vendorID: "0x1af4"
         deviceID: "0x1000"
     nodeSelector:
       node-role.kubernetes.io/worker-cnf: ""

10. Apply the updated performance profile:

    $ oc apply -f <your_profile_name>.yaml

1. The associated machine config pool is in a degraded state:

   # oc get mcp

   Example output

   NAME         CONFIG                                                 UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
   master       rendered-master-2ee57a93fa6c9181b546ca46e1571d2d       True      False      False      3              3                   3                     0                      2d21h
   worker       rendered-worker-d6b2bdc07d9f5a59a6b68950acf25e5f       True      False      False      2              2                   2                     0                      2d21h
   worker-cnf   rendered-worker-cnf-6c838641b8a08fff08dbd8b02fb63f7c   False     True       True       2              1                   1                     1                      2d20h

2. The describe section of the MCP shows the reason:

   # oc describe mcp worker-cnf

   Example output

     Message:               Node node-worker-cnf is reporting: "prepping update:
     machineconfig.machineconfiguration.openshift.io \"rendered-worker-cnf-40b9996919c08e335f3ff230ce1d170\" not
     found"
       Reason:                1 nodes are reporting degraded status on sync

3. The degraded state should also appear under the performance profile status field marked as degraded = true:

   # oc describe performanceprofiles performance

   Example output

   Message: Machine config pool worker-cnf Degraded Reason: 1 nodes are reporting degraded status on sync.
   Machine config pool worker-cnf Degraded Message: Node yquinn-q8s5v-w-b-z5lqn.c.openshift-gce-devel.internal is
   reporting: "prepping update: machineconfig.machineconfiguration.openshift.io
   \"rendered-worker-cnf-40b9996919c08e335f3ff230ce1d170\" not found".    Reason:  MCPDegraded
      Status:  True
      Type:    Degraded

Procedure

1. Navigate to the directory where you want to store the must-gather data.

2. Collect debugging information by running the following command:

   $ oc adm must-gather

   Example output

   [must-gather      ] OUT Using must-gather plug-in image: quay.io/openshift-release
   When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information:
   ClusterID: 829er0fa-1ad8-4e59-a46e-2644921b7eb6
   ClusterVersion: Stable at "<cluster_version>"
   ClusterOperators:
   	All healthy and stable
   
   
   [must-gather      ] OUT namespace/openshift-must-gather-8fh4x created
   [must-gather      ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-rhlgc created
   [must-gather-5564g] POD 2023-07-17T10:17:37.610340849Z Gathering data for ns/openshift-cluster-version...
   [must-gather-5564g] POD 2023-07-17T10:17:38.786591298Z Gathering data for ns/default...
   [must-gather-5564g] POD 2023-07-17T10:17:39.117418660Z Gathering data for ns/openshift...
   [must-gather-5564g] POD 2023-07-17T10:17:39.447592859Z Gathering data for ns/kube-system...
   [must-gather-5564g] POD 2023-07-17T10:17:39.803381143Z Gathering data for ns/openshift-etcd...
   
   ...
   
   Reprinting Cluster State:
   When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information:
   ClusterID: 829er0fa-1ad8-4e59-a46e-2644921b7eb6
   ClusterVersion: Stable at "<cluster_version>"
   ClusterOperators:
   	All healthy and stable

3. Create a compressed file from the must-gather directory that was created in your working directory. For example, on a computer that uses a Linux operating system, run the following command:

   $ tar cvaf must-gather.tar.gz must-gather-local.54213423446277122891

   1

   Replace must-gather-local.5421342344627712289// with the directory name created by the must-gather tool.

   Note

   Create a compressed file to attach the data to a support case or to use with the Performance Profile Creator wrapper script when you create a performance profile.

4. Attach the compressed file to your support case on the Red Hat Customer Portal.