14.4. 运行延迟测试

先决条件

• 已在集群中安装了实时内核。
• 您使用客户门户网站凭证登录到 registry.redhat.io。

流程

• 要运行 hwlatdetect 测试，请运行以下命令，并根据情况替换变量值：

  $ podman run -v $(pwd)/:/kubeconfig:Z -e KUBECONFIG=/kubeconfig/kubeconfig \
  -e LATENCY_TEST_RUN=true -e DISCOVERY_MODE=true -e FEATURES=performance -e ROLE_WORKER_CNF=worker-cnf \
  -e LATENCY_TEST_RUNTIME=600 -e MAXIMUM_LATENCY=20 \
  registry.redhat.io/openshift4/cnf-tests-rhel8:v4.13 \
  /usr/bin/test-run.sh -ginkgo.v -ginkgo.focus="hwlatdetect"

  hwlatdetect 测试运行了 10 分钟 (600 秒)。当最观察到的延迟低于 MAXIMUM_LATENCY (20 FORWARD) 时，测试会成功运行。

  如果结果超过延迟阈值，测试会失败。

  重要

  对于有效结果，测试应至少运行 12 小时。

  失败输出示例

  running /usr/bin/cnftests -ginkgo.v -ginkgo.focus=hwlatdetect
  I0908 15:25:20.023712      27 request.go:601] Waited for 1.046586367s due to client-side throttling, not priority and fairness, request: GET:https://api.hlxcl6.lab.eng.tlv2.redhat.com:6443/apis/imageregistry.operator.openshift.io/v1?timeout=32s
  Running Suite: CNF Features e2e integration tests
  =================================================
  Random Seed: 1662650718
  Will run 1 of 194 specs
  
  [...]
  
  • Failure [283.574 seconds]
  [performance] Latency Test
  /remote-source/app/vendor/github.com/openshift/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/4_latency/latency.go:62
    with the hwlatdetect image
    /remote-source/app/vendor/github.com/openshift/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/4_latency/latency.go:228
      should succeed [It]
      /remote-source/app/vendor/github.com/openshift/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/4_latency/latency.go:236
  
      Log file created at: 2022/09/08 15:25:27
      Running on machine: hwlatdetect-b6n4n
      Binary: Built with gc go1.17.12 for linux/amd64
      Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
      I0908 15:25:27.160620       1 node.go:39] Environment information: /proc/cmdline: BOOT_IMAGE=(hd1,gpt3)/ostree/rhcos-c6491e1eedf6c1f12ef7b95e14ee720bf48359750ac900b7863c625769ef5fb9/vmlinuz-4.18.0-372.19.1.el8_6.x86_64 random.trust_cpu=on console=tty0 console=ttyS0,115200n8 ignition.platform.id=metal ostree=/ostree/boot.1/rhcos/c6491e1eedf6c1f12ef7b95e14ee720bf48359750ac900b7863c625769ef5fb9/0 ip=dhcp root=UUID=5f80c283-f6e6-4a27-9b47-a287157483b2 rw rootflags=prjquota boot=UUID=773bf59a-bafd-48fc-9a87-f62252d739d3 skew_tick=1 nohz=on rcu_nocbs=0-3 tuned.non_isolcpus=0000ffff,ffffffff,fffffff0 systemd.cpu_affinity=4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79 intel_iommu=on iommu=pt isolcpus=managed_irq,0-3 nohz_full=0-3 tsc=nowatchdog nosoftlockup nmi_watchdog=0 mce=off skew_tick=1 rcutree.kthread_prio=11 + +
      I0908 15:25:27.160830       1 node.go:46] Environment information: kernel version 4.18.0-372.19.1.el8_6.x86_64
      I0908 15:25:27.160857       1 main.go:50] running the hwlatdetect command with arguments [/usr/bin/hwlatdetect --threshold 1 --hardlimit 1 --duration 100 --window 10000000us --width 950000us]
      F0908 15:27:10.603523       1 main.go:53] failed to run hwlatdetect command; out: hwlatdetect:  test duration 100 seconds
         detector: tracer
         parameters:
              Latency threshold: 1us 1
              Sample window:     10000000us
              Sample width:      950000us
           Non-sampling period:  9050000us
              Output File:       None
  
      Starting test
      test finished
      Max Latency: 326us 2
      Samples recorded: 5
      Samples exceeding threshold: 5
      ts: 1662650739.017274507, inner:6, outer:6
      ts: 1662650749.257272414, inner:14, outer:326
      ts: 1662650779.977272835, inner:314, outer:12
      ts: 1662650800.457272384, inner:3, outer:9
      ts: 1662650810.697273520, inner:3, outer:2
  
  [...]
  
  JUnit report was created: /junit.xml/cnftests-junit.xml
  
  
  Summarizing 1 Failure:
  
  [Fail] [performance] Latency Test with the hwlatdetect image [It] should succeed
  /remote-source/app/vendor/github.com/openshift/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/4_latency/latency.go:476
  
  Ran 1 of 194 Specs in 365.797 seconds
  FAIL! -- 0 Passed | 1 Failed | 0 Pending | 193 Skipped
  --- FAIL: TestTest (366.08s)
  FAIL

  1

  您可以使用 MAXIMUM_LATENCY 或 HWLATDETECT_MAXIMUM_LATENCY 环境变量来配置延迟阈值。

  2

  测试期间测量的最大延迟值。

• 在每次运行后收集的粗略结果，以便对整个测试过程中所做的任何更改产生影响的历史记录。
• 基本测试和配置设置的组合。

• 测试的持续时间
• CPU 内核数
• 主机固件设置

先决条件

• 您使用客户门户网站凭证登录到 registry.redhat.io。
• 已在集群中安装了实时内核。
• 已使用 Node Tuning Operator 应用了集群性能配置集。

流程

• 要执行 cyclictest，请运行以下命令，并根据情况替换变量值：

  $ podman run -v $(pwd)/:/kubeconfig:Z -e KUBECONFIG=/kubeconfig/kubeconfig \
  -e LATENCY_TEST_RUN=true -e DISCOVERY_MODE=true -e FEATURES=performance -e ROLE_WORKER_CNF=worker-cnf \
  -e LATENCY_TEST_CPUS=10 -e LATENCY_TEST_RUNTIME=600 -e MAXIMUM_LATENCY=20 \
  registry.redhat.io/openshift4/cnf-tests-rhel8:v4.13 \
  /usr/bin/test-run.sh -ginkgo.v -ginkgo.focus="cyclictest"

  该命令运行 cyclictest 工具 10 分钟（600 秒）。当观察到的延迟低于 MAXIMUM_LATENCY 时，测试会成功运行（在本例中，20 TOKENs）。对于电信 RAN 工作负载，对 20 个以上延迟的激增通常并不能接受。

  如果结果超过延迟阈值，测试会失败。

  重要

  对于有效结果，测试应至少运行 12 小时。

  失败输出示例

  running /usr/bin/cnftests -ginkgo.v -ginkgo.focus=cyclictest
  I0908 13:01:59.193776      27 request.go:601] Waited for 1.046228824s due to client-side throttling, not priority and fairness, request: GET:https://api.compute-1.example.com:6443/apis/packages.operators.coreos.com/v1?timeout=32s
  Running Suite: CNF Features e2e integration tests
  =================================================
  Random Seed: 1662642118
  Will run 1 of 194 specs
  
  [...]
  
  Summarizing 1 Failure:
  
  [Fail] [performance] Latency Test with the cyclictest image [It] should succeed
  /remote-source/app/vendor/github.com/openshift/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/4_latency/latency.go:220
  
  Ran 1 of 194 Specs in 161.151 seconds
  FAIL! -- 0 Passed | 1 Failed | 0 Pending | 193 Skipped
  --- FAIL: TestTest (161.48s)
  FAIL

先决条件

• 您使用客户门户网站凭证登录到 registry.redhat.io。
• 已使用 Node Tuning Operator 应用了集群性能配置集。

流程

• 要执行 oslat 测试，请运行以下命令，根据需要替换变量值：

  $ podman run -v $(pwd)/:/kubeconfig:Z -e KUBECONFIG=/kubeconfig/kubeconfig \
  -e LATENCY_TEST_RUN=true -e DISCOVERY_MODE=true -e FEATURES=performance -e ROLE_WORKER_CNF=worker-cnf \
  -e LATENCY_TEST_CPUS=10 -e LATENCY_TEST_RUNTIME=600 -e MAXIMUM_LATENCY=20 \
  registry.redhat.io/openshift4/cnf-tests-rhel8:v4.13 \
  /usr/bin/test-run.sh -ginkgo.v -ginkgo.focus="oslat"

  LATENCY_TEST_CPUS 指定使用 oslat 命令测试的 CPU 列表。

  命令运行 oslat 工具 10 分钟（600 秒）。当最观察到的延迟低于 MAXIMUM_LATENCY (20 FORWARD) 时，测试会成功运行。

  如果结果超过延迟阈值，测试会失败。

  重要

  对于有效结果，测试应至少运行 12 小时。

  失败输出示例

  running /usr/bin/cnftests -ginkgo.v -ginkgo.focus=oslat
  I0908 12:51:55.999393      27 request.go:601] Waited for 1.044848101s due to client-side throttling, not priority and fairness, request: GET:https://compute-1.example.com:6443/apis/machineconfiguration.openshift.io/v1?timeout=32s
  Running Suite: CNF Features e2e integration tests
  =================================================
  Random Seed: 1662641514
  Will run 1 of 194 specs
  
  [...]
  
  • Failure [77.833 seconds]
  [performance] Latency Test
  /remote-source/app/vendor/github.com/openshift/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/4_latency/latency.go:62
    with the oslat image
    /remote-source/app/vendor/github.com/openshift/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/4_latency/latency.go:128
      should succeed [It]
      /remote-source/app/vendor/github.com/openshift/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/4_latency/latency.go:153
  
      The current latency 304 is bigger than the expected one 1 : 1
  
  [...]
  
  Summarizing 1 Failure:
  
  [Fail] [performance] Latency Test with the oslat image [It] should succeed
  /remote-source/app/vendor/github.com/openshift/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/4_latency/latency.go:177
  
  Ran 1 of 194 Specs in 161.091 seconds
  FAIL! -- 0 Passed | 1 Failed | 0 Pending | 193 Skipped
  --- FAIL: TestTest (161.42s)
  FAIL

  1

  在本例中，测量的延迟超出了最大允许的值。