Red Hat Summit Red Hat Summit지원콘솔개발자평가판 시작연락처

Chapter 12. Troubleshoot OVS DPDK PMD CPU Usage with perf and Collect and Send the Troubleshooting Data

  1. Prerequisites Use the steps in this section to install troubleshooting tools.

  2. Install perf on the compute node: 

    yum install perf -y
    Copy to Clipboard Toggle word wrap

  3. Install Open vSwitch debug RPMs: 

    subscription-manager repos --enable=rhel-7-server-openstack-10-debug-rpms
    Copy to Clipboard Toggle word wrap

  4. Install sysstat (needed for the pidstat command): 

    yum install sysstat -y
    Copy to Clipboard Toggle word wrap

12.1. Diagnosis
링크 복사

Use the steps in this section to troubleshoot and collect data.

12.1.1. PMD Threads
링크 복사

  1. Determine the location of PMD threads: 

    IFS=$'\n' ; for l in $(ps -T -p `pidof ovs-vswitchd` | grep pmd);do PID=`echo $l | awk '{print $2}'`; PMD=`echo $l | awk
'{print $NF}'` ; PCPU=`taskset -c -p $PID | awk '{print $NF}'` ; echo "$PMD with PID $PID in on pCPU $PCPU"; done
    Copy to Clipboard Toggle word wrap

    For example: 

    [root@overcloud-compute-1 ~]# IFS=$'\n' ; for l in $(ps -T -p `pidof ovs-vswitchd` | grep pmd);do PID=`echo $l | awk
'{print $2}'`; PMD=`echo $l | awk '{print $NF}'` ; PCPU=`taskset -c -p $PID | awk '{print $NF}'` ; echo "$PMD with PID $PID in on pCPU
$PCPU"; done
pmd545 with PID 412314 in on pCPU 2
pmd555 with PID 412315 in on pCPU 4
pmd550 with PID 412316 in on pCPU 6
pmd551 with PID 412317 in on pCPU 8
pmd553 with PID 412318 in on pCPU 22
pmd554 with PID 412319 in on pCPU 24
pmd549 with PID 412320 in on pCPU 26
pmd556 with PID 412321 in on pCPU 28
pmd546 with PID 412322 in on pCPU 3
pmd548 with PID 412323 in on pCPU 5
pmd547 with PID 412324 in on pCPU 23
pmd552 with PID 412325 in on pCPU 25
    Copy to Clipboard Toggle word wrap

  2. While reproducing the issue, run perf record and perf report and save the output.

    • Create the script gather_perf_data_a.sh: 

      cat<<'EOF'>>gather_perf_data_a.sh
#!/bin/bash -x
IFS=$'\n' ;
dir_name=/tmp/perf_record_a
mkdir ${dir_name}
rm -f ${dir_name}/*

for l in $(ps -T -p `pidof ovs-vswitchd` | grep pmd);do PID=`echo $l | awk '{print $2}'`; PMD=`echo $l | awk '{print $NF}'` ; PCPU=`taskset -c -p $PID | awk '{print $NF}'` ; echo "$PMD with PID $PID in on pCPU $PCPU"; done > ${dir_name}/pmds.txt

for l in $(ps -T -p `pidof ovs-vswitchd` | grep pmd);do
  PID=`echo $l | awk '{print $2}'`;
  PMD=`echo $l | awk '{print $NF}'` ;
  PCPU=`taskset -c -p $PID | awk '{print $NF}'` ;
  echo "$PMD with PID $PID in on pCPU $PCPU";
  date
  perf record -C $PCPU -g -o perf_record_-g_$PCPU sleep 60 &
done

sleep 80

for l in $(ps -T -p `pidof ovs-vswitchd` | grep pmd);do
  PID=`echo $l | awk '{print $2}'`;
  PMD=`echo $l | awk '{print $NF}'` ;
  PCPU=`taskset -c -p $PID | awk '{print $NF}'` ;
  echo "$PMD with PID $PID in on pCPU $PCPU";
  date
  perf record -C $PCPU -o perf_record_$PCPU sleep 60 &
done

sleep 80

for f in perf_record_-g_*;do
  perf report -g -i $f | cat > ${dir_name}/perf_report_$f.txt ;
  rm -f $f
done

for f in perf_record_*;do
  perf report -i $f | cat > ${dir_name}/perf_report_$f.txt ;
  rm -f $f
done

archive_name="${dir_name}_`hostname`_`date '+%F_%H%m%S'`.tar.gz"
tar -czf $archive_name ${dir_name}
echo "Archived all data in archive ${archive_name}"
EOF
      Copy to Clipboard Toggle word wrap

    • Run the script: 

      chmod +x gather_perf_data_a.sh
./gather_perf_data_a.sh
      Copy to Clipboard Toggle word wrap

The report can be read using perf report -i ${archive_name}. If this is for a case that was opened with Red Hat support, attach the resulting tar archive to the case.

12.1.2. Additional Data
링크 복사

  1. Create the script gather_perf_data_b.sh to collect additional data: 

    cat<<'EOF'>>gather_perf_data_b.sh
#!/bin/bash -x
dir_name=/tmp/perf_record_b
mkdir ${dir_name}
rm -f ${dir_name}/*

date > ${dir_name}/pidstat1.txt
pidstat -u -t -p `pidof ovs-vswitchd`,`pidof ovsdb-server` 5 12 >> ${dir_name}/pidstat1.txt &
perf record -p `pidof ovs-vswitchd` -g --call-graph dwarf sleep 60

sleep 20

date > ${dir_name}/pidstat2.txt
pidstat -u -t -p `pidof ovs-vswitchd`,`pidof ovsdb-server` 1 60 >> ${dir_name}/pidstat2.txt

mv perf.data perf.data_openvswitch

perf script -F tid -i perf.data_openvswitch | sort -u | grep -o '[0-9]*' | xargs -n1 -I{} perf report -i perf.data_openvswitch --no-children --percentage relative --stdio --tid {} -g none > ${dir_name}/perf_reports.txt
perf script -F tid -i perf.data_openvswitch | sort -u | grep -o '[0-9]*' | xargs -n1 -I{} perf report -i perf.data_openvswitch --no-children --percentage relative --stdio --tid {}  > ${dir_name}/perf_reports_callgraph.txt

rm -f perf.data_openvswitch

archive_name="${dir_name}_`hostname`_`date '+%F_%H%m%S'`.tar.gz"
tar -czf $archive_name ${dir_name}
echo "Archived all data in archive ${archive_name}"
EOF
    Copy to Clipboard Toggle word wrap

  2. Execute the script: 

    chmod +x gather_perf_data_b.sh
./gather_perf_data_b.sh
    Copy to Clipboard Toggle word wrap
    Note

    Make sure that there is sufficient disk space. The 'perf.data' file can take up several Gigabytes of disk space.

If this is for a Red Hat support ticket, attach the resulting tar archive to the case.

12.1.3. Open vSwitch Logs
링크 복사

  1. Provide all Open vSwitch (OVS) logs. Ensure that /var has sufficient disk space. Use df -h to determine free disk space on /var and du -sh /var/log/openvswitch to determine the total size of OVS logs. 

    tar -cvzf /var/openvswitch_`hostname`_`date +"%F_%H%M%S"`.tar.gz /var/log/openvswitch
    Copy to Clipboard Toggle word wrap
  2. Attach the resulting file, for example, /var/openvswitch_overcloud-compute-0_2018-02-27_153713.tar.gz, to the support case for analysis.

  3. Generate and provide an sosreport. Ensure that /var has sufficient disk space. Use df -h to determine free disk space on /var. 

    sosreport --batch --all-logs
    Copy to Clipboard Toggle word wrap
맨 위로 이동
Red Hat logoGithubredditYoutubeTwitter

자세한 정보

평가판, 구매 및 판매

커뮤니티

Red Hat 문서 정보

Red Hat을 사용하는 고객은 신뢰할 수 있는 콘텐츠가 포함된 제품과 서비스를 통해 혁신하고 목표를 달성할 수 있습니다. 최신 업데이트를 확인하세요.

보다 포괄적 수용을 위한 오픈 소스 용어 교체

Red Hat은 코드, 문서, 웹 속성에서 문제가 있는 언어를 교체하기 위해 최선을 다하고 있습니다. 자세한 내용은 다음을 참조하세요.Red Hat 블로그.

Red Hat 소개

Red Hat은 기업이 핵심 데이터 센터에서 네트워크 에지에 이르기까지 플랫폼과 환경 전반에서 더 쉽게 작업할 수 있도록 강화된 솔루션을 제공합니다.

Theme

© 2026 Red Hat