확장 및 성능


OpenShift Container Platform 4.15

프로덕션 환경에서 OpenShift Container Platform 클러스터 스케일링 및 성능 튜닝

Red Hat OpenShift Documentation Team

초록

이 문서에서는 OpenShift Container Platform 환경의 클러스터를 스케일링하고 성능을 최적화하는 방법을 설명합니다.

1장. OpenShift Container Platform 확장성 및 성능 개요

OpenShift Container Platform은 클러스터의 성능 및 규모를 최적화하는 데 도움이 되는 모범 사례와 툴을 제공합니다. 다음 문서에서는 권장 성능 및 확장성 사례, 참조 설계 사양, 최적화 및 짧은 대기 시간 튜닝에 대한 정보를 제공합니다.

Red Hat 지원에 문의하려면 지원 받기를 참조하십시오.

참고

일부 성능 및 확장성 Operator에는 OpenShift Container Platform 릴리스 사이클과 독립적인 릴리스 사이클이 있습니다. 자세한 내용은 OpenShift Operator 를 참조하십시오.

컨트롤 플레인 권장 사례

인프라 관련 권장 사례

etcd 관련 권장 사례

Telco 참조 설계 사양

Telco RAN DU 사양

계획, 최적화 및 측정

오브젝트 최대값에 따른 환경 계획

IBM Z 및 IBM LinuxONE에 대한 권장 사례

Node Tuning Operator 사용

CPU 관리자 및 토폴로지 관리자 사용

NUMA 인식 워크로드 예약

스토리지, 라우팅, 네트워킹 및 CPU 사용량 최적화

베어 메탈 호스트 및 이벤트 관리

대규모 페이지란 무엇이며 앱에서 사용하는 방법

클러스터 안정성 및 파티셔닝 워크로드를 개선하기 위한 짧은 대기 시간 튜닝

작업자 대기 시간 프로필을 사용하여 대기 시간이 많은 환경에서 클러스터 안정성 개선

워크로드 파티셔닝

Node Observability Operator 사용

3장. 참조 설계 사양

3.1. Telco core 및 RAN DU 참조 설계 사양

Telco 코어 참조 설계 사양(RDS)은 컨트롤 플레인 및 일부 중앙 집중식 데이터 플레인 기능을 포함하여 대규모 통신 애플리케이션을 지원할 수 있는 상용 하드웨어에서 실행되는 OpenShift Container Platform 4.15 클러스터를 설명합니다.

Telco RAN RDS는 RAN(Radio Access Network)에서 5G 워크로드를 호스팅하기 위해 상용 하드웨어에서 실행되는 클러스터의 구성을 설명합니다.

3.1.1. 통신 5G 배포를 위한 참조 설계 사양

Red Hat 및 인증된 파트너는 OpenShift Container Platform 4.15 클러스터에서 telco 애플리케이션을 실행하는 데 필요한 네트워킹 및 운영 기능에 대한 깊은 기술 전문 지식 및 지원을 제공합니다.

Red Hat의 통신 파트너는 엔터프라이즈 5G 솔루션을 위해 대규모로 복제할 수 있는 잘 통합되고 테스트되고 안정적인 환경이 필요합니다. Telco 코어 및 RAN DU 참조 설계 사양(RDS)은 특정 OpenShift Container Platform 버전을 기반으로 하는 권장 솔루션 아키텍처를 간략하게 설명합니다. 각 RDS는 telco 코어 및 RAN DU 모델을 사용하기 위한 테스트되고 검증된 플랫폼 구성을 설명합니다. RDS는 통신 5G 코어 및 RAN DU에 대한 중요한 KPI 세트를 정의하여 애플리케이션을 실행할 때 최적의 환경을 보장합니다. RDS는 높은 심각도의 에스컬레이션을 최소화하고 애플리케이션 안정성을 향상시킵니다.

5G 사용 사례는 계속 증가하고 있으며 워크로드는 지속적으로 변하고 있습니다. Red Hat은 telco 코어 및 RAN DU RDS를 통해 고객 및 파트너 피드백을 기반으로 진화하는 요구 사항을 지원하기 위해 최선을 다하고 있습니다.

3.1.2. 참조 설계 범위

Telco core 및 telco RAN 참조 설계 사양(RDS)은 권장, 테스트 및 지원되는 구성을 캡처하여 Telco core 및 telco RAN 프로필을 실행하는 클러스터에 대해 안정적이고 반복 가능한 성능을 제공합니다.

각 RDS에는 클러스터가 개별 프로필을 실행할 수 있도록 설계 및 검증되는 릴리스된 기능 및 지원되는 구성이 포함되어 있습니다. 이 구성은 기능 및 KPI 대상을 충족하는 기본 OpenShift Container Platform 설치를 제공합니다. 각 RDS는 개별 구성에 대해 예상되는 변형도 설명합니다. 각 RDS의 검증에는 긴 기간과 대규모 테스트가 포함되어 있습니다.

참고

검증된 참조 구성은 OpenShift Container Platform의 각 주요 Y-stream 릴리스에 대해 업데이트됩니다. Z-stream 패치 릴리스는 참조 구성에 대해 주기적으로 다시 테스트됩니다.

3.1.3. 참조 설계의 편차

검증된 Telco 코어 및 telco RAN DU 참조 설계 사양(RDS)을 벗어나면 변경되는 특정 구성 요소 또는 기능 이외의 중요한 영향을 미칠 수 있습니다. 편차는 전체 솔루션의 맥락에서 분석 및 엔지니어링이 필요합니다.

중요

RDS의 모든 편차는 명확한 작업 추적 정보로 분석 및 문서화되어야 합니다. 실사실은 파트너로부터 참조 설계에 맞춰 편차를 가져오는 방법을 이해할 것으로 예상됩니다. 이를 위해서는 파트너사가 Red Hat과 협력하여 플랫폼에서 최상의 결과를 얻을 수 있도록 활용 사례를 제공하기 위해 추가 리소스를 제공해야 할 수 있습니다. 이는 솔루션의 지원 가능성과 Red Hat 및 파트너 간의 조정을 보장하는 데 매우 중요합니다.

RDS의 편차는 다음과 같은 결과 중 일부 또는 모두를 가질 수 있습니다.

  • 문제를 해결하는 데 시간이 더 걸릴 수 있습니다.
  • 프로젝트 서비스 수준 계약(SLA), 프로젝트 기한, 최종 공급자 성능 요구 사항이 누락될 위험이 있습니다.
  • 승인되지 않은 편차는 행정 수준에서 에스컬레이션이 필요할 수 있습니다.

    참고

    Red Hat은 파트너 참여 우선 순위에 따라 편차 요청의 우선 순위를 지정합니다.

3.2. Telco RAN DU 참조 설계 사양

3.2.1. Telco RAN DU 4.15 참조 설계 개요

Telco RAN distributed unit (DU) 4.15 참조 설계는 상용 하드웨어에서 실행되는 OpenShift Container Platform 4.15 클러스터를 구성하여 Telco RAN DU 워크로드를 호스팅합니다. Telco RAN DU 프로필을 실행하는 클러스터에 대해 안정적이고 반복 가능한 성능을 얻기 위해 권장, 테스트 및 지원되는 구성을 캡처합니다.

3.2.1.1. telco RAN DU를 위한 OpenShift Container Platform 4.15 기능

Telco RAN DU 참조 설계 사양 (RDS)의 4.15 버전에는 telco RAN DU RDS의 4.14 버전과 동일한 기능 세트가 있습니다.

자세한 내용은 OpenShift Container Platform 4.14 telco RAN DU 기능을 참조하십시오.

3.2.1.2. 배포 아키텍처 개요

중앙 집중식 관리형 RHACM 허브 클러스터에서 관리되는 클러스터에 telco RAN DU 4.15 참조 구성을 배포합니다. 참조 설계 사양(RDS)에는 관리 클러스터 및 허브 클러스터 구성 요소가 포함되어 있습니다.

그림 3.1. Telco RAN DU 배포 아키텍처 개요

3.2.2. Ttelco RAN DU 사용 모델 개요

다음 정보를 사용하여 허브 클러스터 및 관리형 단일 노드 OpenShift 클러스터에 대한 통신 RAN DU 워크로드, 클러스터 리소스 및 하드웨어 사양을 계획합니다.

3.2.2.1. Telco RAN DU 애플리케이션 워크로드

DU 작업자 노드에는 최대 성능을 위해 조정된 펌웨어가 있는 3세대 Xeon(Ice Lake) 2.20 Cryostat 또는 더 나은 CPU가 있어야 합니다.

5G RAN DU 사용자 애플리케이션 및 워크로드는 다음과 같은 모범 사례 및 애플리케이션 제한을 준수해야 합니다.

  • CNF 모범 사례 가이드의 최신 버전을 준수하는 클라우드 네이티브 네트워크 기능(CNF )을 개발합니다.
  • 고성능 네트워킹을 위해 SR-IOV를 사용합니다.
  • exec 프로브를 사용하여 다른 적절한 옵션을 사용할 수 없는 경우에만 사용

    • CNF에서 CPU 고정을 사용하는 경우 exec 프로브를 사용하지 마십시오. 다른 프로브 구현(예: httpGet 또는 tcpSocket )을 사용합니다.
    • exec 프로브를 사용해야 하는 경우 exec 프로브 빈도 및 수량을 제한합니다. 최대 exec 프로브 수는 10초 미만으로 유지해야 하며 빈도를 10초 미만으로 설정하지 않아야 합니다.
참고

시작 프로브에는 지속적인 상태 작업 중에 최소한의 리소스가 필요합니다. exec 프로브의 제한은 주로 liveness 및 readiness 프로브에 적용됩니다.

3.2.2.2. Telco RAN DU 대표 참조 애플리케이션 워크로드 특성

대표적인 참조 애플리케이션 워크로드에는 다음과 같은 특징이 있습니다.

  • 관리 및 제어 기능을 포함하여 vRAN 애플리케이션에 대한 최대 15개의 Pod 및 30개의 컨테이너가 있습니다.
  • Pod당 최대 2개의 ConfigMap 및 4개의 Secret CR 사용
  • 10초 미만의 빈도로 최대 10 exec 프로브 사용
  • kube-apiserver 에서의 증분 애플리케이션 로드는 클러스터 플랫폼 사용의 10% 미만입니다.

    참고

    플랫폼 지표에서 CPU 로드를 추출할 수 있습니다. 예를 들면 다음과 같습니다.

    query=avg_over_time(pod:container_cpu_usage:sum{namespace="openshift-kube-apiserver"}[30m])
    Copy to Clipboard Toggle word wrap
  • 애플리케이션 로그는 플랫폼 로그 수집기에 의해 수집되지 않습니다.
  • 기본 CNI의 집계 트래픽은 1MBps 미만입니다.
3.2.2.3. Telco RAN 작업자 노드 클러스터 리소스 사용률

애플리케이션 워크로드 및 OpenShift Container Platform Pod를 포함하여 시스템에서 실행 중인 최대 Pod 수는 120입니다.

리소스 사용률

OpenShift Container Platform 리소스 사용률은 다음과 같은 애플리케이션 워크로드 특성을 포함한 다양한 요인에 따라 다릅니다.

  • Pod 수
  • 프로브 유형 및 빈도
  • 커널 네트워킹을 통한 기본 CNI 또는 보조 CNI의 메시징 속도
  • API 액세스 속도
  • 로깅 속도
  • 스토리지 IOPS

클러스터 리소스 요구 사항은 다음 조건에 따라 적용할 수 있습니다.

  • 클러스터는 설명된 대표 애플리케이션 워크로드를 실행하고 있습니다.
  • 클러스터는 "Telco RAN DU 작업자 노드 클러스터 리소스 사용률"에 설명된 제약 조건으로 관리됩니다.
  • RAN DU 사용 모델 구성에서 선택 사항으로 명시된 구성 요소는 적용되지 않습니다.
중요

리소스 사용률에 미치는 영향 및 Telco RAN DU 참조 디자인 범위를 벗어난 구성에 대한 KPI 대상을 충족하기 위해 추가 분석을 수행해야 합니다. 요구 사항에 따라 클러스터에서 추가 리소스를 할당해야 할 수 있습니다.

3.2.2.4. hub 클러스터 관리 특성

RHACM(Red Hat Advanced Cluster Management)은 권장되는 클러스터 관리 솔루션입니다. hub 클러스터에서 다음 제한으로 구성합니다.

  • 규정 준수 평가 간격을 10분 이상 사용하여 최대 5개의 RHACM 정책을 구성합니다.
  • 정책에서 최대 10개의 관리형 클러스터 템플릿을 사용합니다. 가능한 경우 hub-side template을 사용하십시오.
  • policy-controllerobservability-controller 애드온을 제외한 모든 RHACM 애드온을 비활성화합니다. Observability 를 기본 구성으로 설정합니다.

    중요

    선택적 구성 요소 또는 추가 기능을 구성하면 리소스 사용량이 추가되고 전체 시스템 성능이 저하될 수 있습니다.

    자세한 내용은 참조 설계 배포 구성 요소를 참조하십시오.

Expand
표 3.1. 참조 애플리케이션 로드 아래의 OpenShift 플랫폼 리소스 사용률
지표제한참고

CPU 사용량

4000 mc 미만 - 2 코어 (4 하이퍼스레드)

플랫폼 CPU는 예약된 각 코어의 두 하이퍼스레드를 포함하여 예약된 코어에 고정되어 있습니다. 시스템은 주기적인 시스템 작업 및 급증을 허용하도록 안정적인 상태에서 3개의 CPU (3000mc)를 사용하도록 설계되었습니다.

사용된 메모리

16G 미만

 
3.2.2.5. Telco RAN DU RDS 구성 요소

다음 섹션에서는 telco RAN DU 워크로드를 실행하기 위해 클러스터를 구성하고 배포하는 데 사용하는 다양한 OpenShift Container Platform 구성 요소 및 구성에 대해 설명합니다.

그림 3.2. Telco RAN DU 참조 구성 요소

참고

telco RAN DU 프로필에 포함되지 않은 구성 요소가 워크로드 애플리케이션에 할당된 CPU 리소스에 영향을 미치지 않도록 합니다.

중요

트리 부족 드라이버는 지원되지 않습니다.

3.2.3. Telco RAN DU 4.15 참조 설계 구성 요소

다음 섹션에서는 RAN DU 워크로드를 실행하기 위해 클러스터를 구성하고 배포하는 데 사용하는 다양한 OpenShift Container Platform 구성 요소 및 구성에 대해 설명합니다.

3.2.3.1. 호스트 펌웨어 튜닝
이번 릴리스의 새로운 기능
  • 이 릴리스에는 참조 디자인 업데이트가 없습니다
설명

시스템 수준 성능 구성. 권장 설정은 짧은 대기 시간과 고성능은 호스트 펌웨어 구성 을 참조하십시오.

Ironic 검사가 활성화되면 hub 클러스터의 클러스터별 BareMetalHost CR에서 펌웨어 설정 값을 사용할 수 있습니다. 클러스터를 설치하는 데 사용하는 SiteConfig CR의 spec.clusters.nodes 필드에 라벨을 사용하여 Ironic 검사를 활성화합니다. 예를 들면 다음과 같습니다.

nodes:
  - hostName: "example-node1.example.com"
    ironicInspect: "enabled"
Copy to Clipboard Toggle word wrap
참고

Ttelco RAN DU 참조 SiteConfig 는 기본적으로 ironicInspect 필드를 활성화하지 않습니다.

제한 및 요구사항
  • 하이퍼 스레딩을 활성화해야 합니다.
엔지니어링 고려 사항
  • 최대 성능을 위해 모든 설정 조정

    참고

    필요에 따라 성능 저하를 위해 펌웨어 선택을 조정할 수 있습니다.

3.2.3.2. Node Tuning Operator
이번 릴리스의 새로운 기능
  • 이 릴리스에는 참조 디자인 업데이트가 없습니다
설명

성능 프로파일을 생성하여 클러스터 성능을 조정합니다. 성능 프로필을 사용하여 구성하는 설정은 다음과 같습니다.

  • 실시간 또는 비실시간 커널 선택.
  • 예약된 또는 분리된 cpuset 에 코어 할당. 관리 워크로드 파티션에 할당된 OpenShift Container Platform 프로세스는 예약된 세트에 고정됩니다.
  • kubelet 기능 활성화(CPU 관리자, 토폴로지 관리자, 메모리 관리자).
  • 대규모 페이지 구성.
  • 추가 커널 인수 설정.
  • 코어당 전원 튜닝 및 최대 CPU 빈도 설정.
제한 및 요구사항

Node Tuning Operator는 PerformanceProfile CR을 사용하여 클러스터를 구성합니다. RAN DU 프로파일 PerformanceProfile CR에서 다음 설정을 구성해야 합니다.

  • 예약 및 분리된 코어를 선택하고 Intel 3rd Generation Xeon (Ice Lake) 2.20 Cryostat CPU에서 최소 4개의 하이퍼스레드(각주 2개 코어)를 할당하거나 최대 성능을 위해 펌웨어를 튜닝했는지 확인합니다.
  • 포함된 각 코어에 대해 두 개의 하이퍼스레드 형제를 모두 포함하도록 예약된 cpuset 을 설정합니다. 예약되지 않은 코어는 워크로드 예약에 할당 가능한 CPU로 사용할 수 있습니다. 하이퍼스레드 형제가 예약된 코어와 분리된 코어 간에 분할되지 않도록 합니다.
  • 예약 및 분리된 CPU로 설정된 내용에 따라 모든 코어에 모든 스레드를 포함하도록 예약 및 분리된 CPU를 구성합니다.
  • 예약된 CPU 세트에 포함할 각 NUMA 노드의 코어 0을 설정합니다.
  • 대규모 페이지 크기를 1G로 설정합니다.
참고

관리 파티션에 워크로드를 추가해서는 안 됩니다. OpenShift 관리 플랫폼의 일부인 포드만 관리 파티션에 주석을 달아야 합니다.

엔지니어링 고려 사항
  • RT 커널을 사용하여 성능 요구 사항을 충족해야 합니다.

    참고

    필요한 경우 RT가 아닌 커널을 사용할 수 있습니다.

  • 구성하는 대규모 페이지 수는 애플리케이션 워크로드 요구 사항에 따라 다릅니다. 이 매개변수의 변형은 예상되고 허용됩니다.
  • 선택한 하드웨어 및 시스템에서 사용 중인 추가 구성 요소를 기반으로 예약 및 격리된 CPU 세트 구성에서 변동이 예상됩니다. 변형은 지정된 제한을 충족해야 합니다.
  • IRQ 선호도 지원이 없는 하드웨어는 분리된 CPU에 영향을 미칩니다. 보장된 전체 CPU QoS가 있는 Pod가 할당된 CPU를 완전히 사용하도록 하려면 서버의 모든 하드웨어가 IRQ 선호도를 지원해야 합니다. 자세한 내용은 IRQ 선호도 설정 지원 정보를 참조하십시오.
참고

OpenShift Container Platform 4.15에서 클러스터에 구성된 모든 PerformanceProfile CR로 인해 Node Tuning Operator가 cgroup v1을 사용하도록 모든 클러스터 노드를 자동으로 설정합니다.

cgroup에 대한 자세한 내용은 Linux cgroup 구성을 참조하십시오.

3.2.3.3. PTP Operator
이번 릴리스의 새로운 기능
  • 이 릴리스에는 참조 디자인 업데이트가 없습니다
설명

클러스터 노드에서 PTP의 지원 및 구성에 대한 자세한 내용은 PTP 타이밍 을 참조하십시오. DU 노드는 다음 모드에서 실행할 수 있습니다.

  • 일반 클럭 (OC)은 할 마스터 클록 또는 경계 클록 (T-BC)에 동기화됩니다.
  • 단일 또는 듀얼 카드 E810 Westport 채널 NIC를 지원하는 GPS에서 동기화된 마스터 클럭
  • E810 Westport 채널 NIC를 지원하는 이중 경계 클럭( NIC당 하나씩)으로

    참고

    고가용성 경계 클록은 지원되지 않습니다.

  • 선택 사항: 라디오 단위(RU)의 경계 클럭

마스터 클록에 대한 이벤트 및 메트릭은 4.14 telco RAN DU RDS에 추가된 기술 프리뷰 기능입니다. 자세한 내용은 PTP 하드웨어 빠른 이벤트 알림 프레임워크 사용을 참조하십시오.

DU 애플리케이션이 실행 중인 노드에서 발생하는 PTP 이벤트에 애플리케이션을 구독할 수 있습니다.

제한 및 요구사항
  • 듀얼 NIC 구성에서 고가용성은 지원되지 않습니다.
  • E810 Westport 채널 NIC에서는 DPDK(digital Phase-Locked Cryostat) 클럭 동기화가 지원되지 않습니다.
  • GPS 오프셋은 보고되지 않습니다. 기본 오프셋을 5 미만으로 사용합니다.
  • DPLL 오프셋이 보고되지 않습니다. 기본 오프셋을 5 미만으로 사용합니다.
엔지니어링 고려 사항
  • 일반 클럭, 경계 클럭 또는 마스터 클록에 대한 구성이 제공됩니다.
  • PTP 빠른 이벤트 알림은 ConfigMap CR을 사용하여 PTP 이벤트 서브스크립션을 저장
  • GPS 타이밍과 최소 펌웨어 버전 4.40인 PTP 할 마스터 클록에 대해 Intel E810-XXV-4T Westport Channel NIC를 사용하십시오.
3.2.3.4. SR-IOV Operator
이번 릴리스의 새로운 기능
  • 이 릴리스에는 참조 디자인 업데이트가 없습니다
설명
SR-IOV Operator는 SR-IOV CNI 및 장치 플러그인을 프로비저닝하고 구성합니다. netdevice (커널 VF) 및 DPDK( V fio) 장치가 모두 지원됩니다.
엔지니어링 고려 사항
  • 구성 및 SriovNetwork NodePolicy CR(사용자 정의 리소스) 수에 대한 고객 변동이 예상됩니다.
  • IOMMU 커널 명령줄 설정은 설치 시 MachineConfig CR과 함께 적용됩니다. 이렇게 하면 SriovOperator CR에서 노드를 추가할 때 노드가 재부팅되지 않습니다.
3.2.3.5. 로깅
이번 릴리스의 새로운 기능
  • 이 릴리스에는 참조 디자인 업데이트가 없습니다
설명
로깅을 사용하여 원격 분석을 위해 far edge 노드에서 로그를 수집합니다. 권장되는 로그 수집기는 Vector입니다.
엔지니어링 고려 사항
  • 예를 들어, 인프라 및 감사 로그 이외의 로그를 처리하려면 추가 로깅 속도를 기반으로 하는 추가 CPU 및 네트워크 대역폭이 필요합니다.
  • OpenShift Container Platform 4.14부터 Vector는 참조 로그 수집기입니다.

    참고

    RAN 사용 모델에서 fluentd 사용은 더 이상 사용되지 않습니다.

3.2.3.6. SRIOV-FEC Operator
이번 릴리스의 새로운 기능
  • 이 릴리스에는 참조 디자인 업데이트가 없습니다
설명
SRIOV-FEC Operator는 FEC 액셀러레이터 하드웨어를 지원하는 선택적 타사 Certified Operator입니다.
제한 및 요구사항
  • FEC Operator v2.7.0부터 다음을 수행합니다.

    • SecureBoot 지원
    • PFvfio 드라이버를 사용하려면 Pod에 삽입되는 vfio-token 을 사용해야 합니다. VF 토큰은 EAL 매개변수 --vfio-vf-token 을 사용하여 DPDK에 전달할 수 있습니다.
엔지니어링 고려 사항
  • SRIOV-FEC Operator는 분리된 CPU 세트의 CPU 코어를 사용합니다.
  • 예를 들어 검증 정책을 확장하여 FEC 준비 상태를 사전 점검의 일부로 검증할 수 있습니다.
3.2.3.7. Local Storage Operator
이번 릴리스의 새로운 기능
  • 이 릴리스에는 참조 디자인 업데이트가 없습니다
설명
Local Storage Operator를 사용하여 애플리케이션에서 PVC 리소스로 사용할 수 있는 영구 볼륨을 생성할 수 있습니다. 생성하는 PV 리소스의 수 및 유형은 요구 사항에 따라 다릅니다.
엔지니어링 고려 사항
  • PV 를 생성하기 전에 PV CR에 대한 백업 스토리지를 생성합니다. 파티션, 로컬 볼륨, LVM 볼륨 또는 전체 디스크일 수 있습니다.
  • 디스크 및 파티션을 올바르게 할당하도록 각 장치에 액세스하는 데 사용되는 하드웨어 경로에서 LocalVolume CR의 장치 목록을 참조하십시오. 논리 이름(예: /dev/sda)은 노드를 재부팅해도 일관성이 보장되지 않습니다.

    자세한 내용은 장치 식별자에 대한 RHEL 9 설명서 를 참조하십시오.

3.2.3.8. LVMS Operator
이번 릴리스의 새로운 기능
  • 이 릴리스에는 참조 디자인 업데이트가 없습니다
이번 릴리스의 새로운 기능
  • 간소화된 LVMS deviceSelector 논리
  • ext4PV 리소스가 포함된 LVM 스토리지
참고

LVMS Operator는 선택적 구성 요소입니다.

설명

LVMS Operator는 블록 및 파일 스토리지에 대한 동적 프로비저닝을 제공합니다. LVMS Operator는 애플리케이션에서 PVC 리소스로 사용할 수 있는 로컬 장치에서 논리 볼륨을 생성합니다. 볼륨 확장 및 스냅샷도 가능합니다.

다음 예제 구성은 설치 디스크 를 제외한 노드에서 사용 가능한 모든 디스크를 활용하는 Cryostat1 볼륨 그룹을 생성합니다.

StorageLVMCluster.yaml

apiVersion: lvm.topolvm.io/v1alpha1
kind: LVMCluster
metadata:
  name: storage-lvmcluster
  namespace: openshift-storage
  annotations:
    ran.openshift.io/ztp-deploy-wave: "10"
spec:
  storage:
    deviceClasses:
    - name: vg1
      thinPoolConfig:
        name: thin-pool-1
        sizePercent: 90
        overprovisionRatio: 10
Copy to Clipboard Toggle word wrap

제한 및 요구사항
  • 노드가 3개 미만인 클러스터 토폴로지에 사용되는 경우 Ceph가 제외됩니다. 예를 들어 Ceph는 단일 작업자 노드가 있는 단일 노드 OpenShift 클러스터 또는 단일 노드 OpenShift 클러스터에서 제외됩니다.
  • 단일 노드 OpenShift 클러스터에서 영구 스토리지는 둘 다 아닌 LVMS 또는 로컬 스토리지에서 제공해야 합니다.
엔지니어링 고려 사항
  • LVMS Operator는 DU 사용 사례에 대한 참조 스토리지 솔루션이 아닙니다. 애플리케이션 워크로드에 LVMS Operator가 필요한 경우 애플리케이션 코어에 대해 리소스 사용을 고려합니다.
  • 스토리지 요구 사항에 충분한 디스크 또는 파티션을 사용할 수 있는지 확인합니다.
3.2.3.9. 워크로드 파티셔닝
이번 릴리스의 새로운 기능
  • 이 릴리스에는 참조 디자인 업데이트가 없습니다
설명

워크로드 파티셔닝은 DU 프로파일의 일부인 OpenShift 플랫폼과 Day 2 Operator Pod를 예약된 cpuset 에 고정하고 노드 회계에서 예약된 CPU를 제거합니다. 이렇게 하면 사용자 워크로드에서 예약되지 않은 모든 CPU 코어를 사용할 수 있습니다.

OpenShift Container Platform 4.14에서 워크로드 파티셔닝을 활성화하고 구성하는 방법입니다.

4.14 이상
  • 설치 매개변수를 설정하여 파티션을 구성합니다.

    cpuPartitioningMode: AllNodes
    Copy to Clipboard Toggle word wrap
  • PerformanceProfile CR에 예약된 CPU 세트를 사용하여 관리 파티션 코어 구성
4.13 및 이전 버전
  • 설치 시 적용된 추가 MachineConfiguration CR을 사용하여 파티션 구성
제한 및 요구사항
  • Pod 를 관리 파티션에 적용할 수 있도록 네임스페이스 및 Pod CR에 주석을 달 수 있어야 합니다.
  • CPU 제한이 있는 Pod는 파티션에 할당할 수 없습니다. 변경으로 Pod QoS를 변경할 수 있기 때문입니다.
  • 관리 파티션에 할당할 수 있는 최소 CPU 수에 대한 자세한 내용은 Node Tuning Operator 를 참조하십시오.
엔지니어링 고려 사항
  • 워크로드 파티셔닝은 모든 관리 Pod를 예약된 코어에 고정합니다. 워크로드 시작, 노드 재부팅 또는 기타 시스템 이벤트가 발생할 때 발생하는 CPU 사용 급증을 고려하여 예약된 세트에 코어 수를 할당해야 합니다.
3.2.3.10. 클러스터 튜닝
이번 릴리스의 새로운 기능
  • 이 릴리스에는 참조 디자인 업데이트가 없습니다
설명

클러스터 기능 기능에는 제외 시 클러스터에서 다음 Operator 및 해당 리소스를 비활성화하는 MachineAPI 구성 요소가 포함되어 있습니다.

  • openshift/cluster-autoscaler-operator
  • openshift/cluster-control-plane-machine-set-operator
  • openshift/machine-api-operator
참고

클러스터 기능을 사용하여 이미지 레지스트리 Operator를 제거합니다.

제한 및 요구사항
  • 설치 관리자가 프로비저닝한 설치 방법에서는 클러스터 기능을 사용할 수 없습니다.
  • 모든 플랫폼 튜닝 구성을 적용해야 합니다. 다음 표에는 필요한 플랫폼 튜닝 구성이 나열되어 있습니다.

    Expand
    표 3.2. 클러스터 기능 구성
    기능설명

    선택적 클러스터 기능 제거

    단일 노드 OpenShift 클러스터에서만 선택적 클러스터 Operator를 비활성화하여 OpenShift Container Platform 풋프린트를 줄입니다.

    • Marketplace 및 Node Tuning Operator를 제외한 모든 선택적 Operator를 제거합니다.

    클러스터 모니터링 구성

    다음을 수행하여 공간 절약을 위해 모니터링 스택을 구성합니다.

    • 로컬 alertmanager 및 Telemeter 구성 요소를 비활성화 합니다.
    • RHACM 관찰 기능을 사용하는 경우 경고를 허브 클러스터에 전달하려면 적절한 additionalAlertManagerConfigs CR로 CR을 보강해야 합니다.
    • Prometheus 보존 기간을 24시간으로 줄입니다.

      참고

      RHACM 허브 클러스터는 관리되는 클러스터 메트릭을 집계합니다.

    네트워킹 진단 비활성화

    필요하지 않으므로 단일 노드 OpenShift에 대한 네트워킹 진단을 비활성화합니다.

    단일 OperatorHub 카탈로그 소스 구성

    RAN DU 배포에 필요한 Operator만 포함하는 단일 카탈로그 소스를 사용하도록 클러스터를 구성합니다. 각 카탈로그 소스는 클러스터에서 CPU 사용을 늘립니다. 단일 CatalogSource 를 사용하면 플랫폼 CPU 예산에 적합합니다.

3.2.3.11. 머신 구성
이번 릴리스의 새로운 기능
  • 이 릴리스에는 참조 디자인 업데이트가 없습니다
제한 및 요구사항
  • CRI-O wipe disable MachineConfig 는 디스크의 이미지가 정의된 유지 관리 창에서 예약된 유지 관리 중이 아닌 정적이라고 가정합니다. 이미지가 정적임을 확인하려면 Pod imagePullPolicy 필드를 Always 로 설정하지 마십시오.

    Expand
    표 3.3. 머신 구성 옵션
    기능설명

    컨테이너 런타임

    모든 노드 역할에 대해 컨테이너 런타임을 crun 으로 설정합니다.

    kubelet 구성 및 컨테이너 마운트 숨기기

    kubelet 하우스키핑 및 제거 모니터링의 빈도를 줄여 CPU 사용량을 줄입니다. kubelet 및 CRI-O에 표시되는 컨테이너 마운트 네임스페이스를 생성하여 시스템 마운트 검사 리소스 사용량을 줄입니다.

    SCTP

    선택적 구성(기본적으로 활성화)은 SCTP를 활성화합니다. SCTP는 RAN 애플리케이션에 필요하지만 RHCOS에서 기본적으로 비활성화되어 있습니다.

    kdump

    선택적 설정(기본적으로 사용)을 사용하면 커널 패닉이 발생할 때 kdump에서 디버그 정보를 캡처할 수 있습니다.

    CRI-O wipe disable

    클린 종료 후 CRI-O 이미지 캐시 자동 제거 기능을 비활성화합니다.

    SR-IOV 관련 커널 인수

    커널 명령줄에 추가 SR-IOV 관련 인수가 포함됩니다.

    RCU 일반 systemd 서비스

    시스템이 완전히 시작된 후 rcu_normal 를 설정합니다.

    일회성 시간 동기화

    컨트롤 플레인 또는 작업자 노드에 대한 일회성 시스템 시간 동기화 작업을 실행합니다.

3.2.3.12. 참조 설계 배포 구성 요소

다음 섹션에서는 RHACM(Red Hat Advanced Cluster Management)을 사용하여 허브 클러스터를 구성하는 데 사용하는 다양한 OpenShift Container Platform 구성 요소 및 구성에 대해 설명합니다.

3.2.3.12.1. Red Hat Advanced Cluster Management(RHACM)
이번 릴리스의 새로운 기능
  • 이 릴리스에는 참조 디자인 업데이트가 없습니다
설명

RHACM은 배포된 클러스터에 대한 MCE(Multi Cluster Engine) 설치 및 지속적인 라이프사이클 관리 기능을 제공합니다. Policy CR을 사용하여 구성 및 업그레이드를 선언적으로 지정하고 토폴로지 Aware Lifecycle Manager에서 관리하는 대로 RHACM 정책 컨트롤러를 사용하여 클러스터에 정책을 적용합니다.

  • ZTP(ZTP)는 RHACM의 MCE 기능을 사용합니다.
  • 구성, 업그레이드 및 클러스터 상태는 RHACM 정책 컨트롤러로 관리됩니다.

설치 중 RHACM은 site Config CR(사용자 정의 리소스)에 구성된 개별 노드에 레이블을 적용할 수 있습니다.

제한 및 요구사항
  • 단일 허브 클러스터는 5 Policy CR이 각 클러스터에 바인딩된 최대 3500개의 배포된 단일 노드 OpenShift 클러스터를 지원합니다.
엔지니어링 고려 사항
  • RHACM 정책 허브 측 템플릿을 사용하여 클러스터 구성을 보다 효과적으로 확장할 수 있습니다. 단일 그룹 정책 또는 그룹과 클러스터별 값이 템플릿으로 대체되는 일반 그룹 정책 수를 사용하여 정책 수를 크게 줄일 수 있습니다.
  • 클러스터별 구성: 관리 클러스터에는 일반적으로 개별 클러스터에 고유한 몇 가지 구성 값이 있습니다. 이러한 구성은 클러스터 이름을 기반으로 ConfigMap CR에서 가져온 값을 사용하여 RHACM 정책 허브 쪽 템플릿을 사용하여 관리해야 합니다.
  • 관리 클러스터에 CPU 리소스를 저장하려면 클러스터의 GitOps ZTP 설치 후 정적 구성을 적용하는 정책을 관리 클러스터에서 바인딩해야 합니다. 자세한 내용은 영구 볼륨 릴리스를 참조하십시오.
3.2.3.12.2. 토폴로지 인식 라이프사이클 관리자(TALM)
이번 릴리스의 새로운 기능
  • 이 릴리스에는 참조 디자인 업데이트가 없습니다
설명
관리형 업데이트

TALM은 변경 사항(클러스터 및 Operator 업그레이드, 구성 등)이 네트워크에 롤아웃되는 방식을 관리하기 위해 hub 클러스터에서만 실행되는 Operator입니다. TALM은 다음을 수행합니다.

  • Policy CR을 사용하여 사용자 구성 가능한 일괄 처리의 클러스터에 점진적으로 업데이트를 적용합니다.
  • 클러스터별로 ztp-done 레이블 또는 기타 사용자 구성 가능 라벨 추가
단일 노드 OpenShift 클러스터 사전 연결

TALM은 업그레이드를 시작하기 전에 OpenShift Container Platform, OLM Operator 및 추가 사용자 이미지의 선택적 사전 처리를 단일 노드 OpenShift 클러스터에 지원합니다.

  • PreCachingConfig 사용자 지정 리소스는 선택적 사전 캐싱 구성을 지정하는 데 사용할 수 있습니다. 예를 들면 다음과 같습니다.

    apiVersion: ran.openshift.io/v1alpha1
    kind: PreCachingConfig
    metadata:
      name: example-config
      namespace: example-ns
    spec:
      additionalImages:
        - quay.io/foobar/application1@sha256:3d5800990dee7cd4727d3fe238a97e2d2976d3808fc925ada29c559a47e2e
        - quay.io/foobar/application2@sha256:3d5800123dee7cd4727d3fe238a97e2d2976d3808fc925ada29c559a47adf
        - quay.io/foobar/applicationN@sha256:4fe1334adfafadsf987123adfffdaf1243340adfafdedga0991234afdadfs
      spaceRequired: 45 GiB 
    1
    
      overrides:
        preCacheImage: quay.io/test_images/pre-cache:latest
        platformImage: quay.io/openshift-release-dev/ocp-release@sha256:3d5800990dee7cd4727d3fe238a97e2d2976d3808fc925ada29c559a47e2e
      operatorsIndexes:
        - registry.example.com:5000/custom-redhat-operators:1.0.0
      operatorsPackagesAndChannels:
        - local-storage-operator: stable
        - ptp-operator: stable
        - sriov-network-operator: stable
      excludePrecachePatterns: 
    2
    
        - aws
        - vsphere
    Copy to Clipboard Toggle word wrap
    1 1
    구성 가능한 space-required 매개변수를 사용하면 스토리지 공간 전후의 유효성을 검사할 수 있습니다.
    2
    구성 가능한 필터링을 사용하면 사용되지 않는 이미지를 제외할 수 있습니다.
단일 노드 OpenShift의 백업 및 복원
TALM은 로컬 디스크의 전용 파티션으로 클러스터 운영 체제 및 구성의 스냅샷을 작성할 수 있습니다. 클러스터를 백업 상태로 반환하는 복원 스크립트가 제공됩니다.
제한 및 요구사항
  • TALM은 400개의 배치로 동시 클러스터 배포를 지원
  • 사전 캐싱 및 백업 기능은 단일 노드 OpenShift 클러스터에만 사용할 수 있습니다.
엔지니어링 고려 사항
  • PreCachingConfig CR은 선택 사항이며 플랫폼 관련 (OpenShift 및 OLM Operator) 이미지를 사전 캐시하려는 경우 생성할 필요가 없습니다. PreCachingConfig CR은 ClusterGroupUpgrade CR에서 참조하기 전에 적용해야 합니다.
  • TALM 백업 및 복원 기능을 사용하도록 선택한 경우 설치 중에 복구 파티션을 만듭니다.
3.2.3.12.3. GitOps 및 GitOps ZTP 플러그인
이번 릴리스의 새로운 기능
  • 이 릴리스에는 참조 디자인 업데이트가 없습니다
설명

GitOps 및 GitOps ZTP 플러그인은 클러스터 배포 및 구성을 관리하기 위한 GitOps 기반 인프라를 제공합니다. 클러스터 정의 및 구성은 Git에서 선언적 상태로 유지됩니다. ZTP 플러그인은 site Config CR에서 설치 CR을 생성하고 PolicyGenTemplate CR을 기반으로 정책의 구성 CR을 자동으로 래핑할 수 있도록 지원합니다.

기준 참조 구성 CR을 사용하여 관리형 클러스터에서 여러 버전의 OpenShift Container Platform을 배포하고 관리할 수 있습니다. 기본 CR과 함께 사용자 정의 CR을 사용할 수도 있습니다.

제한
  • ArgoCD 애플리케이션당 300개의 site Config CR. 여러 애플리케이션을 사용하여 단일 허브 클러스터에서 지원하는 최대 클러스터 수를 달성할 수 있습니다.
  • Git의 /source-crs 폴더의 콘텐츠는 GitOps ZTP 플러그인 컨테이너에 제공된 콘텐츠를 덮어씁니다. Git이 검색 경로에서 우선합니다.
  • PolicyGenTemplate 을 생성기로 포함하는 kustomization.yaml 파일과 동일한 디렉터리에 /source-crs 폴더를 추가합니다.

    참고

    이 컨텍스트에서 /source-crs 디렉터리의 대체 위치는 지원되지 않습니다.

엔지니어링 고려 사항
  • 콘텐츠를 업데이트할 때 파일의 혼동 또는 의도하지 않은 덮어쓰기를 방지하려면 /source-crs 폴더 및 Git의 추가 매니페스트에서 사용자 제공 CR에 대해 고유하고 구분 가능한 이름을 사용합니다.
  • SiteConfig CR을 사용하면 여러 추가 경로가 허용됩니다. 동일한 이름의 파일이 여러 디렉토리 경로에 있는 경우 마지막으로 발견된 파일이 우선합니다. 이를 통해 Git에 버전별 Day 0 매니페스트(extra-manifests)의 전체 세트를 배치하고 site Config CR에서 참조할 수 있습니다. 이 기능을 사용하면 여러 OpenShift Container Platform 버전을 관리 클러스터에 동시에 배포할 수 있습니다.
  • SiteConfig CR의 extraManifestPath 필드는 OpenShift Container Platform 4.15 이상에서 더 이상 사용되지 않습니다. 대신 새로운 extraManifests.searchPaths 필드를 사용합니다.
3.2.3.12.4. 에이전트 기반 설치 프로그램
이번 릴리스의 새로운 기능
  • 이 릴리스에는 참조 디자인 업데이트가 없습니다
설명

에이전트 기반 설치 관리자(ABI)는 중앙 집중식 인프라 없이 설치 기능을 제공합니다. 설치 프로그램은 서버에 마운트하는 ISO 이미지를 생성합니다. 서버를 부팅하면 OpenShift Container Platform을 설치하고 추가 매니페스트를 제공했습니다.

참고

ABI를 사용하여 허브 클러스터 없이 OpenShift Container Platform 클러스터를 설치할 수도 있습니다. 이러한 방식으로 ABI를 사용할 때 이미지 레지스트리가 계속 필요합니다.

에이전트 기반 설치 관리자(ABI)는 선택적 구성 요소입니다.

제한 및 요구사항
  • 설치 시 제한된 추가 매니페스트 세트를 제공할 수 있습니다.
  • RAN DU 사용 사례에 필요한 MachineConfiguration CR을 포함해야 합니다.
엔지니어링 고려 사항
  • ABI는 기본 OpenShift Container Platform 설치를 제공합니다.
  • 설치 후 Day 2 Operator 및 RAN DU 사용 사례 구성을 설치합니다.
3.2.3.13. 추가 구성 요소
3.2.3.13.1. Bare Metal Event Relay

Bare Metal Event Relay는 관리형 spoke 클러스터에서 독점적으로 실행되는 선택적 Operator입니다. Redfish 하드웨어 이벤트를 클러스터 애플리케이션으로 릴레이합니다.

참고

베어 메탈 이벤트 릴레이는 RAN DU에 포함되지 않으며 모델 참조 구성에 포함되어 있으며 선택적 기능입니다. Bare Metal Event Relay를 사용하려면 애플리케이션 CPU 예산에서 추가 CPU 리소스를 할당합니다.

3.2.4. Ttelco RAN distributed unit (DU) 참조 구성 CR

다음 CR(사용자 정의 리소스)을 사용하여 telco RAN DU 프로필을 사용하여 OpenShift Container Platform 클러스터를 구성하고 배포합니다. 일부 CR은 요구 사항에 따라 선택 사항입니다. 변경할 수 있는 CR 필드에는 YAML 주석을 사용하여 CR에 주석이 추가됩니다.

참고

ztp-site-generate 컨테이너 이미지에서 RAN DU CR의 전체 세트를 추출할 수 있습니다. 자세한 내용은 GitOps ZTP 사이트 구성 리포지토리 준비를 참조하십시오.

3.2.4.1. 2일차 Operator 참조 CR
Expand
표 3.4. 2일차 Operator CR
Component참조 CR선택 사항이번 릴리스의 새로운 기능

클러스터 로깅

ClusterLogForwarder.yaml

없음

없음

클러스터 로깅

ClusterLogging.yaml

없음

없음

클러스터 로깅

ClusterLogNS.yaml

없음

없음

클러스터 로깅

ClusterLogOperGroup.yaml

없음

없음

클러스터 로깅

ClusterLogSubscription.yaml

없음

없음

Local Storage Operator

StorageClass.yaml

제공됨

없음

Local Storage Operator

StorageLV.yaml

제공됨

없음

Local Storage Operator

StorageNS.yaml

제공됨

없음

Local Storage Operator

StorageOperGroup.yaml

제공됨

없음

Local Storage Operator

StorageSubscription.yaml

제공됨

없음

Node Tuning Operator

PerformanceProfile.yaml

없음

없음

Node Tuning Operator

TunedPerformancePatch.yaml

없음

없음

PTP 빠른 이벤트 알림

PtpOperatorConfigForEvent.yaml

제공됨

없음

PTP Operator

PtpConfigBoundary.yaml

없음

없음

PTP Operator

PtpConfigDualCardGmWpc.yaml

없음

없음

PTP Operator

PtpConfigGmWpc.yaml

없음

없음

PTP Operator

PtpConfigSlave.yaml

없음

없음

PTP Operator

SriovFecClusterConfig.yaml

제공됨

없음

SR-IOV Operator

SriovNetwork.yaml

없음

없음

SR-IOV Operator

SriovNetworkNodePolicy.yaml

없음

없음

SR-IOV Operator

SriovOperatorConfig.yaml

없음

없음

SR-IOV Operator

SriovSubscription.yaml

없음

없음

SR-IOV Operator

SriovSubscriptionNS.yaml

없음

없음

SR-IOV Operator

SriovSubscriptionOperGroup.yaml

없음

없음

3.2.4.2. 클러스터 튜닝 참조 CR
Expand
표 3.5.
    

3.2.4.3.
Expand
표 3.6.
    

3.2.4.4.

3.2.4.4.1.

apiVersion: "logging.openshift.io/v1"
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
  annotations: {}
spec:
  outputs: $outputs
  pipelines: $pipelines
Copy to Clipboard Toggle word wrap

apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
  name: instance
  namespace: openshift-logging
  annotations: {}
spec:
  managementState: "Managed"
  collection:
    logs:
      type: "vector"
Copy to Clipboard Toggle word wrap

---
apiVersion: v1
kind: Namespace
metadata:
  name: openshift-logging
  annotations:
    workload.openshift.io/allowed: management
Copy to Clipboard Toggle word wrap

---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: cluster-logging
  namespace: openshift-logging
  annotations: {}
spec:
  targetNamespaces:
    - openshift-logging
Copy to Clipboard Toggle word wrap

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: cluster-logging
  namespace: openshift-logging
  annotations: {}
spec:
  channel: "stable"
  name: cluster-logging
  source: redhat-operators-disconnected
  sourceNamespace: openshift-marketplace
  installPlanApproval: Manual
status:
  state: AtLatestKnown
Copy to Clipboard Toggle word wrap

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations: {}
  name: example-storage-class
provisioner: kubernetes.io/no-provisioner
reclaimPolicy: Delete
Copy to Clipboard Toggle word wrap

apiVersion: "local.storage.openshift.io/v1"
kind: "LocalVolume"
metadata:
  name: "local-disks"
  namespace: "openshift-local-storage"
  annotations: {}
spec:
  logLevel: Normal
  managementState: Managed
  storageClassDevices:
    # The list of storage classes and associated devicePaths need to be specified like this example:
    - storageClassName: "example-storage-class"
      volumeMode: Filesystem
      fsType: xfs
      # The below must be adjusted to the hardware.
      # For stability and reliability, it's recommended to use persistent
      # naming conventions for devicePaths, such as /dev/disk/by-path.
      devicePaths:
        - /dev/disk/by-path/pci-0000:05:00.0-nvme-1
#---
## How to verify
## 1. Create a PVC
# apiVersion: v1
# kind: PersistentVolumeClaim
# metadata:
#   name: local-pvc-name
# spec:
#   accessModes:
#   - ReadWriteOnce
#   volumeMode: Filesystem
#   resources:
#     requests:
#       storage: 100Gi
#   storageClassName: example-storage-class
#---
## 2. Create a pod that mounts it
# apiVersion: v1
# kind: Pod
# metadata:
#   labels:
#     run: busybox
#   name: busybox
# spec:
#   containers:
#   - image: quay.io/quay/busybox:latest
#     name: busybox
#     resources: {}
#     command: ["/bin/sh", "-c", "sleep infinity"]
#     volumeMounts:
#     - name: local-pvc
#       mountPath: /data
#   volumes:
#   - name: local-pvc
#     persistentVolumeClaim:
#       claimName: local-pvc-name
#   dnsPolicy: ClusterFirst
#   restartPolicy: Always
## 3. Run the pod on the cluster and verify the size and access of the `/data` mount
Copy to Clipboard Toggle word wrap

apiVersion: v1
kind: Namespace
metadata:
  name: openshift-local-storage
  annotations:
    workload.openshift.io/allowed: management
Copy to Clipboard Toggle word wrap

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: openshift-local-storage
  namespace: openshift-local-storage
  annotations: {}
spec:
  targetNamespaces:
    - openshift-local-storage
Copy to Clipboard Toggle word wrap

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: local-storage-operator
  namespace: openshift-local-storage
  annotations: {}
spec:
  channel: "stable"
  name: local-storage-operator
  source: redhat-operators-disconnected
  sourceNamespace: openshift-marketplace
  installPlanApproval: Manual
status:
  state: AtLatestKnown
Copy to Clipboard Toggle word wrap

apiVersion: performance.openshift.io/v2
kind: PerformanceProfile
metadata:
  # if you change this name make sure the 'include' line in TunedPerformancePatch.yaml
  # matches this name: include=openshift-node-performance-${PerformanceProfile.metadata.name}
  # Also in file 'validatorCRs/informDuValidator.yaml':
  # name: 50-performance-${PerformanceProfile.metadata.name}
  name: openshift-node-performance-profile
  annotations:
    ran.openshift.io/reference-configuration: "ran-du.redhat.com"
spec:
  additionalKernelArgs:
    - "rcupdate.rcu_normal_after_boot=0"
    - "efi=runtime"
    - "vfio_pci.enable_sriov=1"
    - "vfio_pci.disable_idle_d3=1"
    - "module_blacklist=irdma"
  cpu:
    isolated: $isolated
    reserved: $reserved
  hugepages:
    defaultHugepagesSize: $defaultHugepagesSize
    pages:
      - size: $size
        count: $count
        node: $node
  machineConfigPoolSelector:
    pools.operator.machineconfiguration.openshift.io/$mcp: ""
  nodeSelector:
    node-role.kubernetes.io/$mcp: ''
  numa:
    topologyPolicy: "restricted"
  # To use the standard (non-realtime) kernel, set enabled to false
  realTimeKernel:
    enabled: true
  workloadHints:
    # WorkloadHints defines the set of upper level flags for different type of workloads.
    # See https://github.com/openshift/cluster-node-tuning-operator/blob/master/docs/performanceprofile/performance_profile.md#workloadhints
    # for detailed descriptions of each item.
    # The configuration below is set for a low latency, performance mode.
    realTime: true
    highPowerConsumption: false
    perPodPowerManagement: false
Copy to Clipboard Toggle word wrap

apiVersion: tuned.openshift.io/v1
kind: Tuned
metadata:
  name: performance-patch
  namespace: openshift-cluster-node-tuning-operator
  annotations: {}
spec:
  profile:
    - name: performance-patch
      # Please note:
      # - The 'include' line must match the associated PerformanceProfile name, following below pattern
      #   include=openshift-node-performance-${PerformanceProfile.metadata.name}
      # - When using the standard (non-realtime) kernel, remove the kernel.timer_migration override from
      #   the [sysctl] section and remove the entire section if it is empty.
      data: |
        [main]
        summary=Configuration changes profile inherited from performance created tuned
        include=openshift-node-performance-openshift-node-performance-profile
        [sysctl]
        kernel.timer_migration=1
        [scheduler]
        group.ice-ptp=0:f:10:*:ice-ptp.*
        group.ice-gnss=0:f:10:*:ice-gnss.*
        [service]
        service.stalld=start,enable
        service.chronyd=stop,disable
  recommend:
    - machineConfigLabels:
        machineconfiguration.openshift.io/role: "$mcp"
      priority: 19
      profile: performance-patch
Copy to Clipboard Toggle word wrap

apiVersion: ptp.openshift.io/v1
kind: PtpOperatorConfig
metadata:
  name: default
  namespace: openshift-ptp
  annotations: {}
spec:
  daemonNodeSelector:
    node-role.kubernetes.io/$mcp: ""
  ptpEventConfig:
    enableEventPublisher: true
    transportHost: "http://ptp-event-publisher-service-NODE_NAME.openshift-ptp.svc.cluster.local:9043"
Copy to Clipboard Toggle word wrap

apiVersion: ptp.openshift.io/v1
kind: PtpConfig
metadata:
  name: boundary
  namespace: openshift-ptp
  annotations: {}
spec:
  profile:
    - name: "boundary"
      ptp4lOpts: "-2"
      phc2sysOpts: "-a -r -n 24"
      ptpSchedulingPolicy: SCHED_FIFO
      ptpSchedulingPriority: 10
      ptpSettings:
        logReduce: "true"
      ptp4lConf: |
        # The interface name is hardware-specific
        [$iface_slave]
        masterOnly 0
        [$iface_master_1]
        masterOnly 1
        [$iface_master_2]
        masterOnly 1
        [$iface_master_3]
        masterOnly 1
        [global]
        #
        # Default Data Set
        #
        twoStepFlag 1
        slaveOnly 0
        priority1 128
        priority2 128
        domainNumber 24
        #utc_offset 37
        clockClass 248
        clockAccuracy 0xFE
        offsetScaledLogVariance 0xFFFF
        free_running 0
        freq_est_interval 1
        dscp_event 0
        dscp_general 0
        dataset_comparison G.8275.x
        G.8275.defaultDS.localPriority 128
        #
        # Port Data Set
        #
        logAnnounceInterval -3
        logSyncInterval -4
        logMinDelayReqInterval -4
        logMinPdelayReqInterval -4
        announceReceiptTimeout 3
        syncReceiptTimeout 0
        delayAsymmetry 0
        fault_reset_interval -4
        neighborPropDelayThresh 20000000
        masterOnly 0
        G.8275.portDS.localPriority 128
        #
        # Run time options
        #
        assume_two_step 0
        logging_level 6
        path_trace_enabled 0
        follow_up_info 0
        hybrid_e2e 0
        inhibit_multicast_service 0
        net_sync_monitor 0
        tc_spanning_tree 0
        tx_timestamp_timeout 50
        unicast_listen 0
        unicast_master_table 0
        unicast_req_duration 3600
        use_syslog 1
        verbose 0
        summary_interval 0
        kernel_leap 1
        check_fup_sync 0
        clock_class_threshold 135
        #
        # Servo Options
        #
        pi_proportional_const 0.0
        pi_integral_const 0.0
        pi_proportional_scale 0.0
        pi_proportional_exponent -0.3
        pi_proportional_norm_max 0.7
        pi_integral_scale 0.0
        pi_integral_exponent 0.4
        pi_integral_norm_max 0.3
        step_threshold 2.0
        first_step_threshold 0.00002
        max_frequency 900000000
        clock_servo pi
        sanity_freq_limit 200000000
        ntpshm_segment 0
        #
        # Transport options
        #
        transportSpecific 0x0
        ptp_dst_mac 01:1B:19:00:00:00
        p2p_dst_mac 01:80:C2:00:00:0E
        udp_ttl 1
        udp6_scope 0x0E
        uds_address /var/run/ptp4l
        #
        # Default interface options
        #
        clock_type BC
        network_transport L2
        delay_mechanism E2E
        time_stamping hardware
        tsproc_mode filter
        delay_filter moving_median
        delay_filter_length 10
        egressLatency 0
        ingressLatency 0
        boundary_clock_jbod 0
        #
        # Clock description
        #
        productDescription ;;
        revisionData ;;
        manufacturerIdentity 00:00:00
        userDescription ;
        timeSource 0xA0
  recommend:
    - profile: "boundary"
      priority: 4
      match:
        - nodeLabel: "node-role.kubernetes.io/$mcp"
Copy to Clipboard Toggle word wrap

# In this example 2 cards $iface_master and $iface_master_1 are connected via SMA1 ports by a cable
# and $iface_master_1 receives 1PPS signals from $iface_master
apiVersion: ptp.openshift.io/v1
kind: PtpConfig
metadata:
  name: grandmaster
  namespace: openshift-ptp
  annotations:
    ran.openshift.io/ztp-deploy-wave: "10"
spec:
  profile:
  - name: "grandmaster"
    ptp4lOpts: "-2 --summary_interval -4"
    phc2sysOpts: -r -u 0 -m -w -N 8 -R 16 -s $iface_master -n 24
    ptpSchedulingPolicy: SCHED_FIFO
    ptpSchedulingPriority: 10
    ptpSettings:
      logReduce: "true"
    plugins:
      e810:
        enableDefaultConfig: false
        settings:
          LocalMaxHoldoverOffSet: 1500
          LocalHoldoverTimeout: 14400
          MaxInSpecOffset: 1500
        pins: $e810_pins
        #  "$iface_master":
        #    "U.FL2": "0 2"
        #    "U.FL1": "0 1"
        #    "SMA2": "0 2"
        #    "SMA1": "2 1"
        #  "$iface_master_1":
        #    "U.FL2": "0 2"
        #    "U.FL1": "0 1"
        #    "SMA2": "0 2"
        #    "SMA1": "1 1"
        ublxCmds:
          - args: #ubxtool -P 29.20 -z CFG-HW-ANT_CFG_VOLTCTRL,1
              - "-P"
              - "29.20"
              - "-z"
              - "CFG-HW-ANT_CFG_VOLTCTRL,1"
            reportOutput: false
          - args: #ubxtool -P 29.20 -e GPS
              - "-P"
              - "29.20"
              - "-e"
              - "GPS"
            reportOutput: false
          - args: #ubxtool -P 29.20 -d Galileo
              - "-P"
              - "29.20"
              - "-d"
              - "Galileo"
            reportOutput: false
          - args: #ubxtool -P 29.20 -d GLONASS
              - "-P"
              - "29.20"
              - "-d"
              - "GLONASS"
            reportOutput: false
          - args: #ubxtool -P 29.20 -d BeiDou
              - "-P"
              - "29.20"
              - "-d"
              - "BeiDou"
            reportOutput: false
          - args: #ubxtool -P 29.20 -d SBAS
              - "-P"
              - "29.20"
              - "-d"
              - "SBAS"
            reportOutput: false
          - args: #ubxtool -P 29.20 -t -w 5 -v 1 -e SURVEYIN,600,50000
              - "-P"
              - "29.20"
              - "-t"
              - "-w"
              - "5"
              - "-v"
              - "1"
              - "-e"
              - "SURVEYIN,600,50000"
            reportOutput: true
          - args: #ubxtool -P 29.20 -p MON-HW
              - "-P"
              - "29.20"
              - "-p"
              - "MON-HW"
            reportOutput: true
          - args: #ubxtool -P 29.20 -p CFG-MSG,1,38,248
              - "-P"
              - "29.20"
              - "-p"
              - "CFG-MSG,1,38,248"
            reportOutput: true
    ts2phcOpts: " "
    ts2phcConf: |
      [nmea]
      ts2phc.master 1
      [global]
      use_syslog  0
      verbose 1
      logging_level 7
      ts2phc.pulsewidth 100000000
      #cat /dev/GNSS to find available serial port
      #example value of gnss_serialport is /dev/ttyGNSS_1700_0
      ts2phc.nmea_serialport $gnss_serialport
      leapfile  /usr/share/zoneinfo/leap-seconds.list
      [$iface_master]
      ts2phc.extts_polarity rising
      ts2phc.extts_correction 0
      [$iface_master_1]
      ts2phc.extts_polarity rising
      #this is a measured value in nanoseconds to compensate for SMA cable delay
      ts2phc.extts_correction -10
    ptp4lConf: |
      [$iface_master]
      masterOnly 1
      [$iface_master_1]
      masterOnly 1
      [$iface_master_1_1]
      masterOnly 1
      [$iface_master_1_2]
      masterOnly 1
      [global]
      #
      # Default Data Set
      #
      twoStepFlag 1
      priority1 128
      priority2 128
      domainNumber 24
      #utc_offset 37
      clockClass 6
      clockAccuracy 0x27
      offsetScaledLogVariance 0xFFFF
      free_running 0
      freq_est_interval 1
      dscp_event 0
      dscp_general 0
      dataset_comparison G.8275.x
      G.8275.defaultDS.localPriority 128
      #
      # Port Data Set
      #
      logAnnounceInterval -3
      logSyncInterval -4
      logMinDelayReqInterval -4
      logMinPdelayReqInterval 0
      announceReceiptTimeout 3
      syncReceiptTimeout 0
      delayAsymmetry 0
      fault_reset_interval -4
      neighborPropDelayThresh 20000000
      masterOnly 0
      G.8275.portDS.localPriority 128
      #
      # Run time options
      #
      assume_two_step 0
      logging_level 6
      path_trace_enabled 0
      follow_up_info 0
      hybrid_e2e 0
      inhibit_multicast_service 0
      net_sync_monitor 0
      tc_spanning_tree 0
      tx_timestamp_timeout 50
      unicast_listen 0
      unicast_master_table 0
      unicast_req_duration 3600
      use_syslog 1
      verbose 0
      summary_interval -4
      kernel_leap 1
      check_fup_sync 0
      clock_class_threshold 7
      #
      # Servo Options
      #
      pi_proportional_const 0.0
      pi_integral_const 0.0
      pi_proportional_scale 0.0
      pi_proportional_exponent -0.3
      pi_proportional_norm_max 0.7
      pi_integral_scale 0.0
      pi_integral_exponent 0.4
      pi_integral_norm_max 0.3
      step_threshold 2.0
      first_step_threshold 0.00002
      clock_servo pi
      sanity_freq_limit  200000000
      ntpshm_segment 0
      #
      # Transport options
      #
      transportSpecific 0x0
      ptp_dst_mac 01:1B:19:00:00:00
      p2p_dst_mac 01:80:C2:00:00:0E
      udp_ttl 1
      udp6_scope 0x0E
      uds_address /var/run/ptp4l
      #
      # Default interface options
      #
      clock_type BC
      network_transport L2
      delay_mechanism E2E
      time_stamping hardware
      tsproc_mode filter
      delay_filter moving_median
      delay_filter_length 10
      egressLatency 0
      ingressLatency 0
      boundary_clock_jbod 1
      #
      # Clock description
      #
      productDescription ;;
      revisionData ;;
      manufacturerIdentity 00:00:00
      userDescription ;
      timeSource 0x20
  recommend:
  - profile: "grandmaster"
    priority: 4
    match:
    - nodeLabel: "node-role.kubernetes.io/$mcp"
Copy to Clipboard Toggle word wrap

# The grandmaster profile is provided for testing only
# It is not installed on production clusters
apiVersion: ptp.openshift.io/v1
kind: PtpConfig
metadata:
  name: grandmaster
  namespace: openshift-ptp
  annotations: {}
spec:
  profile:
    - name: "grandmaster"
      ptp4lOpts: "-2 --summary_interval -4"
      phc2sysOpts: -r -u 0 -m -O -37 -N 8 -R 16 -s $iface_master -n 24
      ptpSchedulingPolicy: SCHED_FIFO
      ptpSchedulingPriority: 10
      ptpSettings:
        logReduce: "true"
      plugins:
        e810:
          enableDefaultConfig: false
          settings:
            LocalMaxHoldoverOffSet: 1500
            LocalHoldoverTimeout: 14400
            MaxInSpecOffset: 1500
          pins: $e810_pins
          #  "$iface_master":
          #    "U.FL2": "0 2"
          #    "U.FL1": "0 1"
          #    "SMA2": "0 2"
          #    "SMA1": "0 1"
          ublxCmds:
            - args: #ubxtool -P 29.20 -z CFG-HW-ANT_CFG_VOLTCTRL,1
                - "-P"
                - "29.20"
                - "-z"
                - "CFG-HW-ANT_CFG_VOLTCTRL,1"
              reportOutput: false
            - args: #ubxtool -P 29.20 -e GPS
                - "-P"
                - "29.20"
                - "-e"
                - "GPS"
              reportOutput: false
            - args: #ubxtool -P 29.20 -d Galileo
                - "-P"
                - "29.20"
                - "-d"
                - "Galileo"
              reportOutput: false
            - args: #ubxtool -P 29.20 -d GLONASS
                - "-P"
                - "29.20"
                - "-d"
                - "GLONASS"
              reportOutput: false
            - args: #ubxtool -P 29.20 -d BeiDou
                - "-P"
                - "29.20"
                - "-d"
                - "BeiDou"
              reportOutput: false
            - args: #ubxtool -P 29.20 -d SBAS
                - "-P"
                - "29.20"
                - "-d"
                - "SBAS"
              reportOutput: false
            - args: #ubxtool -P 29.20 -t -w 5 -v 1 -e SURVEYIN,600,50000
                - "-P"
                - "29.20"
                - "-t"
                - "-w"
                - "5"
                - "-v"
                - "1"
                - "-e"
                - "SURVEYIN,600,50000"
              reportOutput: true
            - args: #ubxtool -P 29.20 -p MON-HW
                - "-P"
                - "29.20"
                - "-p"
                - "MON-HW"
              reportOutput: true
            - args: #ubxtool -P 29.20 -p CFG-MSG,1,38,300
                - "-P"
                - "29.20"
                - "-p"
                - "CFG-MSG,1,38,300"
              reportOutput: true
      ts2phcOpts: " "
      ts2phcConf: |
        [nmea]
        ts2phc.master 1
        [global]
        use_syslog  0
        verbose 1
        logging_level 7
        ts2phc.pulsewidth 100000000
        #cat /dev/GNSS to find available serial port
        #example value of gnss_serialport is /dev/ttyGNSS_1700_0
        ts2phc.nmea_serialport $gnss_serialport
        leapfile  /usr/share/zoneinfo/leap-seconds.list
        [$iface_master]
        ts2phc.extts_polarity rising
        ts2phc.extts_correction 0
      ptp4lConf: |
        [$iface_master]
        masterOnly 1
        [$iface_master_1]
        masterOnly 1
        [$iface_master_2]
        masterOnly 1
        [$iface_master_3]
        masterOnly 1
        [global]
        #
        # Default Data Set
        #
        twoStepFlag 1
        priority1 128
        priority2 128
        domainNumber 24
        #utc_offset 37
        clockClass 6
        clockAccuracy 0x27
        offsetScaledLogVariance 0xFFFF
        free_running 0
        freq_est_interval 1
        dscp_event 0
        dscp_general 0
        dataset_comparison G.8275.x
        G.8275.defaultDS.localPriority 128
        #
        # Port Data Set
        #
        logAnnounceInterval -3
        logSyncInterval -4
        logMinDelayReqInterval -4
        logMinPdelayReqInterval 0
        announceReceiptTimeout 3
        syncReceiptTimeout 0
        delayAsymmetry 0
        fault_reset_interval -4
        neighborPropDelayThresh 20000000
        masterOnly 0
        G.8275.portDS.localPriority 128
        #
        # Run time options
        #
        assume_two_step 0
        logging_level 6
        path_trace_enabled 0
        follow_up_info 0
        hybrid_e2e 0
        inhibit_multicast_service 0
        net_sync_monitor 0
        tc_spanning_tree 0
        tx_timestamp_timeout 50
        unicast_listen 0
        unicast_master_table 0
        unicast_req_duration 3600
        use_syslog 1
        verbose 0
        summary_interval -4
        kernel_leap 1
        check_fup_sync 0
        clock_class_threshold 7
        #
        # Servo Options
        #
        pi_proportional_const 0.0
        pi_integral_const 0.0
        pi_proportional_scale 0.0
        pi_proportional_exponent -0.3
        pi_proportional_norm_max 0.7
        pi_integral_scale 0.0
        pi_integral_exponent 0.4
        pi_integral_norm_max 0.3
        step_threshold 2.0
        first_step_threshold 0.00002
        clock_servo pi
        sanity_freq_limit  200000000
        ntpshm_segment 0
        #
        # Transport options
        #
        transportSpecific 0x0
        ptp_dst_mac 01:1B:19:00:00:00
        p2p_dst_mac 01:80:C2:00:00:0E
        udp_ttl 1
        udp6_scope 0x0E
        uds_address /var/run/ptp4l
        #
        # Default interface options
        #
        clock_type BC
        network_transport L2
        delay_mechanism E2E
        time_stamping hardware
        tsproc_mode filter
        delay_filter moving_median
        delay_filter_length 10
        egressLatency 0
        ingressLatency 0
        boundary_clock_jbod 0
        #
        # Clock description
        #
        productDescription ;;
        revisionData ;;
        manufacturerIdentity 00:00:00
        userDescription ;
        timeSource 0x20
  recommend:
    - profile: "grandmaster"
      priority: 4
      match:
        - nodeLabel: "node-role.kubernetes.io/$mcp"
Copy to Clipboard Toggle word wrap

apiVersion: ptp.openshift.io/v1
kind: PtpConfig
metadata:
  name: ordinary
  namespace: openshift-ptp
  annotations: {}
spec:
  profile:
    - name: "ordinary"
      # The interface name is hardware-specific
      interface: $interface
      ptp4lOpts: "-2 -s"
      phc2sysOpts: "-a -r -n 24"
      ptpSchedulingPolicy: SCHED_FIFO
      ptpSchedulingPriority: 10
      ptpSettings:
        logReduce: "true"
      ptp4lConf: |
        [global]
        #
        # Default Data Set
        #
        twoStepFlag 1
        slaveOnly 1
        priority1 128
        priority2 128
        domainNumber 24
        #utc_offset 37
        clockClass 255
        clockAccuracy 0xFE
        offsetScaledLogVariance 0xFFFF
        free_running 0
        freq_est_interval 1
        dscp_event 0
        dscp_general 0
        dataset_comparison G.8275.x
        G.8275.defaultDS.localPriority 128
        #
        # Port Data Set
        #
        logAnnounceInterval -3
        logSyncInterval -4
        logMinDelayReqInterval -4
        logMinPdelayReqInterval -4
        announceReceiptTimeout 3
        syncReceiptTimeout 0
        delayAsymmetry 0
        fault_reset_interval -4
        neighborPropDelayThresh 20000000
        masterOnly 0
        G.8275.portDS.localPriority 128
        #
        # Run time options
        #
        assume_two_step 0
        logging_level 6
        path_trace_enabled 0
        follow_up_info 0
        hybrid_e2e 0
        inhibit_multicast_service 0
        net_sync_monitor 0
        tc_spanning_tree 0
        tx_timestamp_timeout 50
        unicast_listen 0
        unicast_master_table 0
        unicast_req_duration 3600
        use_syslog 1
        verbose 0
        summary_interval 0
        kernel_leap 1
        check_fup_sync 0
        clock_class_threshold 7
        #
        # Servo Options
        #
        pi_proportional_const 0.0
        pi_integral_const 0.0
        pi_proportional_scale 0.0
        pi_proportional_exponent -0.3
        pi_proportional_norm_max 0.7
        pi_integral_scale 0.0
        pi_integral_exponent 0.4
        pi_integral_norm_max 0.3
        step_threshold 2.0
        first_step_threshold 0.00002
        max_frequency 900000000
        clock_servo pi
        sanity_freq_limit 200000000
        ntpshm_segment 0
        #
        # Transport options
        #
        transportSpecific 0x0
        ptp_dst_mac 01:1B:19:00:00:00
        p2p_dst_mac 01:80:C2:00:00:0E
        udp_ttl 1
        udp6_scope 0x0E
        uds_address /var/run/ptp4l
        #
        # Default interface options
        #
        clock_type OC
        network_transport L2
        delay_mechanism E2E
        time_stamping hardware
        tsproc_mode filter
        delay_filter moving_median
        delay_filter_length 10
        egressLatency 0
        ingressLatency 0
        boundary_clock_jbod 0
        #
        # Clock description
        #
        productDescription ;;
        revisionData ;;
        manufacturerIdentity 00:00:00
        userDescription ;
        timeSource 0xA0
  recommend:
    - profile: "ordinary"
      priority: 4
      match:
        - nodeLabel: "node-role.kubernetes.io/$mcp"
Copy to Clipboard Toggle word wrap

---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: ptp-operator-subscription
  namespace: openshift-ptp
  annotations: {}
spec:
  channel: "stable"
  name: ptp-operator
  source: redhat-operators-disconnected
  sourceNamespace: openshift-marketplace
  installPlanApproval: Manual
status:
  state: AtLatestKnown
Copy to Clipboard Toggle word wrap

---
apiVersion: v1
kind: Namespace
metadata:
  name: openshift-ptp
  annotations:
    workload.openshift.io/allowed: management
  labels:
    openshift.io/cluster-monitoring: "true"
Copy to Clipboard Toggle word wrap

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: ptp-operators
  namespace: openshift-ptp
  annotations: {}
spec:
  targetNamespaces:
    - openshift-ptp
Copy to Clipboard Toggle word wrap

apiVersion: v1
kind: Namespace
metadata:
  name: vran-acceleration-operators
  annotations: {}
Copy to Clipboard Toggle word wrap

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: vran-operators
  namespace: vran-acceleration-operators
  annotations: {}
spec:
  targetNamespaces:
    - vran-acceleration-operators
Copy to Clipboard Toggle word wrap

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: sriov-fec-subscription
  namespace: vran-acceleration-operators
  annotations: {}
spec:
  channel: stable
  name: sriov-fec
  source: certified-operators
  sourceNamespace: openshift-marketplace
  installPlanApproval: Manual
status:
  state: AtLatestKnown
Copy to Clipboard Toggle word wrap

apiVersion: sriovfec.intel.com/v2
kind: SriovFecClusterConfig
metadata:
  name: config
  namespace: vran-acceleration-operators
  annotations: {}
spec:
  drainSkip: $drainSkip # true if SNO, false by default
  priority: 1
  nodeSelector:
    node-role.kubernetes.io/master: ""
  acceleratorSelector:
    pciAddress: $pciAddress
  physicalFunction:
    pfDriver: "vfio-pci"
    vfDriver: "vfio-pci"
    vfAmount: 16
    bbDevConfig: $bbDevConfig
#Recommended configuration for Intel ACC100 (Mount Bryce) FPGA here: https://github.com/smart-edge-open/openshift-operator/blob/main/spec/openshift-sriov-fec-operator.md#sample-cr-for-wireless-fec-acc100
#Recommended configuration for Intel N3000 FPGA here: https://github.com/smart-edge-open/openshift-operator/blob/main/spec/openshift-sriov-fec-operator.md#sample-cr-for-wireless-fec-n3000
Copy to Clipboard Toggle word wrap

apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetwork
metadata:
  name: ""
  namespace: openshift-sriov-network-operator
  annotations: {}
spec:
  #  resourceName: ""
  networkNamespace: openshift-sriov-network-operator
#  vlan: ""
#  spoofChk: ""
#  ipam: ""
#  linkState: ""
#  maxTxRate: ""
#  minTxRate: ""
#  vlanQoS: ""
#  trust: ""
#  capabilities: ""
Copy to Clipboard Toggle word wrap

apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
  name: $name
  namespace: openshift-sriov-network-operator
  annotations: {}
spec:
  # The attributes for Mellanox/Intel based NICs as below.
  #     deviceType: netdevice/vfio-pci
  #     isRdma: true/false
  deviceType: $deviceType
  isRdma: $isRdma
  nicSelector:
    # The exact physical function name must match the hardware used
    pfNames: [$pfNames]
  nodeSelector:
    node-role.kubernetes.io/$mcp: ""
  numVfs: $numVfs
  priority: $priority
  resourceName: $resourceName
Copy to Clipboard Toggle word wrap

apiVersion: sriovnetwork.openshift.io/v1
kind: SriovOperatorConfig
metadata:
  name: default
  namespace: openshift-sriov-network-operator
  annotations: {}
spec:
  configDaemonNodeSelector:
    "node-role.kubernetes.io/$mcp": ""
  # Injector and OperatorWebhook pods can be disabled (set to "false") below
  # to reduce the number of management pods. It is recommended to start with the
  # webhook and injector pods enabled, and only disable them after verifying the
  # correctness of user manifests.
  #   If the injector is disabled, containers using sr-iov resources must explicitly assign
  #   them in the  "requests"/"limits" section of the container spec, for example:
  #    containers:
  #    - name: my-sriov-workload-container
  #      resources:
  #        limits:
  #          openshift.io/<resource_name>:  "1"
  #        requests:
  #          openshift.io/<resource_name>:  "1"
  enableInjector: true
  enableOperatorWebhook: true
  logLevel: 0
Copy to Clipboard Toggle word wrap

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: sriov-network-operator-subscription
  namespace: openshift-sriov-network-operator
  annotations: {}
spec:
  channel: "stable"
  name: sriov-network-operator
  source: redhat-operators-disconnected
  sourceNamespace: openshift-marketplace
  installPlanApproval: Manual
status:
  state: AtLatestKnown
Copy to Clipboard Toggle word wrap

apiVersion: v1
kind: Namespace
metadata:
  name: openshift-sriov-network-operator
  annotations:
    workload.openshift.io/allowed: management
Copy to Clipboard Toggle word wrap

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: sriov-network-operators
  namespace: openshift-sriov-network-operator
  annotations: {}
spec:
  targetNamespaces:
    - openshift-sriov-network-operator
Copy to Clipboard Toggle word wrap

3.2.4.4.2.

# example-node1-bmh-secret & assisted-deployment-pull-secret need to be created under same namespace example-sno
---
apiVersion: ran.openshift.io/v1
kind: SiteConfig
metadata:
  name: "example-sno"
  namespace: "example-sno"
spec:
  baseDomain: "example.com"
  pullSecretRef:
    name: "assisted-deployment-pull-secret"
  clusterImageSetNameRef: "openshift-4.10"
  sshPublicKey: "ssh-rsa AAAA..."
  clusters:
  - clusterName: "example-sno"
    networkType: "OVNKubernetes"
    # installConfigOverrides is a generic way of passing install-config
    # parameters through the siteConfig.  The 'capabilities' field configures
    # the composable openshift feature.  In this 'capabilities' setting, we
    # remove all but the marketplace component from the optional set of
    # components.
    # Notes:
    # - OperatorLifecycleManager is needed for 4.15 and later
    # - NodeTuning is needed for 4.13 and later, not for 4.12 and earlier
    installConfigOverrides: |
      {
        "capabilities": {
          "baselineCapabilitySet": "None",
          "additionalEnabledCapabilities": [
            "NodeTuning",
            "OperatorLifecycleManager"
          ]
        }
      }
    # It is strongly recommended to include crun manifests as part of the additional install-time manifests for 4.13+.
    # The crun manifests can be obtained from source-crs/optional-extra-manifest/ and added to the git repo ie.sno-extra-manifest.
    # extraManifestPath: sno-extra-manifest
    clusterLabels:
      # These example cluster labels correspond to the bindingRules in the PolicyGenTemplate examples
      du-profile: "latest"
      # These example cluster labels correspond to the bindingRules in the PolicyGenTemplate examples in ../policygentemplates:
      # ../policygentemplates/common-ranGen.yaml will apply to all clusters with 'common: true'
      common: true
      # ../policygentemplates/group-du-sno-ranGen.yaml will apply to all clusters with 'group-du-sno: ""'
      group-du-sno: ""
      # ../policygentemplates/example-sno-site.yaml will apply to all clusters with 'sites: "example-sno"'
      # Normally this should match or contain the cluster name so it only applies to a single cluster
      sites : "example-sno"
    clusterNetwork:
      - cidr: 1001:1::/48
        hostPrefix: 64
    machineNetwork:
      - cidr: 1111:2222:3333:4444::/64
    serviceNetwork:
      - 1001:2::/112
    additionalNTPSources:
      - 1111:2222:3333:4444::2
    # Initiates the cluster for workload partitioning. Setting specific reserved/isolated CPUSets is done via PolicyTemplate
    # please see Workload Partitioning Feature for a complete guide.
    cpuPartitioningMode: AllNodes
    # Optionally; This can be used to override the KlusterletAddonConfig that is created for this cluster:
    #crTemplates:
    #  KlusterletAddonConfig: "KlusterletAddonConfigOverride.yaml"
    nodes:
      - hostName: "example-node1.example.com"
        role: "master"
        # Optionally; This can be used to configure desired BIOS setting on a host:
        #biosConfigRef:
        #  filePath: "example-hw.profile"
        bmcAddress: "idrac-virtualmedia+https://[1111:2222:3333:4444::bbbb:1]/redfish/v1/Systems/System.Embedded.1"
        bmcCredentialsName:
          name: "example-node1-bmh-secret"
        bootMACAddress: "AA:BB:CC:DD:EE:11"
        # Use UEFISecureBoot to enable secure boot
        bootMode: "UEFI"
        rootDeviceHints:
          deviceName: "/dev/disk/by-path/pci-0000:01:00.0-scsi-0:2:0:0"
        # disk partition at `/var/lib/containers` with ignitionConfigOverride. Some values must be updated. See DiskPartitionContainer.md for more details
        ignitionConfigOverride: |
           {
            "ignition": {
              "version": "3.2.0"
            },
            "storage": {
              "disks": [
                {
                  "device": "/dev/disk/by-path/pci-0000:01:00.0-scsi-0:2:0:0",
                  "partitions": [
                    {
                     "label": "var-lib-containers",
                     "sizeMiB": 0,
                     "startMiB": 250000
                  }
              ],
              "wipeTable": false
             }
           ],
            "filesystems": [
              {
               "device": "/dev/disk/by-partlabel/var-lib-containers",
               "format": "xfs",
               "mountOptions": [
                 "defaults",
                 "prjquota"
                ],
                "path": "/var/lib/containers",
                "wipeFilesystem": true
               }
             ]
           },
           "systemd": {
             "units": [
               {
                "contents": "# Generated by Butane\n[Unit]\nRequires=systemd-fsck@dev-disk-by\\x2dpartlabel-var\\x2dlib\\x2dcontainers.service\nAfter=systemd-fsck@dev-disk-by\\x2dpartlabel-var\\x2dlib\\x2dcontainers.service\n\n[Mount]\nWhere=/var/lib/containers\nWhat=/dev/disk/by-partlabel/var-lib-containers\nType=xfs\nOptions=defaults,prjquota\n\n[Install]\nRequiredBy=local-fs.target",
                "enabled": true,
                "name": "var-lib-containers.mount"
               }
              ]
            }
           }
        nodeNetwork:
          interfaces:
            - name: eno1
              macAddress: "AA:BB:CC:DD:EE:11"
          config:
            interfaces:
              - name: eno1
                type: ethernet
                state: up
                ipv4:
                  enabled: false
                ipv6:
                  enabled: true
                  address:
                  # For SNO sites with static IP addresses, the node-specific,
                  # API and Ingress IPs should all be the same and configured on
                  # the interface
                  - ip: 1111:2222:3333:4444::aaaa:1
                    prefix-length: 64
            dns-resolver:
              config:
                search:
                - example.com
                server:
                - 1111:2222:3333:4444::2
            routes:
              config:
              - destination: ::/0
                next-hop-interface: eno1
                next-hop-address: 1111:2222:3333:4444::1
                table-id: 254
Copy to Clipboard Toggle word wrap

apiVersion: operator.openshift.io/v1
kind: Network
metadata:
  name: cluster
  annotations: {}
spec:
  disableNetworkDiagnostics: true
Copy to Clipboard Toggle word wrap

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    openshift.io/node-selector: ""
    workload.openshift.io/allowed: "management"
  labels:
    openshift.io/cluster-monitoring: "true"
    pod-security.kubernetes.io/enforce: baseline
    pod-security.kubernetes.io/enforce-version: v1.25
    pod-security.kubernetes.io/audit: baseline
    pod-security.kubernetes.io/audit-version: v1.25
    pod-security.kubernetes.io/warn: baseline
    pod-security.kubernetes.io/warn-version: v1.25
  name: "openshift-marketplace"
Copy to Clipboard Toggle word wrap

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
  annotations: {}
data:
  config.yaml: |
    alertmanagerMain:
      enabled: false
    telemeterClient:
      enabled: false
    prometheusK8s:
       retention: 24h
Copy to Clipboard Toggle word wrap

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: default-cat-source
  namespace: openshift-marketplace
  annotations:
    target.workload.openshift.io/management: '{"effect": "PreferredDuringScheduling"}'
spec:
  displayName: default-cat-source
  image: $imageUrl
  publisher: Red Hat
  sourceType: grpc
  updateStrategy:
    registryPoll:
      interval: 1h
status:
  connectionState:
    lastObservedState: READY
Copy to Clipboard Toggle word wrap

apiVersion: v1
kind: ConfigMap
metadata:
  name: collect-profiles-config
  namespace: openshift-operator-lifecycle-manager
  annotations: {}
data:
  pprof-config.yaml: |
    disabled: True
Copy to Clipboard Toggle word wrap

apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
  name: disconnected-internal-icsp
  annotations: {}
spec:
  repositoryDigestMirrors:
    - $mirrors
Copy to Clipboard Toggle word wrap

apiVersion: config.openshift.io/v1
kind: OperatorHub
metadata:
  name: cluster
  annotations: {}
spec:
  disableAllDefaultSources: true
Copy to Clipboard Toggle word wrap

3.2.4.4.3.

apiVersion: machineconfiguration.openshift.io/v1
kind: ContainerRuntimeConfig
metadata:
  name: enable-crun-master
spec:
  machineConfigPoolSelector:
    matchLabels:
      pools.operator.machineconfiguration.openshift.io/master: ""
  containerRuntimeConfig:
    defaultRuntime: crun
Copy to Clipboard Toggle word wrap

apiVersion: machineconfiguration.openshift.io/v1
kind: ContainerRuntimeConfig
metadata:
  name: enable-crun-worker
spec:
  machineConfigPoolSelector:
    matchLabels:
      pools.operator.machineconfiguration.openshift.io/worker: ""
  containerRuntimeConfig:
    defaultRuntime: crun
Copy to Clipboard Toggle word wrap

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: 99-crio-disable-wipe-master
spec:
  config:
    ignition:
      version: 3.2.0
    storage:
      files:
        - contents:
            source: data:text/plain;charset=utf-8;base64,W2NyaW9dCmNsZWFuX3NodXRkb3duX2ZpbGUgPSAiIgo=
          mode: 420
          path: /etc/crio/crio.conf.d/99-crio-disable-wipe.toml
Copy to Clipboard Toggle word wrap

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 99-crio-disable-wipe-worker
spec:
  config:
    ignition:
      version: 3.2.0
    storage:
      files:
        - contents:
            source: data:text/plain;charset=utf-8;base64,W2NyaW9dCmNsZWFuX3NodXRkb3duX2ZpbGUgPSAiIgo=
          mode: 420
          path: /etc/crio/crio.conf.d/99-crio-disable-wipe.toml
Copy to Clipboard Toggle word wrap

apiVersion: config.openshift.io/v1
kind: Node
metadata:
  name: cluster
spec:
  cgroupMode: "v1"
Copy to Clipboard Toggle word wrap

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: 05-kdump-config-master
spec:
  config:
    ignition:
      version: 3.2.0
    systemd:
      units:
        - enabled: true
          name: kdump-remove-ice-module.service
          contents: |
            [Unit]
            Description=Remove ice module when doing kdump
            Before=kdump.service
            [Service]
            Type=oneshot
            RemainAfterExit=true
            ExecStart=/usr/local/bin/kdump-remove-ice-module.sh
            [Install]
            WantedBy=multi-user.target
    storage:
      files:
        - contents:
            source: data:text/plain;charset=utf-8;base64,IyEvdXNyL2Jpbi9lbnYgYmFzaAoKIyBUaGlzIHNjcmlwdCByZW1vdmVzIHRoZSBpY2UgbW9kdWxlIGZyb20ga2R1bXAgdG8gcHJldmVudCBrZHVtcCBmYWlsdXJlcyBvbiBjZXJ0YWluIHNlcnZlcnMuCiMgVGhpcyBpcyBhIHRlbXBvcmFyeSB3b3JrYXJvdW5kIGZvciBSSEVMUExBTi0xMzgyMzYgYW5kIGNhbiBiZSByZW1vdmVkIHdoZW4gdGhhdCBpc3N1ZSBpcwojIGZpeGVkLgoKc2V0IC14CgpTRUQ9Ii91c3IvYmluL3NlZCIKR1JFUD0iL3Vzci9iaW4vZ3JlcCIKCiMgb3ZlcnJpZGUgZm9yIHRlc3RpbmcgcHVycG9zZXMKS0RVTVBfQ09ORj0iJHsxOi0vZXRjL3N5c2NvbmZpZy9rZHVtcH0iClJFTU9WRV9JQ0VfU1RSPSJtb2R1bGVfYmxhY2tsaXN0PWljZSIKCiMgZXhpdCBpZiBmaWxlIGRvZXNuJ3QgZXhpc3QKWyAhIC1mICR7S0RVTVBfQ09ORn0gXSAmJiBleGl0IDAKCiMgZXhpdCBpZiBmaWxlIGFscmVhZHkgdXBkYXRlZAoke0dSRVB9IC1GcSAke1JFTU9WRV9JQ0VfU1RSfSAke0tEVU1QX0NPTkZ9ICYmIGV4aXQgMAoKIyBUYXJnZXQgbGluZSBsb29rcyBzb21ldGhpbmcgbGlrZSB0aGlzOgojIEtEVU1QX0NPTU1BTkRMSU5FX0FQUEVORD0iaXJxcG9sbCBucl9jcHVzPTEgLi4uIGhlc3RfZGlzYWJsZSIKIyBVc2Ugc2VkIHRvIG1hdGNoIGV2ZXJ5dGhpbmcgYmV0d2VlbiB0aGUgcXVvdGVzIGFuZCBhcHBlbmQgdGhlIFJFTU9WRV9JQ0VfU1RSIHRvIGl0CiR7U0VEfSAtaSAncy9eS0RVTVBfQ09NTUFORExJTkVfQVBQRU5EPSJbXiJdKi8mICcke1JFTU9WRV9JQ0VfU1RSfScvJyAke0tEVU1QX0NPTkZ9IHx8IGV4aXQgMAo=
          mode: 448
          path: /usr/local/bin/kdump-remove-ice-module.sh
Copy to Clipboard Toggle word wrap

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 05-kdump-config-worker
spec:
  config:
    ignition:
      version: 3.2.0
    systemd:
      units:
        - enabled: true
          name: kdump-remove-ice-module.service
          contents: |
            [Unit]
            Description=Remove ice module when doing kdump
            Before=kdump.service
            [Service]
            Type=oneshot
            RemainAfterExit=true
            ExecStart=/usr/local/bin/kdump-remove-ice-module.sh
            [Install]
            WantedBy=multi-user.target
    storage:
      files:
        - contents:
            source: data:text/plain;charset=utf-8;base64,IyEvdXNyL2Jpbi9lbnYgYmFzaAoKIyBUaGlzIHNjcmlwdCByZW1vdmVzIHRoZSBpY2UgbW9kdWxlIGZyb20ga2R1bXAgdG8gcHJldmVudCBrZHVtcCBmYWlsdXJlcyBvbiBjZXJ0YWluIHNlcnZlcnMuCiMgVGhpcyBpcyBhIHRlbXBvcmFyeSB3b3JrYXJvdW5kIGZvciBSSEVMUExBTi0xMzgyMzYgYW5kIGNhbiBiZSByZW1vdmVkIHdoZW4gdGhhdCBpc3N1ZSBpcwojIGZpeGVkLgoKc2V0IC14CgpTRUQ9Ii91c3IvYmluL3NlZCIKR1JFUD0iL3Vzci9iaW4vZ3JlcCIKCiMgb3ZlcnJpZGUgZm9yIHRlc3RpbmcgcHVycG9zZXMKS0RVTVBfQ09ORj0iJHsxOi0vZXRjL3N5c2NvbmZpZy9rZHVtcH0iClJFTU9WRV9JQ0VfU1RSPSJtb2R1bGVfYmxhY2tsaXN0PWljZSIKCiMgZXhpdCBpZiBmaWxlIGRvZXNuJ3QgZXhpc3QKWyAhIC1mICR7S0RVTVBfQ09ORn0gXSAmJiBleGl0IDAKCiMgZXhpdCBpZiBmaWxlIGFscmVhZHkgdXBkYXRlZAoke0dSRVB9IC1GcSAke1JFTU9WRV9JQ0VfU1RSfSAke0tEVU1QX0NPTkZ9ICYmIGV4aXQgMAoKIyBUYXJnZXQgbGluZSBsb29rcyBzb21ldGhpbmcgbGlrZSB0aGlzOgojIEtEVU1QX0NPTU1BTkRMSU5FX0FQUEVORD0iaXJxcG9sbCBucl9jcHVzPTEgLi4uIGhlc3RfZGlzYWJsZSIKIyBVc2Ugc2VkIHRvIG1hdGNoIGV2ZXJ5dGhpbmcgYmV0d2VlbiB0aGUgcXVvdGVzIGFuZCBhcHBlbmQgdGhlIFJFTU9WRV9JQ0VfU1RSIHRvIGl0CiR7U0VEfSAtaSAncy9eS0RVTVBfQ09NTUFORExJTkVfQVBQRU5EPSJbXiJdKi8mICcke1JFTU9WRV9JQ0VfU1RSfScvJyAke0tEVU1QX0NPTkZ9IHx8IGV4aXQgMAo=
          mode: 448
          path: /usr/local/bin/kdump-remove-ice-module.sh
Copy to Clipboard Toggle word wrap

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: 06-kdump-enable-master
spec:
  config:
    ignition:
      version: 3.2.0
    systemd:
      units:
        - enabled: true
          name: kdump.service
  kernelArguments:
    - crashkernel=512M
Copy to Clipboard Toggle word wrap

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 06-kdump-enable-worker
spec:
  config:
    ignition:
      version: 3.2.0
    systemd:
      units:
        - enabled: true
          name: kdump.service
  kernelArguments:
    - crashkernel=512M
Copy to Clipboard Toggle word wrap

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: container-mount-namespace-and-kubelet-conf-master
spec:
  config:
    ignition:
      version: 3.2.0
    storage:
      files:
        - contents:
            source: data:text/plain;charset=utf-8;base64,IyEvYmluL2Jhc2gKCmRlYnVnKCkgewogIGVjaG8gJEAgPiYyCn0KCnVzYWdlKCkgewogIGVjaG8gVXNhZ2U6ICQoYmFzZW5hbWUgJDApIFVOSVQgW2VudmZpbGUgW3Zhcm5hbWVdXQogIGVjaG8KICBlY2hvIEV4dHJhY3QgdGhlIGNvbnRlbnRzIG9mIHRoZSBmaXJzdCBFeGVjU3RhcnQgc3RhbnphIGZyb20gdGhlIGdpdmVuIHN5c3RlbWQgdW5pdCBhbmQgcmV0dXJuIGl0IHRvIHN0ZG91dAogIGVjaG8KICBlY2hvICJJZiAnZW52ZmlsZScgaXMgcHJvdmlkZWQsIHB1dCBpdCBpbiB0aGVyZSBpbnN0ZWFkLCBhcyBhbiBlbnZpcm9ubWVudCB2YXJpYWJsZSBuYW1lZCAndmFybmFtZSciCiAgZWNobyAiRGVmYXVsdCAndmFybmFtZScgaXMgRVhFQ1NUQVJUIGlmIG5vdCBzcGVjaWZpZWQiCiAgZXhpdCAxCn0KClVOSVQ9JDEKRU5WRklMRT0kMgpWQVJOQU1FPSQzCmlmIFtbIC16ICRVTklUIHx8ICRVTklUID09ICItLWhlbHAiIHx8ICRVTklUID09ICItaCIgXV07IHRoZW4KICB1c2FnZQpmaQpkZWJ1ZyAiRXh0cmFjdGluZyBFeGVjU3RhcnQgZnJvbSAkVU5JVCIKRklMRT0kKHN5c3RlbWN0bCBjYXQgJFVOSVQgfCBoZWFkIC1uIDEpCkZJTEU9JHtGSUxFI1wjIH0KaWYgW1sgISAtZiAkRklMRSBdXTsgdGhlbgogIGRlYnVnICJGYWlsZWQgdG8gZmluZCByb290IGZpbGUgZm9yIHVuaXQgJFVOSVQgKCRGSUxFKSIKICBleGl0CmZpCmRlYnVnICJTZXJ2aWNlIGRlZmluaXRpb24gaXMgaW4gJEZJTEUiCkVYRUNTVEFSVD0kKHNlZCAtbiAtZSAnL15FeGVjU3RhcnQ9LipcXCQvLC9bXlxcXSQvIHsgcy9eRXhlY1N0YXJ0PS8vOyBwIH0nIC1lICcvXkV4ZWNTdGFydD0uKlteXFxdJC8geyBzL15FeGVjU3RhcnQ9Ly87IHAgfScgJEZJTEUpCgppZiBbWyAkRU5WRklMRSBdXTsgdGhlbgogIFZBUk5BTUU9JHtWQVJOQU1FOi1FWEVDU1RBUlR9CiAgZWNobyAiJHtWQVJOQU1FfT0ke0VYRUNTVEFSVH0iID4gJEVOVkZJTEUKZWxzZQogIGVjaG8gJEVYRUNTVEFSVApmaQo=
          mode: 493
          path: /usr/local/bin/extractExecStart
        - contents:
            source: data:text/plain;charset=utf-8;base64,IyEvYmluL2Jhc2gKbnNlbnRlciAtLW1vdW50PS9ydW4vY29udGFpbmVyLW1vdW50LW5hbWVzcGFjZS9tbnQgIiRAIgo=
          mode: 493
          path: /usr/local/bin/nsenterCmns
    systemd:
      units:
        - contents: |
            [Unit]
            Description=Manages a mount namespace that both kubelet and crio can use to share their container-specific mounts

            [Service]
            Type=oneshot
            RemainAfterExit=yes
            RuntimeDirectory=container-mount-namespace
            Environment=RUNTIME_DIRECTORY=%t/container-mount-namespace
            Environment=BIND_POINT=%t/container-mount-namespace/mnt
            ExecStartPre=bash -c "findmnt ${RUNTIME_DIRECTORY} || mount --make-unbindable --bind ${RUNTIME_DIRECTORY} ${RUNTIME_DIRECTORY}"
            ExecStartPre=touch ${BIND_POINT}
            ExecStart=unshare --mount=${BIND_POINT} --propagation slave mount --make-rshared /
            ExecStop=umount -R ${RUNTIME_DIRECTORY}
          name: container-mount-namespace.service
        - dropins:
            - contents: |
                [Unit]
                Wants=container-mount-namespace.service
                After=container-mount-namespace.service

                [Service]
                ExecStartPre=/usr/local/bin/extractExecStart %n /%t/%N-execstart.env ORIG_EXECSTART
                EnvironmentFile=-/%t/%N-execstart.env
                ExecStart=
                ExecStart=bash -c "nsenter --mount=%t/container-mount-namespace/mnt \
                    ${ORIG_EXECSTART}"
              name: 90-container-mount-namespace.conf
          name: crio.service
        - dropins:
            - contents: |
                [Unit]
                Wants=container-mount-namespace.service
                After=container-mount-namespace.service

                [Service]
                ExecStartPre=/usr/local/bin/extractExecStart %n /%t/%N-execstart.env ORIG_EXECSTART
                EnvironmentFile=-/%t/%N-execstart.env
                ExecStart=
                ExecStart=bash -c "nsenter --mount=%t/container-mount-namespace/mnt \
                    ${ORIG_EXECSTART} --housekeeping-interval=30s"
              name: 90-container-mount-namespace.conf
            - contents: |
                [Service]
                Environment="OPENSHIFT_MAX_HOUSEKEEPING_INTERVAL_DURATION=60s"
                Environment="OPENSHIFT_EVICTION_MONITORING_PERIOD_DURATION=30s"
              name: 30-kubelet-interval-tuning.conf
          name: kubelet.service
Copy to Clipboard Toggle word wrap

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: container-mount-namespace-and-kubelet-conf-worker
spec:
  config:
    ignition:
      version: 3.2.0
    storage:
      files:
        - contents:
            source: data:text/plain;charset=utf-8;base64,IyEvYmluL2Jhc2gKCmRlYnVnKCkgewogIGVjaG8gJEAgPiYyCn0KCnVzYWdlKCkgewogIGVjaG8gVXNhZ2U6ICQoYmFzZW5hbWUgJDApIFVOSVQgW2VudmZpbGUgW3Zhcm5hbWVdXQogIGVjaG8KICBlY2hvIEV4dHJhY3QgdGhlIGNvbnRlbnRzIG9mIHRoZSBmaXJzdCBFeGVjU3RhcnQgc3RhbnphIGZyb20gdGhlIGdpdmVuIHN5c3RlbWQgdW5pdCBhbmQgcmV0dXJuIGl0IHRvIHN0ZG91dAogIGVjaG8KICBlY2hvICJJZiAnZW52ZmlsZScgaXMgcHJvdmlkZWQsIHB1dCBpdCBpbiB0aGVyZSBpbnN0ZWFkLCBhcyBhbiBlbnZpcm9ubWVudCB2YXJpYWJsZSBuYW1lZCAndmFybmFtZSciCiAgZWNobyAiRGVmYXVsdCAndmFybmFtZScgaXMgRVhFQ1NUQVJUIGlmIG5vdCBzcGVjaWZpZWQiCiAgZXhpdCAxCn0KClVOSVQ9JDEKRU5WRklMRT0kMgpWQVJOQU1FPSQzCmlmIFtbIC16ICRVTklUIHx8ICRVTklUID09ICItLWhlbHAiIHx8ICRVTklUID09ICItaCIgXV07IHRoZW4KICB1c2FnZQpmaQpkZWJ1ZyAiRXh0cmFjdGluZyBFeGVjU3RhcnQgZnJvbSAkVU5JVCIKRklMRT0kKHN5c3RlbWN0bCBjYXQgJFVOSVQgfCBoZWFkIC1uIDEpCkZJTEU9JHtGSUxFI1wjIH0KaWYgW1sgISAtZiAkRklMRSBdXTsgdGhlbgogIGRlYnVnICJGYWlsZWQgdG8gZmluZCByb290IGZpbGUgZm9yIHVuaXQgJFVOSVQgKCRGSUxFKSIKICBleGl0CmZpCmRlYnVnICJTZXJ2aWNlIGRlZmluaXRpb24gaXMgaW4gJEZJTEUiCkVYRUNTVEFSVD0kKHNlZCAtbiAtZSAnL15FeGVjU3RhcnQ9LipcXCQvLC9bXlxcXSQvIHsgcy9eRXhlY1N0YXJ0PS8vOyBwIH0nIC1lICcvXkV4ZWNTdGFydD0uKlteXFxdJC8geyBzL15FeGVjU3RhcnQ9Ly87IHAgfScgJEZJTEUpCgppZiBbWyAkRU5WRklMRSBdXTsgdGhlbgogIFZBUk5BTUU9JHtWQVJOQU1FOi1FWEVDU1RBUlR9CiAgZWNobyAiJHtWQVJOQU1FfT0ke0VYRUNTVEFSVH0iID4gJEVOVkZJTEUKZWxzZQogIGVjaG8gJEVYRUNTVEFSVApmaQo=
          mode: 493
          path: /usr/local/bin/extractExecStart
        - contents:
            source: data:text/plain;charset=utf-8;base64,IyEvYmluL2Jhc2gKbnNlbnRlciAtLW1vdW50PS9ydW4vY29udGFpbmVyLW1vdW50LW5hbWVzcGFjZS9tbnQgIiRAIgo=
          mode: 493
          path: /usr/local/bin/nsenterCmns
    systemd:
      units:
        - contents: |
            [Unit]
            Description=Manages a mount namespace that both kubelet and crio can use to share their container-specific mounts

            [Service]
            Type=oneshot
            RemainAfterExit=yes
            RuntimeDirectory=container-mount-namespace
            Environment=RUNTIME_DIRECTORY=%t/container-mount-namespace
            Environment=BIND_POINT=%t/container-mount-namespace/mnt
            ExecStartPre=bash -c "findmnt ${RUNTIME_DIRECTORY} || mount --make-unbindable --bind ${RUNTIME_DIRECTORY} ${RUNTIME_DIRECTORY}"
            ExecStartPre=touch ${BIND_POINT}
            ExecStart=unshare --mount=${BIND_POINT} --propagation slave mount --make-rshared /
            ExecStop=umount -R ${RUNTIME_DIRECTORY}
          name: container-mount-namespace.service
        - dropins:
            - contents: |
                [Unit]
                Wants=container-mount-namespace.service
                After=container-mount-namespace.service

                [Service]
                ExecStartPre=/usr/local/bin/extractExecStart %n /%t/%N-execstart.env ORIG_EXECSTART
                EnvironmentFile=-/%t/%N-execstart.env
                ExecStart=
                ExecStart=bash -c "nsenter --mount=%t/container-mount-namespace/mnt \
                    ${ORIG_EXECSTART}"
              name: 90-container-mount-namespace.conf
          name: crio.service
        - dropins:
            - contents: |
                [Unit]
                Wants=container-mount-namespace.service
                After=container-mount-namespace.service

                [Service]
                ExecStartPre=/usr/local/bin/extractExecStart %n /%t/%N-execstart.env ORIG_EXECSTART
                EnvironmentFile=-/%t/%N-execstart.env
                ExecStart=
                ExecStart=bash -c "nsenter --mount=%t/container-mount-namespace/mnt \
                    ${ORIG_EXECSTART} --housekeeping-interval=30s"
              name: 90-container-mount-namespace.conf
            - contents: |
                [Service]
                Environment="OPENSHIFT_MAX_HOUSEKEEPING_INTERVAL_DURATION=60s"
                Environment="OPENSHIFT_EVICTION_MONITORING_PERIOD_DURATION=30s"
              name: 30-kubelet-interval-tuning.conf
          name: kubelet.service
Copy to Clipboard Toggle word wrap

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: 99-sync-time-once-master
spec:
  config:
    ignition:
      version: 3.2.0
    systemd:
      units:
        - contents: |
            [Unit]
            Description=Sync time once
            After=network.service
            [Service]
            Type=oneshot
            TimeoutStartSec=300
            ExecCondition=/bin/bash -c 'systemctl is-enabled chronyd.service --quiet && exit 1 || exit 0'
            ExecStart=/usr/sbin/chronyd -n -f /etc/chrony.conf -q
            RemainAfterExit=yes
            [Install]
            WantedBy=multi-user.target
          enabled: true
          name: sync-time-once.service
Copy to Clipboard Toggle word wrap

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 99-sync-time-once-worker
spec:
  config:
    ignition:
      version: 3.2.0
    systemd:
      units:
        - contents: |
            [Unit]
            Description=Sync time once
            After=network.service
            [Service]
            Type=oneshot
            TimeoutStartSec=300
            ExecCondition=/bin/bash -c 'systemctl is-enabled chronyd.service --quiet && exit 1 || exit 0'
            ExecStart=/usr/sbin/chronyd -n -f /etc/chrony.conf -q
            RemainAfterExit=yes
            [Install]
            WantedBy=multi-user.target
          enabled: true
          name: sync-time-once.service
Copy to Clipboard Toggle word wrap

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: load-sctp-module-master
spec:
  config:
    ignition:
      version: 2.2.0
    storage:
      files:
        - contents:
            source: data:,
            verification: {}
          filesystem: root
          mode: 420
          path: /etc/modprobe.d/sctp-blacklist.conf
        - contents:
            source: data:text/plain;charset=utf-8,sctp
          filesystem: root
          mode: 420
          path: /etc/modules-load.d/sctp-load.conf
Copy to Clipboard Toggle word wrap

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: load-sctp-module-worker
spec:
  config:
    ignition:
      version: 2.2.0
    storage:
      files:
        - contents:
            source: data:,
            verification: {}
          filesystem: root
          mode: 420
          path: /etc/modprobe.d/sctp-blacklist.conf
        - contents:
            source: data:text/plain;charset=utf-8,sctp
          filesystem: root
          mode: 420
          path: /etc/modules-load.d/sctp-load.conf
Copy to Clipboard Toggle word wrap

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: 08-set-rcu-normal-master
spec:
  config:
    ignition:
      version: 3.2.0
    storage:
      files:
        - contents:
            source: data:text/plain;charset=utf-8;base64,IyEvYmluL2Jhc2gKIwojIERpc2FibGUgcmN1X2V4cGVkaXRlZCBhZnRlciBub2RlIGhhcyBmaW5pc2hlZCBib290aW5nCiMKIyBUaGUgZGVmYXVsdHMgYmVsb3cgY2FuIGJlIG92ZXJyaWRkZW4gdmlhIGVudmlyb25tZW50IHZhcmlhYmxlcwojCgojIERlZmF1bHQgd2FpdCB0aW1lIGlzIDYwMHMgPSAxMG06Ck1BWElNVU1fV0FJVF9USU1FPSR7TUFYSU1VTV9XQUlUX1RJTUU6LTYwMH0KCiMgRGVmYXVsdCBzdGVhZHktc3RhdGUgdGhyZXNob2xkID0gMiUKIyBBbGxvd2VkIHZhbHVlczoKIyAgNCAgLSBhYnNvbHV0ZSBwb2QgY291bnQgKCsvLSkKIyAgNCUgLSBwZXJjZW50IGNoYW5nZSAoKy8tKQojICAtMSAtIGRpc2FibGUgdGhlIHN0ZWFkeS1zdGF0ZSBjaGVjawpTVEVBRFlfU1RBVEVfVEhSRVNIT0xEPSR7U1RFQURZX1NUQVRFX1RIUkVTSE9MRDotMiV9CgojIERlZmF1bHQgc3RlYWR5LXN0YXRlIHdpbmRvdyA9IDYwcwojIElmIHRoZSBydW5uaW5nIHBvZCBjb3VudCBzdGF5cyB3aXRoaW4gdGhlIGdpdmVuIHRocmVzaG9sZCBmb3IgdGhpcyB0aW1lCiMgcGVyaW9kLCByZXR1cm4gQ1BVIHV0aWxpemF0aW9uIHRvIG5vcm1hbCBiZWZvcmUgdGhlIG1heGltdW0gd2FpdCB0aW1lIGhhcwojIGV4cGlyZXMKU1RFQURZX1NUQVRFX1dJTkRPVz0ke1NURUFEWV9TVEFURV9XSU5ET1c6LTYwfQoKIyBEZWZhdWx0IHN0ZWFkeS1zdGF0ZSBhbGxvd3MgYW55IHBvZCBjb3VudCB0byBiZSAic3RlYWR5IHN0YXRlIgojIEluY3JlYXNpbmcgdGhpcyB3aWxsIHNraXAgYW55IHN0ZWFkeS1zdGF0ZSBjaGVja3MgdW50aWwgdGhlIGNvdW50IHJpc2VzIGFib3ZlCiMgdGhpcyBudW1iZXIgdG8gYXZvaWQgZmFsc2UgcG9zaXRpdmVzIGlmIHRoZXJlIGFyZSBzb21lIHBlcmlvZHMgd2hlcmUgdGhlCiMgY291bnQgZG9lc24ndCBpbmNyZWFzZSBidXQgd2Uga25vdyB3ZSBjYW4ndCBiZSBhdCBzdGVhZHktc3RhdGUgeWV0LgpTVEVBRFlfU1RBVEVfTUlOSU1VTT0ke1NURUFEWV9TVEFURV9NSU5JTVVNOi0wfQoKIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwoKd2l0aGluKCkgewogIGxvY2FsIGxhc3Q9JDEgY3VycmVudD0kMiB0aHJlc2hvbGQ9JDMKICBsb2NhbCBkZWx0YT0wIHBjaGFuZ2UKICBkZWx0YT0kKCggY3VycmVudCAtIGxhc3QgKSkKICBpZiBbWyAkY3VycmVudCAtZXEgJGxhc3QgXV07IHRoZW4KICAgIHBjaGFuZ2U9MAogIGVsaWYgW1sgJGxhc3QgLWVxIDAgXV07IHRoZW4KICAgIHBjaGFuZ2U9MTAwMDAwMAogIGVsc2UKICAgIHBjaGFuZ2U9JCgoICggIiRkZWx0YSIgKiAxMDApIC8gbGFzdCApKQogIGZpCiAgZWNobyAtbiAibGFzdDokbGFzdCBjdXJyZW50OiRjdXJyZW50IGRlbHRhOiRkZWx0YSBwY2hhbmdlOiR7cGNoYW5nZX0lOiAiCiAgbG9jYWwgYWJzb2x1dGUgbGltaXQKICBjYXNlICR0aHJlc2hvbGQgaW4KICAgIColKQogICAgICBhYnNvbHV0ZT0ke3BjaGFuZ2UjIy19ICMgYWJzb2x1dGUgdmFsdWUKICAgICAgbGltaXQ9JHt0aHJlc2hvbGQlJSV9CiAgICAgIDs7CiAgICAqKQogICAgICBhYnNvbHV0ZT0ke2RlbHRhIyMtfSAjIGFic29sdXRlIHZhbHVlCiAgICAgIGxpbWl0PSR0aHJlc2hvbGQKICAgICAgOzsKICBlc2FjCiAgaWYgW1sgJGFic29sdXRlIC1sZSAkbGltaXQgXV07IHRoZW4KICAgIGVjaG8gIndpdGhpbiAoKy8tKSR0aHJlc2hvbGQiCiAgICByZXR1cm4gMAogIGVsc2UKICAgIGVjaG8gIm91dHNpZGUgKCsvLSkkdGhyZXNob2xkIgogICAgcmV0dXJuIDEKICBmaQp9CgpzdGVhZHlzdGF0ZSgpIHsKICBsb2NhbCBsYXN0PSQxIGN1cnJlbnQ9JDIKICBpZiBbWyAkbGFzdCAtbHQgJFNURUFEWV9TVEFURV9NSU5JTVVNIF1dOyB0aGVuCiAgICBlY2hvICJsYXN0OiRsYXN0IGN1cnJlbnQ6JGN1cnJlbnQgV2FpdGluZyB0byByZWFjaCAkU1RFQURZX1NUQVRFX01JTklNVU0gYmVmb3JlIGNoZWNraW5nIGZvciBzdGVhZHktc3RhdGUiCiAgICByZXR1cm4gMQogIGZpCiAgd2l0aGluICIkbGFzdCIgIiRjdXJyZW50IiAiJFNURUFEWV9TVEFURV9USFJFU0hPTEQiCn0KCndhaXRGb3JSZWFkeSgpIHsKICBsb2dnZXIgIlJlY292ZXJ5OiBXYWl0aW5nICR7TUFYSU1VTV9XQUlUX1RJTUV9cyBmb3IgdGhlIGluaXRpYWxpemF0aW9uIHRvIGNvbXBsZXRlIgogIGxvY2FsIHQ9MCBzPTEwCiAgbG9jYWwgbGFzdENjb3VudD0wIGNjb3VudD0wIHN0ZWFkeVN0YXRlVGltZT0wCiAgd2hpbGUgW1sgJHQgLWx0ICRNQVhJTVVNX1dBSVRfVElNRSBdXTsgZG8KICAgIHNsZWVwICRzCiAgICAoKHQgKz0gcykpCiAgICAjIERldGVjdCBzdGVhZHktc3RhdGUgcG9kIGNvdW50CiAgICBjY291bnQ9JChjcmljdGwgcHMgMj4vZGV2L251bGwgfCB3YyAtbCkKICAgIGlmIFtbICRjY291bnQgLWd0IDAgXV0gJiYgc3RlYWR5c3RhdGUgIiRsYXN0Q2NvdW50IiAiJGNjb3VudCI7IHRoZW4KICAgICAgKChzdGVhZHlTdGF0ZVRpbWUgKz0gcykpCiAgICAgIGVjaG8gIlN0ZWFkeS1zdGF0ZSBmb3IgJHtzdGVhZHlTdGF0ZVRpbWV9cy8ke1NURUFEWV9TVEFURV9XSU5ET1d9cyIKICAgICAgaWYgW1sgJHN0ZWFkeVN0YXRlVGltZSAtZ2UgJFNURUFEWV9TVEFURV9XSU5ET1cgXV07IHRoZW4KICAgICAgICBsb2dnZXIgIlJlY292ZXJ5OiBTdGVhZHktc3RhdGUgKCsvLSAkU1RFQURZX1NUQVRFX1RIUkVTSE9MRCkgZm9yICR7U1RFQURZX1NUQVRFX1dJTkRPV31zOiBEb25lIgogICAgICAgIHJldHVybiAwCiAgICAgIGZpCiAgICBlbHNlCiAgICAgIGlmIFtbICRzdGVhZHlTdGF0ZVRpbWUgLWd0IDAgXV07IHRoZW4KICAgICAgICBlY2hvICJSZXNldHRpbmcgc3RlYWR5LXN0YXRlIHRpbWVyIgogICAgICAgIHN0ZWFkeVN0YXRlVGltZT0wCiAgICAgIGZpCiAgICBmaQogICAgbGFzdENjb3VudD0kY2NvdW50CiAgZG9uZQogIGxvZ2dlciAiUmVjb3Zlcnk6IFJlY292ZXJ5IENvbXBsZXRlIFRpbWVvdXQiCn0KCnNldFJjdU5vcm1hbCgpIHsKICBlY2hvICJTZXR0aW5nIHJjdV9ub3JtYWwgdG8gMSIKICBlY2hvIDEgPiAvc3lzL2tlcm5lbC9yY3Vfbm9ybWFsCn0KCm1haW4oKSB7CiAgd2FpdEZvclJlYWR5CiAgZWNobyAiV2FpdGluZyBmb3Igc3RlYWR5IHN0YXRlIHRvb2s6ICQoYXdrICd7cHJpbnQgaW50KCQxLzM2MDApImgiLCBpbnQoKCQxJTM2MDApLzYwKSJtIiwgaW50KCQxJTYwKSJzIn0nIC9wcm9jL3VwdGltZSkiCiAgc2V0UmN1Tm9ybWFsCn0KCmlmIFtbICIke0JBU0hfU09VUkNFWzBdfSIgPSAiJHswfSIgXV07IHRoZW4KICBtYWluICIke0B9IgogIGV4aXQgJD8KZmkK
          mode: 493
          path: /usr/local/bin/set-rcu-normal.sh
    systemd:
      units:
        - contents: |
            [Unit]
            Description=Disable rcu_expedited after node has finished booting by setting rcu_normal to 1

            [Service]
            Type=simple
            ExecStart=/usr/local/bin/set-rcu-normal.sh

            # Maximum wait time is 600s = 10m:
            Environment=MAXIMUM_WAIT_TIME=600

            # Steady-state threshold = 2%
            # Allowed values:
            #  4  - absolute pod count (+/-)
            #  4% - percent change (+/-)
            #  -1 - disable the steady-state check
            # Note: '%' must be escaped as '%%' in systemd unit files
            Environment=STEADY_STATE_THRESHOLD=2%%

            # Steady-state window = 120s
            # If the running pod count stays within the given threshold for this time
            # period, return CPU utilization to normal before the maximum wait time has
            # expires
            Environment=STEADY_STATE_WINDOW=120

            # Steady-state minimum = 40
            # Increasing this will skip any steady-state checks until the count rises above
            # this number to avoid false positives if there are some periods where the
            # count doesn't increase but we know we can't be at steady-state yet.
            Environment=STEADY_STATE_MINIMUM=40

            [Install]
            WantedBy=multi-user.target
          enabled: true
          name: set-rcu-normal.service
Copy to Clipboard Toggle word wrap

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 08-set-rcu-normal-worker
spec:
  config:
    ignition:
      version: 3.2.0
    storage:
      files:
        - contents:
            source: data:text/plain;charset=utf-8;base64,IyEvYmluL2Jhc2gKIwojIERpc2FibGUgcmN1X2V4cGVkaXRlZCBhZnRlciBub2RlIGhhcyBmaW5pc2hlZCBib290aW5nCiMKIyBUaGUgZGVmYXVsdHMgYmVsb3cgY2FuIGJlIG92ZXJyaWRkZW4gdmlhIGVudmlyb25tZW50IHZhcmlhYmxlcwojCgojIERlZmF1bHQgd2FpdCB0aW1lIGlzIDYwMHMgPSAxMG06Ck1BWElNVU1fV0FJVF9USU1FPSR7TUFYSU1VTV9XQUlUX1RJTUU6LTYwMH0KCiMgRGVmYXVsdCBzdGVhZHktc3RhdGUgdGhyZXNob2xkID0gMiUKIyBBbGxvd2VkIHZhbHVlczoKIyAgNCAgLSBhYnNvbHV0ZSBwb2QgY291bnQgKCsvLSkKIyAgNCUgLSBwZXJjZW50IGNoYW5nZSAoKy8tKQojICAtMSAtIGRpc2FibGUgdGhlIHN0ZWFkeS1zdGF0ZSBjaGVjawpTVEVBRFlfU1RBVEVfVEhSRVNIT0xEPSR7U1RFQURZX1NUQVRFX1RIUkVTSE9MRDotMiV9CgojIERlZmF1bHQgc3RlYWR5LXN0YXRlIHdpbmRvdyA9IDYwcwojIElmIHRoZSBydW5uaW5nIHBvZCBjb3VudCBzdGF5cyB3aXRoaW4gdGhlIGdpdmVuIHRocmVzaG9sZCBmb3IgdGhpcyB0aW1lCiMgcGVyaW9kLCByZXR1cm4gQ1BVIHV0aWxpemF0aW9uIHRvIG5vcm1hbCBiZWZvcmUgdGhlIG1heGltdW0gd2FpdCB0aW1lIGhhcwojIGV4cGlyZXMKU1RFQURZX1NUQVRFX1dJTkRPVz0ke1NURUFEWV9TVEFURV9XSU5ET1c6LTYwfQoKIyBEZWZhdWx0IHN0ZWFkeS1zdGF0ZSBhbGxvd3MgYW55IHBvZCBjb3VudCB0byBiZSAic3RlYWR5IHN0YXRlIgojIEluY3JlYXNpbmcgdGhpcyB3aWxsIHNraXAgYW55IHN0ZWFkeS1zdGF0ZSBjaGVja3MgdW50aWwgdGhlIGNvdW50IHJpc2VzIGFib3ZlCiMgdGhpcyBudW1iZXIgdG8gYXZvaWQgZmFsc2UgcG9zaXRpdmVzIGlmIHRoZXJlIGFyZSBzb21lIHBlcmlvZHMgd2hlcmUgdGhlCiMgY291bnQgZG9lc24ndCBpbmNyZWFzZSBidXQgd2Uga25vdyB3ZSBjYW4ndCBiZSBhdCBzdGVhZHktc3RhdGUgeWV0LgpTVEVBRFlfU1RBVEVfTUlOSU1VTT0ke1NURUFEWV9TVEFURV9NSU5JTVVNOi0wfQoKIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwoKd2l0aGluKCkgewogIGxvY2FsIGxhc3Q9JDEgY3VycmVudD0kMiB0aHJlc2hvbGQ9JDMKICBsb2NhbCBkZWx0YT0wIHBjaGFuZ2UKICBkZWx0YT0kKCggY3VycmVudCAtIGxhc3QgKSkKICBpZiBbWyAkY3VycmVudCAtZXEgJGxhc3QgXV07IHRoZW4KICAgIHBjaGFuZ2U9MAogIGVsaWYgW1sgJGxhc3QgLWVxIDAgXV07IHRoZW4KICAgIHBjaGFuZ2U9MTAwMDAwMAogIGVsc2UKICAgIHBjaGFuZ2U9JCgoICggIiRkZWx0YSIgKiAxMDApIC8gbGFzdCApKQogIGZpCiAgZWNobyAtbiAibGFzdDokbGFzdCBjdXJyZW50OiRjdXJyZW50IGRlbHRhOiRkZWx0YSBwY2hhbmdlOiR7cGNoYW5nZX0lOiAiCiAgbG9jYWwgYWJzb2x1dGUgbGltaXQKICBjYXNlICR0aHJlc2hvbGQgaW4KICAgIColKQogICAgICBhYnNvbHV0ZT0ke3BjaGFuZ2UjIy19ICMgYWJzb2x1dGUgdmFsdWUKICAgICAgbGltaXQ9JHt0aHJlc2hvbGQlJSV9CiAgICAgIDs7CiAgICAqKQogICAgICBhYnNvbHV0ZT0ke2RlbHRhIyMtfSAjIGFic29sdXRlIHZhbHVlCiAgICAgIGxpbWl0PSR0aHJlc2hvbGQKICAgICAgOzsKICBlc2FjCiAgaWYgW1sgJGFic29sdXRlIC1sZSAkbGltaXQgXV07IHRoZW4KICAgIGVjaG8gIndpdGhpbiAoKy8tKSR0aHJlc2hvbGQiCiAgICByZXR1cm4gMAogIGVsc2UKICAgIGVjaG8gIm91dHNpZGUgKCsvLSkkdGhyZXNob2xkIgogICAgcmV0dXJuIDEKICBmaQp9CgpzdGVhZHlzdGF0ZSgpIHsKICBsb2NhbCBsYXN0PSQxIGN1cnJlbnQ9JDIKICBpZiBbWyAkbGFzdCAtbHQgJFNURUFEWV9TVEFURV9NSU5JTVVNIF1dOyB0aGVuCiAgICBlY2hvICJsYXN0OiRsYXN0IGN1cnJlbnQ6JGN1cnJlbnQgV2FpdGluZyB0byByZWFjaCAkU1RFQURZX1NUQVRFX01JTklNVU0gYmVmb3JlIGNoZWNraW5nIGZvciBzdGVhZHktc3RhdGUiCiAgICByZXR1cm4gMQogIGZpCiAgd2l0aGluICIkbGFzdCIgIiRjdXJyZW50IiAiJFNURUFEWV9TVEFURV9USFJFU0hPTEQiCn0KCndhaXRGb3JSZWFkeSgpIHsKICBsb2dnZXIgIlJlY292ZXJ5OiBXYWl0aW5nICR7TUFYSU1VTV9XQUlUX1RJTUV9cyBmb3IgdGhlIGluaXRpYWxpemF0aW9uIHRvIGNvbXBsZXRlIgogIGxvY2FsIHQ9MCBzPTEwCiAgbG9jYWwgbGFzdENjb3VudD0wIGNjb3VudD0wIHN0ZWFkeVN0YXRlVGltZT0wCiAgd2hpbGUgW1sgJHQgLWx0ICRNQVhJTVVNX1dBSVRfVElNRSBdXTsgZG8KICAgIHNsZWVwICRzCiAgICAoKHQgKz0gcykpCiAgICAjIERldGVjdCBzdGVhZHktc3RhdGUgcG9kIGNvdW50CiAgICBjY291bnQ9JChjcmljdGwgcHMgMj4vZGV2L251bGwgfCB3YyAtbCkKICAgIGlmIFtbICRjY291bnQgLWd0IDAgXV0gJiYgc3RlYWR5c3RhdGUgIiRsYXN0Q2NvdW50IiAiJGNjb3VudCI7IHRoZW4KICAgICAgKChzdGVhZHlTdGF0ZVRpbWUgKz0gcykpCiAgICAgIGVjaG8gIlN0ZWFkeS1zdGF0ZSBmb3IgJHtzdGVhZHlTdGF0ZVRpbWV9cy8ke1NURUFEWV9TVEFURV9XSU5ET1d9cyIKICAgICAgaWYgW1sgJHN0ZWFkeVN0YXRlVGltZSAtZ2UgJFNURUFEWV9TVEFURV9XSU5ET1cgXV07IHRoZW4KICAgICAgICBsb2dnZXIgIlJlY292ZXJ5OiBTdGVhZHktc3RhdGUgKCsvLSAkU1RFQURZX1NUQVRFX1RIUkVTSE9MRCkgZm9yICR7U1RFQURZX1NUQVRFX1dJTkRPV31zOiBEb25lIgogICAgICAgIHJldHVybiAwCiAgICAgIGZpCiAgICBlbHNlCiAgICAgIGlmIFtbICRzdGVhZHlTdGF0ZVRpbWUgLWd0IDAgXV07IHRoZW4KICAgICAgICBlY2hvICJSZXNldHRpbmcgc3RlYWR5LXN0YXRlIHRpbWVyIgogICAgICAgIHN0ZWFkeVN0YXRlVGltZT0wCiAgICAgIGZpCiAgICBmaQogICAgbGFzdENjb3VudD0kY2NvdW50CiAgZG9uZQogIGxvZ2dlciAiUmVjb3Zlcnk6IFJlY292ZXJ5IENvbXBsZXRlIFRpbWVvdXQiCn0KCnNldFJjdU5vcm1hbCgpIHsKICBlY2hvICJTZXR0aW5nIHJjdV9ub3JtYWwgdG8gMSIKICBlY2hvIDEgPiAvc3lzL2tlcm5lbC9yY3Vfbm9ybWFsCn0KCm1haW4oKSB7CiAgd2FpdEZvclJlYWR5CiAgZWNobyAiV2FpdGluZyBmb3Igc3RlYWR5IHN0YXRlIHRvb2s6ICQoYXdrICd7cHJpbnQgaW50KCQxLzM2MDApImgiLCBpbnQoKCQxJTM2MDApLzYwKSJtIiwgaW50KCQxJTYwKSJzIn0nIC9wcm9jL3VwdGltZSkiCiAgc2V0UmN1Tm9ybWFsCn0KCmlmIFtbICIke0JBU0hfU09VUkNFWzBdfSIgPSAiJHswfSIgXV07IHRoZW4KICBtYWluICIke0B9IgogIGV4aXQgJD8KZmkK
          mode: 493
          path: /usr/local/bin/set-rcu-normal.sh
    systemd:
      units:
        - contents: |
            [Unit]
            Description=Disable rcu_expedited after node has finished booting by setting rcu_normal to 1

            [Service]
            Type=simple
            ExecStart=/usr/local/bin/set-rcu-normal.sh

            # Maximum wait time is 600s = 10m:
            Environment=MAXIMUM_WAIT_TIME=600

            # Steady-state threshold = 2%
            # Allowed values:
            #  4  - absolute pod count (+/-)
            #  4% - percent change (+/-)
            #  -1 - disable the steady-state check
            # Note: '%' must be escaped as '%%' in systemd unit files
            Environment=STEADY_STATE_THRESHOLD=2%%

            # Steady-state window = 120s
            # If the running pod count stays within the given threshold for this time
            # period, return CPU utilization to normal before the maximum wait time has
            # expires
            Environment=STEADY_STATE_WINDOW=120

            # Steady-state minimum = 40
            # Increasing this will skip any steady-state checks until the count rises above
            # this number to avoid false positives if there are some periods where the
            # count doesn't increase but we know we can't be at steady-state yet.
            Environment=STEADY_STATE_MINIMUM=40

            [Install]
            WantedBy=multi-user.target
          enabled: true
          name: set-rcu-normal.service
Copy to Clipboard Toggle word wrap

3.2.5.

3.2.5.1.

Expand
표 3.7.
  

Expand
표 3.8.
  

3.3.

3.3.1.

3.3.1.1.

Expand
표 3.9.
  

3.3.2.

3.3.2.1.

3.3.2.1.1.

3.3.2.1.2.

참고

3.3.3.

3.3.3.1.
  • cpu-load-balancing.crio.io: "disable"
    cpu-quota.crio.io: "disable"
    irq-load-balancing.crio.io: "disable"
    Copy to Clipboard Toggle word wrap
  • cpu-c-states.crio.io: "disable"
    cpu-freq-governor.crio.io: "performance"
    Copy to Clipboard Toggle word wrap
3.3.3.2.
3.3.3.3.

3.3.3.3.1.

3.3.3.3.2.

3.3.3.3.3.
3.3.3.3.4.
3.3.3.4.
3.3.3.5.

3.3.3.6.

참고

3.3.3.6.1.
3.3.3.6.2.

3.3.3.7.

3.3.3.8.
3.3.3.9.
,

3.3.3.10.

3.3.3.11.

3.3.3.12.
3.3.3.12.1.

3.3.3.12.2.

참고

3.3.4.

3.3.4.1.
Expand
표 3.10.
    

3.3.4.2.
Expand
표 3.11.
    

3.3.4.3.
Expand
표 3.12.
    

3.3.4.4.
Expand
표 3.13.
    

3.3.4.5.
Expand
표 3.14.
    

3.3.4.6.
3.3.4.6.1.

# optional
# count: 1
apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
  name: autosizing-master
spec:
  autoSizingReserved: true
  machineConfigPoolSelector:
    matchLabels:
      pools.operator.machineconfiguration.openshift.io/master: ""
Copy to Clipboard Toggle word wrap

# optional
# count: 1
apiVersion: machineconfiguration.openshift.io/v1
kind: ContainerRuntimeConfig
metadata:
  name: 99-change-pidslimit-custom
spec:
  machineConfigPoolSelector:
    matchLabels:
      # Set to appropriate MCP
      pools.operator.machineconfiguration.openshift.io/master: ""
  containerRuntimeConfig:
    pidsLimit: $pidsLimit
    # Example:
    #pidsLimit: 4096
Copy to Clipboard Toggle word wrap

3.3.4.6.2.

# required
# count: 1
---
apiVersion: v1
kind: Secret
metadata:
  name: rook-ceph-external-cluster-details
  namespace: openshift-storage
type: Opaque
data:
  # encoded content has been made generic
  external_cluster_details: eyJuYW1lIjoicm9vay1jZXBoLW1vbi1lbmRwb2ludHMiLCJraW5kIjoiQ29uZmlnTWFwIiwiZGF0YSI6eyJkYXRhIjoiY2VwaHVzYTE9MS4yLjMuNDo2Nzg5IiwibWF4TW9uSWQiOiIwIiwibWFwcGluZyI6Int9In19LHsibmFtZSI6InJvb2stY2VwaC1tb24iLCJraW5kIjoiU2VjcmV0IiwiZGF0YSI6eyJhZG1pbi1zZWNyZXQiOiJhZG1pbi1zZWNyZXQiLCJmc2lkIjoiMTExMTExMTEtMTExMS0xMTExLTExMTEtMTExMTExMTExMTExIiwibW9uLXNlY3JldCI6Im1vbi1zZWNyZXQifX0seyJuYW1lIjoicm9vay1jZXBoLW9wZXJhdG9yLWNyZWRzIiwia2luZCI6IlNlY3JldCIsImRhdGEiOnsidXNlcklEIjoiY2xpZW50LmhlYWx0aGNoZWNrZXIiLCJ1c2VyS2V5IjoiYzJWamNtVjAifX0seyJuYW1lIjoibW9uaXRvcmluZy1lbmRwb2ludCIsImtpbmQiOiJDZXBoQ2x1c3RlciIsImRhdGEiOnsiTW9uaXRvcmluZ0VuZHBvaW50IjoiMS4yLjMuNCwxLjIuMy4zLDEuMi4zLjIiLCJNb25pdG9yaW5nUG9ydCI6IjkyODMifX0seyJuYW1lIjoiY2VwaC1yYmQiLCJraW5kIjoiU3RvcmFnZUNsYXNzIiwiZGF0YSI6eyJwb29sIjoib2RmX3Bvb2wifX0seyJuYW1lIjoicm9vay1jc2ktcmJkLW5vZGUiLCJraW5kIjoiU2VjcmV0IiwiZGF0YSI6eyJ1c2VySUQiOiJjc2ktcmJkLW5vZGUiLCJ1c2VyS2V5IjoiIn19LHsibmFtZSI6InJvb2stY3NpLXJiZC1wcm92aXNpb25lciIsImtpbmQiOiJTZWNyZXQiLCJkYXRhIjp7InVzZXJJRCI6ImNzaS1yYmQtcHJvdmlzaW9uZXIiLCJ1c2VyS2V5IjoiYzJWamNtVjAifX0seyJuYW1lIjoicm9vay1jc2ktY2VwaGZzLXByb3Zpc2lvbmVyIiwia2luZCI6IlNlY3JldCIsImRhdGEiOnsiYWRtaW5JRCI6ImNzaS1jZXBoZnMtcHJvdmlzaW9uZXIiLCJhZG1pbktleSI6IiJ9fSx7Im5hbWUiOiJyb29rLWNzaS1jZXBoZnMtbm9kZSIsImtpbmQiOiJTZWNyZXQiLCJkYXRhIjp7ImFkbWluSUQiOiJjc2ktY2VwaGZzLW5vZGUiLCJhZG1pbktleSI6ImMyVmpjbVYwIn19LHsibmFtZSI6ImNlcGhmcyIsImtpbmQiOiJTdG9yYWdlQ2xhc3MiLCJkYXRhIjp7ImZzTmFtZSI6ImNlcGhmcyIsInBvb2wiOiJtYW5pbGFfZGF0YSJ9fQ==
Copy to Clipboard Toggle word wrap

# required
# count: 1
---
apiVersion: ocs.openshift.io/v1
kind: StorageCluster
metadata:
  name: ocs-external-storagecluster
  namespace: openshift-storage
spec:
  externalStorage:
    enable: true
  labelSelector: {}
Copy to Clipboard Toggle word wrap

# required: yes
# count: 1
---
apiVersion: v1
kind: Namespace
metadata:
  name: openshift-storage
  annotations:
    workload.openshift.io/allowed: management
  labels:
    openshift.io/cluster-monitoring: "true"
Copy to Clipboard Toggle word wrap

# required: yes
# count: 1
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: openshift-storage-operatorgroup
  namespace: openshift-storage
spec:
  targetNamespaces:
    - openshift-storage
Copy to Clipboard Toggle word wrap

3.3.4.6.3.

# required
# count: 1
apiVersion: operator.openshift.io/v1
kind: Network
metadata:
  name: cluster
spec:
  defaultNetwork:
    ovnKubernetesConfig:
      gatewayConfig:
        routingViaHost: true
  # additional networks are optional and may alternatively be specified using NetworkAttachmentDefinition CRs
  additionalNetworks: [$additionalNetworks]
  # eg
  #- name: add-net-1
  #  namespace: app-ns-1
  #  rawCNIConfig: '{ "cniVersion": "0.3.1", "name": "add-net-1", "plugins": [{"type": "macvlan", "master": "bond1", "ipam": {}}] }'
  #  type: Raw
  #- name: add-net-2
  #  namespace: app-ns-1
  #  rawCNIConfig: '{ "cniVersion": "0.4.0", "name": "add-net-2", "plugins": [ {"type": "macvlan", "master": "bond1", "mode": "private" },{ "type": "tuning", "name": "tuning-arp" }] }'
  #  type: Raw

  # Enable to use MultiNetworkPolicy CRs
  useMultiNetworkPolicy: true
Copy to Clipboard Toggle word wrap

# optional
# copies: 0-N
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: $name
  namespace: $ns
spec:
  nodeSelector:
    kubernetes.io/hostname: $nodeName
  config: $config
  #eg
  #config: '{
  #  "cniVersion": "0.3.1",
  #  "name": "external-169",
  #  "type": "vlan",
  #  "master": "ens8f0",
  #  "mode": "bridge",
  #  "vlanid": 169,
  #  "ipam": {
  #    "type": "static",
  #  }
  #}'
Copy to Clipboard Toggle word wrap

# required
# count: 1-N
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: $name # eg addresspool3
  namespace: metallb-system
  annotations:
    metallb.universe.tf/address-pool: $name # eg addresspool3
spec:
  ##############
  # Expected variation in this configuration
  addresses: [$pools]
  #- 3.3.3.0/24
  autoAssign: true
  ##############
Copy to Clipboard Toggle word wrap

# required
# count: 1-N
apiVersion: metallb.io/v1beta1
kind: BFDProfile
metadata:
  name: bfdprofile
  namespace: metallb-system
spec:
  ################
  # These values may vary. Recommended values are included as default
  receiveInterval: 150 # default 300ms
  transmitInterval: 150 # default 300ms
  #echoInterval: 300 # default 50ms
  detectMultiplier: 10 # default 3
  echoMode: true
  passiveMode: true
  minimumTtl: 5 # default 254
  #
  ################
Copy to Clipboard Toggle word wrap

# required
# count: 1-N
apiVersion: metallb.io/v1beta1
kind: BGPAdvertisement
metadata:
  name: $name # eg bgpadvertisement-1
  namespace: metallb-system
spec:
  ipAddressPools: [$pool]
  # eg:

  #  - addresspool3
  peers: [$peers]
  # eg:

  #    - peer-one
  communities: [$communities]
  # Note correlation with address pool.
  # eg:

  #    - 65535:65282
  aggregationLength: 32
  aggregationLengthV6: 128
  localPref: 100
Copy to Clipboard Toggle word wrap

# required
# count: 1-N
apiVersion: metallb.io/v1beta1
kind: BGPPeer
metadata:
  name: $name
  namespace: metallb-system
spec:
  peerAddress: $ip # eg 192.168.1.2
  peerASN: $peerasn # eg 64501
  myASN: $myasn # eg 64500
  routerID: $id # eg 10.10.10.10
  bfdProfile: bfdprofile
Copy to Clipboard Toggle word wrap

# required
# count: 1
apiVersion: metallb.io/v1beta1
kind: MetalLB
metadata:
  name: metallb
  namespace: metallb-system
spec:
  nodeSelector:
    node-role.kubernetes.io/worker: ""
Copy to Clipboard Toggle word wrap

# required: yes
# count: 1
---
apiVersion: v1
kind: Namespace
metadata:
  name: metallb-system
  annotations:
    workload.openshift.io/allowed: management
  labels:
    openshift.io/cluster-monitoring: "true"
Copy to Clipboard Toggle word wrap

# required: yes
# count: 1
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: metallb-operator
  namespace: metallb-system
Copy to Clipboard Toggle word wrap

# required: yes
# count: 1
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: metallb-operator-sub
  namespace: metallb-system
spec:
  channel: stable
  name: metallb-operator
  source: redhat-operators-disconnected
  sourceNamespace: openshift-marketplace
  installPlanApproval: Automatic
Copy to Clipboard Toggle word wrap

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 99-worker-setsebool
spec:
  config:
    ignition:
      version: 3.2.0
    systemd:
      units:
        - contents: |
            [Unit]
            Description=Set SELinux boolean for tap cni plugin
            Before=kubelet.service

            [Service]
            Type=oneshot
            ExecStart=/sbin/setsebool container_use_devices=on
            RemainAfterExit=true

            [Install]
            WantedBy=multi-user.target graphical.target
          enabled: true
          name: setsebool.service
Copy to Clipboard Toggle word wrap

apiVersion: nmstate.io/v1
kind: NMState
metadata:
  name: nmstate
spec: {}
Copy to Clipboard Toggle word wrap

apiVersion: v1
kind: Namespace
metadata:
  name: openshift-nmstate
  annotations:
    workload.openshift.io/allowed: management
Copy to Clipboard Toggle word wrap

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: openshift-nmstate
  namespace: openshift-nmstate
spec:
  targetNamespaces:
    - openshift-nmstate
Copy to Clipboard Toggle word wrap

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: kubernetes-nmstate-operator
  namespace: openshift-nmstate
spec:
  channel: "stable"
  name: kubernetes-nmstate-operator
  source: redhat-operators-disconnected
  sourceNamespace: openshift-marketplace
  installPlanApproval: Automatic
status:
  state: AtLatestKnown
Copy to Clipboard Toggle word wrap

# optional (though expected for all)
# count: 0-N
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetwork
metadata:
  name: $name # eg sriov-network-abcd
  namespace: openshift-sriov-network-operator
spec:
  capabilities: "$capabilities" # eg '{"mac": true, "ips": true}'
  ipam: "$ipam" # eg '{ "type": "host-local", "subnet": "10.3.38.0/24" }'
  networkNamespace: $nns # eg cni-test
  resourceName: $resource # eg resourceTest
Copy to Clipboard Toggle word wrap

# optional (though expected in all deployments)
# count: 0-N
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
  name: $name
  namespace: openshift-sriov-network-operator
spec: {} # $spec
# eg
#deviceType: netdevice
#nicSelector:
#  deviceID: "1593"
#  pfNames:
#  - ens8f0np0#0-9
#  rootDevices:
#  - 0000:d8:00.0
#  vendor: "8086"
#nodeSelector:
#  kubernetes.io/hostname: host.sample.lab
#numVfs: 20
#priority: 99
#excludeTopology: true
#resourceName: resourceNameABCD
Copy to Clipboard Toggle word wrap

# required
# count: 1
---
apiVersion: sriovnetwork.openshift.io/v1
kind: SriovOperatorConfig
metadata:
  name: default
  namespace: openshift-sriov-network-operator
spec:
  configDaemonNodeSelector:
    node-role.kubernetes.io/worker: ""
  enableInjector: true
  enableOperatorWebhook: true
Copy to Clipboard Toggle word wrap

# required: yes
# count: 1
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: sriov-network-operator-subscription
  namespace: openshift-sriov-network-operator
spec:
  channel: "stable"
  name: sriov-network-operator
  source: redhat-operators-disconnected
  sourceNamespace: openshift-marketplace
  installPlanApproval: Automatic
Copy to Clipboard Toggle word wrap

# required: yes
# count: 1
apiVersion: v1
kind: Namespace
metadata:
  name: openshift-sriov-network-operator
  annotations:
    workload.openshift.io/allowed: management
Copy to Clipboard Toggle word wrap

# required: yes
# count: 1
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: sriov-network-operators
  namespace: openshift-sriov-network-operator
spec:
  targetNamespaces:
    - openshift-sriov-network-operator
Copy to Clipboard Toggle word wrap

3.3.4.6.4.

# Optional
# count: 1
apiVersion: nodetopology.openshift.io/v1
kind: NUMAResourcesOperator
metadata:
  name: numaresourcesoperator
spec:
  nodeGroups:
    - config:
        # Periodic is the default setting
        infoRefreshMode: Periodic
      machineConfigPoolSelector:
        matchLabels:
          # This label must match the pool(s) you want to run NUMA-aligned workloads
          pools.operator.machineconfiguration.openshift.io/worker: ""
Copy to Clipboard Toggle word wrap

# Optional
# count: 1
apiVersion: nodetopology.openshift.io/v1
kind: NUMAResourcesScheduler
metadata:
  name: numaresourcesscheduler
spec:
  #cacheResyncPeriod: "0"
  # Image spec should be the latest for the release
  imageSpec: "registry.redhat.io/openshift4/noderesourcetopology-scheduler-rhel9:v4.14.0"
  #logLevel: "Trace"
  schedulerName: topo-aware-scheduler
Copy to Clipboard Toggle word wrap

3.3.4.6.5.

# optional
# count: 1
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: 40-load-kernel-modules-control-plane
spec:
  config:
    # Release info found in https://github.com/coreos/butane/releases
    ignition:
      version: 3.2.0
    storage:
      files:
        - contents:
            source: data:,
          mode: 420
          overwrite: true
          path: /etc/modprobe.d/kernel-blacklist.conf
        - contents:
            source: data:text/plain;charset=utf-8;base64,aXBfZ3JlCmlwNl90YWJsZXMKaXA2dF9SRUpFQ1QKaXA2dGFibGVfZmlsdGVyCmlwNnRhYmxlX21hbmdsZQppcHRhYmxlX2ZpbHRlcgppcHRhYmxlX21hbmdsZQppcHRhYmxlX25hdAp4dF9tdWx0aXBvcnQKeHRfb3duZXIKeHRfUkVESVJFQ1QKeHRfc3RhdGlzdGljCnh0X1RDUE1TUwp4dF91MzI=
          mode: 420
          overwrite: true
          path: /etc/modules-load.d/kernel-load.conf
Copy to Clipboard Toggle word wrap

# optional
# count: 1
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: load-sctp-module
spec:
  config:
    ignition:
      version: 2.2.0
    storage:
      files:
        - contents:
            source: data:,
            verification: {}
          filesystem: root
          mode: 420
          path: /etc/modprobe.d/sctp-blacklist.conf
        - contents:
            source: data:text/plain;charset=utf-8;base64,c2N0cA==
          filesystem: root
          mode: 420
          path: /etc/modules-load.d/sctp-load.conf
Copy to Clipboard Toggle word wrap

# optional
# count: 1
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 40-load-kernel-modules-worker
spec:
  config:
    # Release info found in https://github.com/coreos/butane/releases
    ignition:
      version: 3.2.0
    storage:
      files:
        - contents:
            source: data:,
          mode: 420
          overwrite: true
          path: /etc/modprobe.d/kernel-blacklist.conf
        - contents:
            source: data:text/plain;charset=utf-8;base64,aXBfZ3JlCmlwNl90YWJsZXMKaXA2dF9SRUpFQ1QKaXA2dGFibGVfZmlsdGVyCmlwNnRhYmxlX21hbmdsZQppcHRhYmxlX2ZpbHRlcgppcHRhYmxlX21hbmdsZQppcHRhYmxlX25hdAp4dF9tdWx0aXBvcnQKeHRfb3duZXIKeHRfUkVESVJFQ1QKeHRfc3RhdGlzdGljCnh0X1RDUE1TUwp4dF91MzI=
          mode: 420
          overwrite: true
          path: /etc/modules-load.d/kernel-load.conf
Copy to Clipboard Toggle word wrap

# required
# count: 1
apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  outputs:
    - type: "kafka"
      name: kafka-open
      url: tcp://10.11.12.13:9092/test
  pipelines:
    - inputRefs:
        - infrastructure
        #- application
        - audit
      labels:
        label1: test1
        label2: test2
        label3: test3
        label4: test4
        label5: test5
      name: all-to-default
      outputRefs:
        - kafka-open
Copy to Clipboard Toggle word wrap

# required
# count: 1
apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
  name: instance
  namespace: openshift-logging
spec:
  collection:
    type: vector
  managementState: Managed
Copy to Clipboard Toggle word wrap

---
apiVersion: v1
kind: Namespace
metadata:
  name: openshift-logging
  annotations:
    workload.openshift.io/allowed: management
Copy to Clipboard Toggle word wrap

---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: cluster-logging
  namespace: openshift-logging
spec:
  targetNamespaces:
    - openshift-logging
Copy to Clipboard Toggle word wrap

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: cluster-logging
  namespace: openshift-logging
spec:
  channel: "stable"
  name: cluster-logging
  source: redhat-operators-disconnected
  sourceNamespace: openshift-marketplace
  installPlanApproval: Automatic
Copy to Clipboard Toggle word wrap

# required
# count: 1..N
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: redhat-operators-disconnected
  namespace: openshift-marketplace
spec:
  displayName: Red Hat Disconnected Operators Catalog
  image: $imageUrl
  publisher: Red Hat
  sourceType: grpc
#  updateStrategy:
#    registryPoll:
#      interval: 1h
#status:
#    connectionState:
#        lastObservedState: READY
Copy to Clipboard Toggle word wrap

# required
# count: 1
apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
  name: disconnected-internal-icsp
spec:
  repositoryDigestMirrors: []
#    - $mirrors
Copy to Clipboard Toggle word wrap

# required
# count: 1
apiVersion: config.openshift.io/v1
kind: OperatorHub
metadata:
  name: cluster
spec:
  disableAllDefaultSources: true
Copy to Clipboard Toggle word wrap

# optional
# count: 1
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    k8sPrometheusAdapter:
      dedicatedServiceMonitors:
        enabled: true
    prometheusK8s:
      retention: 15d
      volumeClaimTemplate:
        spec:
          storageClassName: ocs-external-storagecluster-ceph-rbd
          resources:
            requests:
              storage: 100Gi
    alertmanagerMain:
      volumeClaimTemplate:
        spec:
          storageClassName: ocs-external-storagecluster-ceph-rbd
          resources:
            requests:
              storage: 20Gi
Copy to Clipboard Toggle word wrap

# required
# count: 1
apiVersion: performance.openshift.io/v2
kind: PerformanceProfile
metadata:
  name: $name
  annotations:
    # Some pods want the kernel stack to ignore IPv6 router Advertisement.
    kubeletconfig.experimental: |
      {"allowedUnsafeSysctls":["net.ipv6.conf.all.accept_ra"]}
spec:
  cpu:
    # node0 CPUs: 0-17,36-53
    # node1 CPUs: 18-34,54-71
    # siblings: (0,36), (1,37)...
    # we want to reserve the first Core of each NUMA socket
    #
    # no CPU left behind! all-cpus == isolated + reserved
    isolated: $isolated # eg 1-17,19-35,37-53,55-71
    reserved: $reserved # eg 0,18,36,54
  # Guaranteed QoS pods will disable IRQ balancing for cores allocated to the pod.
  # default value of globallyDisableIrqLoadBalancing is false
  globallyDisableIrqLoadBalancing: false
  hugepages:
    defaultHugepagesSize: 1G
    pages:
      # 32GB per numa node
      - count: $count # eg 64
        size: 1G
  machineConfigPoolSelector:
    # For SNO: machineconfiguration.openshift.io/role: 'master'
    pools.operator.machineconfiguration.openshift.io/worker: ''
  nodeSelector:
    # For SNO: node-role.kubernetes.io/master: ""
    node-role.kubernetes.io/worker: ""
  workloadHints:
    realTime: false
    highPowerConsumption: false
    perPodPowerManagement: true
  realTimeKernel:
    enabled: false
  numa:
    # All guaranteed QoS containers get resources from a single NUMA node
    topologyPolicy: "single-numa-node"
  net:
    userLevelNetworking: false
Copy to Clipboard Toggle word wrap

4장.

주의

4.1.

참고

Expand
  

4.1.1.

4.2.

4.2.1.

Expand
        

4.2.2.

Expand
      

4.2.3.

Expand
      

4.3.

중요

required pods per cluster / pods per node = total number of nodes needed
Copy to Clipboard Toggle word wrap

2200 / 500 = 4.4
Copy to Clipboard Toggle word wrap

2200 / 20 = 110
Copy to Clipboard Toggle word wrap

required pods per cluster / total number of nodes = expected pods per node
Copy to Clipboard Toggle word wrap

4.4.

Expand
     

Expand
    

---
apiVersion: template.openshift.io/v1
kind: Template
metadata:
  name: deployment-config-template
  creationTimestamp:
  annotations:
    description: This template will create a deploymentConfig with 1 replica, 4 env vars and a service.
    tags: ''
objects:
- apiVersion: apps.openshift.io/v1
  kind: DeploymentConfig
  metadata:
    name: deploymentconfig${IDENTIFIER}
  spec:
    template:
      metadata:
        labels:
          name: replicationcontroller${IDENTIFIER}
      spec:
        enableServiceLinks: false
        containers:
        - name: pause${IDENTIFIER}
          image: "${IMAGE}"
          ports:
          - containerPort: 8080
            protocol: TCP
          env:
          - name: ENVVAR1_${IDENTIFIER}
            value: "${ENV_VALUE}"
          - name: ENVVAR2_${IDENTIFIER}
            value: "${ENV_VALUE}"
          - name: ENVVAR3_${IDENTIFIER}
            value: "${ENV_VALUE}"
          - name: ENVVAR4_${IDENTIFIER}
            value: "${ENV_VALUE}"
          resources: {}
          imagePullPolicy: IfNotPresent
          capabilities: {}
          securityContext:
            capabilities: {}
            privileged: false
        restartPolicy: Always
        serviceAccount: ''
    replicas: 1
    selector:
      name: replicationcontroller${IDENTIFIER}
    triggers:
    - type: ConfigChange
    strategy:
      type: Rolling
- apiVersion: v1
  kind: Service
  metadata:
    name: service${IDENTIFIER}
  spec:
    selector:
      name: replicationcontroller${IDENTIFIER}
    ports:
    - name: serviceport${IDENTIFIER}
      protocol: TCP
      port: 80
      targetPort: 8080
    clusterIP: ''
    type: ClusterIP
    sessionAffinity: None
  status:
    loadBalancer: {}
parameters:
- name: IDENTIFIER
  description: Number to append to the name of resources
  value: '1'
  required: true
- name: IMAGE
  description: Image to use for deploymentConfig
  value: gcr.io/google-containers/pause-amd64:3.0
  required: false
- name: ENV_VALUE
  description: Value to use for environment variables
  generate: expression
  from: "[A-Za-z0-9]{255}"
  required: false
labels:
  template: deployment-config-template
Copy to Clipboard Toggle word wrap

5장.

중요

5.1.

참고

Expand
표 5.1.
  

Expand
표 5.2.
  

Expand
표 5.3.
  

$ oc create quota <name> --hard=count/<resource>.<group>=<quota> 
1
Copy to Clipboard Toggle word wrap
1

5.1.1.

  1. $ oc describe node ip-172-31-27-209.us-west-2.compute.internal | egrep 'Capacity|Allocatable|gpu'
    Copy to Clipboard Toggle word wrap

                        openshift.com/gpu-accelerator=true
    Capacity:
     nvidia.com/gpu:  2
    Allocatable:
     nvidia.com/gpu:  2
     nvidia.com/gpu:  0           0
    Copy to Clipboard Toggle word wrap

  2. $ cat gpu-quota.yaml
    Copy to Clipboard Toggle word wrap

    apiVersion: v1
    kind: ResourceQuota
    metadata:
      name: gpu-quota
      namespace: nvidia
    spec:
      hard:
        requests.nvidia.com/gpu: 1
    Copy to Clipboard Toggle word wrap

  3. $ oc create -f gpu-quota.yaml
    Copy to Clipboard Toggle word wrap

    resourcequota/gpu-quota created
    Copy to Clipboard Toggle word wrap

  4. $ oc describe quota gpu-quota -n nvidia
    Copy to Clipboard Toggle word wrap

    Name:                    gpu-quota
    Namespace:               nvidia
    Resource                 Used  Hard
    --------                 ----  ----
    requests.nvidia.com/gpu  0     1
    Copy to Clipboard Toggle word wrap

  5. $ oc create pod gpu-pod.yaml
    Copy to Clipboard Toggle word wrap

    apiVersion: v1
    kind: Pod
    metadata:
      generateName: gpu-pod-s46h7
      namespace: nvidia
    spec:
      restartPolicy: OnFailure
      containers:
      - name: rhel7-gpu-pod
        image: rhel7
        env:
          - name: NVIDIA_VISIBLE_DEVICES
            value: all
          - name: NVIDIA_DRIVER_CAPABILITIES
            value: "compute,utility"
          - name: NVIDIA_REQUIRE_CUDA
            value: "cuda>=5.0"
    
        command: ["sleep"]
        args: ["infinity"]
    
        resources:
          limits:
            nvidia.com/gpu: 1
    Copy to Clipboard Toggle word wrap

  6. $ oc get pods
    Copy to Clipboard Toggle word wrap

    NAME              READY     STATUS      RESTARTS   AGE
    gpu-pod-s46h7     1/1       Running     0          1m
    Copy to Clipboard Toggle word wrap

  7. $ oc describe quota gpu-quota -n nvidia
    Copy to Clipboard Toggle word wrap

    Name:                    gpu-quota
    Namespace:               nvidia
    Resource                 Used  Hard
    --------                 ----  ----
    requests.nvidia.com/gpu  1     1
    Copy to Clipboard Toggle word wrap

  8. $ oc create -f gpu-pod.yaml
    Copy to Clipboard Toggle word wrap

    Error from server (Forbidden): error when creating "gpu-pod.yaml": pods "gpu-pod-f7z2w" is forbidden: exceeded quota: gpu-quota, requested: requests.nvidia.com/gpu=1, used: requests.nvidia.com/gpu=1, limited: requests.nvidia.com/gpu=1
    Copy to Clipboard Toggle word wrap

5.1.2.

Expand
  

참고

5.2.

5.2.1.

5.2.2.

5.2.3.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: core-object-counts
spec:
  hard:
    configmaps: "10" 
1

    persistentvolumeclaims: "4" 
2

    replicationcontrollers: "20" 
3

    secrets: "10" 
4

    services: "10" 
5
Copy to Clipboard Toggle word wrap

1
2
3
4
5

apiVersion: v1
kind: ResourceQuota
metadata:
  name: openshift-object-counts
spec:
  hard:
    openshift.io/imagestreams: "10" 
1
Copy to Clipboard Toggle word wrap

1

apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-resources
spec:
  hard:
    pods: "4" 
1

    requests.cpu: "1" 
2

    requests.memory: 1Gi 
3

    requests.ephemeral-storage: 2Gi 
4

    limits.cpu: "2" 
5

    limits.memory: 2Gi 
6

    limits.ephemeral-storage: 4Gi 
7
Copy to Clipboard Toggle word wrap

1
2
3
4
5
6
7

apiVersion: v1
kind: ResourceQuota
metadata:
  name: besteffort
spec:
  hard:
    pods: "1" 
1

  scopes:
  - BestEffort 
2
Copy to Clipboard Toggle word wrap

1
2

apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-resources-long-running
spec:
  hard:
    pods: "4" 
1

    limits.cpu: "4" 
2

    limits.memory: "2Gi" 
3

    limits.ephemeral-storage: "4Gi" 
4

  scopes:
  - NotTerminating 
5
Copy to Clipboard Toggle word wrap

1
2
3
4
5

apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-resources-time-bound
spec:
  hard:
    pods: "2" 
1

    limits.cpu: "1" 
2

    limits.memory: "1Gi" 
3

    limits.ephemeral-storage: "1Gi" 
4

  scopes:
  - Terminating 
5
Copy to Clipboard Toggle word wrap

1
2
3
4
5

apiVersion: v1
kind: ResourceQuota
metadata:
  name: storage-consumption
spec:
  hard:
    persistentvolumeclaims: "10" 
1

    requests.storage: "50Gi" 
2

    gold.storageclass.storage.k8s.io/requests.storage: "10Gi" 
3

    silver.storageclass.storage.k8s.io/requests.storage: "20Gi" 
4

    silver.storageclass.storage.k8s.io/persistentvolumeclaims: "5" 
5

    bronze.storageclass.storage.k8s.io/requests.storage: "0" 
6

    bronze.storageclass.storage.k8s.io/persistentvolumeclaims: "0" 
7
Copy to Clipboard Toggle word wrap

1
2
3
4
5
6
7

5.2.4.

$ oc create -f <resource_quota_definition> [-n <project_name>]
Copy to Clipboard Toggle word wrap

$ oc create -f core-object-counts.yaml -n demoproject
Copy to Clipboard Toggle word wrap

5.2.5.

$ oc create quota <name> --hard=count/<resource>.<group>=<quota>,count/<resource>.<group>=<quota>
Copy to Clipboard Toggle word wrap

$ oc create quota test --hard=count/deployments.extensions=2,count/replicasets.extensions=4,count/pods=3,count/secrets=4
resourcequota "test" created

$ oc describe quota test
Name:                         test
Namespace:                    quota
Resource                      Used  Hard
--------                      ----  ----
count/deployments.extensions  0     2
count/pods                    0     3
count/replicasets.extensions  0     4
count/secrets                 0     4
Copy to Clipboard Toggle word wrap

5.2.6.

  1. $ oc get quota -n demoproject
    NAME                AGE
    besteffort          11m
    compute-resources   2m
    core-object-counts  29m
    Copy to Clipboard Toggle word wrap
  2. $ oc describe quota core-object-counts -n demoproject
    Name:			core-object-counts
    Namespace:		demoproject
    Resource		Used	Hard
    --------		----	----
    configmaps		3	10
    persistentvolumeclaims	0	4
    replicationcontrollers	3	20
    secrets			9	10
    services		2	10
    Copy to Clipboard Toggle word wrap

5.2.7.

kubernetesMasterConfig:
  apiLevels:
  - v1beta3
  - v1
  apiServerArguments: null
  controllerArguments:
    resource-quota-sync-period:
      - "10s"
Copy to Clipboard Toggle word wrap

$ master-restart api
$ master-restart controllers
Copy to Clipboard Toggle word wrap

참고

5.2.8.

admissionConfig:
  pluginConfig:
    ResourceQuota:
      configuration:
        apiVersion: resourcequota.admission.k8s.io/v1alpha1
        kind: Configuration
        limitedResources:
        - resource: persistentvolumeclaims 
1

        matchContains:
        - gold.storageclass.storage.k8s.io/requests.storage 
2
Copy to Clipboard Toggle word wrap
1
2

5.3.

apiVersion: "v1"
kind: "LimitRange"
metadata:
  name: "core-resource-limits" 
1

spec:
  limits:
    - type: "Pod"
      max:
        cpu: "2" 
2

        memory: "1Gi" 
3

      min:
        cpu: "200m" 
4

        memory: "6Mi" 
5

    - type: "Container"
      max:
        cpu: "2" 
6

        memory: "1Gi" 
7

      min:
        cpu: "100m" 
8

        memory: "4Mi" 
9

      default:
        cpu: "300m" 
10

        memory: "200Mi" 
11

      defaultRequest:
        cpu: "200m" 
12

        memory: "100Mi" 
13

      maxLimitRequestRatio:
        cpu: "10" 
14
Copy to Clipboard Toggle word wrap

1
2
3
4
5
6
7
8
9
10
11
12
13
14

apiVersion: "v1"
kind: "LimitRange"
metadata:
  name: "openshift-resource-limits"
spec:
  limits:
    - type: openshift.io/Image
      max:
        storage: 1Gi 
1

    - type: openshift.io/ImageStream
      max:
        openshift.io/image-tags: 20 
2

        openshift.io/images: 30 
3

    - type: "Pod"
      max:
        cpu: "2" 
4

        memory: "1Gi" 
5

        ephemeral-storage: "1Gi" 
6

      min:
        cpu: "1" 
7

        memory: "1Gi" 
8
Copy to Clipboard Toggle word wrap

1
2
3
4
5
6
7
8

5.3.1.

Expand
  

5.3.2.

Expand
표 5.4.
  

5.3.3.

Expand
표 5.5.
  

참고

5.3.4.

Expand
표 5.6.
  

5.3.5.

5.3.6.

Expand
표 5.7.
  

{
  "apiVersion": "v1",
  "kind": "LimitRange",
  "metadata": {
    "name": "pvcs" 
1

  },
  "spec": {
    "limits": [{
        "type": "PersistentVolumeClaim",
        "min": {
          "storage": "2Gi" 
2

        },
        "max": {
          "storage": "50Gi" 
3

        }
      }
    ]
  }
}
Copy to Clipboard Toggle word wrap

1
2
3

5.4.

5.4.1.

  1. $ oc create -f <limit_range_file> -n <project>
    Copy to Clipboard Toggle word wrap

5.4.2.

  1. $ oc get limits -n demoproject
    Copy to Clipboard Toggle word wrap

    NAME              AGE
    resource-limits   6d
    Copy to Clipboard Toggle word wrap

  2. $ oc describe limits resource-limits -n demoproject
    Copy to Clipboard Toggle word wrap

    Name:                           resource-limits
    Namespace:                      demoproject
    Type                            Resource                Min     Max     Default Request Default Limit   Max Limit/Request Ratio
    ----                            --------                ---     ---     --------------- -------------   -----------------------
    Pod                             cpu                     200m    2       -               -               -
    Pod                             memory                  6Mi     1Gi     -               -               -
    Container                       cpu                     100m    2       200m            300m            10
    Container                       memory                  4Mi     1Gi     100Mi           200Mi           -
    openshift.io/Image              storage                 -       1Gi     -               -               -
    openshift.io/ImageStream        openshift.io/image      -       12      -               -               -
    openshift.io/ImageStream        openshift.io/image-tags -       10      -               -               -
    Copy to Clipboard Toggle word wrap

5.4.3.

$ oc delete limits <limit_name>
Copy to Clipboard Toggle word wrap

7장.

7.1.

참고

참고

7.2.

  • oc get tuned.tuned.openshift.io/default -o yaml -n openshift-cluster-node-tuning-operator
    Copy to Clipboard Toggle word wrap

주의

7.3.

apiVersion: tuned.openshift.io/v1
kind: Tuned
metadata:
  name: default
  namespace: openshift-cluster-node-tuning-operator
spec:
  profile:
  - data: |
      [main]
      summary=Optimize systems running OpenShift (provider specific parent profile)
      include=-provider-${f:exec:cat:/var/lib/tuned/provider},openshift
    name: openshift
  recommend:
  - profile: openshift-control-plane
    priority: 30
    match:
    - label: node-role.kubernetes.io/master
    - label: node-role.kubernetes.io/infra
  - profile: openshift-node
    priority: 40
Copy to Clipboard Toggle word wrap

$ oc exec $tuned_pod -n openshift-cluster-node-tuning-operator -- find /usr/lib/tuned/openshift{,-control-plane,-node} -name tuned.conf -exec grep -H ^ {} \;
Copy to Clipboard Toggle word wrap

7.4.

$ oc get profile.tuned.openshift.io -n openshift-cluster-node-tuning-operator
Copy to Clipboard Toggle word wrap

NAME             TUNED                     APPLIED   DEGRADED   AGE
master-0         openshift-control-plane   True      False      6h33m
master-1         openshift-control-plane   True      False      6h33m
master-2         openshift-control-plane   True      False      6h33m
worker-a         openshift-node            True      False      6h28m
worker-b         openshift-node            True      False      6h28m
Copy to Clipboard Toggle word wrap

$ oc get co/node-tuning -n openshift-cluster-node-tuning-operator
Copy to Clipboard Toggle word wrap

NAME          VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
node-tuning   4.15.1    True        False         True       60m     1/5 Profiles with bootcmdline conflict
Copy to Clipboard Toggle word wrap

7.5.

profile:
- name: tuned_profile_1
  data: |
    # TuneD profile specification
    [main]
    summary=Description of tuned_profile_1 profile

    [sysctl]
    net.ipv4.ip_forward=1
    # ... other sysctl's or other TuneD daemon plugins supported by the containerized TuneD

# ...

- name: tuned_profile_n
  data: |
    # TuneD profile specification
    [main]
    summary=Description of tuned_profile_n profile

    # tuned_profile_n profile settings
Copy to Clipboard Toggle word wrap

recommend:
<recommend-item-1>
# ...
<recommend-item-n>
Copy to Clipboard Toggle word wrap

- machineConfigLabels: 
1

    <mcLabels> 
2

  match: 
3

    <match> 
4

  priority: <priority> 
5

  profile: <tuned_profile_name> 
6

  operand: 
7

    debug: <bool> 
8

    tunedConfig:
      reapply_sysctl: <bool> 
9
Copy to Clipboard Toggle word wrap
1
2
3
4
5
6
7
8
9

- label: <label_name> 
1

  value: <label_value> 
2

  type: <label_type> 
3

    <match> 
4
Copy to Clipboard Toggle word wrap
1
2
3
4

중요

- match:
  - label: tuned.openshift.io/elasticsearch
    match:
    - label: node-role.kubernetes.io/master
    - label: node-role.kubernetes.io/infra
    type: pod
  priority: 10
  profile: openshift-control-plane-es
- match:
  - label: node-role.kubernetes.io/master
  - label: node-role.kubernetes.io/infra
  priority: 20
  profile: openshift-control-plane
- priority: 30
  profile: openshift-node
Copy to Clipboard Toggle word wrap

apiVersion: tuned.openshift.io/v1
kind: Tuned
metadata:
  name: openshift-node-custom
  namespace: openshift-cluster-node-tuning-operator
spec:
  profile:
  - data: |
      [main]
      summary=Custom OpenShift node profile with an additional kernel parameter
      include=openshift-node
      [bootloader]
      cmdline_openshift_node_custom=+skew_tick=1
    name: openshift-node-custom

  recommend:
  - machineConfigLabels:
      machineconfiguration.openshift.io/role: "worker-custom"
    priority: 20
    profile: openshift-node-custom
Copy to Clipboard Toggle word wrap

apiVersion: tuned.openshift.io/v1
kind: Tuned
metadata:
  name: provider-gce
  namespace: openshift-cluster-node-tuning-operator
spec:
  profile:
  - data: |
      [main]
      summary=GCE Cloud provider-specific profile
      # Your tuning for GCE Cloud provider goes here.
    name: provider-gce
Copy to Clipboard Toggle word wrap

참고

7.6.

apiVersion: tuned.openshift.io/v1
kind: Tuned
metadata:
  name: ingress
  namespace: openshift-cluster-node-tuning-operator
spec:
  profile:
  - data: |
      [main]
      summary=A custom OpenShift ingress profile
      include=openshift-control-plane
      [sysctl]
      net.ipv4.ip_local_port_range="1024 65535"
      net.ipv4.tcp_tw_reuse=1
    name: openshift-ingress
  recommend:
  - match:
    - label: tuned.openshift.io/ingress-node-label
    priority: 10
    profile: openshift-ingress
Copy to Clipboard Toggle word wrap

중요

$ oc exec $tuned_pod -n openshift-cluster-node-tuning-operator -- find /usr/lib/tuned/ -name tuned.conf -printf '%h\n' | sed 's|^.*/||'
Copy to Clipboard Toggle word wrap

apiVersion: tuned.openshift.io/v1
kind: Tuned
metadata:
  name: openshift-node-hpc-compute
  namespace: openshift-cluster-node-tuning-operator
spec:
  profile:
  - data: |
      [main]
      summary=Custom OpenShift node profile for HPC compute workloads
      include=openshift-node,hpc-compute
    name: openshift-node-hpc-compute

  recommend:
  - match:
    - label: tuned.openshift.io/openshift-node-hpc-compute
    priority: 20
    profile: openshift-node-hpc-compute
Copy to Clipboard Toggle word wrap

apiVersion: tuned.openshift.io/v1
kind: Tuned
metadata:
  name: openshift-no-reapply-sysctl
  namespace: openshift-cluster-node-tuning-operator
spec:
  profile:
  - data: |
      [main]
      summary=Custom OpenShift profile
      include=openshift-node
      [sysctl]
      vm.max_map_count=>524288
    name: openshift-no-reapply-sysctl
  recommend:
  - match:
    - label: tuned.openshift.io/openshift-no-reapply-sysctl
    priority: 15
    profile: openshift-no-reapply-sysctl
    operand:
      tunedConfig:
        reapply_sysctl: false
Copy to Clipboard Toggle word wrap

7.7.

참고

7.8.

  1.     apiVersion: v1
        kind: ConfigMap
        metadata:
          name: tuned-1
          namespace: clusters
        data:
          tuning: |
            apiVersion: tuned.openshift.io/v1
            kind: Tuned
            metadata:
              name: tuned-1
              namespace: openshift-cluster-node-tuning-operator
            spec:
              profile:
              - data: |
                  [main]
                  summary=Custom OpenShift profile
                  include=openshift-node
                  [sysctl]
                  vm.dirty_ratio="55"
                name: tuned-1-profile
              recommend:
              - priority: 20
                profile: tuned-1-profile
    Copy to Clipboard Toggle word wrap
    참고

  2. $ oc --kubeconfig="$MGMT_KUBECONFIG" create -f tuned-1.yaml
    Copy to Clipboard Toggle word wrap
  3.     apiVersion: hypershift.openshift.io/v1alpha1
        kind: NodePool
        metadata:
          ...
          name: nodepool-1
          namespace: clusters
        ...
        spec:
          ...
          tuningConfig:
          - name: tuned-1
        status:
        ...
    Copy to Clipboard Toggle word wrap
    참고

  1. $ oc --kubeconfig="$HC_KUBECONFIG" get tuned.tuned.openshift.io -n openshift-cluster-node-tuning-operator
    Copy to Clipboard Toggle word wrap

    NAME       AGE
    default    7m36s
    rendered   7m36s
    tuned-1    65s
    Copy to Clipboard Toggle word wrap

  2. $ oc --kubeconfig="$HC_KUBECONFIG" get profile.tuned.openshift.io -n openshift-cluster-node-tuning-operator
    Copy to Clipboard Toggle word wrap

    NAME                           TUNED            APPLIED   DEGRADED   AGE
    nodepool-1-worker-1            tuned-1-profile  True      False      7m43s
    nodepool-1-worker-2            tuned-1-profile  True      False      7m14s
    Copy to Clipboard Toggle word wrap

    참고

  3. $ oc --kubeconfig="$HC_KUBECONFIG" debug node/nodepool-1-worker-1 -- chroot /host sysctl vm.dirty_ratio
    Copy to Clipboard Toggle word wrap

    vm.dirty_ratio = 55
    Copy to Clipboard Toggle word wrap

7.9.

  1.     apiVersion: v1
        kind: ConfigMap
        metadata:
          name: tuned-hugepages
          namespace: clusters
        data:
          tuning: |
            apiVersion: tuned.openshift.io/v1
            kind: Tuned
            metadata:
              name: hugepages
              namespace: openshift-cluster-node-tuning-operator
            spec:
              profile:
              - data: |
                  [main]
                  summary=Boot time configuration for hugepages
                  include=openshift-node
                  [bootloader]
                  cmdline_openshift_node_hugepages=hugepagesz=2M hugepages=50
                name: openshift-node-hugepages
              recommend:
              - priority: 20
                profile: openshift-node-hugepages
    Copy to Clipboard Toggle word wrap
    참고

  2. $ oc --kubeconfig="<management_cluster_kubeconfig>" create -f tuned-hugepages.yaml 
    1
    Copy to Clipboard Toggle word wrap
    1
  3. $ hcp create nodepool aws \
      --cluster-name <hosted_cluster_name> \
    1
    
      --name <nodepool_name> \
    2
    
      --node-count <nodepool_replicas> \
    3
    
      --instance-type <instance_type> \
    4
    
      --render > hugepages-nodepool.yaml
    Copy to Clipboard Toggle word wrap
    1
    2
    3
    4
    참고

  4.     apiVersion: hypershift.openshift.io/v1alpha1
        kind: NodePool
        metadata:
          name: hugepages-nodepool
          namespace: clusters
          ...
        spec:
          management:
            ...
            upgradeType: InPlace
          ...
          tuningConfig:
          - name: tuned-hugepages
    Copy to Clipboard Toggle word wrap
    참고

  5. $ oc --kubeconfig="<management_cluster_kubeconfig>" create -f hugepages-nodepool.yaml
    Copy to Clipboard Toggle word wrap

  1. $ oc --kubeconfig="<hosted_cluster_kubeconfig>" get tuned.tuned.openshift.io -n openshift-cluster-node-tuning-operator
    Copy to Clipboard Toggle word wrap

    NAME                 AGE
    default              123m
    hugepages-8dfb1fed   1m23s
    rendered             123m
    Copy to Clipboard Toggle word wrap

  2. $ oc --kubeconfig="<hosted_cluster_kubeconfig>" get profile.tuned.openshift.io -n openshift-cluster-node-tuning-operator
    Copy to Clipboard Toggle word wrap

    NAME                           TUNED                      APPLIED   DEGRADED   AGE
    nodepool-1-worker-1            openshift-node             True      False      132m
    nodepool-1-worker-2            openshift-node             True      False      131m
    hugepages-nodepool-worker-1    openshift-node-hugepages   True      False      4m8s
    hugepages-nodepool-worker-2    openshift-node-hugepages   True      False      3m57s
    Copy to Clipboard Toggle word wrap

  3. $ oc --kubeconfig="<hosted_cluster_kubeconfig>" debug node/nodepool-1-worker-1 -- chroot /host cat /proc/cmdline
    Copy to Clipboard Toggle word wrap

    BOOT_IMAGE=(hd0,gpt3)/ostree/rhcos-... hugepagesz=2M hugepages=50
    Copy to Clipboard Toggle word wrap

8장.

8.1.

  1. # oc label node perf-node.example.com cpumanager=true
    Copy to Clipboard Toggle word wrap
  2. # oc edit machineconfigpool worker
    Copy to Clipboard Toggle word wrap
  3. metadata:
      creationTimestamp: 2020-xx-xxx
      generation: 3
      labels:
        custom-kubelet: cpumanager-enabled
    Copy to Clipboard Toggle word wrap
  4. apiVersion: machineconfiguration.openshift.io/v1
    kind: KubeletConfig
    metadata:
      name: cpumanager-enabled
    spec:
      machineConfigPoolSelector:
        matchLabels:
          custom-kubelet: cpumanager-enabled
      kubeletConfig:
         cpuManagerPolicy: static 
    1
    
         cpuManagerReconcilePeriod: 5s 
    2
    Copy to Clipboard Toggle word wrap
    1
    2
  5. # oc create -f cpumanager-kubeletconfig.yaml
    Copy to Clipboard Toggle word wrap

  6. # oc get machineconfig 99-worker-XXXXXX-XXXXX-XXXX-XXXXX-kubelet -o json | grep ownerReference -A7
    Copy to Clipboard Toggle word wrap

           "ownerReferences": [
                {
                    "apiVersion": "machineconfiguration.openshift.io/v1",
                    "kind": "KubeletConfig",
                    "name": "cpumanager-enabled",
                    "uid": "7ed5616d-6b72-11e9-aae1-021e1ce18878"
                }
            ]
    Copy to Clipboard Toggle word wrap

  7. # oc debug node/perf-node.example.com
    sh-4.2# cat /host/etc/kubernetes/kubelet.conf | grep cpuManager
    Copy to Clipboard Toggle word wrap

    cpuManagerPolicy: static        
    1
    
    cpuManagerReconcilePeriod: 5s   
    2
    Copy to Clipboard Toggle word wrap

    1
    2
  8. $ oc new-project <project_name>
    Copy to Clipboard Toggle word wrap
  9. # cat cpumanager-pod.yaml
    Copy to Clipboard Toggle word wrap

    apiVersion: v1
    kind: Pod
    metadata:
      generateName: cpumanager-
    spec:
      securityContext:
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      containers:
      - name: cpumanager
        image: gcr.io/google_containers/pause:3.2
        resources:
          requests:
            cpu: 1
            memory: "1G"
          limits:
            cpu: 1
            memory: "1G"
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop: [ALL]
      nodeSelector:
        cpumanager: "true"
    Copy to Clipboard Toggle word wrap

  10. # oc create -f cpumanager-pod.yaml
    Copy to Clipboard Toggle word wrap

  1. # oc describe pod cpumanager
    Copy to Clipboard Toggle word wrap

    Name:               cpumanager-6cqz7
    Namespace:          default
    Priority:           0
    PriorityClassName:  <none>
    Node:  perf-node.example.com/xxx.xx.xx.xxx
    ...
     Limits:
          cpu:     1
          memory:  1G
        Requests:
          cpu:        1
          memory:     1G
    ...
    QoS Class:       Guaranteed
    Node-Selectors:  cpumanager=true
    Copy to Clipboard Toggle word wrap

  2. # oc describe node --selector='cpumanager=true' | grep -i cpumanager- -B2
    Copy to Clipboard Toggle word wrap

    NAMESPACE    NAME                CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
    cpuman       cpumanager-mlrrz    1 (28%)       1 (28%)     1G (13%)         1G (13%)       27m
    Copy to Clipboard Toggle word wrap

  3. # oc debug node/perf-node.example.com
    Copy to Clipboard Toggle word wrap
    sh-4.2# systemctl status | grep -B5 pause
    Copy to Clipboard Toggle word wrap
    참고

    # ├─init.scope
    │ └─1 /usr/lib/systemd/systemd --switched-root --system --deserialize 17
    └─kubepods.slice
      ├─kubepods-pod69c01f8e_6b74_11e9_ac0f_0a2b62178a22.slice
      │ ├─crio-b5437308f1a574c542bdf08563b865c0345c8f8c0b0a655612c.scope
      │ └─32706 /pause
    Copy to Clipboard Toggle word wrap

  4. # cd /sys/fs/cgroup/kubepods.slice/kubepods-pod69c01f8e_6b74_11e9_ac0f_0a2b62178a22.slice/crio-b5437308f1ad1a7db0574c542bdf08563b865c0345c86e9585f8c0b0a655612c.scope
    Copy to Clipboard Toggle word wrap
    # for i in `ls cpuset.cpus cgroup.procs` ; do echo -n "$i "; cat $i ; done
    Copy to Clipboard Toggle word wrap
    참고

    cpuset.cpus 1
    tasks 32706
    Copy to Clipboard Toggle word wrap

  5. # grep ^Cpus_allowed_list /proc/32706/status
    Copy to Clipboard Toggle word wrap

     Cpus_allowed_list:    1
    Copy to Clipboard Toggle word wrap

  6. # cat /sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc494a073_6b77_11e9_98c0_06bba5c387ea.slice/crio-c56982f57b75a2420947f0afc6cafe7534c5734efc34157525fa9abbf99e3849.scope/cpuset.cpus
    Copy to Clipboard Toggle word wrap
    # oc describe node perf-node.example.com
    Copy to Clipboard Toggle word wrap

    ...
    Capacity:
     attachable-volumes-aws-ebs:  39
     cpu:                         2
     ephemeral-storage:           124768236Ki
     hugepages-1Gi:               0
     hugepages-2Mi:               0
     memory:                      8162900Ki
     pods:                        250
    Allocatable:
     attachable-volumes-aws-ebs:  39
     cpu:                         1500m
     ephemeral-storage:           124768236Ki
     hugepages-1Gi:               0
     hugepages-2Mi:               0
     memory:                      7548500Ki
     pods:                        250
    -------                               ----                           ------------  ----------  ---------------  -------------  ---
      default                                 cpumanager-6cqz7               1 (66%)       1 (66%)     1G (12%)         1G (12%)       29m
    
    Allocated resources:
      (Total limits may be over 100 percent, i.e., overcommitted.)
      Resource                    Requests          Limits
      --------                    --------          ------
      cpu                         1440m (96%)       1 (66%)
    Copy to Clipboard Toggle word wrap

    NAME                    READY   STATUS    RESTARTS   AGE
    cpumanager-6cqz7        1/1     Running   0          33m
    cpumanager-7qc2t        0/1     Pending   0          11s
    Copy to Clipboard Toggle word wrap

8.2.

8.3.

  1. $ oc edit KubeletConfig cpumanager-enabled
    Copy to Clipboard Toggle word wrap
    apiVersion: machineconfiguration.openshift.io/v1
    kind: KubeletConfig
    metadata:
      name: cpumanager-enabled
    spec:
      machineConfigPoolSelector:
        matchLabels:
          custom-kubelet: cpumanager-enabled
      kubeletConfig:
         cpuManagerPolicy: static 
    1
    
         cpuManagerReconcilePeriod: 5s
         topologyManagerPolicy: single-numa-node 
    2
    Copy to Clipboard Toggle word wrap
    1
    2

8.4.

spec:
  containers:
  - name: nginx
    image: nginx
Copy to Clipboard Toggle word wrap

spec:
  containers:
  - name: nginx
    image: nginx
    resources:
      limits:
        memory: "200Mi"
      requests:
        memory: "100Mi"
Copy to Clipboard Toggle word wrap

spec:
  containers:
  - name: nginx
    image: nginx
    resources:
      limits:
        memory: "200Mi"
        cpu: "2"
        example.com/device: "1"
      requests:
        memory: "200Mi"
        cpu: "2"
        example.com/device: "1"
Copy to Clipboard Toggle word wrap

9장.

9.1.

그림 9.1.

참고

9.2.

Expand
표 9.1.
   

Expand
표 9.2.
      

Expand
표 9.3.
    

9.3.

9.3.1.

    1. apiVersion: v1
      kind: Namespace
      metadata:
        name: openshift-numaresources
      Copy to Clipboard Toggle word wrap
    2. $ oc create -f nro-namespace.yaml
      Copy to Clipboard Toggle word wrap
    1. apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: numaresources-operator
        namespace: openshift-numaresources
      spec:
        targetNamespaces:
        - openshift-numaresources
      Copy to Clipboard Toggle word wrap
    2. $ oc create -f nro-operatorgroup.yaml
      Copy to Clipboard Toggle word wrap
    1. apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: numaresources-operator
        namespace: openshift-numaresources
      spec:
        channel: "4.15"
        name: numaresources-operator
        source: redhat-operators
        sourceNamespace: openshift-marketplace
      Copy to Clipboard Toggle word wrap
    2. $ oc create -f nro-sub.yaml
      Copy to Clipboard Toggle word wrap

  1. $ oc get csv -n openshift-numaresources
    Copy to Clipboard Toggle word wrap

    NAME                             DISPLAY                  VERSION   REPLACES   PHASE
    numaresources-operator.v4.15.2   numaresources-operator   4.15.2               Succeeded
    Copy to Clipboard Toggle word wrap

9.3.2.

    1. 참고

9.4.

9.4.1.

    1. apiVersion: nodetopology.openshift.io/v1
      kind: NUMAResourcesOperator
      metadata:
        name: numaresourcesoperator
      spec:
        nodeGroups:
        - machineConfigPoolSelector:
            matchLabels:
              pools.operator.machineconfiguration.openshift.io/worker: "" 
      1
      Copy to Clipboard Toggle word wrap
      1
    2. $ oc create -f nrop.yaml
      Copy to Clipboard Toggle word wrap
      참고

  1. apiVersion: nodetopology.openshift.io/v1
    kind: NUMAResourcesOperator
    metadata:
      name: numaresourcesoperator
    spec:
      logLevel: Normal
      nodeGroups:
        - machineConfigPoolSelector:
            matchLabels:
              machineconfiguration.openshift.io/role: worker-ht
        - machineConfigPoolSelector:
            matchLabels:
              machineconfiguration.openshift.io/role: worker-cnf
        - machineConfigPoolSelector:
            matchLabels:
              machineconfiguration.openshift.io/role: worker-other
    Copy to Clipboard Toggle word wrap

  1. $ oc get numaresourcesoperators.nodetopology.openshift.io
    Copy to Clipboard Toggle word wrap

    NAME                    AGE
    numaresourcesoperator   27s
    Copy to Clipboard Toggle word wrap

  2. $ oc get all -n openshift-numaresources
    Copy to Clipboard Toggle word wrap

    NAME                                                    READY   STATUS    RESTARTS   AGE
    pod/numaresources-controller-manager-7d9d84c58d-qk2mr   1/1     Running   0          12m
    pod/numaresourcesoperator-worker-7d96r                  2/2     Running   0          97s
    pod/numaresourcesoperator-worker-crsht                  2/2     Running   0          97s
    pod/numaresourcesoperator-worker-jp9mw                  2/2     Running   0          97s
    Copy to Clipboard Toggle word wrap

9.4.2.

    1. apiVersion: nodetopology.openshift.io/v1
      kind: NUMAResourcesScheduler
      metadata:
        name: numaresourcesscheduler
      spec:
        imageSpec: "registry.redhat.io/openshift4/noderesourcetopology-scheduler-rhel9:v4.15" 
      1
      Copy to Clipboard Toggle word wrap
      1
    2. $ oc create -f nro-scheduler.yaml
      Copy to Clipboard Toggle word wrap
  1. $ oc get all -n openshift-numaresources
    Copy to Clipboard Toggle word wrap

    NAME                                                    READY   STATUS    RESTARTS   AGE
    pod/numaresources-controller-manager-7d9d84c58d-qk2mr   1/1     Running   0          12m
    pod/numaresourcesoperator-worker-7d96r                  2/2     Running   0          97s
    pod/numaresourcesoperator-worker-crsht                  2/2     Running   0          97s
    pod/numaresourcesoperator-worker-jp9mw                  2/2     Running   0          97s
    pod/secondary-scheduler-847cb74f84-9whlm                1/1     Running   0          10m
    
    NAME                                          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                     AGE
    daemonset.apps/numaresourcesoperator-worker   3         3         3       3            3           node-role.kubernetes.io/worker=   98s
    
    NAME                                               READY   UP-TO-DATE   AVAILABLE   AGE
    deployment.apps/numaresources-controller-manager   1/1     1            1           12m
    deployment.apps/secondary-scheduler                1/1     1            1           10m
    
    NAME                                                          DESIRED   CURRENT   READY   AGE
    replicaset.apps/numaresources-controller-manager-7d9d84c58d   1         1         1       12m
    replicaset.apps/secondary-scheduler-847cb74f84                1         1         1       10m
    Copy to Clipboard Toggle word wrap

9.4.3.

참고

9.4.4.

apiVersion: performance.openshift.io/v2
kind: PerformanceProfile
metadata:
  name: performance
spec:
  cpu:
    isolated: "3"
    reserved: 0-2
  machineConfigPoolSelector:
    pools.operator.machineconfiguration.openshift.io/worker: "" 
1

  nodeSelector:
    node-role.kubernetes.io/worker: ""
  numa:
    topologyPolicy: single-numa-node 
2

  realTimeKernel:
    enabled: true
  workloadHints:
    highPowerConsumption: true
    perPodPowerManagement: false
    realTime: true
Copy to Clipboard Toggle word wrap
1
2

9.4.5.

    1. apiVersion: machineconfiguration.openshift.io/v1
      kind: KubeletConfig
      metadata:
        name: worker-tuning
      spec:
        machineConfigPoolSelector:
          matchLabels:
            pools.operator.machineconfiguration.openshift.io/worker: "" 
      1
      
        kubeletConfig:
          cpuManagerPolicy: "static" 
      2
      
          cpuManagerReconcilePeriod: "5s"
          reservedSystemCPUs: "0,1" 
      3
      
          memoryManagerPolicy: "Static" 
      4
      
          evictionHard:
            memory.available: "100Mi"
          kubeReserved:
            memory: "512Mi"
          reservedMemory:
            - numaNode: 0
              limits:
                memory: "1124Mi"
          systemReserved:
            memory: "512Mi"
          topologyManagerPolicy: "single-numa-node" 
      5
      Copy to Clipboard Toggle word wrap
      1
      2
      3
      4
      5
    2. $ oc create -f nro-kubeletconfig.yaml
      Copy to Clipboard Toggle word wrap
      참고

9.4.6.

  1. $ oc get numaresourcesschedulers.nodetopology.openshift.io numaresourcesscheduler -o json | jq '.status.schedulerName'
    Copy to Clipboard Toggle word wrap

    "topo-aware-scheduler"
    Copy to Clipboard Toggle word wrap

    1. apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: numa-deployment-1
        namespace: openshift-numaresources
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: test
        template:
          metadata:
            labels:
              app: test
          spec:
            schedulerName: topo-aware-scheduler 
      1
      
            containers:
            - name: ctnr
              image: quay.io/openshifttest/hello-openshift:openshift
              imagePullPolicy: IfNotPresent
              resources:
                limits:
                  memory: "100Mi"
                  cpu: "10"
                requests:
                  memory: "100Mi"
                  cpu: "10"
            - name: ctnr2
              image: registry.access.redhat.com/rhel:latest
              imagePullPolicy: IfNotPresent
              command: ["/bin/sh", "-c"]
              args: [ "while true; do sleep 1h; done;" ]
              resources:
                limits:
                  memory: "100Mi"
                  cpu: "8"
                requests:
                  memory: "100Mi"
                  cpu: "8"
      Copy to Clipboard Toggle word wrap
      1
    2. $ oc create -f nro-deployment.yaml
      Copy to Clipboard Toggle word wrap

  1. $ oc get pods -n openshift-numaresources
    Copy to Clipboard Toggle word wrap

    NAME                                                READY   STATUS    RESTARTS   AGE
    numa-deployment-1-6c4f5bdb84-wgn6g                  2/2     Running   0          5m2s
    numaresources-controller-manager-7d9d84c58d-4v65j   1/1     Running   0          18m
    numaresourcesoperator-worker-7d96r                  2/2     Running   4          43m
    numaresourcesoperator-worker-crsht                  2/2     Running   2          43m
    numaresourcesoperator-worker-jp9mw                  2/2     Running   2          43m
    secondary-scheduler-847cb74f84-fpncj                1/1     Running   0          18m
    Copy to Clipboard Toggle word wrap

  2. $ oc describe pod numa-deployment-1-6c4f5bdb84-wgn6g -n openshift-numaresources
    Copy to Clipboard Toggle word wrap

    Events:
      Type    Reason          Age    From                  Message
      ----    ------          ----   ----                  -------
      Normal  Scheduled       4m45s  topo-aware-scheduler  Successfully assigned openshift-numaresources/numa-deployment-1-6c4f5bdb84-wgn6g to worker-1
    Copy to Clipboard Toggle word wrap

    참고

    1. $ oc get pods -n openshift-numaresources -o wide
      Copy to Clipboard Toggle word wrap

      NAME                                 READY   STATUS    RESTARTS   AGE   IP            NODE     NOMINATED NODE   READINESS GATES
      numa-deployment-1-6c4f5bdb84-wgn6g   0/2     Running   0          82m   10.128.2.50   worker-1   <none>  <none>
      Copy to Clipboard Toggle word wrap

    2. $ oc describe noderesourcetopologies.topology.node.k8s.io worker-1
      Copy to Clipboard Toggle word wrap

      ...
      
      Zones:
        Costs:
          Name:   node-0
          Value:  10
          Name:   node-1
          Value:  21
        Name:     node-0
        Resources:
          Allocatable:  39
          Available:    21 
      1
      
          Capacity:     40
          Name:         cpu
          Allocatable:  6442450944
          Available:    6442450944
          Capacity:     6442450944
          Name:         hugepages-1Gi
          Allocatable:  134217728
          Available:    134217728
          Capacity:     134217728
          Name:         hugepages-2Mi
          Allocatable:  262415904768
          Available:    262206189568
          Capacity:     270146007040
          Name:         memory
        Type:           Node
      Copy to Clipboard Toggle word wrap

      1

  3. $ oc get pod numa-deployment-1-6c4f5bdb84-wgn6g -n openshift-numaresources -o jsonpath="{ .status.qosClass }"
    Copy to Clipboard Toggle word wrap

    Guaranteed
    Copy to Clipboard Toggle word wrap

9.5.

  • 참고

  • apiVersion: nodetopology.openshift.io/v1
    kind: NUMAResourcesOperator
    metadata:
      name: numaresourcesoperator
    spec:
      nodeGroups:
      - config:
          infoRefreshMode: Periodic 
    1
    
          infoRefreshPeriod: 10s 
    2
    
          podsFingerprinting: Enabled 
    3
    
        name: worker
    Copy to Clipboard Toggle word wrap
    1
    2
    3

  1. $ oc get numaresop numaresourcesoperator -o json | jq '.status'
    Copy to Clipboard Toggle word wrap

          ...
    
            "config": {
            "infoRefreshMode": "Periodic",
            "infoRefreshPeriod": "10s",
            "podsFingerprinting": "Enabled"
          },
          "name": "worker"
    
          ...
    Copy to Clipboard Toggle word wrap

9.6.

  1. $ oc get crd | grep noderesourcetopologies
    Copy to Clipboard Toggle word wrap

    NAME                                                              CREATED AT
    noderesourcetopologies.topology.node.k8s.io                       2022-01-18T08:28:06Z
    Copy to Clipboard Toggle word wrap

  2. $ oc get numaresourcesschedulers.nodetopology.openshift.io numaresourcesscheduler -o json | jq '.status.schedulerName'
    Copy to Clipboard Toggle word wrap

    topo-aware-scheduler
    Copy to Clipboard Toggle word wrap

  3. $ oc get noderesourcetopologies.topology.node.k8s.io
    Copy to Clipboard Toggle word wrap

    NAME                    AGE
    compute-0.example.com   17h
    compute-1.example.com   17h
    Copy to Clipboard Toggle word wrap

    참고

  4. $ oc get noderesourcetopologies.topology.node.k8s.io -o yaml
    Copy to Clipboard Toggle word wrap

    apiVersion: v1
    items:
    - apiVersion: topology.node.k8s.io/v1
      kind: NodeResourceTopology
      metadata:
        annotations:
          k8stopoawareschedwg/rte-update: periodic
        creationTimestamp: "2022-06-16T08:55:38Z"
        generation: 63760
        name: worker-0
        resourceVersion: "8450223"
        uid: 8b77be46-08c0-4074-927b-d49361471590
      topologyPolicies:
      - SingleNUMANodeContainerLevel
      zones:
      - costs:
        - name: node-0
          value: 10
        - name: node-1
          value: 21
        name: node-0
        resources:
        - allocatable: "38"
          available: "38"
          capacity: "40"
          name: cpu
        - allocatable: "134217728"
          available: "134217728"
          capacity: "134217728"
          name: hugepages-2Mi
        - allocatable: "262352048128"
          available: "262352048128"
          capacity: "270107316224"
          name: memory
        - allocatable: "6442450944"
          available: "6442450944"
          capacity: "6442450944"
          name: hugepages-1Gi
        type: Node
      - costs:
        - name: node-0
          value: 21
        - name: node-1
          value: 10
        name: node-1
        resources:
        - allocatable: "268435456"
          available: "268435456"
          capacity: "268435456"
          name: hugepages-2Mi
        - allocatable: "269231067136"
          available: "269231067136"
          capacity: "270573244416"
          name: memory
        - allocatable: "40"
          available: "40"
          capacity: "40"
          name: cpu
        - allocatable: "1073741824"
          available: "1073741824"
          capacity: "1073741824"
          name: hugepages-1Gi
        type: Node
    - apiVersion: topology.node.k8s.io/v1
      kind: NodeResourceTopology
      metadata:
        annotations:
          k8stopoawareschedwg/rte-update: periodic
        creationTimestamp: "2022-06-16T08:55:37Z"
        generation: 62061
        name: worker-1
        resourceVersion: "8450129"
        uid: e8659390-6f8d-4e67-9a51-1ea34bba1cc3
      topologyPolicies:
      - SingleNUMANodeContainerLevel
      zones: 
    1
    
      - costs:
        - name: node-0
          value: 10
        - name: node-1
          value: 21
        name: node-0
        resources: 
    2
    
        - allocatable: "38"
          available: "38"
          capacity: "40"
          name: cpu
        - allocatable: "6442450944"
          available: "6442450944"
          capacity: "6442450944"
          name: hugepages-1Gi
        - allocatable: "134217728"
          available: "134217728"
          capacity: "134217728"
          name: hugepages-2Mi
        - allocatable: "262391033856"
          available: "262391033856"
          capacity: "270146301952"
          name: memory
        type: Node
      - costs:
        - name: node-0
          value: 21
        - name: node-1
          value: 10
        name: node-1
        resources:
        - allocatable: "40"
          available: "40"
          capacity: "40"
          name: cpu
        - allocatable: "1073741824"
          available: "1073741824"
          capacity: "1073741824"
          name: hugepages-1Gi
        - allocatable: "268435456"
          available: "268435456"
          capacity: "268435456"
          name: hugepages-2Mi
        - allocatable: "269192085504"
          available: "269192085504"
          capacity: "270534262784"
          name: memory
        type: Node
    kind: List
    metadata:
      resourceVersion: ""
      selfLink: ""
    Copy to Clipboard Toggle word wrap

    1
    2

9.6.1.

    1. $ oc get NUMAResourcesScheduler
      Copy to Clipboard Toggle word wrap

      NAME                     AGE
      numaresourcesscheduler   92m
      Copy to Clipboard Toggle word wrap

    2. $ oc delete NUMAResourcesScheduler numaresourcesscheduler
      Copy to Clipboard Toggle word wrap

      numaresourcesscheduler.nodetopology.openshift.io "numaresourcesscheduler" deleted
      Copy to Clipboard Toggle word wrap

  1. apiVersion: nodetopology.openshift.io/v1
    kind: NUMAResourcesScheduler
    metadata:
      name: numaresourcesscheduler
    spec:
      imageSpec: "registry.redhat.io/openshift4/noderesourcetopology-scheduler-container-rhel8:v4.15"
      cacheResyncPeriod: "5s" 
    1
    Copy to Clipboard Toggle word wrap
    1
  2. $ oc create -f nro-scheduler-cacheresync.yaml
    Copy to Clipboard Toggle word wrap

    numaresourcesscheduler.nodetopology.openshift.io/numaresourcesscheduler created
    Copy to Clipboard Toggle word wrap

    1. $ oc get crd | grep numaresourcesschedulers
      Copy to Clipboard Toggle word wrap

      NAME                                                              CREATED AT
      numaresourcesschedulers.nodetopology.openshift.io                 2022-02-25T11:57:03Z
      Copy to Clipboard Toggle word wrap

    2. $ oc get numaresourcesschedulers.nodetopology.openshift.io
      Copy to Clipboard Toggle word wrap

      NAME                     AGE
      numaresourcesscheduler   3h26m
      Copy to Clipboard Toggle word wrap

    1. $ oc get pods -n openshift-numaresources
      Copy to Clipboard Toggle word wrap

      NAME                                               READY   STATUS    RESTARTS   AGE
      numaresources-controller-manager-d87d79587-76mrm   1/1     Running   0          46h
      numaresourcesoperator-worker-5wm2k                 2/2     Running   0          45h
      numaresourcesoperator-worker-pb75c                 2/2     Running   0          45h
      secondary-scheduler-7976c4d466-qm4sc               1/1     Running   0          21m
      Copy to Clipboard Toggle word wrap

    2. $ oc logs secondary-scheduler-7976c4d466-qm4sc -n openshift-numaresources
      Copy to Clipboard Toggle word wrap

      ...
      I0223 11:04:55.614788       1 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.Namespace total 11 items received
      I0223 11:04:56.609114       1 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.ReplicationController total 10 items received
      I0223 11:05:22.626818       1 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.StorageClass total 7 items received
      I0223 11:05:31.610356       1 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.PodDisruptionBudget total 7 items received
      I0223 11:05:31.713032       1 eventhandlers.go:186] "Add event for scheduled pod" pod="openshift-marketplace/certified-operators-thtvq"
      I0223 11:05:53.461016       1 eventhandlers.go:244] "Delete event for scheduled pod" pod="openshift-marketplace/certified-operators-thtvq"
      Copy to Clipboard Toggle word wrap

9.6.2.

    1. $ oc get NUMAResourcesScheduler
      Copy to Clipboard Toggle word wrap

      NAME                     AGE
      numaresourcesscheduler   92m
      Copy to Clipboard Toggle word wrap

    2. $ oc delete NUMAResourcesScheduler numaresourcesscheduler
      Copy to Clipboard Toggle word wrap

      numaresourcesscheduler.nodetopology.openshift.io "numaresourcesscheduler" deleted
      Copy to Clipboard Toggle word wrap

  1. apiVersion: nodetopology.openshift.io/v1
    kind: NUMAResourcesScheduler
    metadata:
      name: numaresourcesscheduler
    spec:
      imageSpec: "registry.redhat.io/openshift4/noderesourcetopology-scheduler-container-rhel8:v{product-version}"
      scoringStrategy:
            type: "MostAllocated" 
    1
    Copy to Clipboard Toggle word wrap
    1
  2. $ oc create -f nro-scheduler-mostallocated.yaml
    Copy to Clipboard Toggle word wrap

    numaresourcesscheduler.nodetopology.openshift.io/numaresourcesscheduler created
    Copy to Clipboard Toggle word wrap

    1. $ oc get crd | grep numaresourcesschedulers
      Copy to Clipboard Toggle word wrap

      NAME                                                              CREATED AT
      numaresourcesschedulers.nodetopology.openshift.io                 2022-02-25T11:57:03Z
      Copy to Clipboard Toggle word wrap

    2. $ oc get numaresourcesschedulers.nodetopology.openshift.io
      Copy to Clipboard Toggle word wrap

      NAME                     AGE
      numaresourcesscheduler   3h26m
      Copy to Clipboard Toggle word wrap

  1. $ oc get -n openshift-numaresources cm topo-aware-scheduler-config -o yaml | grep scoring -A 1
    Copy to Clipboard Toggle word wrap

    scoringStrategy:
      type: MostAllocated
    Copy to Clipboard Toggle word wrap

9.6.3.

참고

    1. $ oc get NUMAResourcesScheduler
      Copy to Clipboard Toggle word wrap

      NAME                     AGE
      numaresourcesscheduler   90m
      Copy to Clipboard Toggle word wrap

    2. $ oc delete NUMAResourcesScheduler numaresourcesscheduler
      Copy to Clipboard Toggle word wrap

      numaresourcesscheduler.nodetopology.openshift.io "numaresourcesscheduler" deleted
      Copy to Clipboard Toggle word wrap

  1. apiVersion: nodetopology.openshift.io/v1
    kind: NUMAResourcesScheduler
    metadata:
      name: numaresourcesscheduler
    spec:
      imageSpec: "registry.redhat.io/openshift4/noderesourcetopology-scheduler-container-rhel8:v4.15"
      logLevel: Debug
    Copy to Clipboard Toggle word wrap
  2. $ oc create -f nro-scheduler-debug.yaml
    Copy to Clipboard Toggle word wrap

    numaresourcesscheduler.nodetopology.openshift.io/numaresourcesscheduler created
    Copy to Clipboard Toggle word wrap

    1. $ oc get crd | grep numaresourcesschedulers
      Copy to Clipboard Toggle word wrap

      NAME                                                              CREATED AT
      numaresourcesschedulers.nodetopology.openshift.io                 2022-02-25T11:57:03Z
      Copy to Clipboard Toggle word wrap

    2. $ oc get numaresourcesschedulers.nodetopology.openshift.io
      Copy to Clipboard Toggle word wrap

      NAME                     AGE
      numaresourcesscheduler   3h26m
      Copy to Clipboard Toggle word wrap

    1. $ oc get pods -n openshift-numaresources
      Copy to Clipboard Toggle word wrap

      NAME                                               READY   STATUS    RESTARTS   AGE
      numaresources-controller-manager-d87d79587-76mrm   1/1     Running   0          46h
      numaresourcesoperator-worker-5wm2k                 2/2     Running   0          45h
      numaresourcesoperator-worker-pb75c                 2/2     Running   0          45h
      secondary-scheduler-7976c4d466-qm4sc               1/1     Running   0          21m
      Copy to Clipboard Toggle word wrap

    2. $ oc logs secondary-scheduler-7976c4d466-qm4sc -n openshift-numaresources
      Copy to Clipboard Toggle word wrap

      ...
      I0223 11:04:55.614788       1 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.Namespace total 11 items received
      I0223 11:04:56.609114       1 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.ReplicationController total 10 items received
      I0223 11:05:22.626818       1 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.StorageClass total 7 items received
      I0223 11:05:31.610356       1 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.PodDisruptionBudget total 7 items received
      I0223 11:05:31.713032       1 eventhandlers.go:186] "Add event for scheduled pod" pod="openshift-marketplace/certified-operators-thtvq"
      I0223 11:05:53.461016       1 eventhandlers.go:244] "Delete event for scheduled pod" pod="openshift-marketplace/certified-operators-thtvq"
      Copy to Clipboard Toggle word wrap

9.6.4.

참고

  1. $ oc get numaresourcesoperators.nodetopology.openshift.io numaresourcesoperator -o jsonpath="{.status.daemonsets[0]}"
    Copy to Clipboard Toggle word wrap

    {"name":"numaresourcesoperator-worker","namespace":"openshift-numaresources"}
    Copy to Clipboard Toggle word wrap

  2. $ oc get ds -n openshift-numaresources numaresourcesoperator-worker -o jsonpath="{.spec.selector.matchLabels}"
    Copy to Clipboard Toggle word wrap

    {"name":"resource-topology"}
    Copy to Clipboard Toggle word wrap

  3. $ oc get pods -n openshift-numaresources -l name=resource-topology -o wide
    Copy to Clipboard Toggle word wrap

    NAME                                 READY   STATUS    RESTARTS   AGE    IP            NODE
    numaresourcesoperator-worker-5wm2k   2/2     Running   0          2d1h   10.135.0.64   compute-0.example.com
    numaresourcesoperator-worker-pb75c   2/2     Running   0          2d1h   10.132.2.33   compute-1.example.com
    Copy to Clipboard Toggle word wrap

  4. $ oc logs -n openshift-numaresources -c resource-topology-exporter numaresourcesoperator-worker-pb75c
    Copy to Clipboard Toggle word wrap

    I0221 13:38:18.334140       1 main.go:206] using sysinfo:
    reservedCpus: 0,1
    reservedMemory:
      "0": 1178599424
    I0221 13:38:18.334370       1 main.go:67] === System information ===
    I0221 13:38:18.334381       1 sysinfo.go:231] cpus: reserved "0-1"
    I0221 13:38:18.334493       1 sysinfo.go:237] cpus: online "0-103"
    I0221 13:38:18.546750       1 main.go:72]
    cpus: allocatable "2-103"
    hugepages-1Gi:
      numa cell 0 -> 6
      numa cell 1 -> 1
    hugepages-2Mi:
      numa cell 0 -> 64
      numa cell 1 -> 128
    memory:
      numa cell 0 -> 45758Mi
      numa cell 1 -> 48372Mi
    Copy to Clipboard Toggle word wrap

9.6.5.

Info: couldn't find configuration in "/etc/resource-topology-exporter/config.yaml"
Copy to Clipboard Toggle word wrap

$ oc get configmap
Copy to Clipboard Toggle word wrap

NAME                           DATA   AGE
0e2a6bd3.openshift-kni.io      0      6d21h
kube-root-ca.crt               1      6d21h
openshift-service-ca.crt       1      6d21h
topo-aware-scheduler-config    1      6d18h
Copy to Clipboard Toggle word wrap

    1. $ oc get kubeletconfig -o yaml
      Copy to Clipboard Toggle word wrap

      machineConfigPoolSelector:
        matchLabels:
          cnf-worker-tuning: enabled
      Copy to Clipboard Toggle word wrap

    2. $ oc get mcp worker -o yaml
      Copy to Clipboard Toggle word wrap

      labels:
        machineconfiguration.openshift.io/mco-built-in: ""
        pools.operator.machineconfiguration.openshift.io/worker: ""
      Copy to Clipboard Toggle word wrap

  1. $ oc edit mcp worker -o yaml
    Copy to Clipboard Toggle word wrap

    labels:
      machineconfiguration.openshift.io/mco-built-in: ""
      pools.operator.machineconfiguration.openshift.io/worker: ""
      cnf-worker-tuning: enabled
    Copy to Clipboard Toggle word wrap

  • $ oc get configmap
    Copy to Clipboard Toggle word wrap

    NAME                           DATA   AGE
    0e2a6bd3.openshift-kni.io      0      6d21h
    kube-root-ca.crt               1      6d21h
    numaresourcesoperator-worker   1      5m
    openshift-service-ca.crt       1      6d21h
    topo-aware-scheduler-config    1      6d18h
    Copy to Clipboard Toggle word wrap

9.6.6.

  • $ oc adm must-gather --image=registry.redhat.io/numaresources-must-gather/numaresources-must-gather-rhel9:v4.15
    Copy to Clipboard Toggle word wrap

10장.

10.1.

10.1.1.

Expand
표 10.1.
   

10.1.3.

Expand
표 10.3.
    

10.1.4.

10.2.

10.2.1.

Expand
   

Expand
   

Expand
  

10.2.2.

Expand
  

중요

$ oc -n openshift-ingress patch deploy/router-default --type=strategic --patch='{"spec":{"template":{"spec":{"containers":[{"name":"router","livenessProbe":{"timeoutSeconds":5},"readinessProbe":{"timeoutSeconds":5}}]}}}}'
Copy to Clipboard Toggle word wrap

$ oc -n openshift-ingress describe deploy/router-default | grep -e Liveness: -e Readiness:
    Liveness:   http-get http://:1936/healthz delay=0s timeout=5s period=10s #success=1 #failure=3
    Readiness:  http-get http://:1936/healthz/ready delay=0s timeout=5s period=10s #success=1 #failure=3
Copy to Clipboard Toggle word wrap

10.2.3.

주의

  • $ oc -n openshift-ingress-operator patch ingresscontrollers/default --type=merge --patch='{"spec":{"tuningOptions":{"reloadInterval":"15s"}}}'
    Copy to Clipboard Toggle word wrap

10.3.

10.3.1.

중요

참고

참고

10.3.3.

10.4.

중요

10.4.1.

10.4.2.

참고

  1. apiVersion: machineconfiguration.openshift.io/v1
    kind: MachineConfig
    metadata:
      labels:
        machineconfiguration.openshift.io/role: master
      name: 99-kubens-master
    spec:
      config:
        ignition:
          version: 3.2.0
        systemd:
          units:
          - enabled: true
            name: kubens.service
    ---
    apiVersion: machineconfiguration.openshift.io/v1
    kind: MachineConfig
    metadata:
      labels:
        machineconfiguration.openshift.io/role: worker
      name: 99-kubens-worker
    spec:
      config:
        ignition:
          version: 3.2.0
        systemd:
          units:
          - enabled: true
            name: kubens.service
    Copy to Clipboard Toggle word wrap
  2. $ oc apply -f mount_namespace_config.yaml
    Copy to Clipboard Toggle word wrap

    machineconfig.machineconfiguration.openshift.io/99-kubens-master created
    machineconfig.machineconfiguration.openshift.io/99-kubens-worker created
    Copy to Clipboard Toggle word wrap

  3. $ oc get mcp
    Copy to Clipboard Toggle word wrap

    NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
    master   rendered-master-03d4bc4befb0f4ed3566a2c8f7636751   False     True       False      3              0                   0                     0                      45m
    worker   rendered-worker-10577f6ab0117ed1825f8af2ac687ddf   False     True       False      3              1                   1
    Copy to Clipboard Toggle word wrap

  4. $ oc wait --for=condition=Updated mcp --all --timeout=30m
    Copy to Clipboard Toggle word wrap

    machineconfigpool.machineconfiguration.openshift.io/master condition met
    machineconfigpool.machineconfiguration.openshift.io/worker condition met
    Copy to Clipboard Toggle word wrap

  1. $ oc debug node/<node_name>
    Copy to Clipboard Toggle word wrap
  2. sh-4.4# chroot /host
    Copy to Clipboard Toggle word wrap
  3. sh-4.4# readlink /proc/1/ns/mnt
    Copy to Clipboard Toggle word wrap

    mnt:[4026531953]
    Copy to Clipboard Toggle word wrap

  4. sh-4.4# readlink /proc/$(pgrep kubelet)/ns/mnt
    Copy to Clipboard Toggle word wrap

    mnt:[4026531840]
    Copy to Clipboard Toggle word wrap

  5. sh-4.4# readlink /proc/$(pgrep crio)/ns/mnt
    Copy to Clipboard Toggle word wrap

    mnt:[4026531840]
    Copy to Clipboard Toggle word wrap

10.4.3.

참고

  1. $ ssh core@<node_name>
    Copy to Clipboard Toggle word wrap
  2. [core@control-plane-1 ~]$ sudo kubensenter findmnt
    Copy to Clipboard Toggle word wrap

    kubensenter: Autodetect: kubens.service namespace found at /run/kubens/mnt
    TARGET                                SOURCE                 FSTYPE     OPTIONS
    /                                     /dev/sda4[/ostree/deploy/rhcos/deploy/32074f0e8e5ec453e56f5a8a7bc9347eaa4172349ceab9c22b709d9d71a3f4b0.0]
    |                                                            xfs        rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota
                                          shm                    tmpfs
    ...
    Copy to Clipboard Toggle word wrap

  3. [core@control-plane-1 ~]$ sudo kubensenter
    Copy to Clipboard Toggle word wrap

    kubensenter: Autodetect: kubens.service namespace found at /run/kubens/mnt
    Copy to Clipboard Toggle word wrap

10.4.4.

[Unit]
Description=Example service
[Service]
ExecStart=/usr/bin/kubensenter /path/to/original/command arg1 arg2
Copy to Clipboard Toggle word wrap

11장.

11.1.

11.2.

참고

11.2.1.

참고

11.2.2.

  1. apiVersion: metal3.io/v1alpha1
    kind: BareMetalHost
    metadata:
      name: <bare_metal_host_name>
    spec:
      online: true
      bmc:
        address: <bmc_address>
        credentialsName: <secret_credentials_name>  
    1
    
        disableCertificateVerification: True 
    2
    
      bootMACAddress: <host_boot_mac_address>
    Copy to Clipboard Toggle word wrap
    1
    2
  2. 참고

11.2.3.

  1. $ oc annotate machineset <machineset> -n openshift-machine-api 'metal3.io/autoscale-to-hosts=<any_value>'
    Copy to Clipboard Toggle word wrap

참고

11.2.4.

참고

  1. $ oc annotate machineset <machineset> -n openshift-machine-api 'baremetalhost.metal3.io/detached'
    Copy to Clipboard Toggle word wrap

    참고

  2. $ oc annotate machineset <machineset> -n openshift-machine-api 'baremetalhost.metal3.io/detached-'
    Copy to Clipboard Toggle word wrap

11.2.5.

    1. $ oc get baremetalhosts -n openshift-machine-api -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.provisioning.state}{"\n"}{end}'
      Copy to Clipboard Toggle word wrap

      master-0.example.com  managed
      master-1.example.com  managed
      master-2.example.com  managed
      worker-0.example.com  managed
      worker-1.example.com  managed
      worker-2.example.com  managed
      Copy to Clipboard Toggle word wrap

    2. $ oc adm cordon <bare_metal_host> 
      1
      Copy to Clipboard Toggle word wrap
      1
    3. $ oc adm drain <bare_metal_host> --force=true
      Copy to Clipboard Toggle word wrap

    4. $ oc patch <bare_metal_host> --type json -p '[{"op": "replace", "path": "/spec/online", "value": false}]'
      Copy to Clipboard Toggle word wrap
    5. $ oc adm uncordon <bare_metal_host>
      Copy to Clipboard Toggle word wrap

12장.

중요

12.1.

중요

참고

12.2.

12.2.1.

그림 12.1.

12.2.1.1.

12.2.1.2.

12.2.1.3.

12.2.1.4.

12.2.1.5.

참고

12.2.1.6.

12.2.2.

12.2.3.

    1. apiVersion: v1
      kind: Namespace
      metadata:
        name: openshift-bare-metal-events
        labels:
          name: openshift-bare-metal-events
          openshift.io/cluster-monitoring: "true"
      Copy to Clipboard Toggle word wrap
    2. $ oc create -f bare-metal-events-namespace.yaml
      Copy to Clipboard Toggle word wrap
    1. apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: bare-metal-event-relay-group
        namespace: openshift-bare-metal-events
      spec:
        targetNamespaces:
        - openshift-bare-metal-events
      Copy to Clipboard Toggle word wrap
    2. $ oc create -f bare-metal-events-operatorgroup.yaml
      Copy to Clipboard Toggle word wrap
    1. apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: bare-metal-event-relay-subscription
        namespace: openshift-bare-metal-events
      spec:
        channel: "stable"
        name: bare-metal-event-relay
        source: redhat-operators
        sourceNamespace: openshift-marketplace
      Copy to Clipboard Toggle word wrap
    2. $ oc create -f bare-metal-events-sub.yaml
      Copy to Clipboard Toggle word wrap

$ oc get csv -n openshift-bare-metal-events -o custom-columns=Name:.metadata.name,Phase:.status.phase
Copy to Clipboard Toggle word wrap

12.2.4.

  1. 참고

12.3.

참고

  1. $ oc get pods -n amq-interconnect
    Copy to Clipboard Toggle word wrap

    NAME                                    READY   STATUS    RESTARTS   AGE
    amq-interconnect-645db76c76-k8ghs       1/1     Running   0          23h
    interconnect-operator-5cb5fc7cc-4v7qm   1/1     Running   0          23h
    Copy to Clipboard Toggle word wrap

  2. $ oc get pods -n openshift-bare-metal-events
    Copy to Clipboard Toggle word wrap

    NAME                                                            READY   STATUS    RESTARTS   AGE
    hw-event-proxy-operator-controller-manager-74d5649b7c-dzgtl     2/2     Running   0          25s
    Copy to Clipboard Toggle word wrap

12.4.

12.4.1.

중요

참고

  • 참고

  1. $ curl https://<bmc_ip_address>/redfish/v1/EventService --insecure -H 'Content-Type: application/json' -u "<bmc_username>:<password>"
    Copy to Clipboard Toggle word wrap

    {
       "@odata.context": "/redfish/v1/$metadata#EventService.EventService",
       "@odata.id": "/redfish/v1/EventService",
       "@odata.type": "#EventService.v1_0_2.EventService",
       "Actions": {
          "#EventService.SubmitTestEvent": {
             "EventType@Redfish.AllowableValues": ["StatusChange", "ResourceUpdated", "ResourceAdded", "ResourceRemoved", "Alert"],
             "target": "/redfish/v1/EventService/Actions/EventService.SubmitTestEvent"
          }
       },
       "DeliveryRetryAttempts": 3,
       "DeliveryRetryIntervalSeconds": 30,
       "Description": "Event Service represents the properties for the service",
       "EventTypesForSubscription": ["StatusChange", "ResourceUpdated", "ResourceAdded", "ResourceRemoved", "Alert"],
       "EventTypesForSubscription@odata.count": 5,
       "Id": "EventService",
       "Name": "Event Service",
       "ServiceEnabled": true,
       "Status": {
          "Health": "OK",
          "HealthRollup": "OK",
          "State": "Enabled"
       },
       "Subscriptions": {
          "@odata.id": "/redfish/v1/EventService/Subscriptions"
       }
    }
    Copy to Clipboard Toggle word wrap

  2. $ oc get route -n openshift-bare-metal-events
    Copy to Clipboard Toggle word wrap

    NAME            HOST/PORT   PATH                                                                    SERVICES                 PORT   TERMINATION   WILDCARD
    hw-event-proxy              hw-event-proxy-openshift-bare-metal-events.apps.compute-1.example.com   hw-event-proxy-service   9087   edge          None
    Copy to Clipboard Toggle word wrap

    1. apiVersion: metal3.io/v1alpha1
      kind: BMCEventSubscription
      metadata:
        name: sub-01
        namespace: openshift-machine-api
      spec:
         hostName: <hostname> 
      1
      
         destination: <proxy_service_url> 
      2
      
         context: ''
      Copy to Clipboard Toggle word wrap
      1
      2
    2. $ oc create -f bmc_sub.yaml
      Copy to Clipboard Toggle word wrap
  3. $ oc delete -f bmc_sub.yaml
    Copy to Clipboard Toggle word wrap
  4. $ curl -i -k -X POST -H "Content-Type: application/json"  -d '{"Destination": "https://<proxy_service_url>", "Protocol" : "Redfish", "EventTypes": ["Alert"], "Context": "root"}' -u <bmc_username>:<password> 'https://<bmc_ip_address>/redfish/v1/EventService/Subscriptions' –v
    Copy to Clipboard Toggle word wrap

    HTTP/1.1 201 Created
    Server: AMI MegaRAC Redfish Service
    Location: /redfish/v1/EventService/Subscriptions/1
    Allow: GET, POST
    Access-Control-Allow-Origin: *
    Access-Control-Expose-Headers: X-Auth-Token
    Access-Control-Allow-Headers: X-Auth-Token
    Access-Control-Allow-Credentials: true
    Cache-Control: no-cache, must-revalidate
    Link: <http://redfish.dmtf.org/schemas/v1/EventDestination.v1_6_0.json>; rel=describedby
    Link: <http://redfish.dmtf.org/schemas/v1/EventDestination.v1_6_0.json>
    Link: </redfish/v1/EventService/Subscriptions>; path=
    ETag: "1651135676"
    Content-Type: application/json; charset=UTF-8
    OData-Version: 4.0
    Content-Length: 614
    Date: Thu, 28 Apr 2022 08:47:57 GMT
    Copy to Clipboard Toggle word wrap

12.4.2.

  1. $ curl --globoff -H "Content-Type: application/json" -k -X GET --user <bmc_username>:<password> https://<bmc_ip_address>/redfish/v1/EventService/Subscriptions
    Copy to Clipboard Toggle word wrap

    % Total % Received % Xferd Average Speed Time Time Time Current
    Dload Upload Total Spent Left Speed
    100 435 100 435 0 0 399 0 0:00:01 0:00:01 --:--:-- 399
    {
      "@odata.context": "/redfish/v1/$metadata#EventDestinationCollection.EventDestinationCollection",
      "@odata.etag": ""
      1651137375 "",
      "@odata.id": "/redfish/v1/EventService/Subscriptions",
      "@odata.type": "#EventDestinationCollection.EventDestinationCollection",
      "Description": "Collection for Event Subscriptions",
      "Members": [
      {
        "@odata.id": "/redfish/v1/EventService/Subscriptions/1"
      }],
      "Members@odata.count": 1,
      "Name": "Event Subscriptions Collection"
    }
    Copy to Clipboard Toggle word wrap

  2. $ curl --globoff -L -w "%{http_code} %{url_effective}\n" -k -u <bmc_username>:<password >-H "Content-Type: application/json" -d '{}' -X DELETE https://<bmc_ip_address>/redfish/v1/EventService/Subscriptions/1
    Copy to Clipboard Toggle word wrap

12.4.3.

  1. 참고

    1. apiVersion: "event.redhat-cne.org/v1alpha1"
      kind: "HardwareEvent"
      metadata:
        name: "hardware-event"
      spec:
        nodeSelector:
          node-role.kubernetes.io/hw-event: "" 
      1
      
        logLevel: "debug" 
      2
      
        msgParserTimeout: "10" 
      3
      Copy to Clipboard Toggle word wrap
      1
      참고

      2
      3
    2. $ oc create -f hardware-event.yaml
      Copy to Clipboard Toggle word wrap
    1. apiVersion: v1
      kind: Secret
      metadata:
        name: redfish-basic-auth
      type: Opaque
      stringData: 
      1
      
        username: <bmc_username>
        password: <bmc_password>
        # BMC host DNS or IP address
        hostaddr: <bmc_host_ip_address>
      Copy to Clipboard Toggle word wrap
      1
    2. $ oc create -f hw-event-bmc-secret.yaml
      Copy to Clipboard Toggle word wrap

12.5.

참고

[
 {
  "id": "ca11ab76-86f9-428c-8d3a-666c24e34d32",
  "endpointUri": "http://localhost:9089/api/ocloudNotifications/v1/dummy",
  "uriLocation": "http://localhost:8089/api/ocloudNotifications/v1/subscriptions/ca11ab76-86f9-428c-8d3a-666c24e34d32",
  "resource": "/cluster/node/openshift-worker-0.openshift.example.com/redfish/event"
 }
]
Copy to Clipboard Toggle word wrap

Expand
표 12.1.
  

{
  "uriLocation": "http://localhost:8089/api/ocloudNotifications/v1/subscriptions",
  "resource": "/cluster/node/openshift-worker-0.openshift.example.com/redfish/event"
}
Copy to Clipboard Toggle word wrap

Expand
표 12.2.
  

{
  "id":"ca11ab76-86f9-428c-8d3a-666c24e34d32",
  "endpointUri":"http://localhost:9089/api/ocloudNotifications/v1/dummy",
  "uriLocation":"http://localhost:8089/api/ocloudNotifications/v1/subscriptions/ca11ab76-86f9-428c-8d3a-666c24e34d32",
  "resource":"/cluster/node/openshift-worker-0.openshift.example.com/redfish/event"
}
Copy to Clipboard Toggle word wrap

OK
Copy to Clipboard Toggle word wrap

12.6.

  1. containers:
      - name: cloud-event-sidecar
        image: cloud-event-sidecar
        args:
          - "--metrics-addr=127.0.0.1:9091"
          - "--store-path=/store"
          - "--transport-host=consumer-events-subscription-service.cloud-events.svc.cluster.local:9043"
          - "--http-event-publishers=ptp-event-publisher-service-NODE_NAME.openshift-ptp.svc.cluster.local:9043" 
    1
    
          - "--api-port=8089"
    Copy to Clipboard Toggle word wrap
    1

  2. apiVersion: v1
    kind: Service
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        service.alpha.openshift.io/serving-cert-secret-name: sidecar-consumer-secret
      name: consumer-events-subscription-service
      namespace: cloud-events
      labels:
        app: consumer-service
    spec:
      ports:
        - name: sub-port
          port: 9043
      selector:
        app: consumer
      clusterIP: None
      sessionAffinity: None
      type: ClusterIP
    Copy to Clipboard Toggle word wrap

13장.

13.1.

13.2.

apiVersion: v1
kind: Pod
metadata:
  generateName: hugepages-volume-
spec:
  containers:
  - securityContext:
      privileged: true
    image: rhel7:latest
    command:
    - sleep
    - inf
    name: example
    volumeMounts:
    - mountPath: /dev/hugepages
      name: hugepage
    resources:
      limits:
        hugepages-2Mi: 100Mi 
1

        memory: "1Gi"
        cpu: "1"
  volumes:
  - name: hugepage
    emptyDir:
      medium: HugePages
Copy to Clipboard Toggle word wrap
1

13.3.

  1. apiVersion: v1
    kind: Pod
    metadata:
      generateName: hugepages-volume-
      labels:
        app: hugepages-example
    spec:
      containers:
      - securityContext:
          capabilities:
            add: [ "IPC_LOCK" ]
        image: rhel7:latest
        command:
        - sleep
        - inf
        name: example
        volumeMounts:
        - mountPath: /dev/hugepages
          name: hugepage
        - mountPath: /etc/podinfo
          name: podinfo
        resources:
          limits:
            hugepages-1Gi: 2Gi
            memory: "1Gi"
            cpu: "1"
          requests:
            hugepages-1Gi: 2Gi
        env:
        - name: REQUESTS_HUGEPAGES_1GI <.>
          valueFrom:
            resourceFieldRef:
              containerName: example
              resource: requests.hugepages-1Gi
      volumes:
      - name: hugepage
        emptyDir:
          medium: HugePages
      - name: podinfo
        downwardAPI:
          items:
            - path: "hugepages_1G_request" <.>
              resourceFieldRef:
                containerName: example
                resource: requests.hugepages-1Gi
                divisor: 1Gi
    Copy to Clipboard Toggle word wrap

  2. $ oc create -f hugepages-volume-pod.yaml
    Copy to Clipboard Toggle word wrap

  1. $ oc exec -it $(oc get pods -l app=hugepages-example -o jsonpath='{.items[0].metadata.name}') \
         -- env | grep REQUESTS_HUGEPAGES_1GI
    Copy to Clipboard Toggle word wrap

    REQUESTS_HUGEPAGES_1GI=2147483648
    Copy to Clipboard Toggle word wrap

  2. $ oc exec -it $(oc get pods -l app=hugepages-example -o jsonpath='{.items[0].metadata.name}') \
         -- cat /etc/podinfo/hugepages_1G_request
    Copy to Clipboard Toggle word wrap

    2
    Copy to Clipboard Toggle word wrap

13.4.

  1. $ oc label node <node_using_hugepages> node-role.kubernetes.io/worker-hp=
    Copy to Clipboard Toggle word wrap
  2. apiVersion: tuned.openshift.io/v1
    kind: Tuned
    metadata:
      name: hugepages 
    1
    
      namespace: openshift-cluster-node-tuning-operator
    spec:
      profile: 
    2
    
      - data: |
          [main]
          summary=Boot time configuration for hugepages
          include=openshift-node
          [bootloader]
          cmdline_openshift_node_hugepages=hugepagesz=2M hugepages=50 
    3
    
        name: openshift-node-hugepages
    
      recommend:
      - machineConfigLabels: 
    4
    
          machineconfiguration.openshift.io/role: "worker-hp"
        priority: 30
        profile: openshift-node-hugepages
    Copy to Clipboard Toggle word wrap
    1
    2
    3
    4
  3. $ oc create -f hugepages-tuned-boottime.yaml
    Copy to Clipboard Toggle word wrap
  4. apiVersion: machineconfiguration.openshift.io/v1
    kind: MachineConfigPool
    metadata:
      name: worker-hp
      labels:
        worker-hp: ""
    spec:
      machineConfigSelector:
        matchExpressions:
          - {key: machineconfiguration.openshift.io/role, operator: In, values: [worker,worker-hp]}
      nodeSelector:
        matchLabels:
          node-role.kubernetes.io/worker-hp: ""
    Copy to Clipboard Toggle word wrap
  5. $ oc create -f hugepages-mcp.yaml
    Copy to Clipboard Toggle word wrap

$ oc get node <node_using_hugepages> -o jsonpath="{.status.allocatable.hugepages-2Mi}"
100Mi
Copy to Clipboard Toggle word wrap
참고

13.5.

  1. apiVersion: tuned.openshift.io/v1
    kind: Tuned
    metadata:
      name: thp-workers-profile
      namespace: openshift-cluster-node-tuning-operator
    spec:
      profile:
      - data: |
          [main]
          summary=Custom tuned profile for OpenShift to turn off THP on worker nodes
          include=openshift-node
    
          [vm]
          transparent_hugepages=never
        name: openshift-thp-never-worker
    
      recommend:
      - match:
        - label: node-role.kubernetes.io/worker
        priority: 25
        profile: openshift-thp-never-worker
    Copy to Clipboard Toggle word wrap
  2. $ oc create -f thp-disable-tuned.yaml
    Copy to Clipboard Toggle word wrap
  3. $ oc get profile -n openshift-cluster-node-tuning-operator
    Copy to Clipboard Toggle word wrap

  • $ cat /sys/kernel/mm/transparent_hugepage/enabled
    Copy to Clipboard Toggle word wrap

    always madvise [never]
    Copy to Clipboard Toggle word wrap

14장.

14.1.

중요

참고

14.2.

참고

15장.

15.1.

참고

참고

15.1.1.

참고

15.1.2.

  1. $ oc label node <node_name> node-role.kubernetes.io/worker-cnf="" 
    1
    Copy to Clipboard Toggle word wrap
    1
    1. apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfigPool
      metadata:
        name: worker-cnf 
      1
      
        labels:
          machineconfiguration.openshift.io/role: worker-cnf 
      2
      
      spec:
        machineConfigSelector:
          matchExpressions:
            - {
                 key: machineconfiguration.openshift.io/role,
                 operator: In,
                 values: [worker, worker-cnf],
              }
        paused: false
        nodeSelector:
          matchLabels:
            node-role.kubernetes.io/worker-cnf: "" 
      3
      Copy to Clipboard Toggle word wrap

      1
      2
      3
    2. $ oc apply -f mcp-worker-cnf.yaml
      Copy to Clipboard Toggle word wrap

      machineconfigpool.machineconfiguration.openshift.io/worker-cnf created
      Copy to Clipboard Toggle word wrap

  • $ oc get mcp
    Copy to Clipboard Toggle word wrap

    NAME         CONFIG                                                 UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
    master       rendered-master-58433c7c3c1b4ed5ffef95234d451490       True      False      False      3              3                   3                     0                      6h46m
    worker       rendered-worker-168f52b168f151e4f853259729b6azc4       True      False      False      2              2                   2                     0                      6h46m
    worker-cnf   rendered-worker-cnf-168f52b168f151e4f853259729b6azc4   True      False      False      1              1                   1                     0                      73s
    Copy to Clipboard Toggle word wrap

15.1.3.

  1. $ oc adm must-gather
    Copy to Clipboard Toggle word wrap

  2. $ tar cvaf must-gather.tar.gz <must_gather_folder> 
    1
    Copy to Clipboard Toggle word wrap
    1
    참고

15.1.4.

중요

  1. $ oc get mcp
    Copy to Clipboard Toggle word wrap

    NAME         CONFIG                                                 UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
    master       rendered-master-58433c8c3c0b4ed5feef95434d455490       True      False      False      3              3                   3                     0                      8h
    worker       rendered-worker-668f56a164f151e4a853229729b6adc4       True      False      False      2              2                   2                     0                      8h
    worker-cnf   rendered-worker-cnf-668f56a164f151e4a853229729b6adc4   True      False      False      1              1                   1                     0                      79m
    Copy to Clipboard Toggle word wrap

  2. $ podman login registry.redhat.io
    Copy to Clipboard Toggle word wrap
    Username: <user_name>
    Password: <password>
    Copy to Clipboard Toggle word wrap
  3. $ podman run --rm --entrypoint performance-profile-creator registry.redhat.io/openshift4/ose-cluster-node-tuning-rhel9-operator:v4.15 -h
    Copy to Clipboard Toggle word wrap

    A tool that automates creation of Performance Profiles
    
    Usage:
      performance-profile-creator [flags]
    
    Flags:
          --disable-ht                        Disable Hyperthreading
      -h, --help                              help for performance-profile-creator
          --info string                       Show cluster information; requires --must-gather-dir-path, ignore the other arguments. [Valid values: log, json] (default "log")
          --mcp-name string                   MCP name corresponding to the target machines (required)
          --must-gather-dir-path string       Must gather directory path (default "must-gather")
          --offlined-cpu-count int            Number of offlined CPUs
          --per-pod-power-management          Enable Per Pod Power Management
          --power-consumption-mode string     The power consumption mode.  [Valid values: default, low-latency, ultra-low-latency] (default "default")
          --profile-name string               Name of the performance profile to be created (default "performance")
          --reserved-cpu-count int            Number of reserved CPUs (required)
          --rt-kernel                         Enable Real Time Kernel (required)
          --split-reserved-cpus-across-numa   Split the Reserved CPUs across NUMA nodes
          --topology-manager-policy string    Kubelet Topology Manager Policy of the performance profile to be created. [Valid values: single-numa-node, best-effort, restricted] (default "restricted")
          --user-level-networking             Run with User level Networking(DPDK) enabled
    Copy to Clipboard Toggle word wrap

  4. $ podman run --entrypoint performance-profile-creator -v <path_to_must_gather>:/must-gather:z registry.redhat.io/openshift4/ose-cluster-node-tuning-rhel9-operator:v4.15 --info log --must-gather-dir-path /must-gather
    Copy to Clipboard Toggle word wrap
    • level=info msg="Cluster info:"
      level=info msg="MCP 'master' nodes:"
      level=info msg=---
      level=info msg="MCP 'worker' nodes:"
      level=info msg="Node: host.example.com (NUMA cells: 1, HT: true)"
      level=info msg="NUMA cell 0 : [0 1 2 3]"
      level=info msg="CPU(s): 4"
      level=info msg="Node: host1.example.com (NUMA cells: 1, HT: true)"
      level=info msg="NUMA cell 0 : [0 1 2 3]"
      level=info msg="CPU(s): 4"
      level=info msg=---
      level=info msg="MCP 'worker-cnf' nodes:"
      level=info msg="Node: host2.example.com (NUMA cells: 1, HT: true)"
      level=info msg="NUMA cell 0 : [0 1 2 3]"
      level=info msg="CPU(s): 4"
      level=info msg=---
      Copy to Clipboard Toggle word wrap

  5. $ podman run --entrypoint performance-profile-creator -v <path_to_must_gather>:/must-gather:z registry.redhat.io/openshift4/ose-cluster-node-tuning-rhel9-operator:v4.15 --mcp-name=worker-cnf --reserved-cpu-count=1 --rt-kernel=true --split-reserved-cpus-across-numa=false --must-gather-dir-path /must-gather --power-consumption-mode=ultra-low-latency --offlined-cpu-count=1 > my-performance-profile.yaml
    Copy to Clipboard Toggle word wrap
    • 참고

      level=info msg="Nodes targeted by worker-cnf MCP are: [worker-2]"
      level=info msg="NUMA cell(s): 1"
      level=info msg="NUMA cell 0 : [0 1 2 3]"
      level=info msg="CPU(s): 4"
      level=info msg="1 reserved CPUs allocated: 0 "
      level=info msg="2 isolated CPUs allocated: 2-3"
      level=info msg="Additional Kernel Args based on configuration: []"
      Copy to Clipboard Toggle word wrap

  6. $ cat my-performance-profile.yaml
    Copy to Clipboard Toggle word wrap

    ---
    apiVersion: performance.openshift.io/v2
    kind: PerformanceProfile
    metadata:
      name: performance
    spec:
      cpu:
        isolated: 2-3
        offlined: "1"
        reserved: "0"
      machineConfigPoolSelector:
        machineconfiguration.openshift.io/role: worker-cnf
      nodeSelector:
        node-role.kubernetes.io/worker-cnf: ""
      numa:
        topologyPolicy: restricted
      realTimeKernel:
        enabled: true
      workloadHints:
        highPowerConsumption: true
        perPodPowerManagement: false
        realTime: true
    Copy to Clipboard Toggle word wrap

  7. $ oc apply -f my-performance-profile.yaml
    Copy to Clipboard Toggle word wrap

    performanceprofile.performance.openshift.io/performance created
    Copy to Clipboard Toggle word wrap

15.1.5.

중요

  1. $ vi run-perf-profile-creator.sh
    Copy to Clipboard Toggle word wrap
  2. #!/bin/bash
    
    readonly CONTAINER_RUNTIME=${CONTAINER_RUNTIME:-podman}
    readonly CURRENT_SCRIPT=$(basename "$0")
    readonly CMD="${CONTAINER_RUNTIME} run --entrypoint performance-profile-creator"
    readonly IMG_EXISTS_CMD="${CONTAINER_RUNTIME} image exists"
    readonly IMG_PULL_CMD="${CONTAINER_RUNTIME} image pull"
    readonly MUST_GATHER_VOL="/must-gather"
    
    NTO_IMG="registry.redhat.io/openshift4/ose-cluster-node-tuning-rhel9-operator:v4.15"
    MG_TARBALL=""
    DATA_DIR=""
    
    usage() {
      print "Wrapper usage:"
      print "  ${CURRENT_SCRIPT} [-h] [-p image][-t path] -- [performance-profile-creator flags]"
      print ""
      print "Options:"
      print "   -h                 help for ${CURRENT_SCRIPT}"
      print "   -p                 Node Tuning Operator image"
      print "   -t                 path to a must-gather tarball"
    
      ${IMG_EXISTS_CMD} "${NTO_IMG}" && ${CMD} "${NTO_IMG}" -h
    }
    
    function cleanup {
      [ -d "${DATA_DIR}" ] && rm -rf "${DATA_DIR}"
    }
    trap cleanup EXIT
    
    exit_error() {
      print "error: $*"
      usage
      exit 1
    }
    
    print() {
      echo  "$*" >&2
    }
    
    check_requirements() {
      ${IMG_EXISTS_CMD} "${NTO_IMG}" || ${IMG_PULL_CMD} "${NTO_IMG}" || \
          exit_error "Node Tuning Operator image not found"
    
      [ -n "${MG_TARBALL}" ] || exit_error "Must-gather tarball file path is mandatory"
      [ -f "${MG_TARBALL}" ] || exit_error "Must-gather tarball file not found"
    
      DATA_DIR=$(mktemp -d -t "${CURRENT_SCRIPT}XXXX") || exit_error "Cannot create the data directory"
      tar -zxf "${MG_TARBALL}" --directory "${DATA_DIR}" || exit_error "Cannot decompress the must-gather tarball"
      chmod a+rx "${DATA_DIR}"
    
      return 0
    }
    
    main() {
      while getopts ':hp:t:' OPT; do
        case "${OPT}" in
          h)
            usage
            exit 0
            ;;
          p)
            NTO_IMG="${OPTARG}"
            ;;
          t)
            MG_TARBALL="${OPTARG}"
            ;;
          ?)
            exit_error "invalid argument: ${OPTARG}"
            ;;
        esac
      done
      shift $((OPTIND - 1))
    
      check_requirements || exit 1
    
      ${CMD} -v "${DATA_DIR}:${MUST_GATHER_VOL}:z" "${NTO_IMG}" "$@" --must-gather-dir-path "${MUST_GATHER_VOL}"
      echo "" 1>&2
    }
    
    main "$@"
    Copy to Clipboard Toggle word wrap
  3. $ chmod a+x run-perf-profile-creator.sh
    Copy to Clipboard Toggle word wrap
  4. $ podman login registry.redhat.io
    Copy to Clipboard Toggle word wrap
    Username: <user_name>
    Password: <password>
    Copy to Clipboard Toggle word wrap
  5. $ ./run-perf-profile-creator.sh -h
    Copy to Clipboard Toggle word wrap

    Wrapper usage:
      run-perf-profile-creator.sh [-h] [-p image][-t path] -- [performance-profile-creator flags]
    
    Options:
       -h                 help for run-perf-profile-creator.sh
       -p                 Node Tuning Operator image
       -t                 path to a must-gather tarball
    A tool that automates creation of Performance Profiles
    
    Usage:
      performance-profile-creator [flags]
    
    Flags:
          --disable-ht                        Disable Hyperthreading
      -h, --help                              help for performance-profile-creator
          --info string                       Show cluster information; requires --must-gather-dir-path, ignore the other arguments. [Valid values: log, json] (default "log")
          --mcp-name string                   MCP name corresponding to the target machines (required)
          --must-gather-dir-path string       Must gather directory path (default "must-gather")
          --offlined-cpu-count int            Number of offlined CPUs
          --per-pod-power-management          Enable Per Pod Power Management
          --power-consumption-mode string     The power consumption mode.  [Valid values: default, low-latency, ultra-low-latency] (default "default")
          --profile-name string               Name of the performance profile to be created (default "performance")
          --reserved-cpu-count int            Number of reserved CPUs (required)
          --rt-kernel                         Enable Real Time Kernel (required)
          --split-reserved-cpus-across-numa   Split the Reserved CPUs across NUMA nodes
          --topology-manager-policy string    Kubelet Topology Manager Policy of the performance profile to be created. [Valid values: single-numa-node, best-effort, restricted] (default "restricted")
          --user-level-networking             Run with User level Networking(DPDK) enabled
    Copy to Clipboard Toggle word wrap

    참고

  6. $ ./run-perf-profile-creator.sh -t /<path_to_must_gather_dir>/must-gather.tar.gz -- --info=log
    Copy to Clipboard Toggle word wrap
    • level=info msg="Cluster info:"
      level=info msg="MCP 'master' nodes:"
      level=info msg=---
      level=info msg="MCP 'worker' nodes:"
      level=info msg="Node: host.example.com (NUMA cells: 1, HT: true)"
      level=info msg="NUMA cell 0 : [0 1 2 3]"
      level=info msg="CPU(s): 4"
      level=info msg="Node: host1.example.com (NUMA cells: 1, HT: true)"
      level=info msg="NUMA cell 0 : [0 1 2 3]"
      level=info msg="CPU(s): 4"
      level=info msg=---
      level=info msg="MCP 'worker-cnf' nodes:"
      level=info msg="Node: host2.example.com (NUMA cells: 1, HT: true)"
      level=info msg="NUMA cell 0 : [0 1 2 3]"
      level=info msg="CPU(s): 4"
      level=info msg=---
      Copy to Clipboard Toggle word wrap

  7. $ ./run-perf-profile-creator.sh -t /path-to-must-gather/must-gather.tar.gz -- --mcp-name=worker-cnf --reserved-cpu-count=1 --rt-kernel=true --split-reserved-cpus-across-numa=false --power-consumption-mode=ultra-low-latency --offlined-cpu-count=1 > my-performance-profile.yaml
    Copy to Clipboard Toggle word wrap

    • 참고

  8. $ cat my-performance-profile.yaml
    Copy to Clipboard Toggle word wrap

    ---
    apiVersion: performance.openshift.io/v2
    kind: PerformanceProfile
    metadata:
      name: performance
    spec:
      cpu:
        isolated: 2-3
        offlined: "1"
        reserved: "0"
      machineConfigPoolSelector:
        machineconfiguration.openshift.io/role: worker-cnf
      nodeSelector:
        node-role.kubernetes.io/worker-cnf: ""
      numa:
        topologyPolicy: restricted
      realTimeKernel:
        enabled: true
      workloadHints:
        highPowerConsumption: true
        perPodPowerManagement: false
        realTime: true
    Copy to Clipboard Toggle word wrap

  9. $ oc apply -f my-performance-profile.yaml
    Copy to Clipboard Toggle word wrap

    performanceprofile.performance.openshift.io/performance created
    Copy to Clipboard Toggle word wrap

15.1.6.

Expand
표 15.1.
  

Expand
표 15.2.
  

주의

참고

Error: failed to compute the reserved and isolated CPUs: please ensure that reserved-cpu-count plus offlined-cpu-count should be in the range [0,1]
Copy to Clipboard Toggle word wrap
Error: failed to compute the reserved and isolated CPUs: please specify the offlined CPU count in the range [0,1]
Copy to Clipboard Toggle word wrap

15.1.7.

15.1.7.1.

apiVersion: performance.openshift.io/v2
kind: PerformanceProfile
metadata:
  name: cnf-performanceprofile
spec:
  additionalKernelArgs:
    - nmi_watchdog=0
    - audit=0
    - mce=off
    - processor.max_cstate=1
    - idle=poll
    - intel_idle.max_cstate=0
    - default_hugepagesz=1GB
    - hugepagesz=1G
    - intel_iommu=on
  cpu:
    isolated: <CPU_ISOLATED>
    reserved: <CPU_RESERVED>
  hugepages:
    defaultHugepagesSize: 1G
    pages:
      - count: <HUGEPAGES_COUNT>
        node: 0
        size: 1G
  nodeSelector:
    node-role.kubernetes.io/worker: ''
  globallyDisableIrqLoadBalancing: true
  realTimeKernel:
    enabled: false
Copy to Clipboard Toggle word wrap

15.1.7.2.

apiVersion: performance.openshift.io/v2
kind: PerformanceProfile
metadata:
  # if you change this name make sure the 'include' line in TunedPerformancePatch.yaml
  # matches this name: include=openshift-node-performance-${PerformanceProfile.metadata.name}
  # Also in file 'validatorCRs/informDuValidator.yaml':
  # name: 50-performance-${PerformanceProfile.metadata.name}
  name: openshift-node-performance-profile
  annotations:
    ran.openshift.io/reference-configuration: "ran-du.redhat.com"
spec:
  additionalKernelArgs:
    - "rcupdate.rcu_normal_after_boot=0"
    - "efi=runtime"
    - "vfio_pci.enable_sriov=1"
    - "vfio_pci.disable_idle_d3=1"
    - "module_blacklist=irdma"
  cpu:
    isolated: $isolated
    reserved: $reserved
  hugepages:
    defaultHugepagesSize: $defaultHugepagesSize
    pages:
      - size: $size
        count: $count
        node: $node
  machineConfigPoolSelector:
    pools.operator.machineconfiguration.openshift.io/$mcp: ""
  nodeSelector:
    node-role.kubernetes.io/$mcp: ''
  numa:
    topologyPolicy: "restricted"
  # To use the standard (non-realtime) kernel, set enabled to false
  realTimeKernel:
    enabled: true
  workloadHints:
    # WorkloadHints defines the set of upper level flags for different type of workloads.
    # See https://github.com/openshift/cluster-node-tuning-operator/blob/master/docs/performanceprofile/performance_profile.md#workloadhints
    # for detailed descriptions of each item.
    # The configuration below is set for a low latency, performance mode.
    realTime: true
    highPowerConsumption: false
    perPodPowerManagement: false
Copy to Clipboard Toggle word wrap

15.1.7.3.

apiVersion: performance.openshift.io/v2
kind: PerformanceProfile
metadata:
  # if you change this name make sure the 'include' line in TunedPerformancePatch.yaml
  # matches this name: include=openshift-node-performance-${PerformanceProfile.metadata.name}
  # Also in file 'validatorCRs/informDuValidator.yaml':
  # name: 50-performance-${PerformanceProfile.metadata.name}
  name: openshift-node-performance-profile
  annotations:
    ran.openshift.io/reference-configuration: "ran-du.redhat.com"
spec:
  additionalKernelArgs:
    - "rcupdate.rcu_normal_after_boot=0"
    - "efi=runtime"
    - "vfio_pci.enable_sriov=1"
    - "vfio_pci.disable_idle_d3=1"
    - "module_blacklist=irdma"
  cpu:
    isolated: $isolated
    reserved: $reserved
  hugepages:
    defaultHugepagesSize: $defaultHugepagesSize
    pages:
      - size: $size
        count: $count
        node: $node
  machineConfigPoolSelector:
    pools.operator.machineconfiguration.openshift.io/$mcp: ""
  nodeSelector:
    node-role.kubernetes.io/$mcp: ''
  numa:
    topologyPolicy: "restricted"
  # To use the standard (non-realtime) kernel, set enabled to false
  realTimeKernel:
    enabled: true
  workloadHints:
    # WorkloadHints defines the set of upper level flags for different type of workloads.
    # See https://github.com/openshift/cluster-node-tuning-operator/blob/master/docs/performanceprofile/performance_profile.md#workloadhints
    # for detailed descriptions of each item.
    # The configuration below is set for a low latency, performance mode.
    realTime: true
    highPowerConsumption: false
    perPodPowerManagement: false
Copy to Clipboard Toggle word wrap

15.2.

참고

15.3.

Expand
표 15.3.
    

workloadHints:
highPowerConsumption: false
realTime: false
Copy to Clipboard Toggle word wrap

workloadHints:
highPowerConsumption: false
realTime: true
Copy to Clipboard Toggle word wrap

workloadHints:
highPowerConsumption: true
realTime: true
Copy to Clipboard Toggle word wrap

workloadHints:
realTime: true
highPowerConsumption: false
perPodPowerManagement: true
Copy to Clipboard Toggle word wrap

    apiVersion: performance.openshift.io/v2
    kind: PerformanceProfile
    metadata:
      name: workload-hints
    spec:
      ...
      workloadHints:
        realTime: true
        highPowerConsumption: false
        perPodPowerManagement: false 
1
Copy to Clipboard Toggle word wrap
1
참고

15.4.

중요

  1. $ podman run --entrypoint performance-profile-creator -v \
    /must-gather:/must-gather:z registry.redhat.io/openshift4/ose-cluster-node-tuning-rhel9-operator:v4.15 \
    --mcp-name=worker-cnf --reserved-cpu-count=20 --rt-kernel=true \
    --split-reserved-cpus-across-numa=false --topology-manager-policy=single-numa-node \
    --must-gather-dir-path /must-gather --power-consumption-mode=low-latency \ 
    1
    
    --per-pod-power-management=true > my-performance-profile.yaml
    Copy to Clipboard Toggle word wrap
    1

    apiVersion: performance.openshift.io/v2
    kind: PerformanceProfile
    metadata:
         name: performance
    spec:
        [.....]
        workloadHints:
            realTime: true
            highPowerConsumption: false
            perPodPowerManagement: true
    Copy to Clipboard Toggle word wrap

  2. apiVersion: performance.openshift.io/v2
    kind: PerformanceProfile
    metadata:
         name: performance
    spec:
        ...
        additionalKernelArgs:
        - cpufreq.default_governor=schedutil 
    1
    Copy to Clipboard Toggle word wrap
    1
  3. spec:
      profile:
      - data: |
          [sysfs]
          /sys/devices/system/cpu/intel_pstate/max_perf_pct = <x> 
    1
    Copy to Clipboard Toggle word wrap
    1

15.5.

Expand
표 15.4.
  

중요

  1. apiVersion: performance.openshift.io/v2
    kind: PerformanceProfile
    metadata:
      name: infra-cpus
    spec:
      cpu:
        reserved: "0-4,9" 
    1
    
        isolated: "5-8" 
    2
    
      nodeSelector: 
    3
    
        node-role.kubernetes.io/worker: ""
    Copy to Clipboard Toggle word wrap
    1
    2
    3

15.6.

참고

주의

  1. $ lscpu --all --extended
    Copy to Clipboard Toggle word wrap

    CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE MAXMHZ    MINMHZ
    0   0    0      0    0:0:0:0       yes    4800.0000 400.0000
    1   0    0      1    1:1:1:0       yes    4800.0000 400.0000
    2   0    0      2    2:2:2:0       yes    4800.0000 400.0000
    3   0    0      3    3:3:3:0       yes    4800.0000 400.0000
    4   0    0      0    0:0:0:0       yes    4800.0000 400.0000
    5   0    0      1    1:1:1:0       yes    4800.0000 400.0000
    6   0    0      2    2:2:2:0       yes    4800.0000 400.0000
    7   0    0      3    3:3:3:0       yes    4800.0000 400.0000
    Copy to Clipboard Toggle word wrap

    $ cat /sys/devices/system/cpu/cpu0/topology/thread_siblings_list
    Copy to Clipboard Toggle word wrap

    0-4
    Copy to Clipboard Toggle word wrap

  2. ...
      cpu:
        isolated: 0,4
        reserved: 1-3,5-7
    ...
    Copy to Clipboard Toggle word wrap
    참고

중요

15.6.1.

  1. apiVersion: performance.openshift.io/v2
    kind: PerformanceProfile
    metadata:
      name: example-performanceprofile
    spec:
      additionalKernelArgs:
        - nmi_watchdog=0
        - audit=0
        - mce=off
        - processor.max_cstate=1
        - idle=poll
        - intel_idle.max_cstate=0
        - nosmt
      cpu:
        isolated: 2-3
        reserved: 0-1
      hugepages:
        defaultHugepagesSize: 1G
        pages:
          - count: 2
            node: 0
            size: 1G
      nodeSelector:
        node-role.kubernetes.io/performance: ''
      realTimeKernel:
        enabled: true
    Copy to Clipboard Toggle word wrap
    참고

15.7.

15.7.1.

참고

$ find /proc/irq -name effective_affinity -printf "%p: " -exec cat {} \;
Copy to Clipboard Toggle word wrap

/proc/irq/0/effective_affinity: 1
/proc/irq/1/effective_affinity: 8
/proc/irq/2/effective_affinity: 0
/proc/irq/3/effective_affinity: 1
/proc/irq/4/effective_affinity: 2
/proc/irq/5/effective_affinity: 1
/proc/irq/6/effective_affinity: 1
/proc/irq/7/effective_affinity: 1
/proc/irq/8/effective_affinity: 1
/proc/irq/9/effective_affinity: 2
/proc/irq/10/effective_affinity: 1
/proc/irq/11/effective_affinity: 1
/proc/irq/12/effective_affinity: 4
/proc/irq/13/effective_affinity: 1
/proc/irq/14/effective_affinity: 1
/proc/irq/15/effective_affinity: 1
/proc/irq/24/effective_affinity: 2
/proc/irq/25/effective_affinity: 4
/proc/irq/26/effective_affinity: 2
/proc/irq/27/effective_affinity: 1
/proc/irq/28/effective_affinity: 8
/proc/irq/29/effective_affinity: 4
/proc/irq/30/effective_affinity: 4
/proc/irq/31/effective_affinity: 8
/proc/irq/32/effective_affinity: 8
/proc/irq/33/effective_affinity: 1
/proc/irq/34/effective_affinity: 2
Copy to Clipboard Toggle word wrap

15.7.2.

  1. apiVersion: performance.openshift.io/v2
    kind: PerformanceProfile
    metadata:
      name: dynamic-irq-profile
    spec:
      cpu:
        isolated: 2-5
        reserved: 0-1
    ...
    Copy to Clipboard Toggle word wrap
    참고

15.8.

hugepages:
   defaultHugepagesSize: "1G"
   pages:
   - size:  "1G"
     count:  4
     node:  0 
1
Copy to Clipboard Toggle word wrap
1
참고

  • $ oc debug node/ip-10-0-141-105.ec2.internal
    Copy to Clipboard Toggle word wrap
    # grep -i huge /proc/meminfo
    Copy to Clipboard Toggle word wrap

    AnonHugePages:    ###### ##
    ShmemHugePages:        0 kB
    HugePages_Total:       2
    HugePages_Free:        2
    HugePages_Rsvd:        0
    HugePages_Surp:        0
    Hugepagesize:       #### ##
    Hugetlb:            #### ##
    Copy to Clipboard Toggle word wrap

  • $ oc describe node worker-0.ocp4poc.example.com | grep -i huge
    Copy to Clipboard Toggle word wrap

                                       hugepages-1g=true
     hugepages-###:  ###
     hugepages-###:  ###
    Copy to Clipboard Toggle word wrap

15.8.1.

spec:
  hugepages:
    defaultHugepagesSize: 1G
    pages:
    - count: 1024
      node: 0
      size: 2M
    - count: 4
      node: 1
      size: 1G
Copy to Clipboard Toggle word wrap

15.9.

15.9.1.

  1. $ oc edit -f <your_profile_name>.yaml
    Copy to Clipboard Toggle word wrap
      • 참고

  2. apiVersion: performance.openshift.io/v2
    kind: PerformanceProfile
    metadata:
      name: manual
    spec:
      cpu:
        isolated: 3-51,55-103
        reserved: 0-2,52-54
      net:
        userLevelNetworking: true
      nodeSelector:
        node-role.kubernetes.io/worker-cnf: ""
    Copy to Clipboard Toggle word wrap
  3. apiVersion: performance.openshift.io/v2
    kind: PerformanceProfile
    metadata:
      name: manual
    spec:
      cpu:
        isolated: 3-51,55-103
        reserved: 0-2,52-54
      net:
        userLevelNetworking: true
        devices:
        - interfaceName: "eth0"
        - interfaceName: "eth1"
        - vendorID: "0x1af4"
          deviceID: "0x1000"
      nodeSelector:
        node-role.kubernetes.io/worker-cnf: ""
    Copy to Clipboard Toggle word wrap
  4. apiVersion: performance.openshift.io/v2
    kind: PerformanceProfile
    metadata:
      name: manual
    spec:
      cpu:
        isolated: 3-51,55-103
        reserved: 0-2,52-54
      net:
        userLevelNetworking: true
        devices:
        - interfaceName: "eth*"
      nodeSelector:
        node-role.kubernetes.io/worker-cnf: ""
    Copy to Clipboard Toggle word wrap
  5. apiVersion: performance.openshift.io/v2
    kind: PerformanceProfile
    metadata:
      name: manual
    spec:
      cpu:
        isolated: 3-51,55-103
        reserved: 0-2,52-54
      net:
        userLevelNetworking: true
        devices:
        - interfaceName: "!eno1"
      nodeSelector:
        node-role.kubernetes.io/worker-cnf: ""
    Copy to Clipboard Toggle word wrap
  6. apiVersion: performance.openshift.io/v2
    kind: PerformanceProfile
    metadata:
      name: manual
    spec:
      cpu:
        isolated: 3-51,55-103
        reserved: 0-2,52-54
      net:
        userLevelNetworking: true
        devices:
        - interfaceName: "eth0"
        - vendorID: "0x1af4"
          deviceID: "0x1000"
      nodeSelector:
        node-role.kubernetes.io/worker-cnf: ""
    Copy to Clipboard Toggle word wrap
  7. $ oc apply -f <your_profile_name>.yaml
    Copy to Clipboard Toggle word wrap

15.9.2.

apiVersion: performance.openshift.io/v2
metadata:
  name: performance
spec:
  kind: PerformanceProfile
  spec:
    cpu:
      reserved: 0-1  #total = 2
      isolated: 2-8
    net:
      userLevelNetworking: true
# ...
Copy to Clipboard Toggle word wrap
  • 참고

    $ ethtool -l <device>
    Copy to Clipboard Toggle word wrap
  • $ ethtool -l ens4
    Copy to Clipboard Toggle word wrap

    Channel parameters for ens4:
    Pre-set maximums:
    RX:         0
    TX:         0
    Other:      0
    Combined:   4
    Current hardware settings:
    RX:         0
    TX:         0
    Other:      0
    Combined:   4
    Copy to Clipboard Toggle word wrap

  • $ ethtool -l ens4
    Copy to Clipboard Toggle word wrap

    Channel parameters for ens4:
    Pre-set maximums:
    RX:         0
    TX:         0
    Other:      0
    Combined:   4
    Current hardware settings:
    RX:         0
    TX:         0
    Other:      0
    Combined:   2 
    1
    Copy to Clipboard Toggle word wrap

1

apiVersion: performance.openshift.io/v2
metadata:
  name: performance
spec:
  kind: PerformanceProfile
  spec:
    cpu:
      reserved: 0-1  #total = 2
      isolated: 2-8
    net:
      userLevelNetworking: true
      devices:
      - vendorID = 0x1af4
# ...
Copy to Clipboard Toggle word wrap
  • 참고

    $ ethtool -l <device>
    Copy to Clipboard Toggle word wrap
  • $ ethtool -l ens4
    Copy to Clipboard Toggle word wrap

    Channel parameters for ens4:
    Pre-set maximums:
    RX:         0
    TX:         0
    Other:      0
    Combined:   4
    Current hardware settings:
    RX:         0
    TX:         0
    Other:      0
    Combined:   2 
    1
    Copy to Clipboard Toggle word wrap

1

# udevadm info -p /sys/class/net/ens4
...
E: ID_MODEL_ID=0x1000
E: ID_VENDOR_ID=0x1af4
E: INTERFACE=ens4
...
Copy to Clipboard Toggle word wrap
# udevadm info -p /sys/class/net/eth0
...
E: ID_MODEL_ID=0x1002
E: ID_VENDOR_ID=0x1001
E: INTERFACE=eth0
...
Copy to Clipboard Toggle word wrap
  • apiVersion: performance.openshift.io/v2
    metadata:
      name: performance
    spec:
      kind: PerformanceProfile
        spec:
          cpu:
            reserved: 0-1  #total = 2
            isolated: 2-8
          net:
            userLevelNetworking: true
            devices:
            - interfaceName = eth0
            - vendorID = 0x1af4
    ...
    Copy to Clipboard Toggle word wrap
  • $ ethtool -l ens4
    Copy to Clipboard Toggle word wrap

    Channel parameters for ens4:
    Pre-set maximums:
    RX:         0
    TX:         0
    Other:      0
    Combined:   4
    Current hardware settings:
    RX:         0
    TX:         0
    Other:      0
    Combined:   2 
    1
    Copy to Clipboard Toggle word wrap

    1

15.9.3.

  • INFO tuned.plugins.base: instance net_test (net): assigning devices ens1, ens2, ens3
    Copy to Clipboard Toggle word wrap
  • WARNING  tuned.plugins.base: instance net_test: no matching devices available
    Copy to Clipboard Toggle word wrap

16장.

참고

16.1.

참고

  1. apiVersion: v1
    kind: Pod
    metadata:
      name: dynamic-low-latency-pod
      annotations:
        cpu-quota.crio.io: "disable" 
    1
    
        cpu-load-balancing.crio.io: "disable" 
    2
    
        irq-load-balancing.crio.io: "disable" 
    3
    
    spec:
      securityContext:
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      containers:
      - name: dynamic-low-latency-pod
        image: "registry.redhat.io/openshift4/cnf-tests-rhel8:v4.15"
        command: ["sleep", "10h"]
        resources:
          requests:
            cpu: 2
            memory: "200M"
          limits:
            cpu: 2
            memory: "200M"
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop: [ALL]
      nodeSelector:
        node-role.kubernetes.io/worker-cnf: "" 
    4
    
      runtimeClassName: performance-dynamic-low-latency-profile 
    5
    
    # ...
    Copy to Clipboard Toggle word wrap

    1
    2
    3
    4
    5
  2. $ oc get pod -o wide
    Copy to Clipboard Toggle word wrap

    NAME                     READY   STATUS    RESTARTS   AGE     IP           NODE
    dynamic-low-latency-pod  1/1     Running   0          5h33m   10.131.0.10  cnf-worker.example.com
    Copy to Clipboard Toggle word wrap

  3. $ oc exec -it dynamic-low-latency-pod -- /bin/bash -c "grep Cpus_allowed_list /proc/self/status | awk '{print $2}'"
    Copy to Clipboard Toggle word wrap

    Cpus_allowed_list:  2-3
    Copy to Clipboard Toggle word wrap

  1. $ oc debug node/<node-name>
    Copy to Clipboard Toggle word wrap
  2. sh-4.4# chroot /host
    Copy to Clipboard Toggle word wrap

    sh-4.4#
    Copy to Clipboard Toggle word wrap

  3. sh-4.4# cat /proc/irq/default_smp_affinity
    Copy to Clipboard Toggle word wrap

    33
    Copy to Clipboard Toggle word wrap

  4. sh-4.4# find /proc/irq/ -name smp_affinity_list -exec sh -c 'i="$1"; mask=$(cat $i); file=$(echo $i); echo $file: $mask' _ {} \;
    Copy to Clipboard Toggle word wrap

    /proc/irq/0/smp_affinity_list: 0-5
    /proc/irq/1/smp_affinity_list: 5
    /proc/irq/2/smp_affinity_list: 0-5
    /proc/irq/3/smp_affinity_list: 0-5
    /proc/irq/4/smp_affinity_list: 0
    /proc/irq/5/smp_affinity_list: 0-5
    /proc/irq/6/smp_affinity_list: 0-5
    /proc/irq/7/smp_affinity_list: 0-5
    /proc/irq/8/smp_affinity_list: 4
    /proc/irq/9/smp_affinity_list: 4
    /proc/irq/10/smp_affinity_list: 0-5
    /proc/irq/11/smp_affinity_list: 0
    /proc/irq/12/smp_affinity_list: 1
    /proc/irq/13/smp_affinity_list: 0-5
    /proc/irq/14/smp_affinity_list: 1
    /proc/irq/15/smp_affinity_list: 0
    /proc/irq/24/smp_affinity_list: 1
    /proc/irq/25/smp_affinity_list: 1
    /proc/irq/26/smp_affinity_list: 1
    /proc/irq/27/smp_affinity_list: 5
    /proc/irq/28/smp_affinity_list: 1
    /proc/irq/29/smp_affinity_list: 0
    /proc/irq/30/smp_affinity_list: 0-5
    Copy to Clipboard Toggle word wrap

주의

16.2.

apiVersion: v1
kind: Pod
metadata:
  name: qos-demo
  namespace: qos-example
spec:
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: qos-demo-ctr
    image: <image-pull-spec>
    resources:
      limits:
        memory: "200Mi"
        cpu: "1"
      requests:
        memory: "200Mi"
        cpu: "1"
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop: [ALL]
Copy to Clipboard Toggle word wrap
  1. $ oc  apply -f qos-pod.yaml --namespace=qos-example
    Copy to Clipboard Toggle word wrap
  2. $ oc get pod qos-demo --namespace=qos-example --output=yaml
    Copy to Clipboard Toggle word wrap

    spec:
      containers:
        ...
    status:
      qosClass: Guaranteed
    Copy to Clipboard Toggle word wrap

    참고

16.3.

  • apiVersion: performance.openshift.io/v2
    kind: PerformanceProfile
    ...
    status:
      ...
      runtimeClass: performance-manual
    Copy to Clipboard Toggle word wrap
참고

apiVersion: v1
kind: Pod
metadata:
  #...
  annotations:
    #...
    cpu-load-balancing.crio.io: "disable"
    #...
  #...
spec:
  #...
  runtimeClassName: performance-<profile_name>
  #...
Copy to Clipboard Toggle word wrap
참고

16.4.

Expand
표 16.1.
   

  1. apiVersion: v1
    kind: Pod
    metadata:
      #...
      annotations:
        #...
        cpu-c-states.crio.io: "disable"
        cpu-freq-governor.crio.io: "performance"
        #...
      #...
    spec:
      #...
      runtimeClassName: performance-<profile_name>
      #...
    Copy to Clipboard Toggle word wrap

16.5.

apiVersion: v1
kind: Pod
metadata:
  annotations:
      cpu-quota.crio.io: "disable"
spec:
    runtimeClassName: performance-<profile_name>
#...
Copy to Clipboard Toggle word wrap

참고

16.6.

apiVersion: performance.openshift.io/v2
kind: Pod
metadata:
  annotations:
      irq-load-balancing.crio.io: "disable"
spec:
    runtimeClassName: performance-<profile_name>
...
Copy to Clipboard Toggle word wrap

17장.

17.1.

Status:
  Conditions:
    Last Heartbeat Time:   2020-06-02T10:01:24Z
    Last Transition Time:  2020-06-02T10:01:24Z
    Status:                True
    Type:                  Available
    Last Heartbeat Time:   2020-06-02T10:01:24Z
    Last Transition Time:  2020-06-02T10:01:24Z
    Status:                True
    Type:                  Upgradeable
    Last Heartbeat Time:   2020-06-02T10:01:24Z
    Last Transition Time:  2020-06-02T10:01:24Z
    Status:                False
    Type:                  Progressing
    Last Heartbeat Time:   2020-06-02T10:01:24Z
    Last Transition Time:  2020-06-02T10:01:24Z
    Status:                False
    Type:                  Degraded
Copy to Clipboard Toggle word wrap

17.1.1.

  1. # oc get mcp
    Copy to Clipboard Toggle word wrap

    NAME         CONFIG                                                 UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
    master       rendered-master-2ee57a93fa6c9181b546ca46e1571d2d       True      False      False      3              3                   3                     0                      2d21h
    worker       rendered-worker-d6b2bdc07d9f5a59a6b68950acf25e5f       True      False      False      2              2                   2                     0                      2d21h
    worker-cnf   rendered-worker-cnf-6c838641b8a08fff08dbd8b02fb63f7c   False     True       True       2              1                   1                     1                      2d20h
    Copy to Clipboard Toggle word wrap

  2. # oc describe mcp worker-cnf
    Copy to Clipboard Toggle word wrap

      Message:               Node node-worker-cnf is reporting: "prepping update:
      machineconfig.machineconfiguration.openshift.io \"rendered-worker-cnf-40b9996919c08e335f3ff230ce1d170\" not
      found"
        Reason:                1 nodes are reporting degraded status on sync
    Copy to Clipboard Toggle word wrap

  3. # oc describe performanceprofiles performance
    Copy to Clipboard Toggle word wrap

    Message: Machine config pool worker-cnf Degraded Reason: 1 nodes are reporting degraded status on sync.
    Machine config pool worker-cnf Degraded Message: Node yquinn-q8s5v-w-b-z5lqn.c.openshift-gce-devel.internal is
    reporting: "prepping update: machineconfig.machineconfiguration.openshift.io
    \"rendered-worker-cnf-40b9996919c08e335f3ff230ce1d170\" not found".    Reason:  MCPDegraded
       Status:  True
       Type:    Degraded
    Copy to Clipboard Toggle word wrap

17.2.

17.2.1.

17.2.2.

  1. $ oc adm must-gather
    Copy to Clipboard Toggle word wrap

    [must-gather      ] OUT Using must-gather plug-in image: quay.io/openshift-release
    When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information:
    ClusterID: 829er0fa-1ad8-4e59-a46e-2644921b7eb6
    ClusterVersion: Stable at "<cluster_version>"
    ClusterOperators:
    	All healthy and stable
    
    
    [must-gather      ] OUT namespace/openshift-must-gather-8fh4x created
    [must-gather      ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-rhlgc created
    [must-gather-5564g] POD 2023-07-17T10:17:37.610340849Z Gathering data for ns/openshift-cluster-version...
    [must-gather-5564g] POD 2023-07-17T10:17:38.786591298Z Gathering data for ns/default...
    [must-gather-5564g] POD 2023-07-17T10:17:39.117418660Z Gathering data for ns/openshift...
    [must-gather-5564g] POD 2023-07-17T10:17:39.447592859Z Gathering data for ns/kube-system...
    [must-gather-5564g] POD 2023-07-17T10:17:39.803381143Z Gathering data for ns/openshift-etcd...
    
    ...
    
    Reprinting Cluster State:
    When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information:
    ClusterID: 829er0fa-1ad8-4e59-a46e-2644921b7eb6
    ClusterVersion: Stable at "<cluster_version>"
    ClusterOperators:
    	All healthy and stable
    Copy to Clipboard Toggle word wrap

  2. $ tar cvaf must-gather.tar.gz must-gather-local.5421342344627712289
    1
    Copy to Clipboard Toggle word wrap
    1
    참고

18장.

18.1.

18.2.

Expand
표 18.1.
  

참고

참고

18.3.

참고

  1. 참고

  2. $ podman run -v $(pwd)/:/kubeconfig:Z -e KUBECONFIG=/kubeconfig/kubeconfig \
    -e LATENCY_TEST_RUNTIME=600\
    -e MAXIMUM_LATENCY=20 \
    registry.redhat.io/openshift4/cnf-tests-rhel8:v4.15 /usr/bin/test-run.sh \
    --ginkgo.v --ginkgo.timeout="24h"
    Copy to Clipboard Toggle word wrap

  3. 중요

18.3.1.

참고

  • $ podman run -v $(pwd)/:/kubeconfig:Z -e KUBECONFIG=/kubeconfig/kubeconfig \
    -e LATENCY_TEST_RUNTIME=600 -e MAXIMUM_LATENCY=20 \
    registry.redhat.io/openshift4/cnf-tests-rhel8:v4.15 \
    /usr/bin/test-run.sh --ginkgo.focus="hwlatdetect" --ginkgo.v --ginkgo.timeout="24h"
    Copy to Clipboard Toggle word wrap

    중요

    running /usr/bin/cnftests -ginkgo.v -ginkgo.focus=hwlatdetect
    I0908 15:25:20.023712      27 request.go:601] Waited for 1.046586367s due to client-side throttling, not priority and fairness, request: GET:https://api.hlxcl6.lab.eng.tlv2.redhat.com:6443/apis/imageregistry.operator.openshift.io/v1?timeout=32s
    Running Suite: CNF Features e2e integration tests
    =================================================
    Random Seed: 1662650718
    Will run 1 of 3 specs
    
    [...]
    
    • Failure [283.574 seconds]
    [performance] Latency Test
    /remote-source/app/vendor/github.com/openshift/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/4_latency/latency.go:62
      with the hwlatdetect image
      /remote-source/app/vendor/github.com/openshift/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/4_latency/latency.go:228
        should succeed [It]
        /remote-source/app/vendor/github.com/openshift/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/4_latency/latency.go:236
    
        Log file created at: 2022/09/08 15:25:27
        Running on machine: hwlatdetect-b6n4n
        Binary: Built with gc go1.17.12 for linux/amd64
        Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
        I0908 15:25:27.160620       1 node.go:39] Environment information: /proc/cmdline: BOOT_IMAGE=(hd1,gpt3)/ostree/rhcos-c6491e1eedf6c1f12ef7b95e14ee720bf48359750ac900b7863c625769ef5fb9/vmlinuz-4.18.0-372.19.1.el8_6.x86_64 random.trust_cpu=on console=tty0 console=ttyS0,115200n8 ignition.platform.id=metal ostree=/ostree/boot.1/rhcos/c6491e1eedf6c1f12ef7b95e14ee720bf48359750ac900b7863c625769ef5fb9/0 ip=dhcp root=UUID=5f80c283-f6e6-4a27-9b47-a287157483b2 rw rootflags=prjquota boot=UUID=773bf59a-bafd-48fc-9a87-f62252d739d3 skew_tick=1 nohz=on rcu_nocbs=0-3 tuned.non_isolcpus=0000ffff,ffffffff,fffffff0 systemd.cpu_affinity=4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79 intel_iommu=on iommu=pt isolcpus=managed_irq,0-3 nohz_full=0-3 tsc=nowatchdog nosoftlockup nmi_watchdog=0 mce=off skew_tick=1 rcutree.kthread_prio=11 + +
        I0908 15:25:27.160830       1 node.go:46] Environment information: kernel version 4.18.0-372.19.1.el8_6.x86_64
        I0908 15:25:27.160857       1 main.go:50] running the hwlatdetect command with arguments [/usr/bin/hwlatdetect --threshold 1 --hardlimit 1 --duration 100 --window 10000000us --width 950000us]
        F0908 15:27:10.603523       1 main.go:53] failed to run hwlatdetect command; out: hwlatdetect:  test duration 100 seconds
           detector: tracer
           parameters:
                Latency threshold: 1us 
    1
    
                Sample window:     10000000us
                Sample width:      950000us
             Non-sampling period:  9050000us
                Output File:       None
    
        Starting test
        test finished
        Max Latency: 326us 
    2
    
        Samples recorded: 5
        Samples exceeding threshold: 5
        ts: 1662650739.017274507, inner:6, outer:6
        ts: 1662650749.257272414, inner:14, outer:326
        ts: 1662650779.977272835, inner:314, outer:12
        ts: 1662650800.457272384, inner:3, outer:9
        ts: 1662650810.697273520, inner:3, outer:2
    
    [...]
    
    JUnit report was created: /junit.xml/cnftests-junit.xml
    
    
    Summarizing 1 Failure:
    
    [Fail] [performance] Latency Test with the hwlatdetect image [It] should succeed
    /remote-source/app/vendor/github.com/openshift/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/4_latency/latency.go:476
    
    Ran 1 of 194 Specs in 365.797 seconds
    FAIL! -- 0 Passed | 1 Failed | 0 Pending | 2 Skipped
    --- FAIL: TestTest (366.08s)
    FAIL
    Copy to Clipboard Toggle word wrap

    1
    2

hwlatdetect: test duration 3600 seconds
detector: tracer
parameters:
Latency threshold: 10us
Sample window: 1000000us
Sample width: 950000us
Non-sampling period: 50000us
Output File: None

Starting test
test finished
Max Latency: Below threshold
Samples recorded: 0
Copy to Clipboard Toggle word wrap

hwlatdetect: test duration 3600 seconds
detector: tracer
parameters:Latency threshold: 10usSample window: 1000000us
Sample width: 950000usNon-sampling period: 50000usOutput File: None

Starting tests:1610542421.275784439, inner:78, outer:81
ts: 1610542444.330561619, inner:27, outer:28
ts: 1610542445.332549975, inner:39, outer:38
ts: 1610542541.568546097, inner:47, outer:32
ts: 1610542590.681548531, inner:13, outer:17
ts: 1610543033.818801482, inner:29, outer:30
ts: 1610543080.938801990, inner:90, outer:76
ts: 1610543129.065549639, inner:28, outer:39
ts: 1610543474.859552115, inner:28, outer:35
ts: 1610543523.973856571, inner:52, outer:49
ts: 1610543572.089799738, inner:27, outer:30
ts: 1610543573.091550771, inner:34, outer:28
ts: 1610543574.093555202, inner:116, outer:63
Copy to Clipboard Toggle word wrap

주의

18.3.2.

참고

  • $ podman run -v $(pwd)/:/kubeconfig:Z -e KUBECONFIG=/kubeconfig/kubeconfig \
    -e LATENCY_TEST_CPUS=10 -e LATENCY_TEST_RUNTIME=600 -e MAXIMUM_LATENCY=20 \
    registry.redhat.io/openshift4/cnf-tests-rhel8:v4.15 \
    /usr/bin/test-run.sh --ginkgo.focus="cyclictest" --ginkgo.v --ginkgo.timeout="24h"
    Copy to Clipboard Toggle word wrap

    중요

    running /usr/bin/cnftests -ginkgo.v -ginkgo.focus=cyclictest
    I0908 13:01:59.193776      27 request.go:601] Waited for 1.046228824s due to client-side throttling, not priority and fairness, request: GET:https://api.compute-1.example.com:6443/apis/packages.operators.coreos.com/v1?timeout=32s
    Running Suite: CNF Features e2e integration tests
    =================================================
    Random Seed: 1662642118
    Will run 1 of 3 specs
    
    [...]
    
    Summarizing 1 Failure:
    
    [Fail] [performance] Latency Test with the cyclictest image [It] should succeed
    /remote-source/app/vendor/github.com/openshift/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/4_latency/latency.go:220
    
    Ran 1 of 194 Specs in 161.151 seconds
    FAIL! -- 0 Passed | 1 Failed | 0 Pending | 2 Skipped
    --- FAIL: TestTest (161.48s)
    FAIL
    Copy to Clipboard Toggle word wrap

running cmd: cyclictest -q -D 10m -p 1 -t 16 -a 2,4,6,8,10,12,14,16,54,56,58,60,62,64,66,68 -h 30 -i 1000 -m
# Histogram
000000 000000   000000  000000  000000  000000  000000  000000  000000  000000  000000  000000  000000  000000  000000  000000  000000
000001 000000   000000  000000  000000  000000  000000  000000  000000  000000  000000  000000  000000  000000  000000  000000  000000
000002 579506   535967  418614  573648  532870  529897  489306  558076  582350  585188  583793  223781  532480  569130  472250  576043
More histogram entries ...
# Total: 000600000 000600000 000600000 000599999 000599999 000599999 000599998 000599998 000599998 000599997 000599997 000599996 000599996 000599995 000599995 000599995
# Min Latencies: 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002
# Avg Latencies: 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002
# Max Latencies: 00005 00005 00004 00005 00004 00004 00005 00005 00006 00005 00004 00005 00004 00004 00005 00004
# Histogram Overflows: 00000 00000 00000 00000 00000 00000 00000 00000 00000 00000 00000 00000 00000 00000 00000 00000
# Histogram Overflow at cycle number:
# Thread 0:
# Thread 1:
# Thread 2:
# Thread 3:
# Thread 4:
# Thread 5:
# Thread 6:
# Thread 7:
# Thread 8:
# Thread 9:
# Thread 10:
# Thread 11:
# Thread 12:
# Thread 13:
# Thread 14:
# Thread 15:
Copy to Clipboard Toggle word wrap

running cmd: cyclictest -q -D 10m -p 1 -t 16 -a 2,4,6,8,10,12,14,16,54,56,58,60,62,64,66,68 -h 30 -i 1000 -m
# Histogram
000000 000000   000000  000000  000000  000000  000000  000000  000000  000000  000000  000000  000000  000000  000000  000000  000000
000001 000000   000000  000000  000000  000000  000000  000000  000000  000000  000000  000000  000000  000000  000000  000000  000000
000002 564632   579686  354911  563036  492543  521983  515884  378266  592621  463547  482764  591976  590409  588145  589556  353518
More histogram entries ...
# Total: 000599999 000599999 000599999 000599997 000599997 000599998 000599998 000599997 000599997 000599996 000599995 000599996 000599995 000599995 000599995 000599993
# Min Latencies: 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002
# Avg Latencies: 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002 00002
# Max Latencies: 00493 00387 00271 00619 00541 00513 00009 00389 00252 00215 00539 00498 00363 00204 00068 00520
# Histogram Overflows: 00001 00001 00001 00002 00002 00001 00000 00001 00001 00001 00002 00001 00001 00001 00001 00002
# Histogram Overflow at cycle number:
# Thread 0: 155922
# Thread 1: 110064
# Thread 2: 110064
# Thread 3: 110063 155921
# Thread 4: 110063 155921
# Thread 5: 155920
# Thread 6:
# Thread 7: 110062
# Thread 8: 110062
# Thread 9: 155919
# Thread 10: 110061 155919
# Thread 11: 155918
# Thread 12: 155918
# Thread 13: 110060
# Thread 14: 110060
# Thread 15: 110059 155917
Copy to Clipboard Toggle word wrap

18.3.3.

참고

  • $ podman run -v $(pwd)/:/kubeconfig:Z -e KUBECONFIG=/kubeconfig/kubeconfig \
    -e LATENCY_TEST_CPUS=10 -e LATENCY_TEST_RUNTIME=600 -e MAXIMUM_LATENCY=20 \
    registry.redhat.io/openshift4/cnf-tests-rhel8:v4.15 \
    /usr/bin/test-run.sh --ginkgo.focus="oslat" --ginkgo.v --ginkgo.timeout="24h"
    Copy to Clipboard Toggle word wrap

    중요

    running /usr/bin/cnftests -ginkgo.v -ginkgo.focus=oslat
    I0908 12:51:55.999393      27 request.go:601] Waited for 1.044848101s due to client-side throttling, not priority and fairness, request: GET:https://compute-1.example.com:6443/apis/machineconfiguration.openshift.io/v1?timeout=32s
    Running Suite: CNF Features e2e integration tests
    =================================================
    Random Seed: 1662641514
    Will run 1 of 3 specs
    
    [...]
    
    • Failure [77.833 seconds]
    [performance] Latency Test
    /remote-source/app/vendor/github.com/openshift/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/4_latency/latency.go:62
      with the oslat image
      /remote-source/app/vendor/github.com/openshift/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/4_latency/latency.go:128
        should succeed [It]
        /remote-source/app/vendor/github.com/openshift/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/4_latency/latency.go:153
    
        The current latency 304 is bigger than the expected one 1 : 
    1
    
    
    [...]
    
    Summarizing 1 Failure:
    
    [Fail] [performance] Latency Test with the oslat image [It] should succeed
    /remote-source/app/vendor/github.com/openshift/cluster-node-tuning-operator/test/e2e/performanceprofile/functests/4_latency/latency.go:177
    
    Ran 1 of 194 Specs in 161.091 seconds
    FAIL! -- 0 Passed | 1 Failed | 0 Pending | 2 Skipped
    --- FAIL: TestTest (161.42s)
    FAIL
    Copy to Clipboard Toggle word wrap

    1

18.4.

  • $ podman run -v $(pwd)/:/kubeconfig:Z -v $(pwd)/reportdest:<report_folder_path> \
    -e KUBECONFIG=/kubeconfig/kubeconfig registry.redhat.io/openshift4/cnf-tests-rhel8:v4.15 \
    /usr/bin/test-run.sh --report <report_folder_path> --ginkgo.v
    Copy to Clipboard Toggle word wrap

18.5.

  • 참고

    $ podman run -v $(pwd)/:/kubeconfig:Z -v $(pwd)/junit:/junit \
    -e KUBECONFIG=/kubeconfig/kubeconfig registry.redhat.io/openshift4/cnf-tests-rhel8:v4.15 \
    /usr/bin/test-run.sh --ginkgo.junit-report junit/<file-name>.xml --ginkgo.v
    Copy to Clipboard Toggle word wrap

18.6.

참고

  • $ podman run -v $(pwd)/:/kubeconfig:Z -e KUBECONFIG=/kubeconfig/kubeconfig \
    -e LATENCY_TEST_RUNTIME=<time_in_seconds> registry.redhat.io/openshift4/cnf-tests-rhel8:v4.15 \
    /usr/bin/test-run.sh --ginkgo.v --ginkgo.timeout="24h"
    Copy to Clipboard Toggle word wrap
    참고

18.7.

  1. $ podman run -v $(pwd)/:/kubeconfig:Z -e KUBECONFIG=/kubeconfig/kubeconfig \
    registry.redhat.io/openshift4/cnf-tests-rhel8:v4.15 \
    /usr/bin/mirror -registry <disconnected_registry> | oc image mirror -f -
    Copy to Clipboard Toggle word wrap

  2. podman run -v $(pwd)/:/kubeconfig:Z -e KUBECONFIG=/kubeconfig/kubeconfig \
    -e IMAGE_REGISTRY="<disconnected_registry>" \
    -e CNF_TESTS_IMAGE="cnf-tests-rhel8:v4.15" \
    -e LATENCY_TEST_RUNTIME=<time_in_seconds> \
    <disconnected_registry>/cnf-tests-rhel8:v4.15 /usr/bin/test-run.sh --ginkgo.v --ginkgo.timeout="24h"
    Copy to Clipboard Toggle word wrap

  • $ podman run -v $(pwd)/:/kubeconfig:Z -e KUBECONFIG=/kubeconfig/kubeconfig \
    -e IMAGE_REGISTRY="<custom_image_registry>" \
    -e CNF_TESTS_IMAGE="<custom_cnf-tests_image>" \
    -e LATENCY_TEST_RUNTIME=<time_in_seconds> \
    registry.redhat.io/openshift4/cnf-tests-rhel8:v4.15 /usr/bin/test-run.sh --ginkgo.v --ginkgo.timeout="24h"
    Copy to Clipboard Toggle word wrap

  1. $ oc patch configs.imageregistry.operator.openshift.io/cluster --patch '{"spec":{"defaultRoute":true}}' --type=merge
    Copy to Clipboard Toggle word wrap
  2. $ REGISTRY=$(oc get route default-route -n openshift-image-registry --template='{{ .spec.host }}')
    Copy to Clipboard Toggle word wrap
  3. $ oc create ns cnftests
    Copy to Clipboard Toggle word wrap
  4. $ oc policy add-role-to-user system:image-puller system:serviceaccount:cnf-features-testing:default --namespace=cnftests
    Copy to Clipboard Toggle word wrap
    $ oc policy add-role-to-user system:image-puller system:serviceaccount:performance-addon-operators-testing:default --namespace=cnftests
    Copy to Clipboard Toggle word wrap
  5. $ SECRET=$(oc -n cnftests get secret | grep builder-docker | awk {'print $1'}
    Copy to Clipboard Toggle word wrap
    $ TOKEN=$(oc -n cnftests get secret $SECRET -o jsonpath="{.data['\.dockercfg']}" | base64 --decode | jq '.["image-registry.openshift-image-registry.svc:5000"].auth')
    Copy to Clipboard Toggle word wrap
  6. $ echo "{\"auths\": { \"$REGISTRY\": { \"auth\": $TOKEN } }}" > dockerauth.json
    Copy to Clipboard Toggle word wrap
  7. $ podman run -v $(pwd)/:/kubeconfig:Z -e KUBECONFIG=/kubeconfig/kubeconfig \
    registry.redhat.io/openshift4/cnf-tests-rhel8:4.15 \
    /usr/bin/mirror -registry $REGISTRY/cnftests |  oc image mirror --insecure=true \
    -a=$(pwd)/dockerauth.json -f -
    Copy to Clipboard Toggle word wrap
  8. $ podman run -v $(pwd)/:/kubeconfig:Z -e KUBECONFIG=/kubeconfig/kubeconfig \
    -e LATENCY_TEST_RUNTIME=<time_in_seconds> \
    -e IMAGE_REGISTRY=image-registry.openshift-image-registry.svc:5000/cnftests cnf-tests-local:latest /usr/bin/test-run.sh --ginkgo.v --ginkgo.timeout="24h"
    Copy to Clipboard Toggle word wrap

  1. [
        {
            "registry": "public.registry.io:5000",
            "image": "imageforcnftests:4.15"
        }
    ]
    Copy to Clipboard Toggle word wrap
  2. $ podman run -v $(pwd)/:/kubeconfig:Z -e KUBECONFIG=/kubeconfig/kubeconfig \
    registry.redhat.io/openshift4/cnf-tests-rhel8:v4.15 /usr/bin/mirror \
    --registry "my.local.registry:5000/" --images "/kubeconfig/images.json" \
    |  oc image mirror -f -
    Copy to Clipboard Toggle word wrap

18.8.

  • $ podman run -v $(pwd)/:/kubeconfig:Z -e KUBECONFIG=/kubeconfig/kubeconfig \
    registry.redhat.io/openshift4/cnf-tests-rhel8:v4.15 \
    oc get nodes
    Copy to Clipboard Toggle word wrap

19장.

19.1.

중요

Expand
    

Expand
    

Expand
    

참고

19.2.

중요

  • $ openshift-install create manifests --dir=<cluster-install-dir>
    Copy to Clipboard Toggle word wrap
  • $ vi <cluster-install-dir>/manifests/config-node-default-profile.yaml
    Copy to Clipboard Toggle word wrap

    apiVersion: config.openshift.io/v1
    kind: Node
    metadata:
    name: cluster
    spec:
    workerLatencyProfile: "Default"
    Copy to Clipboard Toggle word wrap

19.3.

참고

    1. $ oc edit nodes.config/cluster
      Copy to Clipboard Toggle word wrap
    2. apiVersion: config.openshift.io/v1
      kind: Node
      metadata:
        annotations:
          include.release.openshift.io/ibm-cloud-managed: "true"
          include.release.openshift.io/self-managed-high-availability: "true"
          include.release.openshift.io/single-node-developer: "true"
          release.openshift.io/create-only: "true"
        creationTimestamp: "2022-07-08T16:02:51Z"
        generation: 1
        name: cluster
        ownerReferences:
        - apiVersion: config.openshift.io/v1
          kind: ClusterVersion
          name: version
          uid: 36282574-bf9f-409e-a6cd-3032939293eb
        resourceVersion: "1865"
        uid: 0c0f7a4c-4307-4187-b591-6155695ac85b
      spec:
        workerLatencyProfile: MediumUpdateAverageReaction 
      1
      
      
      # ...
      Copy to Clipboard Toggle word wrap

      1

    1. $ oc edit nodes.config/cluster
      Copy to Clipboard Toggle word wrap
    2. apiVersion: config.openshift.io/v1
      kind: Node
      metadata:
        annotations:
          include.release.openshift.io/ibm-cloud-managed: "true"
          include.release.openshift.io/self-managed-high-availability: "true"
          include.release.openshift.io/single-node-developer: "true"
          release.openshift.io/create-only: "true"
        creationTimestamp: "2022-07-08T16:02:51Z"
        generation: 1
        name: cluster
        ownerReferences:
        - apiVersion: config.openshift.io/v1
          kind: ClusterVersion
          name: version
          uid: 36282574-bf9f-409e-a6cd-3032939293eb
        resourceVersion: "1865"
        uid: 0c0f7a4c-4307-4187-b591-6155695ac85b
      spec:
        workerLatencyProfile: LowUpdateSlowReaction 
      1
      
      
      # ...
      Copy to Clipboard Toggle word wrap

      1

  • $ oc get KubeControllerManager -o yaml | grep -i workerlatency -A 5 -B 5
    Copy to Clipboard Toggle word wrap

    # ...
        - lastTransitionTime: "2022-07-11T19:47:10Z"
          reason: ProfileUpdated
          status: "False"
          type: WorkerLatencyProfileProgressing
        - lastTransitionTime: "2022-07-11T19:47:10Z" 
    1
    
          message: all static pod revision(s) have updated latency profile
          reason: ProfileUpdated
          status: "True"
          type: WorkerLatencyProfileComplete
        - lastTransitionTime: "2022-07-11T19:20:11Z"
          reason: AsExpected
          status: "False"
          type: WorkerLatencyProfileDegraded
        - lastTransitionTime: "2022-07-11T19:20:36Z"
          status: "False"
    # ...
    Copy to Clipboard Toggle word wrap

    1

19.4.

  1. $ oc get KubeAPIServer -o yaml | grep -A 1 default-
    Copy to Clipboard Toggle word wrap

    default-not-ready-toleration-seconds:
    - "300"
    default-unreachable-toleration-seconds:
    - "300"
    Copy to Clipboard Toggle word wrap

  2. $ oc get KubeControllerManager -o yaml | grep -A 1 node-monitor
    Copy to Clipboard Toggle word wrap

    node-monitor-grace-period:
    - 40s
    Copy to Clipboard Toggle word wrap

  3. $ oc debug node/<worker-node-name>
    $ chroot /host
    # cat /etc/kubernetes/kubelet.conf|grep nodeStatusUpdateFrequency
    Copy to Clipboard Toggle word wrap

      “nodeStatusUpdateFrequency”: “10s”
    Copy to Clipboard Toggle word wrap

20장.

  1. apiVersion: v1
    baseDomain: devcluster.openshift.com
    cpuPartitioningMode: AllNodes 
    1
    
    compute:
      - architecture: amd64
        hyperthreading: Enabled
        name: worker
        platform: {}
        replicas: 3
    controlPlane:
      architecture: amd64
      hyperthreading: Enabled
      name: master
      platform: {}
      replicas: 3
    Copy to Clipboard Toggle word wrap
    1
    참고

  2. apiVersion: performance.openshift.io/v2
    kind: PerformanceProfile
    metadata:
      # if you change this name make sure the 'include' line in TunedPerformancePatch.yaml
      # matches this name: include=openshift-node-performance-${PerformanceProfile.metadata.name}
      # Also in file 'validatorCRs/informDuValidator.yaml':
      # name: 50-performance-${PerformanceProfile.metadata.name}
      name: openshift-node-performance-profile
      annotations:
        ran.openshift.io/reference-configuration: "ran-du.redhat.com"
    spec:
      additionalKernelArgs:
        - "rcupdate.rcu_normal_after_boot=0"
        - "efi=runtime"
        - "vfio_pci.enable_sriov=1"
        - "vfio_pci.disable_idle_d3=1"
        - "module_blacklist=irdma"
      cpu:
        isolated: $isolated
        reserved: $reserved
      hugepages:
        defaultHugepagesSize: $defaultHugepagesSize
        pages:
          - size: $size
            count: $count
            node: $node
      machineConfigPoolSelector:
        pools.operator.machineconfiguration.openshift.io/$mcp: ""
      nodeSelector:
        node-role.kubernetes.io/$mcp: ''
      numa:
        topologyPolicy: "restricted"
      # To use the standard (non-realtime) kernel, set enabled to false
      realTimeKernel:
        enabled: true
      workloadHints:
        # WorkloadHints defines the set of upper level flags for different type of workloads.
        # See https://github.com/openshift/cluster-node-tuning-operator/blob/master/docs/performanceprofile/performance_profile.md#workloadhints
        # for detailed descriptions of each item.
        # The configuration below is set for a low latency, performance mode.
        realTime: true
        highPowerConsumption: false
        perPodPowerManagement: false
    Copy to Clipboard Toggle word wrap

    Expand
    표 20.1.
      

    중요

21장.

중요

21.1.

21.2.

21.2.1.

  1. $ oc get packagemanifests -n openshift-marketplace node-observability-operator
    Copy to Clipboard Toggle word wrap

    NAME                            CATALOG                AGE
    node-observability-operator     Red Hat Operators      9h
    Copy to Clipboard Toggle word wrap

  2. $ oc new-project node-observability-operator
    Copy to Clipboard Toggle word wrap
  3. cat <<EOF | oc apply -f -
    apiVersion: operators.coreos.com/v1
    kind: OperatorGroup
    metadata:
      name: node-observability-operator
      namespace: node-observability-operator
    spec:
      targetNamespaces: []
    EOF
    Copy to Clipboard Toggle word wrap
  4. cat <<EOF | oc apply -f -
    apiVersion: operators.coreos.com/v1alpha1
    kind: Subscription
    metadata:
      name: node-observability-operator
      namespace: node-observability-operator
    spec:
      channel: alpha
      name: node-observability-operator
      source: redhat-operators
      sourceNamespace: openshift-marketplace
    EOF
    Copy to Clipboard Toggle word wrap

  1. $ oc -n node-observability-operator get sub node-observability-operator -o yaml | yq '.status.installplan.name'
    Copy to Clipboard Toggle word wrap

    install-dt54w
    Copy to Clipboard Toggle word wrap

  2. $ oc -n node-observability-operator get ip <install_plan_name> -o yaml | yq '.status.phase'
    Copy to Clipboard Toggle word wrap

    COMPLETE
    Copy to Clipboard Toggle word wrap

  3. $ oc get deploy -n node-observability-operator
    Copy to Clipboard Toggle word wrap

    NAME                                            READY   UP-TO-DATE  AVAILABLE   AGE
    node-observability-operator-controller-manager  1/1     1           1           40h
    Copy to Clipboard Toggle word wrap

21.2.2.

21.3.

21.3.1.

중요

참고

  1. $ oc login -u kubeadmin https://<HOSTNAME>:6443
    Copy to Clipboard Toggle word wrap
  2. $ oc project node-observability-operator
    Copy to Clipboard Toggle word wrap
  3.     apiVersion: nodeobservability.olm.openshift.io/v1alpha2
        kind: NodeObservability
        metadata:
          name: cluster 
    1
    
        spec:
          nodeSelector:
            kubernetes.io/hostname: <node_hostname> 
    2
    
          type: crio-kubelet
    Copy to Clipboard Toggle word wrap
    1
    2
  4. oc apply -f nodeobservability.yaml
    Copy to Clipboard Toggle word wrap

    nodeobservability.olm.openshift.io/cluster created
    Copy to Clipboard Toggle word wrap

  5. $ oc get nob/cluster -o yaml | yq '.status.conditions'
    Copy to Clipboard Toggle word wrap

    conditions:
      conditions:
      - lastTransitionTime: "2022-07-05T07:33:54Z"
        message: 'DaemonSet node-observability-ds ready: true NodeObservabilityMachineConfig
          ready: true'
        reason: Ready
        status: "True"
        type: Ready
    Copy to Clipboard Toggle word wrap

21.3.2.

중요

  1. apiVersion: nodeobservability.olm.openshift.io/v1alpha2
    kind: NodeObservabilityRun
    metadata:
      name: nodeobservabilityrun
    spec:
      nodeObservabilityRef:
        name: cluster
    Copy to Clipboard Toggle word wrap
  2. $ oc apply -f nodeobservabilityrun.yaml
    Copy to Clipboard Toggle word wrap
  3. $ oc get nodeobservabilityrun nodeobservabilityrun -o yaml  | yq '.status.conditions'
    Copy to Clipboard Toggle word wrap

    conditions:
    - lastTransitionTime: "2022-07-07T14:57:34Z"
      message: Ready to start profiling
      reason: Ready
      status: "True"
      type: Ready
    - lastTransitionTime: "2022-07-07T14:58:10Z"
      message: Profiling query done
      reason: Finished
      status: "True"
      type: Finished
    Copy to Clipboard Toggle word wrap

  4. for a in $(oc get nodeobservabilityrun nodeobservabilityrun -o yaml | yq .status.agents[].name); do
      echo "agent ${a}"
      mkdir -p "/tmp/${a}"
      for p in $(oc exec "${a}" -c node-observability-agent -- bash -c "ls /run/node-observability/*.pprof"); do
        f="$(basename ${p})"
        echo "copying ${f} to /tmp/${a}/${f}"
        oc exec "${a}" -c node-observability-agent -- cat "${p}" > "/tmp/${a}/${f}"
      done
    done
    Copy to Clipboard Toggle word wrap

21.4.

21.4.1.

  1. $ oc login -u kubeadmin https://<host_name>:6443
    Copy to Clipboard Toggle word wrap
  2. $ oc project node-observability-operator
    Copy to Clipboard Toggle word wrap
  3.     apiVersion: nodeobservability.olm.openshift.io/v1alpha2
        kind: NodeObservability
        metadata:
          name: cluster 
    1
    
        spec:
          nodeSelector:
            kubernetes.io/hostname: <node_hostname> 
    2
    
          type: scripting 
    3
    Copy to Clipboard Toggle word wrap
    1
    2
    3
  4. $ oc apply -f nodeobservability.yaml
    Copy to Clipboard Toggle word wrap

    nodeobservability.olm.openshift.io/cluster created
    Copy to Clipboard Toggle word wrap

  5. $ oc get nob/cluster -o yaml | yq '.status.conditions'
    Copy to Clipboard Toggle word wrap

    conditions:
      conditions:
      - lastTransitionTime: "2022-07-05T07:33:54Z"
        message: 'DaemonSet node-observability-ds ready: true NodeObservabilityScripting
          ready: true'
        reason: Ready
        status: "True"
        type: Ready
    Copy to Clipboard Toggle word wrap

21.4.2.

  1. apiVersion: nodeobservability.olm.openshift.io/v1alpha2
    kind: NodeObservabilityRun
    metadata:
      name: nodeobservabilityrun-script
      namespace: node-observability-operator
    spec:
      nodeObservabilityRef:
        name: cluster
        type: scripting
    Copy to Clipboard Toggle word wrap
    중요

  2. $ oc apply -f nodeobservabilityrun-script.yaml
    Copy to Clipboard Toggle word wrap
  3. $ oc get nodeobservabilityrun nodeobservabilityrun-script -o yaml  | yq '.status.conditions'
    Copy to Clipboard Toggle word wrap

    Status:
      Agents:
        Ip:    10.128.2.252
        Name:  node-observability-agent-n2fpm
        Port:  8443
        Ip:    10.131.0.186
        Name:  node-observability-agent-wcc8p
        Port:  8443
      Conditions:
        Conditions:
          Last Transition Time:  2023-12-19T15:10:51Z
          Message:               Ready to start profiling
          Reason:                Ready
          Status:                True
          Type:                  Ready
          Last Transition Time:  2023-12-19T15:11:01Z
          Message:               Profiling query done
          Reason:                Finished
          Status:                True
          Type:                  Finished
      Finished Timestamp:        2023-12-19T15:11:01Z
      Start Timestamp:           2023-12-19T15:10:51Z
    Copy to Clipboard Toggle word wrap

  4. #!/bin/bash
    
    RUN=$(oc get nodeobservabilityrun --no-headers | awk '{print $1}')
    
    for a in $(oc get nodeobservabilityruns.nodeobservability.olm.openshift.io/${RUN} -o json | jq .status.agents[].name); do
      echo "agent ${a}"
      agent=$(echo ${a} | tr -d "\"\'\`")
      base_dir=$(oc exec "${agent}" -c node-observability-agent -- bash -c "ls -t | grep node-observability-agent" | head -1)
      echo "${base_dir}"
      mkdir -p "/tmp/${agent}"
      for p in $(oc exec "${agent}" -c node-observability-agent -- bash -c "ls ${base_dir}"); do
        f="/${base_dir}/${p}"
        echo "copying ${f} to /tmp/${agent}/${p}"
        oc exec "${agent}" -c node-observability-agent -- cat ${f} > "/tmp/${agent}/${p}"
      done
    done
    Copy to Clipboard Toggle word wrap

Legal Notice

Copyright © 2025 Red Hat

OpenShift documentation is licensed under the Apache License 2.0 (https://www.apache.org/licenses/LICENSE-2.0).

Modified versions must remove all Red Hat trademarks.

Portions adapted from https://github.com/kubernetes-incubator/service-catalog/ with modifications by Red Hat.

Red Hat, Red Hat Enterprise Linux, the Red Hat logo, the Shadowman logo, JBoss, OpenShift, Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries.

Linux® is the registered trademark of Linus Torvalds in the United States and other countries.

Java® is a registered trademark of Oracle and/or its affiliates.

XFS® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries.

MySQL® is a registered trademark of MySQL AB in the United States, the European Union and other countries.

Node.js® is an official trademark of Joyent. Red Hat Software Collections is not formally related to or endorsed by the official Joyent Node.js open source or commercial project.

The OpenStack® Word Mark and OpenStack logo are either registered trademarks/service marks or trademarks/service marks of the OpenStack Foundation, in the United States and other countries and are used with the OpenStack Foundation’s permission. We are not affiliated with, endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.

All other trademarks are the property of their respective owners.

맨 위로 이동
Red Hat logoGithubredditYoutubeTwitter

자세한 정보

평가판, 구매 및 판매

커뮤니티

Red Hat 문서 정보

Red Hat을 사용하는 고객은 신뢰할 수 있는 콘텐츠가 포함된 제품과 서비스를 통해 혁신하고 목표를 달성할 수 있습니다. 최신 업데이트를 확인하세요.

보다 포괄적 수용을 위한 오픈 소스 용어 교체

Red Hat은 코드, 문서, 웹 속성에서 문제가 있는 언어를 교체하기 위해 최선을 다하고 있습니다. 자세한 내용은 다음을 참조하세요.Red Hat 블로그.

Red Hat 소개

Red Hat은 기업이 핵심 데이터 센터에서 네트워크 에지에 이르기까지 플랫폼과 환경 전반에서 더 쉽게 작업할 수 있도록 강화된 솔루션을 제공합니다.

Theme

© 2025 Red Hat