베어 메탈에 설치 프로그램이 프로비저닝 한 클러스터 배포


OpenShift Container Platform 4.17

베어 메탈에 설치 관리자 프로비저닝 OpenShift Container Platform 클러스터 배포

Red Hat OpenShift Documentation Team

초록

이 문서에서는 설치 관리자 프로비저닝 인프라를 사용하여 베어 메탈에 OpenShift Container Platform 클러스터를 배포하는 방법을 설명합니다.

1장. 개요

베어 메탈 노드에 설치 프로그램이 프로비저닝한 설치는 OpenShift Container Platform 클러스터가 실행되는 인프라를 배포하고 구성합니다. 이 가이드에서는 설치 관리자가 프로비저닝한 베어 메탈 설치를 성공적으로 수행하는 방법을 제공합니다. 다음 다이어그램은 배포 1단계에서 설치 환경을 보여줍니다.

설치의 경우 이전 다이어그램의 주요 요소는 다음과 같습니다.

  • provisioner: 설치 프로그램을 실행하고 새로운 OpenShift Container Platform 클러스터의 컨트롤 플레인을 배포하는 부트스트랩 VM을 호스팅하는 물리적 머신입니다.
  • 부트스트랩 VM: OpenShift Container Platform 클러스터 배포 프로세스에서 사용되는 가상 머신입니다.
  • 네트워크 브리지: 부트스트랩 VM은 베어 메탈 네트워크와 provisioning 네트워크(있는 경우 네트워크 브리지, eno1eno2 를 통해)에 연결됩니다.
  • API VIP: API 가상 IP 주소(VIP)는 컨트롤 플레인 노드에서 API 서버의 페일오버를 제공하는 데 사용됩니다. API VIP는 먼저 부트스트랩 VM에 있습니다. 스크립트는 서비스를 시작하기 전에 keepalived.conf 구성 파일을 생성합니다. 부트스트랩 프로세스가 완료되면 VIP가 컨트롤 플레인 노드 중 하나로 이동하고 부트스트랩 VM이 중지됩니다.

배포 2단계에서 프로비저너는 부트스트랩 VM을 자동으로 제거하고 VIP(가상 IP 주소)를 적절한 노드로 이동합니다.

keepalived.conf 파일은 부트스트랩 VM보다 낮은 VRRP(Virtual Router Redundancy Protocol) 우선 순위가 낮은 컨트롤 플레인 머신을 설정하여 API VIP가 부트스트랩 VM에서 컨트롤 플레인으로 이동하기 전에 컨트롤 플레인 시스템의 API가 완전히 작동하는지 확인합니다. API VIP가 컨트롤 플레인 노드 중 하나로 이동하면 외부 클라이언트에서 API VIP 경로로 전송된 트래픽이 해당 컨트롤 플레인 노드에서 실행되는 haproxy 로드 밸런서로 이동합니다. haproxy 의 이 인스턴스는 컨트롤 플레인 노드에서 API VIP 트래픽의 균형을 조정합니다.

Ingress VIP는 컴퓨팅 노드로 이동합니다. keepalived 인스턴스는 Ingress VIP도 관리합니다.

다음 다이어그램에서는 배포 2단계에 대해 설명합니다.

이 시점 이후 프로비저너에서 사용하는 노드를 제거하거나 용도 변경할 수 있습니다. 여기에서 모든 추가 프로비저닝 작업은 컨트롤 플레인에서 수행합니다.

참고

설치 관리자 프로비저닝 인프라 설치의 경우 CoreDNS는 노드 수준에서 포트 53을 노출하여 다른 라우팅 가능한 네트워크에서 액세스할 수 있습니다.

중요

provisioning 네트워크는 선택 사항이지만 PXE 부팅에는 필요합니다. provisioning 네트워크없이 배포하는 경우 redfish-virtualmedia 또는 idrac-virtualmedia 와 같은 BMC(가상 미디어 베이스 보드 관리 컨트롤러) 주소 지정 옵션을 사용해야 합니다.

2장. 사전 요구 사항

OpenShift Container Platform 설치 프로그램으로 프로비저닝된 설치에는 다음이 필요합니다.

  1. RHEL(Red Hat Enterprise Linux) 9.x가 설치된 프로비저너 노드 1개 설치 후 프로비저너를 제거할 수 있습니다.
  2. 컨트롤 플레인 노드 세 개
  3. 각 노드에 대한 BMC(Baseboard Management Controller) 액세스
  4. 하나 이상의 네트워크:

    1. 라우팅 가능한 필수 네트워크 1개
    2. 선택적 프로비저닝 네트워크 1개
    3. 선택적 관리 네트워크 1개

OpenShift Container Platform 설치 프로그램으로 프로비저닝 설치를 시작하기 전에 하드웨어 환경이 다음 요구 사항을 충족하는지 확인합니다.

2.1. 노드 요구 사항

설치 프로그램에서 제공하는 설치에는 여러 하드웨어 노드 요구 사항이 있습니다.

  • CPU 아키텍처: 모든 노드는 x86_64 또는 aarch64 CPU 아키텍처를 사용해야 합니다.
  • 유사한 노드: Red Hat은 노드가 역할별로 동일한 구성을 지정할 것을 권장합니다. 즉, Red Hat은 동일한 CPU, 메모리, 스토리지 설정의 브랜드 및 모델의 노드를 사용할 것을 권장하고 있습니다.
  • 베이스 보드 관리 컨트롤러 : provisioner 노드는 각 OpenShift Container Platform 클러스터 노드의 베이스 보드 관리 컨트롤러 (BMC)에 액세스할 수 있습니다. IPMI, Redfish 또는 전용 프로토콜을 사용할 수 있습니다.
  • 최근 생성: 노드는 최근 생성된 노드여야합니다. 설치 프로그램에서 제공하는 설치는 노드간에 호환되어야 하는 BMC 프로토콜을 사용합니다. 또한 RHEL 9.x에는 RAID 컨트롤러용 최신 드라이버가 포함되어 있습니다. 노드가 프로비저너 노드에 대해 RHEL 9.x를 지원할 수 있을 만큼 최근 노드가 있고 컨트롤 플레인 및 작업자 노드에 대해 RHCOS 9.x가 있는지 확인합니다.
  • 레지스트리 노드: (선택 사항) 연결이 끊어진 미러링된 레지스트리를 설정하는 경우 레지스트리가 자체 노드에 상주하는 것이 좋습니다.
  • 프로비저너 노드 : 설치 프로그램이 제공하는 설치에는 하나의 provisioner 노드가 필요합니다.
  • 컨트롤 플레인: 설치 프로그램에서 프로비저닝한 설치에는 고가용성을 위해 3 개의 컨트롤 플레인 노드가 필요합니다. 컨트롤 플레인 노드 3개만 사용하여 OpenShift Container Platform 클러스터를 배포하여 컨트롤 플레인 노드를 작업자 노드로 예약할 수 있습니다. 소규모 클러스터는 개발, 프로덕션 및 테스트 중에 관리자와 개발자에게 더 많은 리소스를 제공합니다.
  • 작업자 노드: 필수는 아니지만 일반적인 프로덕션 클러스터에는 두 개 이상의 작업자 노드가 있습니다.

    중요

    클러스터가 성능 저하된 상태로 라우터 및 인그레스 트래픽으로 배포되므로 하나의 작업자 노드로만 클러스터를 배포하지 마십시오.

  • 네트워크 인터페이스: 각 노드에는 라우팅 가능한 baremetal 네트워크에 대해 하나 이상의 네트워크 인터페이스가 있어야 합니다. 배포에 provisioning 네트워크를 사용할 때 각 노드에는 provisioning 네트워크에 대해 하나의 네트워크 인터페이스가 있어야합니다. provisioning 네트워크를 사용하는 것이 기본 구성입니다.

    참고

    동일한 서브넷의 NIC(네트워크 카드) 하나만 게이트웨이를 통해 트래픽을 라우팅할 수 있습니다. 기본적으로 ARP(Address Resolution Protocol)는 가장 낮은 번호의 NIC를 사용합니다. 동일한 서브넷의 각 노드에 대해 단일 NIC를 사용하여 네트워크 로드 밸런싱이 예상대로 작동하는지 확인합니다. 동일한 서브넷의 노드에 여러 NIC를 사용하는 경우 단일 본딩 또는 팀 인터페이스를 사용합니다. 그런 다음 별칭 IP 주소 형식으로 해당 인터페이스에 다른 IP 주소를 추가합니다. 네트워크 인터페이스 수준에서 내결함성 또는 로드 밸런싱이 필요한 경우 본딩 또는 팀 인터페이스에서 별칭 IP 주소를 사용합니다. 또는 동일한 서브넷에서 보조 NIC를 비활성화하거나 IP 주소가 없는지 확인할 수 있습니다.

  • UEFI (Unified Extensible Firmware Interface): 설치 프로그램이 프로비저닝한 설치에는 provisioning 네트워크에서 IPv6 주소를 사용하는 경우 모든 OpenShift Container Platform 노드에서 UEFI 부팅이 필요합니다. 또한 provisioning 네트워크 NIC에서 IPv6 프로토콜을 사용하도록 UEFI 장치 PXE 설정을 설정해야하지만 provisioning 네트워크를 생략하면 이 요구 사항이 제거됩니다.

    중요

    ISO 이미지와 같은 가상 미디어에서 설치를 시작할 때 이전 UEFI 부팅 테이블 항목을 모두 삭제합니다. 부팅 테이블에 펌웨어에서 제공하는 일반 항목이 아닌 항목이 포함된 경우 설치에 실패할 수 있습니다.

  • Secure Boot: Secure Boot가 활성화된 노드를 사용하려면 UEFI 펌웨어 드라이버, EFI 애플리케이션 및 운영 체제와 같은 신뢰할 수 있는 소프트웨어에서만 노드를 부팅해야 합니다. Secure Boot를 사용하여 수동으로 배포하거나 관리할 수 있습니다.

    1. 수동형: Secure Boot를 사용하여 OpenShift Container Platform 클러스터를 수동으로 배포하려면 각 컨트롤 플레인 노드와 각 작업자 노드에서 UEFI 부팅 모드 및 Secure Boot를 활성화해야 합니다. Red Hat은 설치 관리자 프로비저닝 설치에서 Redfish 가상 미디어를 사용하는 경우에만 수동으로 활성화된 UEFI 및 Secure Boot를 사용하여 Secure Boot를 지원합니다. 자세한 내용은 "노드 구성" 섹션의 "Secure Boot을 위해 수동으로 노드 구성"을 참조하십시오.
    2. 관리형: Secure Boot를 사용하여 OpenShift Container Platform 클러스터를 배포하려면 install-config.yaml 파일에서 bootMode 값을 UEFISecureBoot로 설정해야 합니다. Red Hat은 펌웨어 버전 2.75.75.75 이상을 실행하는 10세대 HPE 하드웨어와 13세대 Dell 하드웨어에 대한 관리형 Secure Boot를 사용하는 설치 관리자 프로비저닝 설치를 지원합니다. 관리형 Secure Boot를 사용하여 배포하는 경우 Redfish 가상 미디어가 필요하지 않습니다. 자세한 내용은 "OpenShift 설치를 위한 환경 설정" 섹션의 "관리형 Secure Boot 구성"을 참조하십시오.

      참고

      Red Hat은 Secure Boot의 자체 생성 키 또는 기타 키 관리를 지원하지 않습니다.

2.2. 클러스터 설치를 위한 최소 리소스 요구 사항

각 클러스터 시스템이 다음과 같은 최소 요구사항을 충족해야 합니다.

Expand
표 2.1. 최소 리소스 요구사항
머신운영 체제CPU [1]RAM스토리지초당 입력/출력(IOPS)[2]

부트스트랩

RHEL

4

16GB

100GB

300

컨트롤 플레인

RHCOS

4

16GB

100GB

300

Compute

RHCOS

2

8GB

100GB

300

  1. SMT(동시 멀티스레딩) 또는 Hyper-Threading이 활성화되지 않은 경우 하나의 CPU가 하나의 물리적 코어와 동일합니다. 활성화하면 다음 공식을 사용하여 해당 비율을 계산합니다. (코어당 스레드 수 × 코어 수) × 소켓 = CPU입니다.
  2. OpenShift Container Platform 및 Kubernetes는 디스크 성능에 민감하며 특히 컨트롤 플레인 노드의 etcd에 더 빠른 스토리지를 사용하는 것이 좋습니다. 많은 클라우드 플랫폼에서 스토리지 크기와 IOPS를 함께 확장되므로 충분한 성능을 얻으려면 스토리지 볼륨을 과도하게 할당해야 할 수 있습니다.
참고

OpenShift Container Platform 버전 4.13부터 RHCOS는 RHEL 버전 9.2를 기반으로 하며 마이크로 아키텍처 요구 사항을 업데이트합니다. 다음 목록에는 각 아키텍처에 필요한 최소 명령 세트 아키텍처(ISA)가 포함되어 있습니다.

  • x86-64 아키텍처에는 x86-64-v2 ISA가 필요합니다.
  • ARM64 아키텍처에는 ARMv8.0-A ISA가 필요합니다.
  • IBM Power 아키텍처에는 Power 9 ISA가 필요합니다.
  • s390x 아키텍처에는 z14 ISA가 필요합니다.

자세한 내용은 아키텍처 (RHEL 문서)를 참조하십시오.

플랫폼의 인스턴스 유형이 클러스터 머신의 최소 요구 사항을 충족하는 경우 OpenShift Container Platform에서 사용할 수 있습니다.

2.3. OpenShift Virtualization을 위한 베어 메탈 클러스터 계획

OpenShift Virtualization을 사용하는 경우 베어 메탈 클러스터를 설치하기 전에 여러 요구 사항을 알고 있어야 합니다.

  • 실시간 마이그레이션 기능을 사용하려면 클러스터 설치 시 여러 개의 작업자 노드가 있어야 합니다. 이는 실시간 마이그레이션에 클러스터 수준 HA(고가용성) 플래그를 true로 설정해야 하기 때문입니다. HA 플래그는 클러스터가 설치될 때 설정되며 나중에 변경할 수 없습니다. 클러스터를 설치할 때 두 개 미만의 작업자 노드가 정의되어 있는 경우 클러스터 수명 동안 HA 플래그가 false로 설정됩니다.

    참고

    단일 노드 클러스터에 OpenShift Virtualization을 설치할 수 있지만 단일 노드 OpenShift는 고가용성을 지원하지 않습니다.

  • 실시간 마이그레이션에는 공유 스토리지가 필요합니다. OpenShift Virtualization용 스토리지는 RWX(ReadWriteMany) 액세스 모드를 지원하고 사용해야 합니다.
  • SR-IOV(Single Root I/O Virtualization)를 사용하려는 경우 OpenShift Container Platform에서 NIC(네트워크 인터페이스 컨트롤러)를 지원하는지 확인합니다.

2.4. 가상 미디어를 사용하여 설치를 위한 펌웨어 요구 사항

설치 관리자 프로비저닝 OpenShift Container Platform 클러스터용 설치 프로그램은 Redfish 가상 미디어와의 하드웨어 및 펌웨어 호환성을 검증합니다. 노드 펌웨어가 호환되지 않는 경우 설치 프로그램이 노드에 설치를 시작하지 않습니다. 다음 표에는 Redfish 가상 미디어를 사용하여 배포된 설치 관리자 프로비저닝 OpenShift Container Platform 클러스터에서 테스트 및 검증된 최소 펌웨어 버전이 나열되어 있습니다.

참고

Red Hat은 펌웨어, 하드웨어 또는 기타 타사 구성 요소의 모든 조합을 테스트하지 않습니다. 타사 지원에 대한 자세한 내용은 Red Hat 타사 지원 정책을 참조하십시오. 펌웨어 업데이트에 대한 자세한 내용은 노드의 하드웨어 설명서를 참조하거나 하드웨어 공급 업체에 문의하십시오.

Expand
표 2.2. Redfish 가상 미디어를 사용한 HP 하드웨어의 펌웨어 호환성
모델관리펌웨어 버전

11세대

iLO6

1.57 이상

10세대

iLO5

2.63 이상

Expand
표 2.3. Redfish 가상 미디어를 사용하여 Dell 하드웨어의 펌웨어 호환성
모델관리펌웨어 버전

16세대

iDRAC 9

v7.10.70.00

15세대

iDRAC 9

v6.10.30.00 및 v7.10.70.00

14세대

iDRAC 9

v6.10.30.00

Expand
표 2.4. Redfish 가상 미디어를 사용한 Cisco UCS 하드웨어의 펌웨어 호환성
모델관리펌웨어 버전

UCS X-Series 서버

Intersight Managed Mode

5.2(2) 이상

FI-Attached UCS C-Series 서버

Intersight Managed Mode

4.3 이상

독립형 UCS C-Series 서버

독립 실행형 / Intersight

4.3 이상

참고

항상 서버가 UCSHCL 에서 RHCOS(Red Hat Enterprise Linux CoreOS)를 지원하는지 확인합니다.

2.5. 네트워크 요구 사항

OpenShift Container Platform의 설치 관리자 프로비저닝 설치에는 여러 네트워크 요구 사항이 필요합니다. 먼저 설치 프로그램에서 프로비저닝한 설치에는 각 베어 메탈 노드에서 운영 체제를 프로비저닝하기 위한 라우팅 불가능한 프로비저닝 네트워크가 필요합니다. 그리고 설치 프로그램에서 프로비저닝하는 설치에는 라우팅 가능한 baremetal 네트워크가 포함됩니다.

2.5.1. 필요한 포트가 열려 있는지 확인

설치 관리자가 프로비저닝한 설치를 완료하려면 특정 포트가 클러스터 노드 간에 열려 있어야 합니다. 멀리 엣지 작업자 노드에 별도의 서브넷을 사용하는 것과 같은 특정 상황에서는 이러한 서브넷의 노드가 다음과 같은 필수 포트의 다른 서브넷의 노드와 통신할 수 있는지 확인해야 합니다.

Expand
표 2.5. 필수 포트
포트설명

67,68

프로비저닝 네트워크를 사용하는 경우 클러스터 노드는 포트 6768 을 사용하여 프로비저닝 네트워크 인터페이스를 통해 dnsmasq DHCP 서버에 액세스합니다.

69

provisioning 네트워크를 사용하는 경우 클러스터 노드는 provisioning 네트워크 인터페이스를 사용하여 포트 69 의 TFTP 서버와 통신합니다. TFTP 서버는 부트스트랩 VM에서 실행됩니다. 부트스트랩 VM은 프로비저너 노드에서 실행됩니다.

80

이미지 캐싱 옵션을 사용하지 않는 경우 또는 가상 미디어를 사용하는 경우 프로비저너 노드에 RHCOS(Red Hat Enterprise Linux CoreOS) 이미지를 프로비저너 노드에서 클러스터 노드로 스트리밍하려면 baremetal 머신 네트워크 인터페이스에 포트 80 이 열려 있어야 합니다.

123

클러스터 노드는 baremetal 시스템 네트워크를 사용하여 포트 123 의 NTP 서버에 액세스해야 합니다.

5050

Ironic Inspector API는 컨트롤 플레인 노드에서 실행되며 포트 5050 에서 수신 대기합니다. Inspector API는 하드웨어 인트로스펙션을 담당하며 베어 메탈 노드의 하드웨어 특성에 대한 정보를 수집합니다.

5051

포트 5050 은 포트 5051 을 프록시로 사용합니다.

6180

가상 미디어를 사용하여 배포하고 TLS를 사용하지 않는 경우 작업자 노드의 BMC(Baseboard Management Controller)가 RHCOS 이미지에 액세스할 수 있도록 프로비저너 노드와 컨트롤 플레인 노드에는 baremetal 머신 네트워크 인터페이스에 포트 6180 이 열려 있어야 합니다. OpenShift Container Platform 4.13부터 기본 HTTP 포트는 6180 입니다.

6183

가상 미디어를 사용하여 배포하고 TLS를 사용하는 경우 작업자 노드의 BMC가 RHCOS 이미지에 액세스할 수 있도록 프로비저너 노드와 컨트롤 플레인 노드에 포트 6183baremetal 머신 네트워크 인터페이스에 열려 있어야 합니다.

6385

Ironic API 서버는 처음에 부트스트랩 VM에서 실행되고 나중에 컨트롤 플레인 노드에서 실행되며 포트 6385 에서 수신 대기합니다. Ironic API를 사용하면 클라이언트는 새 노드 등록, 전원 상태 관리, 이미지 배포, 하드웨어 정리 등의 작업을 포함하여 베어 메탈 노드 프로비저닝 및 관리를 위해 Ironic과 상호 작용할 수 있습니다.

6388

포트 6385 은 포트 6388 을 프록시로 사용합니다.

8080

TLS 없이 이미지 캐싱을 사용하는 경우, 프로비저너 노드에서 포트 8080 을 열고 클러스터 노드의 BMC 인터페이스에서 액세스할 수 있어야 합니다.

8083

TLS와 함께 이미지 캐싱 옵션을 사용하는 경우 8083 포트를 프로비저너 노드에서 열고 클러스터 노드의 BMC 인터페이스에서 액세스할 수 있어야 합니다.

9999

기본적으로 Ironic Python Agent(IPA)는 Ironic conductor 서비스에서 API 호출을 위해 TCP 포트 9999 에서 수신 대기합니다. IPA가 실행 중인 베어 메탈 노드 간 통신과 Ironic conductor 서비스는 이 포트를 사용합니다.

2.5.2. 네트워크 MTU 증가

OpenShift Container Platform을 배포하기 전에 네트워크 최대 전송 단위(MTU)를 1500 이상으로 늘립니다. MTU가 1500 미만이면 노드를 부팅하는 데 사용되는 Ironic 이미지가 Ironic 검사기 Pod와 통신하지 못할 수 있으며 검사가 실패합니다. 이 경우 노드에 설치할 수 없기 때문에 설치가 중지됩니다.

2.5.3. NIC 설정

OpenShift Container Platform은 다음 두 가지 네트워크를 사용하여 배포합니다.

  • provisioning: provisioning 네트워크는 OpenShift Container Platform 클러스터의 일부인 각 노드에서 기본 운영 체제를 프로비저닝하는데 사용되는 선택 옵션인 라우팅할 수 없는 네트워크입니다. 각 클러스터 노드에서 provisioning 네트워크의 네트워크 인터페이스에는 PXE 부팅으로 구성된 BIOS 또는 UEFI가 있어야 합니다.

    provisioningNetworkInterface 구성 설정은 컨트롤 플레인 노드에서 동일한 provisioning 네트워크 NIC 이름을 지정합니다. bootMACAddress 구성 설정은 provisioning 네트워크에 대해 각 노드에서 특정 NIC를 지정하는 수단을 제공합니다.

    provisioning 네트워크는 선택 사항이지만 PXE 부팅에는 필요합니다. provisioning 네트워크없이 배포하는 경우 redfish-virtualmedia 또는 idrac-virtualmedia 와 같은 가상 미디어 BMC 주소 지정 옵션을 사용해야 합니다.

  • baremetal: baremetal 네트워크는 라우팅 가능한 네트워크입니다. NIC가 provisioning 네트워크를 사용하도록 구성되지 않은 경우 모든 NIC를 사용하여 baremetal 네트워크와 상호 작용할 수 있습니다.
중요

VLAN을 사용하는 경우 각 NIC는 적절한 네트워크에 해당하는 별도의 VLAN에 있어야 합니다.

2.5.4. DNS 요구 사항

클라이언트는 baremetal 네트워크를 통해 OpenShift Container Platform 클러스터 노드에 액세스합니다. 네트워크 관리자는 정식 이름 확장이 클러스터 이름인 하위 도메인 또는 하위 영역을 구성해야 합니다.

<cluster_name>.<base_domain>
Copy to Clipboard Toggle word wrap

예를 들면 다음과 같습니다.

test-cluster.example.com
Copy to Clipboard Toggle word wrap

OpenShift Container Platform에는 클러스터 멤버십 정보를 사용하여 A/AAAA 레코드를 생성하는 기능이 포함되어 있습니다. 이렇게 하면 노드 이름이 해당 IP 주소로 확인됩니다. 노드가 API에 등록되면 클러스터에서 CoreDNS-mDNS를 사용하지 않고 노드 정보를 분산할 수 있습니다. 그러면 멀티캐스트 DNS와 연결된 네트워크 트래픽이 제거됩니다.

CoreDNS가 올바르게 작동하려면 업스트림 DNS 서버에 TCP 및 UDP 연결이 모두 필요합니다. 업스트림 DNS 서버가 OpenShift Container Platform 클러스터 노드에서 TCP 및 UDP 연결을 모두 수신할 수 있는지 확인합니다.

OpenShift Container Platform 배포의 경우 다음 구성 요소에 DNS 이름을 확인해야 합니다.

  • Kubernetes API
  • OpenShift Container Platform 애플리케이션 와일드카드 수신 API

A/AAAA 레코드는 이름 확인에 사용되며 PTR 레코드는 역방향 이름 확인에 사용됩니다. RHCOS(Red Hat Enterprise Linux CoreOS)는 역방향 레코드 또는 DHCP를 사용하여 모든 노드의 호스트 이름을 설정합니다.

설치 프로그램에서 제공하는 설치에는 클러스터 멤버십 정보를 사용하여 A/AAAA 레코드를 생성하는 기능이 포함됩니다. 이렇게 하면 노드 이름이 해당 IP 주소로 확인됩니다. 각 레코드에서 <cluster_name>은 클러스터 이름이고 <base_domain>install-config.yaml 파일에서 지정하는 기반 도메인입니다. 전체 DNS 레코드는 <component>.<cluster_name>.<base_domain> 형식입니다.

Expand
표 2.6. 필수 DNS 레코드
구성 요소레코드설명

Kubernetes API

api.<cluster_name>.<base_domain>.

A/AAAA 레코드와 PTR 레코드는 API 로드 밸런서를 식별합니다. 이 레코드는 클러스터 외부의 클라이언트와 클러스터 내의 모든 노드에서 확인할 수 있어야 합니다.

라우트

*.apps.<cluster_name>.<base_domain>.

와일드카드 A/AAAA 레코드는 애플리케이션 인그레스 로드 밸런서를 나타냅니다. 애플리케이션 인그레스 로드 밸런서는 Ingress 컨트롤러 Pod를 실행하는 노드를 대상으로 합니다. Ingress 컨트롤러 Pod는 기본적으로 작업자 노드에서 실행됩니다. 이 레코드는 클러스터 외부의 클라이언트와 클러스터 내의 모든 노드에서 확인할 수 있어야 합니다.

예를 들어 console-openshift-console.apps.<cluster_name>.<base_domain>은 OpenShift Container Platform 콘솔의 와일드카드 경로로 사용됩니다.

작은 정보

dig 명령을 사용하여 DNS 확인을 확인할 수 있습니다.

2.5.5. DHCP(Dynamic Host Configuration Protocol) 요구 사항

기본적으로 설치 프로그램에서 제공하는 설치는 provisioning 네트워크에 DHCP가 활성화된 ironic-dnsmasq를 배포합니다. provisioningNetwork 구성 설정이 기본값인 managed로 설정된 경우 provisioning 네트워크에서 다른 DHCP 서버가 실행되고 있지 않아야 합니다. provisioning 네트워크에서 실행 중인 DHCP 서버가 있는 경우 install-config.yaml 파일에서 provisioningNetwork 구성 설정을 Unmanaged로 설정해야 합니다.

네트워크 관리자는 외부 DHCP 서버의 baremetal 네트워크에 대해 OpenShift Container Platform 클러스터의 각 노드에 대한 IP 주소를 예약해야 합니다.

2.5.6. DHCP 서버를 사용하여 노드의 IP 주소 예약

baremetal 네트워크의 경우 네트워크 관리자는 다음을 포함하여 여러 IP 주소를 예약해야 합니다.

  1. 두 개의 고유한 가상 IP 주소입니다.

    • API 엔드 포인트에 대한 하나의 가상 IP 주소입니다.
    • 와일드카드 인그레스 끝점에 대한 하나의 가상 IP 주소입니다.
  2. 프로비저너 노드 중 하나의 IP 주소.
  3. 각 컨트롤 플레인 노드에 대해 하나의 IP 주소입니다.
  4. 각 작업자 노드에 대해 하나의 IP 주소 (해당되는 경우)
고정 IP 주소가 되도록 IP 주소 예약

일부 관리자는 각 노드의 IP 주소가 DHCP 서버에서 일정하게 유지되도록 고정 IP 주소를 사용하는 것을 선호합니다. NMState를 사용하여 고정 IP 주소를 구성하려면 "OpenShift 설치를 위한 환경 설정" 섹션의 "(선택 사항) 노드 네트워크 인터페이스 구성을 참조하십시오.

외부 로드 밸런서와 컨트롤 플레인 노드 간 네트워킹

외부 로드 밸런싱 서비스와 컨트롤 플레인 노드는 동일한 L2 네트워크에서 실행해야 하며 VLAN을 사용하여 로드 밸런싱 서비스와 컨트롤 플레인 노드 간에 트래픽을 라우팅할 때 동일한 VLAN에서 실행해야 합니다.

중요

스토리지 인터페이스에 DHCP 예약 또는 고정 IP가 필요합니다.

다음 표에서는 정규화된 도메인 이름의 구체적 구현을 제공합니다. API 및 이름 서버 주소는 표준 이름 확장으로 시작됩니다. 컨트롤 플레인 및 작업자 노드의 호스트 이름은 예외이므로 원하는 호스트 이름 지정 규칙을 사용할 수 있습니다.

Expand
사용법호스트 이름IP

API

api.<cluster_name>.<base_domain>

<ip>

Ingress LB (apps)

*.apps.<cluster_name>.<base_domain>

<ip>

Provisioner node

provisioner.<cluster_name>.<base_domain>

<ip>

Control-plane-0

openshift-control-plane-0.<cluster_name>.<base_domain>

<ip>

Control-plane-1

openshift-control-plane-1.<cluster_name>-.<base_domain>

<ip>

Control-plane-2

openshift-control-plane-2.<cluster_name>.<base_domain>

<ip>

Worker-0

openshift-worker-0.<cluster_name>.<base_domain>

<ip>

Worker-1

openshift-worker-1.<cluster_name>.<base_domain>

<ip>

Worker-n

openshift-worker-n.<cluster_name>.<base_domain>

참고

2.5.7.

2.5.8.

중요

2.5.9.

2.6.

주의

Expand
   

Expand
  

참고

Expand
  

Expand
   

중요

참고

중요

2.7.

참고

2.8.

2.9.

3장.

3.1.

3.2.

  1. # useradd kni
    Copy to Clipboard Toggle word wrap
    # passwd kni
    Copy to Clipboard Toggle word wrap
    # echo "kni ALL=(root) NOPASSWD:ALL" | tee -a /etc/sudoers.d/kni
    Copy to Clipboard Toggle word wrap
    # chmod 0440 /etc/sudoers.d/kni
    Copy to Clipboard Toggle word wrap
  2. # su - kni -c "ssh-keygen -t ed25519 -f /home/kni/.ssh/id_rsa -N ''"
    Copy to Clipboard Toggle word wrap
  3. # su - kni
    Copy to Clipboard Toggle word wrap
  4. $ sudo subscription-manager register --username=<user> --password=<pass> --auto-attach
    Copy to Clipboard Toggle word wrap
    $ sudo subscription-manager repos --enable=rhel-9-for-<architecture>-appstream-rpms --enable=rhel-9-for-<architecture>-baseos-rpms
    Copy to Clipboard Toggle word wrap
    참고

  5. $ sudo dnf install -y libvirt qemu-kvm mkisofs python3-devel jq ipmitool
    Copy to Clipboard Toggle word wrap
  6. $ sudo usermod --append --groups libvirt <user>
    Copy to Clipboard Toggle word wrap
  7. $ sudo systemctl start firewalld
    Copy to Clipboard Toggle word wrap
    $ sudo firewall-cmd --zone=public --add-service=http --permanent
    Copy to Clipboard Toggle word wrap
    $ sudo firewall-cmd --reload
    Copy to Clipboard Toggle word wrap
  8. $ sudo systemctl enable libvirtd --now
    Copy to Clipboard Toggle word wrap
  9. $ sudo virsh pool-define-as --name default --type dir --target /var/lib/libvirt/images
    Copy to Clipboard Toggle word wrap
    $ sudo virsh pool-start default
    Copy to Clipboard Toggle word wrap
    $ sudo virsh pool-autostart default
    Copy to Clipboard Toggle word wrap
  10. $ vim pull-secret.txt
    Copy to Clipboard Toggle word wrap

3.3.

  1. $ chronyc sources
    Copy to Clipboard Toggle word wrap

    MS Name/IP address         Stratum Poll Reach LastRx Last sample
    ===============================================================================
    ^+ time.cloudflare.com           3  10   377   187   -209us[ -209us] +/-   32ms
    ^+ t1.time.ir2.yahoo.com         2  10   377   185  -4382us[-4382us] +/-   23ms
    ^+ time.cloudflare.com           3  10   377   198   -996us[-1220us] +/-   33ms
    ^* brenbox.westnet.ie            1  10   377   193  -9538us[-9761us] +/-   24ms
    Copy to Clipboard Toggle word wrap

  2. $ ping time.cloudflare.com
    Copy to Clipboard Toggle word wrap

    PING time.cloudflare.com (162.159.200.123) 56(84) bytes of data.
    64 bytes from time.cloudflare.com (162.159.200.123): icmp_seq=1 ttl=54 time=32.3 ms
    64 bytes from time.cloudflare.com (162.159.200.123): icmp_seq=2 ttl=54 time=30.9 ms
    64 bytes from time.cloudflare.com (162.159.200.123): icmp_seq=3 ttl=54 time=36.7 ms
    ...
    Copy to Clipboard Toggle word wrap

3.4.

참고

  1. $ export PUB_CONN=<baremetal_nic_name>
    Copy to Clipboard Toggle word wrap
  2. 참고

    1. $ sudo nohup bash -c "
          nmcli con down \"$PUB_CONN\"
          nmcli con delete \"$PUB_CONN\"
          # RHEL 8.1 appends the word \"System\" in front of the connection, delete in case it exists
          nmcli con down \"System $PUB_CONN\"
          nmcli con delete \"System $PUB_CONN\"
          nmcli connection add ifname baremetal type bridge <con_name> baremetal bridge.stp no 
      1
      
          nmcli con add type bridge-slave ifname \"$PUB_CONN\" master baremetal
          pkill dhclient;dhclient baremetal
      "
      Copy to Clipboard Toggle word wrap
      1
    2. $ sudo nohup bash -c "
          nmcli con down \"$PUB_CONN\"
          nmcli con delete \"$PUB_CONN\"
          # RHEL 8.1 appends the word \"System\" in front of the connection, delete in case it exists
          nmcli con down \"System $PUB_CONN\"
          nmcli con delete \"System $PUB_CONN\"
          nmcli connection add ifname baremetal type bridge con-name baremetal bridge.stp no ipv4.method manual ipv4.addr "x.x.x.x/yy" ipv4.gateway "a.a.a.a" ipv4.dns "b.b.b.b" 
      1
      
          nmcli con add type bridge-slave ifname \"$PUB_CONN\" master baremetal
          nmcli con up baremetal
      "
      Copy to Clipboard Toggle word wrap
      1
  3. $ export PROV_CONN=<prov_nic_name>
    Copy to Clipboard Toggle word wrap
  4. $ sudo nohup bash -c "
        nmcli con down \"$PROV_CONN\"
        nmcli con delete \"$PROV_CONN\"
        nmcli connection add ifname provisioning type bridge con-name provisioning
        nmcli con add type bridge-slave ifname \"$PROV_CONN\" master provisioning
        nmcli connection modify provisioning ipv6.addresses fd00:1101::1/64 ipv6.method manual
        nmcli con down provisioning
        nmcli con up provisioning
    "
    Copy to Clipboard Toggle word wrap
    참고

  5. $ nmcli connection modify provisioning ipv4.addresses 172.22.0.254/24 ipv4.method manual
    Copy to Clipboard Toggle word wrap
  6. # ssh kni@provisioner.<cluster-name>.<domain>
    Copy to Clipboard Toggle word wrap
  7. $ sudo nmcli con show
    Copy to Clipboard Toggle word wrap

    NAME               UUID                                  TYPE      DEVICE
    baremetal          4d5133a5-8351-4bb9-bfd4-3af264801530  bridge    baremetal
    provisioning       43942805-017f-4d7d-a2c2-7cb3324482ed  bridge    provisioning
    virbr0             d9bca40f-eee1-410b-8879-a2d4bb0465e7  bridge    virbr0
    bridge-slave-eno1  76a8ed50-c7e5-4999-b4f6-6d9014dd0812  ethernet  eno1
    bridge-slave-eno2  f31c3353-54b7-48de-893a-02d2b34c4736  ethernet  eno2
    Copy to Clipboard Toggle word wrap

3.5.

참고

  1. interfaces:
    - name: enp2s0 
    1
    
      type: ethernet 
    2
    
      state: up 
    3
    
      ipv4:
        enabled: false 
    4
    
      ipv6:
        enabled: false
    - name: br-ex
      type: ovs-bridge
      state: up
      ipv4:
        enabled: false
        dhcp: false
      ipv6:
        enabled: false
        dhcp: false
      bridge:
        port:
        - name: enp2s0 
    5
    
        - name: br-ex
    - name: br-ex
      type: ovs-interface
      state: up
      copy-mac-from: enp2s0
      ipv4:
        enabled: true
        dhcp: true
      ipv6:
        enabled: false
        dhcp: false
    # ...
    Copy to Clipboard Toggle word wrap

    1
    2
    3
    4
    5
  2. $ cat <nmstate_configuration>.yaml | base64 
    1
    Copy to Clipboard Toggle word wrap
    1
  3. apiVersion: machineconfiguration.openshift.io/v1
    kind: MachineConfig
    metadata:
      labels:
        machineconfiguration.openshift.io/role: worker 
    1
    
      name: 10-br-ex-worker 
    2
    
    spec:
      config:
        ignition:
          version: 3.2.0
        storage:
          files:
          - contents:
              source: data:text/plain;charset=utf-8;base64,<base64_encoded_nmstate_configuration> 
    3
    
            mode: 0644
            overwrite: true
            path: /etc/nmstate/openshift/cluster.yml
    # ...
    Copy to Clipboard Toggle word wrap
    1
    2
    3

3.5.1.

  1. $ oc edit mc <machineconfig_custom_resource_name>
    Copy to Clipboard Toggle word wrap
  2. $ oc apply -f ./extraworker-secret.yaml
    Copy to Clipboard Toggle word wrap
  3. apiVersion: metal3.io/v1alpha1
    kind: BareMetalHost
    spec:
    # ...
      preprovisioningNetworkDataName: ostest-extraworker-0-network-config-secret
    # ...
    Copy to Clipboard Toggle word wrap

  4. $ oc project openshift-machine-api
    Copy to Clipboard Toggle word wrap
  5. $ oc get machinesets
    Copy to Clipboard Toggle word wrap
  6. $ oc scale machineset <machineset_name> --replicas=<n> 
    1
    Copy to Clipboard Toggle word wrap
    1

3.6.

참고

    1. $ sudo su -
      Copy to Clipboard Toggle word wrap
    2. # nmcli dev status
      Copy to Clipboard Toggle word wrap
    3. # nmcli connection modify <interface_name> +ipv4.routes "192.168.0.0/24 via <gateway>"
      Copy to Clipboard Toggle word wrap

      # nmcli connection modify eth0 +ipv4.routes "192.168.0.0/24 via 192.168.0.1"
      Copy to Clipboard Toggle word wrap

    4. # nmcli connection up <interface_name>
      Copy to Clipboard Toggle word wrap

    5. # ip route
      Copy to Clipboard Toggle word wrap
    6. 참고

    1. $ sudo su -
      Copy to Clipboard Toggle word wrap
    2. # nmcli dev status
      Copy to Clipboard Toggle word wrap
    3. # nmcli connection modify <interface_name> +ipv4.routes "10.0.0.0/24 via <gateway>"
      Copy to Clipboard Toggle word wrap

      # nmcli connection modify eth0 +ipv4.routes "10.0.0.0/24 via 10.0.0.1"
      Copy to Clipboard Toggle word wrap

    4. # nmcli connection up <interface_name>
      Copy to Clipboard Toggle word wrap

    5. # ip route
      Copy to Clipboard Toggle word wrap
    6. 참고

    1. $ ping <remote_node_ip_address>
      Copy to Clipboard Toggle word wrap

    2. $ ping <control_plane_node_ip_address>
      Copy to Clipboard Toggle word wrap

3.7.

$ export VERSION=stable-4.17
Copy to Clipboard Toggle word wrap
$ export RELEASE_ARCH=<architecture>
Copy to Clipboard Toggle word wrap
$ export RELEASE_IMAGE=$(curl -s https://mirror.openshift.com/pub/openshift-v4/$RELEASE_ARCH/clients/ocp/$VERSION/release.txt | grep 'Pull From: quay.io' | awk -F ' ' '{print $3}')
Copy to Clipboard Toggle word wrap

3.8.

  1. $ export cmd=openshift-baremetal-install
    Copy to Clipboard Toggle word wrap
    $ export pullsecret_file=~/pull-secret.txt
    Copy to Clipboard Toggle word wrap
    $ export extract_dir=$(pwd)
    Copy to Clipboard Toggle word wrap
  2. $ curl -s https://mirror.openshift.com/pub/openshift-v4/clients/ocp/$VERSION/openshift-client-linux.tar.gz | tar zxvf - oc
    Copy to Clipboard Toggle word wrap
  3. $ sudo cp oc /usr/local/bin
    Copy to Clipboard Toggle word wrap
    $ oc adm release extract --registry-config "${pullsecret_file}" --command=$cmd --to "${extract_dir}" ${RELEASE_IMAGE}
    Copy to Clipboard Toggle word wrap
    $ sudo cp openshift-baremetal-install /usr/local/bin
    Copy to Clipboard Toggle word wrap

3.9.

참고

주의

  1. $ sudo dnf install -y podman
    Copy to Clipboard Toggle word wrap
  2. $ sudo firewall-cmd --add-port=8080/tcp --zone=public --permanent
    Copy to Clipboard Toggle word wrap
    $ sudo firewall-cmd --reload
    Copy to Clipboard Toggle word wrap
  3. $ mkdir /home/kni/rhcos_image_cache
    Copy to Clipboard Toggle word wrap
  4. $ sudo semanage fcontext -a -t httpd_sys_content_t "/home/kni/rhcos_image_cache(/.*)?"
    Copy to Clipboard Toggle word wrap
    $ sudo restorecon -Rv /home/kni/rhcos_image_cache/
    Copy to Clipboard Toggle word wrap
  5. $ export RHCOS_QEMU_URI=$(/usr/local/bin/openshift-baremetal-install coreos print-stream-json | jq -r --arg ARCH "$(arch)" '.architectures[$ARCH].artifacts.qemu.formats["qcow2.gz"].disk.location')
    Copy to Clipboard Toggle word wrap
  6. $ export RHCOS_QEMU_NAME=${RHCOS_QEMU_URI##*/}
    Copy to Clipboard Toggle word wrap
  7. $ export RHCOS_QEMU_UNCOMPRESSED_SHA256=$(/usr/local/bin/openshift-baremetal-install coreos print-stream-json | jq -r --arg ARCH "$(arch)" '.architectures[$ARCH].artifacts.qemu.formats["qcow2.gz"].disk["uncompressed-sha256"]')
    Copy to Clipboard Toggle word wrap
  8. $ curl -L ${RHCOS_QEMU_URI} -o /home/kni/rhcos_image_cache/${RHCOS_QEMU_NAME}
    Copy to Clipboard Toggle word wrap
  9. $ ls -Z /home/kni/rhcos_image_cache
    Copy to Clipboard Toggle word wrap
  10. $ podman run -d --name rhcos_image_cache \
    1
    
    -v /home/kni/rhcos_image_cache:/var/www/html \
    -p 8080:8080/tcp \
    registry.access.redhat.com/ubi9/httpd-24
    Copy to Clipboard Toggle word wrap
    1
  11. $ export BAREMETAL_IP=$(ip addr show dev baremetal | awk '/inet /{print $2}' | cut -d"/" -f1)
    Copy to Clipboard Toggle word wrap
    $ export BOOTSTRAP_OS_IMAGE="http://${BAREMETAL_IP}:8080/${RHCOS_QEMU_NAME}?sha256=${RHCOS_QEMU_UNCOMPRESSED_SHA256}"
    Copy to Clipboard Toggle word wrap
    $ echo "    bootstrapOSImage=${BOOTSTRAP_OS_IMAGE}"
    Copy to Clipboard Toggle word wrap
  12. platform:
      baremetal:
        bootstrapOSImage: <bootstrap_os_image>  
    1
    Copy to Clipboard Toggle word wrap
    1

3.10.

중요

그림 3.1.

그림 3.2.

그림 3.3.

  • 작은 정보

3.10.1.

중요

참고

Path: HTTPS:6443/readyz
Healthy threshold: 2
Unhealthy threshold: 2
Timeout: 10
Interval: 10
Copy to Clipboard Toggle word wrap

Path: HTTPS:22623/healthz
Healthy threshold: 2
Unhealthy threshold: 2
Timeout: 10
Interval: 10
Copy to Clipboard Toggle word wrap

Path: HTTP:1936/healthz/ready
Healthy threshold: 2
Unhealthy threshold: 2
Timeout: 5
Interval: 10
Copy to Clipboard Toggle word wrap

  1. # ...
    listen my-cluster-api-6443
        bind 192.168.1.100:6443
        mode tcp
        balance roundrobin
      option httpchk
      http-check connect
      http-check send meth GET uri /readyz
      http-check expect status 200
        server my-cluster-master-2 192.168.1.101:6443 check inter 10s rise 2 fall 2
        server my-cluster-master-0 192.168.1.102:6443 check inter 10s rise 2 fall 2
        server my-cluster-master-1 192.168.1.103:6443 check inter 10s rise 2 fall 2
    
    listen my-cluster-machine-config-api-22623
        bind 192.168.1.100:22623
        mode tcp
        balance roundrobin
      option httpchk
      http-check connect
      http-check send meth GET uri /healthz
      http-check expect status 200
        server my-cluster-master-2 192.168.1.101:22623 check inter 10s rise 2 fall 2
        server my-cluster-master-0 192.168.1.102:22623 check inter 10s rise 2 fall 2
        server my-cluster-master-1 192.168.1.103:22623 check inter 10s rise 2 fall 2
    
    listen my-cluster-apps-443
        bind 192.168.1.100:443
        mode tcp
        balance roundrobin
      option httpchk
      http-check connect
      http-check send meth GET uri /healthz/ready
      http-check expect status 200
        server my-cluster-worker-0 192.168.1.111:443 check port 1936 inter 10s rise 2 fall 2
        server my-cluster-worker-1 192.168.1.112:443 check port 1936 inter 10s rise 2 fall 2
        server my-cluster-worker-2 192.168.1.113:443 check port 1936 inter 10s rise 2 fall 2
    
    listen my-cluster-apps-80
       bind 192.168.1.100:80
       mode tcp
       balance roundrobin
      option httpchk
      http-check connect
      http-check send meth GET uri /healthz/ready
      http-check expect status 200
        server my-cluster-worker-0 192.168.1.111:80 check port 1936 inter 10s rise 2 fall 2
        server my-cluster-worker-1 192.168.1.112:80 check port 1936 inter 10s rise 2 fall 2
        server my-cluster-worker-2 192.168.1.113:80 check port 1936 inter 10s rise 2 fall 2
    # ...
    Copy to Clipboard Toggle word wrap

    # ...
    listen api-server-6443
        bind *:6443
        mode tcp
          server master-00 192.168.83.89:6443 check inter 1s
          server master-01 192.168.84.90:6443 check inter 1s
          server master-02 192.168.85.99:6443 check inter 1s
          server bootstrap 192.168.80.89:6443 check inter 1s
    
    listen machine-config-server-22623
        bind *:22623
        mode tcp
          server master-00 192.168.83.89:22623 check inter 1s
          server master-01 192.168.84.90:22623 check inter 1s
          server master-02 192.168.85.99:22623 check inter 1s
          server bootstrap 192.168.80.89:22623 check inter 1s
    
    listen ingress-router-80
        bind *:80
        mode tcp
        balance source
          server worker-00 192.168.83.100:80 check inter 1s
          server worker-01 192.168.83.101:80 check inter 1s
    
    listen ingress-router-443
        bind *:443
        mode tcp
        balance source
          server worker-00 192.168.83.100:443 check inter 1s
          server worker-01 192.168.83.101:443 check inter 1s
    
    listen ironic-api-6385
        bind *:6385
        mode tcp
        balance source
          server master-00 192.168.83.89:6385 check inter 1s
          server master-01 192.168.84.90:6385 check inter 1s
          server master-02 192.168.85.99:6385 check inter 1s
          server bootstrap 192.168.80.89:6385 check inter 1s
    
    listen inspector-api-5050
        bind *:5050
        mode tcp
        balance source
          server master-00 192.168.83.89:5050 check inter 1s
          server master-01 192.168.84.90:5050 check inter 1s
          server master-02 192.168.85.99:5050 check inter 1s
          server bootstrap 192.168.80.89:5050 check inter 1s
    # ...
    Copy to Clipboard Toggle word wrap

    1. $ curl https://<loadbalancer_ip_address>:6443/version --insecure
      Copy to Clipboard Toggle word wrap

      {
        "major": "1",
        "minor": "11+",
        "gitVersion": "v1.11.0+ad103ed",
        "gitCommit": "ad103ed",
        "gitTreeState": "clean",
        "buildDate": "2019-01-09T06:44:10Z",
        "goVersion": "go1.10.3",
        "compiler": "gc",
        "platform": "linux/amd64"
      }
      Copy to Clipboard Toggle word wrap
    2. $ curl -v https://<loadbalancer_ip_address>:22623/healthz --insecure
      Copy to Clipboard Toggle word wrap

      HTTP/1.1 200 OK
      Content-Length: 0
      Copy to Clipboard Toggle word wrap
    3. $ curl -I -L -H "Host: console-openshift-console.apps.<cluster_name>.<base_domain>" http://<load_balancer_front_end_IP_address>
      Copy to Clipboard Toggle word wrap

      HTTP/1.1 302 Found
      content-length: 0
      location: https://console-openshift-console.apps.ocp4.private.opequon.net/
      cache-control: no-cache
      Copy to Clipboard Toggle word wrap
    4. $ curl -I -L --insecure --resolve console-openshift-console.apps.<cluster_name>.<base_domain>:443:<Load Balancer Front End IP Address> https://console-openshift-console.apps.<cluster_name>.<base_domain>
      Copy to Clipboard Toggle word wrap

      HTTP/1.1 200 OK
      referrer-policy: strict-origin-when-cross-origin
      set-cookie: csrf-token=UlYWOyQ62LWjw2h003xtYSKlh1a0Py2hhctw0WmV2YEdhJjFyQwWcGBsja261dGLgaYO0nxzVErhiXt6QepA7g==; Path=/; Secure; SameSite=Lax
      x-content-type-options: nosniff
      x-dns-prefetch-control: off
      x-frame-options: DENY
      x-xss-protection: 1; mode=block
      date: Wed, 04 Oct 2023 16:29:38 GMT
      content-type: text/html; charset=utf-8
      set-cookie: 1e2670d92730b515ce3a1bb65da45062=1bf5e9573c9a2760c964ed1659cc1673; path=/; HttpOnly; Secure; SameSite=None
      cache-control: private
      Copy to Clipboard Toggle word wrap
  2. <load_balancer_ip_address>  A  api.<cluster_name>.<base_domain>
    A record pointing to Load Balancer Front End
    Copy to Clipboard Toggle word wrap

    <load_balancer_ip_address>   A apps.<cluster_name>.<base_domain>
    A record pointing to Load Balancer Front End
    Copy to Clipboard Toggle word wrap
    중요

  3. # ...
    platform:
      baremetal:
        loadBalancer:
          type: UserManaged 
    1
    
        apiVIPs:
        - <api_ip> 
    2
    
        ingressVIPs:
        - <ingress_ip> 
    3
    
    # ...
    Copy to Clipboard Toggle word wrap
    1
    2
    3

    1. $ curl https://api.<cluster_name>.<base_domain>:6443/version --insecure
      Copy to Clipboard Toggle word wrap

      {
        "major": "1",
        "minor": "11+",
        "gitVersion": "v1.11.0+ad103ed",
        "gitCommit": "ad103ed",
        "gitTreeState": "clean",
        "buildDate": "2019-01-09T06:44:10Z",
        "goVersion": "go1.10.3",
        "compiler": "gc",
        "platform": "linux/amd64"
        }
      Copy to Clipboard Toggle word wrap
    2. $ curl -v https://api.<cluster_name>.<base_domain>:22623/healthz --insecure
      Copy to Clipboard Toggle word wrap

      HTTP/1.1 200 OK
      Content-Length: 0
      Copy to Clipboard Toggle word wrap
    3. $ curl http://console-openshift-console.apps.<cluster_name>.<base_domain> -I -L --insecure
      Copy to Clipboard Toggle word wrap

      HTTP/1.1 302 Found
      content-length: 0
      location: https://console-openshift-console.apps.<cluster-name>.<base domain>/
      cache-control: no-cacheHTTP/1.1 200 OK
      referrer-policy: strict-origin-when-cross-origin
      set-cookie: csrf-token=39HoZgztDnzjJkq/JuLJMeoKNXlfiVv2YgZc09c3TBOBU4NI6kDXaJH1LdicNhN1UsQWzon4Dor9GWGfopaTEQ==; Path=/; Secure
      x-content-type-options: nosniff
      x-dns-prefetch-control: off
      x-frame-options: DENY
      x-xss-protection: 1; mode=block
      date: Tue, 17 Nov 2020 08:42:10 GMT
      content-type: text/html; charset=utf-8
      set-cookie: 1e2670d92730b515ce3a1bb65da45062=9b714eb87e93cf34853e87a92d6894be; path=/; HttpOnly; Secure; SameSite=None
      cache-control: private
      Copy to Clipboard Toggle word wrap
    4. $ curl https://console-openshift-console.apps.<cluster_name>.<base_domain> -I -L --insecure
      Copy to Clipboard Toggle word wrap

      HTTP/1.1 200 OK
      referrer-policy: strict-origin-when-cross-origin
      set-cookie: csrf-token=UlYWOyQ62LWjw2h003xtYSKlh1a0Py2hhctw0WmV2YEdhJjFyQwWcGBsja261dGLgaYO0nxzVErhiXt6QepA7g==; Path=/; Secure; SameSite=Lax
      x-content-type-options: nosniff
      x-dns-prefetch-control: off
      x-frame-options: DENY
      x-xss-protection: 1; mode=block
      date: Wed, 04 Oct 2023 16:29:38 GMT
      content-type: text/html; charset=utf-8
      set-cookie: 1e2670d92730b515ce3a1bb65da45062=1bf5e9573c9a2760c964ed1659cc1673; path=/; HttpOnly; Secure; SameSite=None
      cache-control: private
      Copy to Clipboard Toggle word wrap

3.11.

작은 정보

3.12.

3.12.1.

참고

  1. apiVersion: v1
    baseDomain: <domain>
    metadata:
      name: <cluster_name>
    networking:
      machineNetwork:
      - cidr: <public_cidr>
      networkType: OVNKubernetes
    compute:
    - name: worker
      replicas: 2 
    1
    
    controlPlane:
      name: master
      replicas: 3
      platform:
        baremetal: {}
    platform:
      baremetal:
        apiVIPs:
          - <api_ip>
        ingressVIPs:
          - <wildcard_ip>
        provisioningNetworkCIDR: <CIDR>
        bootstrapExternalStaticIP: <bootstrap_static_ip_address> 
    2
    
        bootstrapExternalStaticGateway: <bootstrap_static_gateway> 
    3
    
        bootstrapExternalStaticDNS: <bootstrap_static_dns> 
    4
    
        hosts:
          - name: openshift-master-0
            role: master
            bmc:
              address: ipmi://<out_of_band_ip> 
    5
    
              username: <user>
              password: <password>
            bootMACAddress: <NIC1_mac_address>
            rootDeviceHints:
             deviceName: "<installation_disk_drive_path>" 
    6
    
          - name: <openshift_master_1>
            role: master
            bmc:
              address: ipmi://<out_of_band_ip>
              username: <user>
              password: <password>
            bootMACAddress: <NIC1_mac_address>
            rootDeviceHints:
             deviceName: "<installation_disk_drive_path>"
          - name: <openshift_master_2>
            role: master
            bmc:
              address: ipmi://<out_of_band_ip>
              username: <user>
              password: <password>
            bootMACAddress: <NIC1_mac_address>
            rootDeviceHints:
             deviceName: "<installation_disk_drive_path>"
          - name: <openshift_worker_0>
            role: worker
            bmc:
              address: ipmi://<out_of_band_ip>
              username: <user>
              password: <password>
            bootMACAddress: <NIC1_mac_address>
          - name: <openshift_worker_1>
            role: worker
            bmc:
              address: ipmi://<out_of_band_ip>
              username: <user>
              password: <password>
            bootMACAddress: <NIC1_mac_address>
            rootDeviceHints:
             deviceName: "<installation_disk_drive_path>"
    pullSecret: '<pull_secret>'
    sshKey: '<ssh_pub_key>'
    Copy to Clipboard Toggle word wrap
    1
    2
    3
    4
    5
    6
    중요

    ironic-inspector inspection failed: No disks satisfied root device hints
    Copy to Clipboard Toggle word wrap
    참고

  2. $ mkdir ~/clusterconfigs
    Copy to Clipboard Toggle word wrap
  3. $ cp install-config.yaml ~/clusterconfigs
    Copy to Clipboard Toggle word wrap
  4. $ ipmitool -I lanplus -U <user> -P <password> -H <management-server-ip> power off
    Copy to Clipboard Toggle word wrap
  5. for i in $(sudo virsh list | tail -n +3 | grep bootstrap | awk {'print $2'});
    do
      sudo virsh destroy $i;
      sudo virsh undefine $i;
      sudo virsh vol-delete $i --pool $i;
      sudo virsh vol-delete $i.ign --pool $i;
      sudo virsh pool-destroy $i;
      sudo virsh pool-undefine $i;
    done
    Copy to Clipboard Toggle word wrap

3.12.2.

Expand
표 3.1.
   

 

 

 

 

 

 

metadata:
    name:
Copy to Clipboard Toggle word wrap
 

networking:
    machineNetwork:
    - cidr:
Copy to Clipboard Toggle word wrap
 

compute:
  - name: worker
Copy to Clipboard Toggle word wrap
 

compute:
    replicas: 2
Copy to Clipboard Toggle word wrap
 

controlPlane:
    name: master
Copy to Clipboard Toggle word wrap
 

controlPlane:
    replicas: 3
Copy to Clipboard Toggle word wrap
 

 

 

 

참고

 

참고

Expand
표 3.2.
   

 

 

 

 

 

 

 

Expand
표 3.3.
   

 

 

 

 

참고

 

3.12.3.

platform:
  baremetal:
    hosts:
      - name: openshift-master-0
        role: master
        bmc:
          address: ipmi://<out-of-band-ip>
          username: <user>
          password: <password>
Copy to Clipboard Toggle word wrap
중요

platform:
  baremetal:
    hosts:
      - name: openshift-master-0
        role: master
        bmc:
          address: redfish://<out-of-band-ip>/redfish/v1/Systems/1
          username: <user>
          password: <password>
Copy to Clipboard Toggle word wrap

platform:
  baremetal:
    hosts:
      - name: openshift-master-0
        role: master
        bmc:
          address: redfish://<out-of-band-ip>/redfish/v1/Systems/1
          username: <user>
          password: <password>
          disableCertificateVerification: True
Copy to Clipboard Toggle word wrap

3.12.4.

  1. $ export SERVER=<ip_address> 
    1
    Copy to Clipboard Toggle word wrap
    1
  2. $ export SystemID=<system_id> 
    1
    Copy to Clipboard Toggle word wrap
    1

  1. $ curl -u $USER:$PASS -X POST -H'Content-Type: application/json' -H'Accept: application/json' -d '{"ResetType": "On"}' https://$SERVER/redfish/v1/Systems/$SystemID/Actions/ComputerSystem.Reset
    Copy to Clipboard Toggle word wrap
  2. $ curl -u $USER:$PASS -X POST -H'Content-Type: application/json' -H'Accept: application/json' -d '{"ResetType": "ForceOff"}' https://$SERVER/redfish/v1/Systems/$SystemID/Actions/ComputerSystem.Reset
    Copy to Clipboard Toggle word wrap
  3. $ curl -u $USER:$PASS -X PATCH -H "Content-Type: application/json" -H "If-Match: <ETAG>"  https://$Server/redfish/v1/Systems/$SystemID/ -d '{"Boot": {"BootSourceOverrideTarget": "pxe", "BootSourceOverrideEnabled": "Once"}}
    Copy to Clipboard Toggle word wrap
  4. $ curl -u $USER:$PASS -X PATCH -H "Content-Type: application/json" -H "If-Match: <ETAG>"  https://$Server/redfish/v1/Systems/$SystemID/ -d '{"Boot": {"BootSourceOverrideMode":"UEFI"}}
    Copy to Clipboard Toggle word wrap

  1. $ curl -u $USER:$PASS -X PATCH -H "Content-Type: application/json" -H "If-Match: <ETAG>" https://$Server/redfish/v1/Systems/$SystemID/ -d '{"Boot": {"BootSourceOverrideTarget": "cd", "BootSourceOverrideEnabled": "Once"}}'
    Copy to Clipboard Toggle word wrap
  2. $ curl -u $USER:$PASS -X POST -H "Content-Type: application/json" https://$Server/redfish/v1/Managers/$ManagerID/VirtualMedia/$VmediaId -d '{"Image": "https://example.com/test.iso", "TransferProtocolType": "HTTPS", "UserName": "", "Password":""}'
    Copy to Clipboard Toggle word wrap
    $ curl -u $USER:$PASS -X PATCH -H "Content-Type: application/json" -H "If-Match: <ETAG>" https://$Server/redfish/v1/Managers/$ManagerID/VirtualMedia/$VmediaId -d '{"Image": "https://example.com/test.iso", "TransferProtocolType": "HTTPS", "UserName": "", "Password":""}'
    Copy to Clipboard Toggle word wrap
참고

중요

3.12.5.

platform:
  baremetal:
    hosts:
      - name: <hostname>
        role: <master | worker>
        bmc:
          address: <address> 
1

          username: <user> 
2

          password: <password>
Copy to Clipboard Toggle word wrap
1
2

Expand
  

중요

참고

platform:
  baremetal:
    hosts:
      - name: openshift-master-0
        role: master
        bmc:
          address: idrac-virtualmedia://<out_of_band_ip>/redfish/v1/Systems/System.Embedded.1
          username: <user>
          password: <password>
Copy to Clipboard Toggle word wrap

참고

platform:
  baremetal:
    hosts:
      - name: openshift-master-0
        role: master
        bmc:
          address: idrac-virtualmedia://<out_of_band_ip>/redfish/v1/Systems/System.Embedded.1
          username: <user>
          password: <password>
          disableCertificateVerification: True
Copy to Clipboard Toggle word wrap

platform:
  baremetal:
    hosts:
      - name: openshift-master-0
        role: master
        bmc:
          address: redfish://<out_of_band_ip>/redfish/v1/Systems/System.Embedded.1
          username: <user>
          password: <password>
Copy to Clipboard Toggle word wrap

platform:
  baremetal:
    hosts:
      - name: openshift-master-0
        role: master
        bmc:
          address: redfish://<out_of_band_ip>/redfish/v1/Systems/System.Embedded.1
          username: <user>
          password: <password>
          disableCertificateVerification: True
Copy to Clipboard Toggle word wrap
참고

3.12.6.

platform:
  baremetal:
    hosts:
      - name: <hostname>
        role: <master | worker>
        bmc:
          address: <address> 
1

          username: <user>
          password: <password>
Copy to Clipboard Toggle word wrap
1

Expand
표 3.4.
  

platform:
  baremetal:
    hosts:
      - name: openshift-master-0
        role: master
        bmc:
          address: redfish-virtualmedia://<out-of-band-ip>/redfish/v1/Systems/1
          username: <user>
          password: <password>
Copy to Clipboard Toggle word wrap

platform:
  baremetal:
    hosts:
      - name: openshift-master-0
        role: master
        bmc:
          address: redfish-virtualmedia://<out-of-band-ip>/redfish/v1/Systems/1
          username: <user>
          password: <password>
          disableCertificateVerification: True
Copy to Clipboard Toggle word wrap
참고

platform:
  baremetal:
    hosts:
      - name: openshift-master-0
        role: master
        bmc:
          address: redfish://<out-of-band-ip>/redfish/v1/Systems/1
          username: <user>
          password: <password>
Copy to Clipboard Toggle word wrap

platform:
  baremetal:
    hosts:
      - name: openshift-master-0
        role: master
        bmc:
          address: redfish://<out-of-band-ip>/redfish/v1/Systems/1
          username: <user>
          password: <password>
          disableCertificateVerification: True
Copy to Clipboard Toggle word wrap

3.12.7.

platform:
  baremetal:
    hosts:
      - name: <hostname>
        role: <master | worker>
        bmc:
          address: <address> 
1

          username: <user>
          password: <password>
Copy to Clipboard Toggle word wrap
1

Expand
표 3.5.
  

platform:
  baremetal:
    hosts:
      - name: openshift-master-0
        role: master
        bmc:
          address: irmc://<out-of-band-ip>
          username: <user>
          password: <password>
Copy to Clipboard Toggle word wrap
참고

3.12.8.

platform:
  baremetal:
    hosts:
      - name: <hostname>
        role: <master | worker>
        bmc:
          address: <address> 
1

          username: <user>
          password: <password>
Copy to Clipboard Toggle word wrap
1

Expand
표 3.6.
  

platform:
  baremetal:
    hosts:
      - name: openshift-master-0
        role: master
        bmc:
          address: redfish-virtualmedia://<server_kvm_ip>/redfish/v1/Systems/<serial_number>
          username: <user>
          password: <password>
Copy to Clipboard Toggle word wrap

platform:
  baremetal:
    hosts:
      - name: openshift-master-0
        role: master
        bmc:
          address: redfish-virtualmedia://<server_kvm_ip>/redfish/v1/Systems/<serial_number>
          username: <user>
          password: <password>
          disableCertificateVerification: True
Copy to Clipboard Toggle word wrap

3.12.9.

Expand
표 3.7.
  

     - name: master-0
       role: master
       bmc:
         address: ipmi://10.10.0.3:6203
         username: admin
         password: redhat
       bootMACAddress: de:ad:be:ef:00:40
       rootDeviceHints:
         deviceName: "/dev/sda"
Copy to Clipboard Toggle word wrap

3.12.10.

  1. apiVersion: v1
    baseDomain: <domain>
    proxy:
      httpProxy: http://USERNAME:PASSWORD@proxy.example.com:PORT
      httpsProxy: https://USERNAME:PASSWORD@proxy.example.com:PORT
      noProxy: <WILDCARD_OF_DOMAIN>,<PROVISIONING_NETWORK/CIDR>,<BMC_ADDRESS_RANGE/CIDR>
    Copy to Clipboard Toggle word wrap

    noProxy: .example.com,172.22.0.0/24,10.10.0.0/24
    Copy to Clipboard Toggle word wrap

3.12.11.

platform:
  baremetal:
    apiVIPs:
      - <api_VIP>
    ingressVIPs:
      - <ingress_VIP>
    provisioningNetwork: "Disabled" 
1
Copy to Clipboard Toggle word wrap
1
중요

3.12.12.

machineNetwork:
- cidr: {{ extcidrnet }}
- cidr: {{ extcidrnet6 }}
clusterNetwork:
- cidr: 10.128.0.0/14
  hostPrefix: 23
- cidr: fd02::/48
  hostPrefix: 64
serviceNetwork:
- 172.30.0.0/16
- fd03::/112
Copy to Clipboard Toggle word wrap
중요

networkConfig:
  nmstate:
    interfaces:
    - name: <interface_name>
# ...
      wait-ip: ipv4+ipv6
# ...
Copy to Clipboard Toggle word wrap

platform:
  baremetal:
    apiVIPs:
      - <api_ipv4>
      - <api_ipv6>
    ingressVIPs:
      - <wildcard_ipv4>
      - <wildcard_ipv6>
Copy to Clipboard Toggle word wrap
참고

3.12.13.

  1. 참고

    1. interfaces:
      - name: <nic1_name> 
      1
      
        type: ethernet
        state: up
        ipv4:
          address:
          - ip: <ip_address> 
      2
      
            prefix-length: 24
          enabled: true
      dns-resolver:
        config:
          server:
          - <dns_ip_address> 
      3
      
      routes:
        config:
        - destination: 0.0.0.0/0
          next-hop-address: <next_hop_ip_address> 
      4
      
          next-hop-interface: <next_hop_nic1_name> 
      5
      Copy to Clipboard Toggle word wrap
      1 2 3 4 5
    2. $ nmstatectl gc <nmstate_yaml_file>
      Copy to Clipboard Toggle word wrap

  2.     hosts:
          - name: openshift-master-0
            role: master
            bmc:
              address: redfish+http://<out_of_band_ip>/redfish/v1/Systems/
              username: <user>
              password: <password>
              disableCertificateVerification: null
            bootMACAddress: <NIC1_mac_address>
            bootMode: UEFI
            rootDeviceHints:
              deviceName: "/dev/sda"
            networkConfig: 
    1
    
              interfaces:
              - name: <nic1_name> 
    2
    
                type: ethernet
                state: up
                ipv4:
                  address:
                  - ip: <ip_address> 
    3
    
                    prefix-length: 24
                  enabled: true
              dns-resolver:
                config:
                  server:
                  - <dns_ip_address> 
    4
    
              routes:
                config:
                - destination: 0.0.0.0/0
                  next-hop-address: <next_hop_ip_address> 
    5
    
                  next-hop-interface: <next_hop_nic1_name> 
    6
    Copy to Clipboard Toggle word wrap
    1
    2 3 4 5 6
    중요

3.12.14.

중요

참고

  1. networking:
      machineNetwork:
      - cidr: 10.0.0.0/24
      - cidr: 192.168.0.0/24
      networkType: OVNKubernetes
    Copy to Clipboard Toggle word wrap
  2. networkConfig:
      interfaces:
      - name: <interface_name> 
    1
    
        type: ethernet
        state: up
        ipv4:
          enabled: true
          dhcp: false
          address:
          - ip: <node_ip> 
    2
    
            prefix-length: 24
          gateway: <gateway_ip> 
    3
    
      dns-resolver:
        config:
          server:
          - <dns_ip> 
    4
    Copy to Clipboard Toggle word wrap
    1
    2
    3
    4

3.12.15.

    1. interfaces:
      - name: eth0
        ipv6:
          addr-gen-mode: <address_mode> 
      1
      Copy to Clipboard Toggle word wrap
      1
    2. $ nmstatectl gc <nmstate_yaml_file> 
      1
      Copy to Clipboard Toggle word wrap
      1
  1.     hosts:
          - name: openshift-master-0
            role: master
            bmc:
              address: redfish+http://<out_of_band_ip>/redfish/v1/Systems/
              username: <user>
              password: <password>
              disableCertificateVerification: null
            bootMACAddress: <NIC1_mac_address>
            bootMode: UEFI
            rootDeviceHints:
              deviceName: "/dev/sda"
            networkConfig:
              interfaces:
              - name: eth0
                ipv6:
                  addr-gen-mode: <address_mode> 
    1
    
    ...
    Copy to Clipboard Toggle word wrap
    1

3.12.16.

참고

  1.     hosts:
          - name: worker-0
            role: worker
            bmc:
              address: redfish+http://<out_of_band_ip>/redfish/v1/Systems/
              username: <user>
              password: <password>
              disableCertificateVerification: false
            bootMACAddress: <NIC1_mac_address>
            bootMode: UEFI
            networkConfig: 
    1
    
              interfaces: 
    2
    
               - name: eno1 
    3
    
                 type: ethernet 
    4
    
                 state: up
                 mac-address: 0c:42:a1:55:f3:06
                 ipv4:
                   enabled: true
                   dhcp: false 
    5
    
                 ethernet:
                   sr-iov:
                     total-vfs: 2 
    6
    
                 ipv6:
                   enabled: false
                   dhcp: false
               - name: sriov:eno1:0
                 type: ethernet
                 state: up 
    7
    
                 ipv4:
                   enabled: false 
    8
    
                 ipv6:
                   enabled: false
               - name: sriov:eno1:1
                 type: ethernet
                 state: down
               - name: eno2
                 type: ethernet
                 state: up
                 mac-address: 0c:42:a1:55:f3:07
                 ipv4:
                   enabled: true
                 ethernet:
                   sr-iov:
                     total-vfs: 2
                 ipv6:
                   enabled: false
               - name: sriov:eno2:0
                 type: ethernet
                 state: up
                 ipv4:
                   enabled: false
                 ipv6:
                   enabled: false
               - name: sriov:eno2:1
                 type: ethernet
                 state: down
               - name: bond0
                 type: bond
                 state: up
                 min-tx-rate: 100 
    9
    
                 max-tx-rate: 200 
    10
    
                 link-aggregation:
                   mode: active-backup 
    11
    
                   options:
                     primary: sriov:eno1:0 
    12
    
                   port:
                     - sriov:eno1:0
                     - sriov:eno2:0
                 ipv4:
                   address:
                     - ip: 10.19.16.57 
    13
    
                       prefix-length: 23
                   dhcp: false
                   enabled: true
                 ipv6:
                   enabled: false
              dns-resolver:
                config:
                  server:
                    - 10.11.5.160
                    - 10.2.70.215
              routes:
                config:
                  - destination: 0.0.0.0/0
                    next-hop-address: 10.19.17.254
                    next-hop-interface: bond0 
    14
    
                    table-id: 254
    Copy to Clipboard Toggle word wrap
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    중요

3.12.17.

hosts:
- name: ostest-master-0
 [...]
 networkConfig: &BOND
   interfaces:
   - name: bond0
     type: bond
     state: up
     ipv4:
       dhcp: true
       enabled: true
     link-aggregation:
       mode: active-backup
       port:
       - enp2s0
       - enp3s0
- name: ostest-master-1
 [...]
 networkConfig: *BOND
- name: ostest-master-2
 [...]
 networkConfig: *BOND
Copy to Clipboard Toggle word wrap
참고

3.12.18.

hosts:
  - name: openshift-master-0
    role: master
    bmc:
      address: redfish://<out_of_band_ip> 
1

      username: <username>
      password: <password>
    bootMACAddress: <NIC1_mac_address>
    rootDeviceHints:
     deviceName: "/dev/sda"
    bootMode: UEFISecureBoot 
2
Copy to Clipboard Toggle word wrap

1
2
참고

참고

3.13.

3.13.1.

  1. $ ./openshift-baremetal-install --dir ~/clusterconfigs create manifests
    Copy to Clipboard Toggle word wrap
    INFO Consuming Install Config from target directory
    WARNING Making control-plane schedulable by setting MastersSchedulable to true for Scheduler cluster settings
    WARNING Discarding the OpenShift Manifest that was provided in the target directory because its dependencies are dirty and it needs to be regenerated
    Copy to Clipboard Toggle word wrap

3.13.2.

  1. $ sudo dnf -y install butane
    Copy to Clipboard Toggle word wrap
  2. 참고

    variant: openshift
    version: 4.17.0
    metadata:
      name: 99-master-chrony-conf-override
      labels:
        machineconfiguration.openshift.io/role: master
    storage:
      files:
        - path: /etc/chrony.conf
          mode: 0644
          overwrite: true
          contents:
            inline: |
              # Use public servers from the pool.ntp.org project.
              # Please consider joining the pool (https://www.pool.ntp.org/join.html).
    
              # The Machine Config Operator manages this file
              server openshift-master-0.<cluster-name>.<domain> iburst 
    1
    
              server openshift-master-1.<cluster-name>.<domain> iburst
              server openshift-master-2.<cluster-name>.<domain> iburst
    
              stratumweight 0
              driftfile /var/lib/chrony/drift
              rtcsync
              makestep 10 3
              bindcmdaddress 127.0.0.1
              bindcmdaddress ::1
              keyfile /etc/chrony.keys
              commandkey 1
              generatecommandkey
              noclientlog
              logchange 0.5
              logdir /var/log/chrony
    
              # Configure the control plane nodes to serve as local NTP servers
              # for all compute nodes, even if they are not in sync with an
              # upstream NTP server.
    
              # Allow NTP client access from the local network.
              allow all
              # Serve time even if not synchronized to a time source.
              local stratum 3 orphan
    Copy to Clipboard Toggle word wrap

    1
  3. $ butane 99-master-chrony-conf-override.bu -o 99-master-chrony-conf-override.yaml
    Copy to Clipboard Toggle word wrap
  4. variant: openshift
    version: 4.17.0
    metadata:
      name: 99-worker-chrony-conf-override
      labels:
        machineconfiguration.openshift.io/role: worker
    storage:
      files:
        - path: /etc/chrony.conf
          mode: 0644
          overwrite: true
          contents:
            inline: |
              # The Machine Config Operator manages this file.
              server openshift-master-0.<cluster-name>.<domain> iburst 
    1
    
              server openshift-master-1.<cluster-name>.<domain> iburst
              server openshift-master-2.<cluster-name>.<domain> iburst
    
              stratumweight 0
              driftfile /var/lib/chrony/drift
              rtcsync
              makestep 10 3
              bindcmdaddress 127.0.0.1
              bindcmdaddress ::1
              keyfile /etc/chrony.keys
              commandkey 1
              generatecommandkey
              noclientlog
              logchange 0.5
              logdir /var/log/chrony
    Copy to Clipboard Toggle word wrap

    1
  5. $ butane 99-worker-chrony-conf-override.bu -o 99-worker-chrony-conf-override.yaml
    Copy to Clipboard Toggle word wrap

3.13.3.

중요

  1. $ cd ~/clusterconfigs
    Copy to Clipboard Toggle word wrap
  2. $ cd manifests
    Copy to Clipboard Toggle word wrap
  3. $ touch cluster-network-avoid-workers-99-config.yaml
    Copy to Clipboard Toggle word wrap
  4. apiVersion: machineconfiguration.openshift.io/v1
    kind: MachineConfig
    metadata:
      name: 50-worker-fix-ipi-rwn
      labels:
        machineconfiguration.openshift.io/role: worker
    spec:
      config:
        ignition:
          version: 3.2.0
        storage:
          files:
            - path: /etc/kubernetes/manifests/keepalived.yaml
              mode: 0644
              contents:
                source: data:,
    Copy to Clipboard Toggle word wrap

  5. apiVersion: operator.openshift.io/v1
    kind: IngressController
    metadata:
      name: default
      namespace: openshift-ingress-operator
    spec:
      nodePlacement:
        nodeSelector:
          matchLabels:
            node-role.kubernetes.io/master: ""
    Copy to Clipboard Toggle word wrap
  6. $ sed -i "s;mastersSchedulable: false;mastersSchedulable: true;g" clusterconfigs/manifests/cluster-scheduler-02-config.yml
    Copy to Clipboard Toggle word wrap
    참고

3.13.4.

중요

참고

  1. apiVersion: operator.openshift.io/v1
    kind: IngressController
    metadata:
      name: default
      namespace: openshift-ingress-operator
    spec:
      replicas: <num-of-router-pods>
      endpointPublishingStrategy:
        type: HostNetwork
      nodePlacement:
        nodeSelector:
          matchLabels:
            node-role.kubernetes.io/worker: ""
    Copy to Clipboard Toggle word wrap
    참고

  2. $ cp ~/router-replicas.yaml clusterconfigs/openshift/99_router-replicas.yaml
    Copy to Clipboard Toggle word wrap

3.13.5.

  1. $ vim clusterconfigs/openshift/99_openshift-cluster-api_hosts-*.yaml
    Copy to Clipboard Toggle word wrap
  2. spec:
      firmware:
        simultaneousMultithreadingEnabled: true
        sriovEnabled: true
        virtualizationEnabled: true
    Copy to Clipboard Toggle word wrap
    참고

3.13.6.

참고

Expand
표 3.8.
    

  1. $ vim clusterconfigs/openshift/99_openshift-cluster-api_hosts-*.yaml
    Copy to Clipboard Toggle word wrap
    참고

    1. spec:
        raid:
          hardwareRAIDVolumes:
          - level: "0" 
      1
      
            name: "sda"
            numberOfPhysicalDisks: 1
            rotational: true
            sizeGibibytes: 0
      Copy to Clipboard Toggle word wrap
      1
    2. spec:
        raid:
          hardwareRAIDVolumes: []
      Copy to Clipboard Toggle word wrap

3.13.7.

  1. apiVersion: machineconfiguration.openshift.io/v1
    kind: MachineConfig
    metadata:
      labels:
        machineconfiguration.openshift.io/role: primary
      name: 10_primary_storage_config
    spec:
      config:
        ignition:
          version: 3.2.0
        storage:
          disks:
            - device: </dev/xxyN>
              partitions:
                - label: recovery
                  startMiB: 32768
                  sizeMiB: 16384
          filesystems:
            - device: /dev/disk/by-partlabel/recovery
              label: recovery
              format: xfs
    Copy to Clipboard Toggle word wrap
  2. $ cp ~/<MachineConfig_manifest> ~/clusterconfigs/openshift
    Copy to Clipboard Toggle word wrap

3.14.

참고

3.14.1.

  1. $ sudo firewall-cmd --add-port=5000/tcp --zone=libvirt  --permanent
    Copy to Clipboard Toggle word wrap
    $ sudo firewall-cmd --add-port=5000/tcp --zone=public   --permanent
    Copy to Clipboard Toggle word wrap
    $ sudo firewall-cmd --reload
    Copy to Clipboard Toggle word wrap
  2. $ sudo yum -y install python3 podman httpd httpd-tools jq
    Copy to Clipboard Toggle word wrap
  3. $ sudo mkdir -p /opt/registry/{auth,certs,data}
    Copy to Clipboard Toggle word wrap

3.14.2.

    1. $ OCP_RELEASE=<release_version>
      Copy to Clipboard Toggle word wrap

    2. $ LOCAL_REGISTRY='<local_registry_host_name>:<local_registry_host_port>'
      Copy to Clipboard Toggle word wrap

    3. $ LOCAL_REPOSITORY='<local_repository_name>'
      Copy to Clipboard Toggle word wrap

    4. $ PRODUCT_REPO='openshift-release-dev'
      Copy to Clipboard Toggle word wrap

    5. $ LOCAL_SECRET_JSON='<path_to_pull_secret>'
      Copy to Clipboard Toggle word wrap

    6. $ RELEASE_NAME="ocp-release"
      Copy to Clipboard Toggle word wrap

    7. $ ARCHITECTURE=<cluster_architecture> 
      1
      Copy to Clipboard Toggle word wrap
      1
    8. $ REMOVABLE_MEDIA_PATH=<path> 
      1
      Copy to Clipboard Toggle word wrap
      1
      1. $ oc adm release mirror -a ${LOCAL_SECRET_JSON}  \
             --from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE} \
             --to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} \
             --to-release-image=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE} --dry-run
        Copy to Clipboard Toggle word wrap
      2. $ oc adm release mirror -a ${LOCAL_SECRET_JSON} --to-dir=${REMOVABLE_MEDIA_PATH}/mirror quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE}
        Copy to Clipboard Toggle word wrap
      3. $ oc image mirror -a ${LOCAL_SECRET_JSON} --from-dir=${REMOVABLE_MEDIA_PATH}/mirror "file://openshift/release:${OCP_RELEASE}*" ${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} 
        1
        Copy to Clipboard Toggle word wrap
        1
      1. $ oc adm release mirror -a ${LOCAL_SECRET_JSON}  \
             --from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE} \
             --to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} \
             --to-release-image=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE}
        Copy to Clipboard Toggle word wrap

      2. 참고

    • $ oc adm release extract -a ${LOCAL_SECRET_JSON} --command=openshift-baremetal-install "${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}"
      Copy to Clipboard Toggle word wrap
    • $ oc adm release extract -a ${LOCAL_SECRET_JSON} --command=openshift-baremetal-install "${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE}"
      Copy to Clipboard Toggle word wrap
      중요

  1. $ openshift-baremetal-install
    Copy to Clipboard Toggle word wrap

3.14.3.

  1. $ echo "additionalTrustBundle: |" >> install-config.yaml
    Copy to Clipboard Toggle word wrap

    $ sed -e 's/^/  /' /opt/registry/certs/domain.crt >> install-config.yaml
    Copy to Clipboard Toggle word wrap
  2. $ echo "imageContentSources:" >> install-config.yaml
    Copy to Clipboard Toggle word wrap
    $ echo "- mirrors:" >> install-config.yaml
    Copy to Clipboard Toggle word wrap
    $ echo "  - registry.example.com:5000/ocp4/openshift4" >> install-config.yaml
    Copy to Clipboard Toggle word wrap

    $ echo "  source: quay.io/openshift-release-dev/ocp-release" >> install-config.yaml
    Copy to Clipboard Toggle word wrap
    $ echo "- mirrors:" >> install-config.yaml
    Copy to Clipboard Toggle word wrap
    $ echo "  - registry.example.com:5000/ocp4/openshift4" >> install-config.yaml
    Copy to Clipboard Toggle word wrap

    $ echo "  source: quay.io/openshift-release-dev/ocp-v4.0-art-dev" >> install-config.yaml
    Copy to Clipboard Toggle word wrap

3.15.

4장.

4.1.

  1. $ ipmitool -I lanplus -U <user> -P <password> -H <management_server_ip> power off
    Copy to Clipboard Toggle word wrap
  2. for i in $(sudo virsh list | tail -n +3 | grep bootstrap | awk {'print $2'});
    do
      sudo virsh destroy $i;
      sudo virsh undefine $i;
      sudo virsh vol-delete $i --pool $i;
      sudo virsh vol-delete $i.ign --pool $i;
      sudo virsh pool-destroy $i;
      sudo virsh pool-undefine $i;
    done
    Copy to Clipboard Toggle word wrap
  3. $ cd ; /bin/rm -rf auth/ bootstrap.ign master.ign worker.ign metadata.json \
    .openshift_install.log .openshift_install_state.json
    Copy to Clipboard Toggle word wrap
  4. $ ./openshift-baremetal-install --dir ~/clusterconfigs create manifests
    Copy to Clipboard Toggle word wrap

4.2.

$ ./openshift-baremetal-install --dir ~/clusterconfigs --log-level debug create cluster
Copy to Clipboard Toggle word wrap

4.3.

$ tail -f /path/to/install-dir/.openshift_install.log
Copy to Clipboard Toggle word wrap

4.4.

참고

5장.

5.1.

5.2.

  1. $ curl -s -o /dev/null -I -w "%{http_code}\n" http://webserver.example.com:8080/rhcos-44.81.202004250133-0-qemu.<architecture>.qcow2.gz?sha256=7d884b46ee54fe87bbc3893bf2aa99af3b2d31f2e19ab5529c60636fbd0f1ce7
    Copy to Clipboard Toggle word wrap

5.3.

  1. $ sudo virsh list
    Copy to Clipboard Toggle word wrap
     Id    Name                           State
     --------------------------------------------
     12    openshift-xf6fq-bootstrap      running
    Copy to Clipboard Toggle word wrap
    참고

  2. $ systemctl status libvirtd
    Copy to Clipboard Toggle word wrap
    ● libvirtd.service - Virtualization daemon
       Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
       Active: active (running) since Tue 2020-03-03 21:21:07 UTC; 3 weeks 5 days ago
         Docs: man:libvirtd(8)
               https://libvirt.org
     Main PID: 9850 (libvirtd)
        Tasks: 20 (limit: 32768)
       Memory: 74.8M
       CGroup: /system.slice/libvirtd.service
               ├─ 9850 /usr/sbin/libvirtd
    Copy to Clipboard Toggle word wrap

  3. $ sudo virsh console example.com
    Copy to Clipboard Toggle word wrap
    Connected to domain example.com
    Escape character is ^]
    Red Hat Enterprise Linux CoreOS 43.81.202001142154.0 (Ootpa) 4.3
    SSH host key: SHA256:BRWJktXZgQQRY5zjuAV0IKZ4WM7i4TiUyMVanqu9Pqg (ED25519)
    SSH host key: SHA256:7+iKGA7VtG5szmk2jB5gl/5EZ+SNcJ3a2g23o0lnIio (ECDSA)
    SSH host key: SHA256:DH5VWhvhvagOTaLsYiVNse9ca+ZSW/30OOMed8rIGOc (RSA)
    ens3:  fd35:919d:4042:2:c7ed:9a9f:a9ec:7
    ens4: 172.22.0.2 fe80::1d05:e52e:be5d:263f
    localhost login:
    Copy to Clipboard Toggle word wrap
    중요

  4. 참고

    $ ssh core@172.22.0.2
    Copy to Clipboard Toggle word wrap

5.3.1.

  1. $ ssh core@172.22.0.2
    Copy to Clipboard Toggle word wrap
  2. [core@localhost ~]$ sudo podman logs -f <container_name>
    Copy to Clipboard Toggle word wrap

$ ipmitool -I lanplus -U root -P <password> -H <out_of_band_ip> power off
Copy to Clipboard Toggle word wrap

5.3.2.

bootstrapOSImage: http://<ip:port>/rhcos-43.81.202001142154.0-qemu.<architecture>.qcow2.gz?sha256=9d999f55ff1d44f7ed7c106508e5deecd04dc3c06095d34d36bf1cd127837e0c
clusterOSImage: http://<ip:port>/rhcos-43.81.202001142154.0-openstack.<architecture>.qcow2.gz?sha256=a1bda656fa0892f7b936fdc6b6a6086bddaed5dafacedcd7a1e811abb78fe3b0
Copy to Clipboard Toggle word wrap

  1. $ ssh core@172.22.0.2
    Copy to Clipboard Toggle word wrap
  2. [core@localhost ~]$ sudo podman logs -f coreos-downloader
    Copy to Clipboard Toggle word wrap

  3. [core@localhost ~]$ journalctl -xe
    Copy to Clipboard Toggle word wrap
    [core@localhost ~]$ journalctl -b -f -u bootkube.service
    Copy to Clipboard Toggle word wrap
  4. [core@localhost ~]$ sudo podman ps
    Copy to Clipboard Toggle word wrap
  5. [core@localhost ~]$ sudo podman logs ironic
    Copy to Clipboard Toggle word wrap

5.4.

  1. $ sudo crictl logs $(sudo crictl ps --pod=$(sudo crictl pods --name=etcd-member --quiet) --quiet)
    Copy to Clipboard Toggle word wrap
  2. $ sudo crictl pods --name=etcd-member
    Copy to Clipboard Toggle word wrap

  3. $ hostname
    Copy to Clipboard Toggle word wrap

    $ sudo hostnamectl set-hostname <hostname>
    Copy to Clipboard Toggle word wrap
  4. $ dig api.<cluster_name>.example.com
    Copy to Clipboard Toggle word wrap
    ; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el8 <<>> api.<cluster_name>.example.com
    ;; global options: +cmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37551
    ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 2
    
    ;; OPT PSEUDOSECTION:
    ; EDNS: version: 0, flags:; udp: 4096
    ; COOKIE: 866929d2f8e8563582af23f05ec44203d313e50948d43f60 (good)
    ;; QUESTION SECTION:
    ;api.<cluster_name>.example.com. IN A
    
    ;; ANSWER SECTION:
    api.<cluster_name>.example.com. 10800 IN	A 10.19.13.86
    
    ;; AUTHORITY SECTION:
    <cluster_name>.example.com. 10800 IN NS	<cluster_name>.example.com.
    
    ;; ADDITIONAL SECTION:
    <cluster_name>.example.com. 10800 IN A	10.19.14.247
    
    ;; Query time: 0 msec
    ;; SERVER: 10.19.14.247#53(10.19.14.247)
    ;; WHEN: Tue May 19 20:30:59 UTC 2020
    ;; MSG SIZE  rcvd: 140
    Copy to Clipboard Toggle word wrap

5.5.

  1. $ oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig get clusterversion -o yaml
    Copy to Clipboard Toggle word wrap

    apiVersion: config.openshift.io/v1
    kind: ClusterVersion
    metadata:
      creationTimestamp: 2019-02-27T22:24:21Z
      generation: 1
      name: version
      resourceVersion: "19927"
      selfLink: /apis/config.openshift.io/v1/clusterversions/version
      uid: 6e0f4cf8-3ade-11e9-9034-0a923b47ded4
    spec:
      channel: stable-4.1
      clusterID: 5ec312f9-f729-429d-a454-61d4906896ca
    status:
      availableUpdates: null
      conditions:
      - lastTransitionTime: 2019-02-27T22:50:30Z
        message: Done applying 4.1.1
        status: "True"
        type: Available
      - lastTransitionTime: 2019-02-27T22:50:30Z
        status: "False"
        type: Failing
      - lastTransitionTime: 2019-02-27T22:50:30Z
        message: Cluster version is 4.1.1
        status: "False"
        type: Progressing
      - lastTransitionTime: 2019-02-27T22:24:31Z
        message: 'Unable to retrieve available updates: unknown version 4.1.1
        reason: RemoteFailed
        status: "False"
        type: RetrievedUpdates
      desired:
        image: registry.svc.ci.openshift.org/openshift/origin-release@sha256:91e6f754975963e7db1a9958075eb609ad226968623939d262d1cf45e9dbc39a
        version: 4.1.1
      history:
      - completionTime: 2019-02-27T22:50:30Z
        image: registry.svc.ci.openshift.org/openshift/origin-release@sha256:91e6f754975963e7db1a9958075eb609ad226968623939d262d1cf45e9dbc39a
        startedTime: 2019-02-27T22:24:31Z
        state: Completed
        version: 4.1.1
      observedGeneration: 1
      versionHash: Wa7as_ik1qE=
    Copy to Clipboard Toggle word wrap

  2. $ oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig get clusterversion version \
         -o=jsonpath='{range .status.conditions[*]}{.type}{" "}{.status}{" "}{.message}{"\n"}{end}'
    Copy to Clipboard Toggle word wrap

    Available True Done applying 4.1.1
    Failing False
    Progressing False Cluster version is 4.0.0-0.alpha-2019-02-26-194020
    RetrievedUpdates False Unable to retrieve available updates: unknown version 4.1.1
    Copy to Clipboard Toggle word wrap

  3. $ oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig get clusteroperator
    Copy to Clipboard Toggle word wrap

    NAME                                  VERSION   AVAILABLE   PROGRESSING   FAILING   SINCE
    cluster-baremetal-operator                      True        False         False     17m
    cluster-autoscaler                              True        False         False     17m
    cluster-storage-operator                        True        False         False     10m
    console                                         True        False         False     7m21s
    dns                                             True        False         False     31m
    image-registry                                  True        False         False     9m58s
    ingress                                         True        False         False     10m
    kube-apiserver                                  True        False         False     28m
    kube-controller-manager                         True        False         False     21m
    kube-scheduler                                  True        False         False     25m
    machine-api                                     True        False         False     17m
    machine-config                                  True        False         False     17m
    marketplace-operator                            True        False         False     10m
    monitoring                                      True        False         False     8m23s
    network                                         True        False         False     13m
    node-tuning                                     True        False         False     11m
    openshift-apiserver                             True        False         False     15m
    openshift-authentication                        True        False         False     20m
    openshift-cloud-credential-operator             True        False         False     18m
    openshift-controller-manager                    True        False         False     10m
    openshift-samples                               True        False         False     8m42s
    operator-lifecycle-manager                      True        False         False     17m
    service-ca                                      True        False         False     30m
    Copy to Clipboard Toggle word wrap

  4. $ oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig get clusteroperator <operator> -oyaml 
    1
    Copy to Clipboard Toggle word wrap
    1

    apiVersion: config.openshift.io/v1
    kind: ClusterOperator
    metadata:
      creationTimestamp: 2019-02-27T22:47:04Z
      generation: 1
      name: monitoring
      resourceVersion: "24677"
      selfLink: /apis/config.openshift.io/v1/clusteroperators/monitoring
      uid: 9a6a5ef9-3ae1-11e9-bad4-0a97b6ba9358
    spec: {}
    status:
      conditions:
      - lastTransitionTime: 2019-02-27T22:49:10Z
        message: Successfully rolled out the stack.
        status: "True"
        type: Available
      - lastTransitionTime: 2019-02-27T22:49:10Z
        status: "False"
        type: Progressing
      - lastTransitionTime: 2019-02-27T22:49:10Z
        status: "False"
        type: Failing
      extension: null
      relatedObjects: null
      version: ""
    Copy to Clipboard Toggle word wrap

  5. $ oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig get clusteroperator <operator> \
         -o=jsonpath='{range .status.conditions[*]}{.type}{" "}{.status}{" "}{.message}{"\n"}{end}'
    Copy to Clipboard Toggle word wrap

    Available True Successfully rolled out the stack
    Progressing False
    Failing False
    Copy to Clipboard Toggle word wrap

  6. oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig get clusteroperator kube-apiserver \
       -o=jsonpath='{.status.relatedObjects}'
    Copy to Clipboard Toggle word wrap

    [map[resource:kubeapiservers group:operator.openshift.io name:cluster] map[group: name:openshift-config resource:namespaces] map[group: name:openshift-config-managed resource:namespaces] map[group: name:openshift-kube-apiserver-operator resource:namespaces] map[group: name:openshift-kube-apiserver resource:namespaces]]
    Copy to Clipboard Toggle word wrap

5.6.

  1. $ oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig get clusteroperator console -oyaml
    Copy to Clipboard Toggle word wrap
    apiVersion: config.openshift.io/v1
    kind: ClusterOperator
    metadata:
      creationTimestamp: 2019-02-27T22:46:57Z
      generation: 1
      name: console
      resourceVersion: "19682"
      selfLink: /apis/config.openshift.io/v1/clusteroperators/console
      uid: 960364aa-3ae1-11e9-bad4-0a97b6ba9358
    spec: {}
    status:
      conditions:
      - lastTransitionTime: 2019-02-27T22:46:58Z
        status: "False"
        type: Failing
      - lastTransitionTime: 2019-02-27T22:50:12Z
        status: "False"
        type: Progressing
      - lastTransitionTime: 2019-02-27T22:50:12Z
        status: "True"
        type: Available
      - lastTransitionTime: 2019-02-27T22:46:57Z
        status: "True"
        type: Upgradeable
      extension: null
      relatedObjects:
      - group: operator.openshift.io
        name: cluster
        resource: consoles
      - group: config.openshift.io
        name: cluster
        resource: consoles
      - group: oauth.openshift.io
        name: console
        resource: oauthclients
      - group: ""
        name: openshift-console-operator
        resource: namespaces
      - group: ""
        name: openshift-console
        resource: namespaces
      versions: null
    Copy to Clipboard Toggle word wrap
  2. $ oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig get route console -n openshift-console \
         -o=jsonpath='{.spec.host}' console-openshift-console.apps.adahiya-1.devcluster.openshift.com
    Copy to Clipboard Toggle word wrap

5.7.

  1. $ oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig get configmaps default-ingress-cert \
         -n openshift-config-managed -o=jsonpath='{.data.ca-bundle\.crt}'
    Copy to Clipboard Toggle word wrap
    -----BEGIN CERTIFICATE-----
    MIIC/TCCAeWgAwIBAgIBATANBgkqhkiG9w0BAQsFADAuMSwwKgYDVQQDDCNjbHVz
    dGVyLWluZ3Jlc3Mtb3BlcmF0b3JAMTU1MTMwNzU4OTAeFw0xOTAyMjcyMjQ2Mjha
    Fw0yMTAyMjYyMjQ2MjlaMC4xLDAqBgNVBAMMI2NsdXN0ZXItaW5ncmVzcy1vcGVy
    YXRvckAxNTUxMzA3NTg5MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA
    uCA4fQ+2YXoXSUL4h/mcvJfrgpBfKBW5hfB8NcgXeCYiQPnCKblH1sEQnI3VC5Pk
    2OfNCF3PUlfm4i8CHC95a7nCkRjmJNg1gVrWCvS/ohLgnO0BvszSiRLxIpuo3C4S
    EVqqvxValHcbdAXWgZLQoYZXV7RMz8yZjl5CfhDaaItyBFj3GtIJkXgUwp/5sUfI
    LDXW8MM6AXfuG+kweLdLCMm3g8WLLfLBLvVBKB+4IhIH7ll0buOz04RKhnYN+Ebw
    tcvFi55vwuUCWMnGhWHGEQ8sWm/wLnNlOwsUz7S1/sW8nj87GFHzgkaVM9EOnoNI
    gKhMBK9ItNzjrP6dgiKBCQIDAQABoyYwJDAOBgNVHQ8BAf8EBAMCAqQwEgYDVR0T
    AQH/BAgwBgEB/wIBADANBgkqhkiG9w0BAQsFAAOCAQEAq+vi0sFKudaZ9aUQMMha
    CeWx9CZvZBblnAWT/61UdpZKpFi4eJ2d33lGcfKwHOi2NP/iSKQBebfG0iNLVVPz
    vwLbSG1i9R9GLdAbnHpPT9UG6fLaDIoKpnKiBfGENfxeiq5vTln2bAgivxrVlyiq
    +MdDXFAWb6V4u2xh6RChI7akNsS3oU9PZ9YOs5e8vJp2YAEphht05X0swA+X8V8T
    C278FFifpo0h3Q0Dbv8Rfn4UpBEtN4KkLeS+JeT+0o2XOsFZp7Uhr9yFIodRsnNo
    H/Uwmab28ocNrGNiEVaVH6eTTQeeZuOdoQzUbClElpVmkrNGY0M42K0PvOQ/e7+y
    AQ==
    -----END CERTIFICATE-----
    Copy to Clipboard Toggle word wrap

5.8.

5.9.

  1. bootMACAddress: 24:6E:96:1B:96:90 # MAC of bootable provisioning NIC
    Copy to Clipboard Toggle word wrap

    bootMACAddress: 24:6E:96:1B:96:90 # MAC of bootable provisioning NIC
    Copy to Clipboard Toggle word wrap

5.10.

  1. $ oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig \
       --namespace=openshift-machine-api get deployments
    Copy to Clipboard Toggle word wrap

    NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
    cluster-autoscaler-operator   1/1     1            1           86m
    cluster-baremetal-operator    1/1     1            1           86m
    machine-api-controllers       1/1     1            1           85m
    machine-api-operator          1/1     1            1           86m
    Copy to Clipboard Toggle word wrap

  2. $ oc --kubeconfig=${INSTALL_DIR}/auth/kubeconfig \
         --namespace=openshift-machine-api logs deployments/machine-api-controllers \
         --container=machine-controller
    Copy to Clipboard Toggle word wrap

5.11.

  1. $ oc get network -o yaml cluster
    Copy to Clipboard Toggle word wrap

    $ openshift-install create manifests
    Copy to Clipboard Toggle word wrap

  2. $ oc get po -n openshift-network-operator
    Copy to Clipboard Toggle word wrap

5.12.

ProvisioningError 51s metal3-baremetal-controller Image provisioning failed: Deploy step deploy.deploy failed with BadRequestError: HTTP POST
https://<bmc_address>/redfish/v1/Managers/iDRAC.Embedded.1/VirtualMedia/CD/Actions/VirtualMedia.InsertMedia
returned code 400.
Base.1.8.GeneralError: A general error has occurred. See ExtendedInfo for more information
Extended information: [
  {
    "Message": "Unable to mount remote share https://<ironic_address>/redfish/boot-<uuid>.iso.",
    "MessageArgs": [
      "https://<ironic_address>/redfish/boot-<uuid>.iso"
    ],
    "MessageArgs@odata.count": 1,
    "MessageId": "IDRAC.2.5.RAC0720",
    "RelatedProperties": [
      "#/Image"
    ],
    "RelatedProperties@odata.count": 1,
    "Resolution": "Retry the operation.",
    "Severity": "Informational"
  }
].
Copy to Clipboard Toggle word wrap

참고

5.13.

  1. $ sudo nano /etc/dnsmasq.conf
    Copy to Clipboard Toggle word wrap
    address=/api-int.<cluster_name>.<base_domain>/<IP_address>
    address=/api-int.mycluster.example.com/192.168.1.10
    address=/api-int.mycluster.example.com/2001:0db8:85a3:0000:0000:8a2e:0370:7334
    Copy to Clipboard Toggle word wrap
  2. $ sudo nano /etc/dnsmasq.conf
    Copy to Clipboard Toggle word wrap
    ptr-record=<IP_address>.in-addr.arpa,api-int.<cluster_name>.<base_domain>
    ptr-record=10.1.168.192.in-addr.arpa,api-int.mycluster.example.com
    Copy to Clipboard Toggle word wrap
  3. $ sudo systemctl restart dnsmasq
    Copy to Clipboard Toggle word wrap

5.14.

  1. $ ipmitool -I lanplus -U <user> -P <password> -H <management_server_ip> power off
    Copy to Clipboard Toggle word wrap
  2. for i in $(sudo virsh list | tail -n +3 | grep bootstrap | awk {'print $2'});
    do
      sudo virsh destroy $i;
      sudo virsh undefine $i;
      sudo virsh vol-delete $i --pool $i;
      sudo virsh vol-delete $i.ign --pool $i;
      sudo virsh pool-destroy $i;
      sudo virsh pool-undefine $i;
    done
    Copy to Clipboard Toggle word wrap
  3. $ cd ; /bin/rm -rf auth/ bootstrap.ign master.ign worker.ign metadata.json \
    .openshift_install.log .openshift_install_state.json
    Copy to Clipboard Toggle word wrap
  4. $ ./openshift-baremetal-install --dir ~/clusterconfigs create manifests
    Copy to Clipboard Toggle word wrap

5.15.

  1. $ /usr/local/bin/oc adm release mirror \
      -a pull-secret-update.json
      --from=$UPSTREAM_REPO \
      --to-release-image=$LOCAL_REG/$LOCAL_REPO:${VERSION} \
      --to=$LOCAL_REG/$LOCAL_REPO
    Copy to Clipboard Toggle word wrap
    참고

    UPSTREAM_REPO=${RELEASE_IMAGE}
    LOCAL_REG=<registry_FQDN>:<registry_port>
    LOCAL_REPO='ocp4/openshift4'
    Copy to Clipboard Toggle word wrap

  2. $ curl -k -u <user>:<password> https://registry.example.com:<registry_port>/v2/_catalog
    {"repositories":["<Repo_Name>"]}
    Copy to Clipboard Toggle word wrap

5.16.

5.16.1.

`runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: Missing CNI default network`
Copy to Clipboard Toggle word wrap

  1. $ oc get all -n openshift-network-operator
    Copy to Clipboard Toggle word wrap
    NAME                                    READY STATUS            RESTARTS   AGE
    pod/network-operator-69dfd7b577-bg89v   0/1   ContainerCreating 0          149m
    Copy to Clipboard Toggle word wrap
  2. $ kubectl get network.config.openshift.io cluster -oyaml
    Copy to Clipboard Toggle word wrap
    apiVersion: config.openshift.io/v1
    kind: Network
    metadata:
      name: cluster
    spec:
      serviceNetwork:
      - 172.30.0.0/16
      clusterNetwork:
      - cidr: 10.128.0.0/14
        hostPrefix: 23
      networkType: OVNKubernetes
    Copy to Clipboard Toggle word wrap

    $ openshift-install create manifests
    Copy to Clipboard Toggle word wrap
  3. $ kubectl -n openshift-network-operator get pods
    Copy to Clipboard Toggle word wrap
  4. $ kubectl -n openshift-network-operator logs -l "name=network-operator"
    Copy to Clipboard Toggle word wrap

5.16.2.

No disk found with matching rootDeviceHints
Copy to Clipboard Toggle word wrap

$ udevadm info /dev/sda
Copy to Clipboard Toggle word wrap

5.16.3.

  1. # This is a dnsmasq dhcp reservation, 'id:00:03:00:01' is the client id and '18:db:f2:8c:d5:9f' is the MAC Address for the NIC
    id:00:03:00:01:18:db:f2:8c:d5:9f,openshift-master-1,[2620:52:0:1302::6]
    Copy to Clipboard Toggle word wrap

5.16.4.

Failed Units: 2
  NetworkManager-wait-online.service
  nodeip-configuration.service
Copy to Clipboard Toggle word wrap

  1. [core@master-X ~]$ hostname
    Copy to Clipboard Toggle word wrap

    참고

  2. [core@master-X ~]$ sudo nmcli con up "<bare_metal_nic>"
    Copy to Clipboard Toggle word wrap

  3. [core@master-X ~]$ hostname
    Copy to Clipboard Toggle word wrap
  4. [core@master-X ~]$ sudo systemctl restart NetworkManager
    Copy to Clipboard Toggle word wrap
  5. [core@master-X ~]$ sudo systemctl restart nodeip-configuration.service
    Copy to Clipboard Toggle word wrap

  6. [core@master-X ~]$ sudo systemctl daemon-reload
    Copy to Clipboard Toggle word wrap
  7. [core@master-X ~]$ sudo systemctl restart kubelet.service
    Copy to Clipboard Toggle word wrap
  8. [core@master-X ~]$ sudo journalctl -fu kubelet.service
    Copy to Clipboard Toggle word wrap

  1. $ oc get csr
    Copy to Clipboard Toggle word wrap
  2. $ oc get csr <pending_csr> -o jsonpath='{.spec.request}' | base64 --decode | openssl req -noout -text
    Copy to Clipboard Toggle word wrap
  3. $ oc delete csr <wrong_csr>
    Copy to Clipboard Toggle word wrap

5.16.5.

  1. $ oc get route oauth-openshift
    Copy to Clipboard Toggle word wrap
  2. $ oc get svc oauth-openshift
    Copy to Clipboard Toggle word wrap
    NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
    oauth-openshift   ClusterIP   172.30.19.162   <none>        443/TCP   59m
    Copy to Clipboard Toggle word wrap
  3. [core@master0 ~]$ curl -k https://172.30.19.162
    Copy to Clipboard Toggle word wrap
    {
      "kind": "Status",
      "apiVersion": "v1",
      "metadata": {
      },
      "status": "Failure",
      "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
      "reason": "Forbidden",
      "details": {
      },
      "code": 403
    Copy to Clipboard Toggle word wrap
  4. $ oc logs deployment/authentication-operator -n openshift-authentication-operator
    Copy to Clipboard Toggle word wrap
    Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"225c5bd5-b368-439b-9155-5fd3c0459d98", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/authentication changed: Degraded message changed from "IngressStateEndpointsDegraded: All 2 endpoints for oauth-server are reporting"
    Copy to Clipboard Toggle word wrap

5.16.6.

  1. Failed Units: 1
      machine-config-daemon-firstboot.service
    Copy to Clipboard Toggle word wrap
  2. [core@worker-X ~]$ sudo systemctl restart machine-config-daemon-firstboot.service
    Copy to Clipboard Toggle word wrap

5.16.7.

  1. $ oc get nodes
    Copy to Clipboard Toggle word wrap
    NAME                         STATUS   ROLES    AGE   VERSION
    master-0.cloud.example.com   Ready    master   145m   v1.30.3
    master-1.cloud.example.com   Ready    master   135m   v1.30.3
    master-2.cloud.example.com   Ready    master   145m   v1.30.3
    worker-2.cloud.example.com   Ready    worker   100m   v1.30.3
    Copy to Clipboard Toggle word wrap
  2. $ oc get bmh -n openshift-machine-api
    Copy to Clipboard Toggle word wrap
    master-1   error registering master-1  ipmi://<out_of_band_ip>
    Copy to Clipboard Toggle word wrap
    $ sudo timedatectl
    Copy to Clipboard Toggle word wrap
                   Local time: Tue 2020-03-10 18:20:02 UTC
               Universal time: Tue 2020-03-10 18:20:02 UTC
                     RTC time: Tue 2020-03-10 18:36:53
                    Time zone: UTC (UTC, +0000)
    System clock synchronized: no
                  NTP service: active
              RTC in local TZ: no
    Copy to Clipboard Toggle word wrap

  1. 참고

    variant: openshift
    version: 4.17.0
    metadata:
      name: 99-master-chrony
      labels:
        machineconfiguration.openshift.io/role: master
    storage:
      files:
      - path: /etc/chrony.conf
        mode: 0644
        overwrite: true
        contents:
          inline: |
            server <NTP_server> iburst 
    1
    
            stratumweight 0
            driftfile /var/lib/chrony/drift
            rtcsync
            makestep 10 3
            bindcmdaddress 127.0.0.1
            bindcmdaddress ::1
            keyfile /etc/chrony.keys
            commandkey 1
            generatecommandkey
            noclientlog
            logchange 0.5
            logdir /var/log/chrony
    Copy to Clipboard Toggle word wrap
    1
  2. $ butane 99-master-chrony.bu -o 99-master-chrony.yaml
    Copy to Clipboard Toggle word wrap
  3. $ oc apply -f 99-master-chrony.yaml
    Copy to Clipboard Toggle word wrap
  4. $ sudo timedatectl
    Copy to Clipboard Toggle word wrap
                   Local time: Tue 2020-03-10 19:10:02 UTC
               Universal time: Tue 2020-03-10 19:10:02 UTC
                     RTC time: Tue 2020-03-10 19:36:53
                    Time zone: UTC (UTC, +0000)
    System clock synchronized: yes
                  NTP service: active
              RTC in local TZ: no
    Copy to Clipboard Toggle word wrap

    $ cp chrony-masters.yaml ~/clusterconfigs/openshift/99_masters-chrony-configuration.yaml
    Copy to Clipboard Toggle word wrap

5.17.

  1. $ oc get nodes
    Copy to Clipboard Toggle word wrap
    NAME                   STATUS   ROLES           AGE  VERSION
    master-0.example.com   Ready    master,worker   4h   v1.30.3
    master-1.example.com   Ready    master,worker   4h   v1.30.3
    master-2.example.com   Ready    master,worker   4h   v1.30.3
    Copy to Clipboard Toggle word wrap
  2. $ oc get pods --all-namespaces | grep -iv running | grep -iv complete
    Copy to Clipboard Toggle word wrap

6장.

6.1.

  1. $ sudo dnf -y install butane
    Copy to Clipboard Toggle word wrap
  2. 참고

    variant: openshift
    version: 4.17.0
    metadata:
      name: 99-master-chrony-conf-override
      labels:
        machineconfiguration.openshift.io/role: master
    storage:
      files:
        - path: /etc/chrony.conf
          mode: 0644
          overwrite: true
          contents:
            inline: |
              # Use public servers from the pool.ntp.org project.
              # Please consider joining the pool (https://www.pool.ntp.org/join.html).
    
              # The Machine Config Operator manages this file
              server openshift-master-0.<cluster-name>.<domain> iburst 
    1
    
              server openshift-master-1.<cluster-name>.<domain> iburst
              server openshift-master-2.<cluster-name>.<domain> iburst
    
              stratumweight 0
              driftfile /var/lib/chrony/drift
              rtcsync
              makestep 10 3
              bindcmdaddress 127.0.0.1
              bindcmdaddress ::1
              keyfile /etc/chrony.keys
              commandkey 1
              generatecommandkey
              noclientlog
              logchange 0.5
              logdir /var/log/chrony
    
              # Configure the control plane nodes to serve as local NTP servers
              # for all compute nodes, even if they are not in sync with an
              # upstream NTP server.
    
              # Allow NTP client access from the local network.
              allow all
              # Serve time even if not synchronized to a time source.
              local stratum 3 orphan
    Copy to Clipboard Toggle word wrap

    1
  3. $ butane 99-master-chrony-conf-override.bu -o 99-master-chrony-conf-override.yaml
    Copy to Clipboard Toggle word wrap
  4. variant: openshift
    version: 4.17.0
    metadata:
      name: 99-worker-chrony-conf-override
      labels:
        machineconfiguration.openshift.io/role: worker
    storage:
      files:
        - path: /etc/chrony.conf
          mode: 0644
          overwrite: true
          contents:
            inline: |
              # The Machine Config Operator manages this file.
              server openshift-master-0.<cluster-name>.<domain> iburst 
    1
    
              server openshift-master-1.<cluster-name>.<domain> iburst
              server openshift-master-2.<cluster-name>.<domain> iburst
    
              stratumweight 0
              driftfile /var/lib/chrony/drift
              rtcsync
              makestep 10 3
              bindcmdaddress 127.0.0.1
              bindcmdaddress ::1
              keyfile /etc/chrony.keys
              commandkey 1
              generatecommandkey
              noclientlog
              logchange 0.5
              logdir /var/log/chrony
    Copy to Clipboard Toggle word wrap

    1
  5. $ butane 99-worker-chrony-conf-override.bu -o 99-worker-chrony-conf-override.yaml
    Copy to Clipboard Toggle word wrap
  6. $ oc apply -f 99-master-chrony-conf-override.yaml
    Copy to Clipboard Toggle word wrap

    machineconfig.machineconfiguration.openshift.io/99-master-chrony-conf-override created
    Copy to Clipboard Toggle word wrap

  7. $ oc apply -f 99-worker-chrony-conf-override.yaml
    Copy to Clipboard Toggle word wrap

    machineconfig.machineconfiguration.openshift.io/99-worker-chrony-conf-override created
    Copy to Clipboard Toggle word wrap

  8. $ oc describe machineconfigpool
    Copy to Clipboard Toggle word wrap

6.2.

  1. $ oc get provisioning -o yaml > enable-provisioning-nw.yaml
    Copy to Clipboard Toggle word wrap
  2. $ vim ~/enable-provisioning-nw.yaml
    Copy to Clipboard Toggle word wrap

    apiVersion: v1
    items:
    - apiVersion: metal3.io/v1alpha1
      kind: Provisioning
      metadata:
        name: provisioning-configuration
      spec:
        provisioningNetwork: 
    1
    
        provisioningIP: 
    2
    
        provisioningNetworkCIDR: 
    3
    
        provisioningDHCPRange: 
    4
    
        provisioningInterface: 
    5
    
        watchAllNameSpaces: 
    6
    Copy to Clipboard Toggle word wrap
    1
    2
    3
    4
    5
    6
  3. $ oc apply -f enable-provisioning-nw.yaml
    Copy to Clipboard Toggle word wrap

6.3.

  • 중요

    apiVersion: nmstate.io/v1
    kind: NodeNetworkConfigurationPolicy
    metadata:
      name: worker-0-br-ex 
    1
    
    spec:
      nodeSelector:
        kubernetes.io/hostname: worker-0
        desiredState:
        interfaces:
        - name: enp2s0 
    2
    
          type: ethernet 
    3
    
          state: up 
    4
    
          ipv4:
            enabled: false 
    5
    
          ipv6:
            enabled: false
        - name: br-ex
          type: ovs-bridge
          state: up
          ipv4:
            enabled: false
            dhcp: false
          ipv6:
            enabled: false
            dhcp: false
          bridge:
            port:
            - name: enp2s0 
    6
    
            - name: br-ex
        - name: br-ex
          type: ovs-interface
          state: up
          copy-mac-from: enp2s0
          ipv4:
            enabled: true
            dhcp: true
            address:
            - ip: "169.254.169.2"
              prefix-length: 29
          ipv6:
            enabled: false
            dhcp: false
            address:
            - ip: "fd69::2"
            prefix-length: 125
    Copy to Clipboard Toggle word wrap

    1
    2
    3
    4
    5
    6

6.4.

중요

그림 6.1.

그림 6.2.

그림 6.3.

  • 작은 정보

6.4.1.

중요

참고

Path: HTTPS:6443/readyz
Healthy threshold: 2
Unhealthy threshold: 2
Timeout: 10
Interval: 10
Copy to Clipboard Toggle word wrap

Path: HTTPS:22623/healthz
Healthy threshold: 2
Unhealthy threshold: 2
Timeout: 10
Interval: 10
Copy to Clipboard Toggle word wrap

Path: HTTP:1936/healthz/ready
Healthy threshold: 2
Unhealthy threshold: 2
Timeout: 5
Interval: 10
Copy to Clipboard Toggle word wrap

  1. # ...
    listen my-cluster-api-6443
        bind 192.168.1.100:6443
        mode tcp
        balance roundrobin
      option httpchk
      http-check connect
      http-check send meth GET uri /readyz
      http-check expect status 200
        server my-cluster-master-2 192.168.1.101:6443 check inter 10s rise 2 fall 2
        server my-cluster-master-0 192.168.1.102:6443 check inter 10s rise 2 fall 2
        server my-cluster-master-1 192.168.1.103:6443 check inter 10s rise 2 fall 2
    
    listen my-cluster-machine-config-api-22623
        bind 192.168.1.100:22623
        mode tcp
        balance roundrobin
      option httpchk
      http-check connect
      http-check send meth GET uri /healthz
      http-check expect status 200
        server my-cluster-master-2 192.168.1.101:22623 check inter 10s rise 2 fall 2
        server my-cluster-master-0 192.168.1.102:22623 check inter 10s rise 2 fall 2
        server my-cluster-master-1 192.168.1.103:22623 check inter 10s rise 2 fall 2
    
    listen my-cluster-apps-443
        bind 192.168.1.100:443
        mode tcp
        balance roundrobin
      option httpchk
      http-check connect
      http-check send meth GET uri /healthz/ready
      http-check expect status 200
        server my-cluster-worker-0 192.168.1.111:443 check port 1936 inter 10s rise 2 fall 2
        server my-cluster-worker-1 192.168.1.112:443 check port 1936 inter 10s rise 2 fall 2
        server my-cluster-worker-2 192.168.1.113:443 check port 1936 inter 10s rise 2 fall 2
    
    listen my-cluster-apps-80
       bind 192.168.1.100:80
       mode tcp
       balance roundrobin
      option httpchk
      http-check connect
      http-check send meth GET uri /healthz/ready
      http-check expect status 200
        server my-cluster-worker-0 192.168.1.111:80 check port 1936 inter 10s rise 2 fall 2
        server my-cluster-worker-1 192.168.1.112:80 check port 1936 inter 10s rise 2 fall 2
        server my-cluster-worker-2 192.168.1.113:80 check port 1936 inter 10s rise 2 fall 2
    # ...
    Copy to Clipboard Toggle word wrap

    # ...
    listen api-server-6443
        bind *:6443
        mode tcp
          server master-00 192.168.83.89:6443 check inter 1s
          server master-01 192.168.84.90:6443 check inter 1s
          server master-02 192.168.85.99:6443 check inter 1s
          server bootstrap 192.168.80.89:6443 check inter 1s
    
    listen machine-config-server-22623
        bind *:22623
        mode tcp
          server master-00 192.168.83.89:22623 check inter 1s
          server master-01 192.168.84.90:22623 check inter 1s
          server master-02 192.168.85.99:22623 check inter 1s
          server bootstrap 192.168.80.89:22623 check inter 1s
    
    listen ingress-router-80
        bind *:80
        mode tcp
        balance source
          server worker-00 192.168.83.100:80 check inter 1s
          server worker-01 192.168.83.101:80 check inter 1s
    
    listen ingress-router-443
        bind *:443
        mode tcp
        balance source
          server worker-00 192.168.83.100:443 check inter 1s
          server worker-01 192.168.83.101:443 check inter 1s
    
    listen ironic-api-6385
        bind *:6385
        mode tcp
        balance source
          server master-00 192.168.83.89:6385 check inter 1s
          server master-01 192.168.84.90:6385 check inter 1s
          server master-02 192.168.85.99:6385 check inter 1s
          server bootstrap 192.168.80.89:6385 check inter 1s
    
    listen inspector-api-5050
        bind *:5050
        mode tcp
        balance source
          server master-00 192.168.83.89:5050 check inter 1s
          server master-01 192.168.84.90:5050 check inter 1s
          server master-02 192.168.85.99:5050 check inter 1s
          server bootstrap 192.168.80.89:5050 check inter 1s
    # ...
    Copy to Clipboard Toggle word wrap

    1. $ curl https://<loadbalancer_ip_address>:6443/version --insecure
      Copy to Clipboard Toggle word wrap

      {
        "major": "1",
        "minor": "11+",
        "gitVersion": "v1.11.0+ad103ed",
        "gitCommit": "ad103ed",
        "gitTreeState": "clean",
        "buildDate": "2019-01-09T06:44:10Z",
        "goVersion": "go1.10.3",
        "compiler": "gc",
        "platform": "linux/amd64"
      }
      Copy to Clipboard Toggle word wrap
    2. $ curl -v https://<loadbalancer_ip_address>:22623/healthz --insecure
      Copy to Clipboard Toggle word wrap

      HTTP/1.1 200 OK
      Content-Length: 0
      Copy to Clipboard Toggle word wrap
    3. $ curl -I -L -H "Host: console-openshift-console.apps.<cluster_name>.<base_domain>" http://<load_balancer_front_end_IP_address>
      Copy to Clipboard Toggle word wrap

      HTTP/1.1 302 Found
      content-length: 0
      location: https://console-openshift-console.apps.ocp4.private.opequon.net/
      cache-control: no-cache
      Copy to Clipboard Toggle word wrap
    4. $ curl -I -L --insecure --resolve console-openshift-console.apps.<cluster_name>.<base_domain>:443:<Load Balancer Front End IP Address> https://console-openshift-console.apps.<cluster_name>.<base_domain>
      Copy to Clipboard Toggle word wrap

      HTTP/1.1 200 OK
      referrer-policy: strict-origin-when-cross-origin
      set-cookie: csrf-token=UlYWOyQ62LWjw2h003xtYSKlh1a0Py2hhctw0WmV2YEdhJjFyQwWcGBsja261dGLgaYO0nxzVErhiXt6QepA7g==; Path=/; Secure; SameSite=Lax
      x-content-type-options: nosniff
      x-dns-prefetch-control: off
      x-frame-options: DENY
      x-xss-protection: 1; mode=block
      date: Wed, 04 Oct 2023 16:29:38 GMT
      content-type: text/html; charset=utf-8
      set-cookie: 1e2670d92730b515ce3a1bb65da45062=1bf5e9573c9a2760c964ed1659cc1673; path=/; HttpOnly; Secure; SameSite=None
      cache-control: private
      Copy to Clipboard Toggle word wrap
  2. <load_balancer_ip_address>  A  api.<cluster_name>.<base_domain>
    A record pointing to Load Balancer Front End
    Copy to Clipboard Toggle word wrap

    <load_balancer_ip_address>   A apps.<cluster_name>.<base_domain>
    A record pointing to Load Balancer Front End
    Copy to Clipboard Toggle word wrap
    중요

  3. # ...
    platform:
        loadBalancer:
          type: UserManaged 
    1
    
        apiVIPs:
        - <api_ip> 
    2
    
        ingressVIPs:
        - <ingress_ip> 
    3
    
    # ...
    Copy to Clipboard Toggle word wrap
    1
    2
    3

    1. $ curl https://api.<cluster_name>.<base_domain>:6443/version --insecure
      Copy to Clipboard Toggle word wrap

      {
        "major": "1",
        "minor": "11+",
        "gitVersion": "v1.11.0+ad103ed",
        "gitCommit": "ad103ed",
        "gitTreeState": "clean",
        "buildDate": "2019-01-09T06:44:10Z",
        "goVersion": "go1.10.3",
        "compiler": "gc",
        "platform": "linux/amd64"
        }
      Copy to Clipboard Toggle word wrap
    2. $ curl -v https://api.<cluster_name>.<base_domain>:22623/healthz --insecure
      Copy to Clipboard Toggle word wrap

      HTTP/1.1 200 OK
      Content-Length: 0
      Copy to Clipboard Toggle word wrap
    3. $ curl http://console-openshift-console.apps.<cluster_name>.<base_domain> -I -L --insecure
      Copy to Clipboard Toggle word wrap

      HTTP/1.1 302 Found
      content-length: 0
      location: https://console-openshift-console.apps.<cluster-name>.<base domain>/
      cache-control: no-cacheHTTP/1.1 200 OK
      referrer-policy: strict-origin-when-cross-origin
      set-cookie: csrf-token=39HoZgztDnzjJkq/JuLJMeoKNXlfiVv2YgZc09c3TBOBU4NI6kDXaJH1LdicNhN1UsQWzon4Dor9GWGfopaTEQ==; Path=/; Secure
      x-content-type-options: nosniff
      x-dns-prefetch-control: off
      x-frame-options: DENY
      x-xss-protection: 1; mode=block
      date: Tue, 17 Nov 2020 08:42:10 GMT
      content-type: text/html; charset=utf-8
      set-cookie: 1e2670d92730b515ce3a1bb65da45062=9b714eb87e93cf34853e87a92d6894be; path=/; HttpOnly; Secure; SameSite=None
      cache-control: private
      Copy to Clipboard Toggle word wrap
    4. $ curl https://console-openshift-console.apps.<cluster_name>.<base_domain> -I -L --insecure
      Copy to Clipboard Toggle word wrap

      HTTP/1.1 200 OK
      referrer-policy: strict-origin-when-cross-origin
      set-cookie: csrf-token=UlYWOyQ62LWjw2h003xtYSKlh1a0Py2hhctw0WmV2YEdhJjFyQwWcGBsja261dGLgaYO0nxzVErhiXt6QepA7g==; Path=/; Secure; SameSite=Lax
      x-content-type-options: nosniff
      x-dns-prefetch-control: off
      x-frame-options: DENY
      x-xss-protection: 1; mode=block
      date: Wed, 04 Oct 2023 16:29:38 GMT
      content-type: text/html; charset=utf-8
      set-cookie: 1e2670d92730b515ce3a1bb65da45062=1bf5e9573c9a2760c964ed1659cc1673; path=/; HttpOnly; Secure; SameSite=None
      cache-control: private
      Copy to Clipboard Toggle word wrap

6.5.

6.5.1.

참고

6.5.2.

6.5.2.1.

Expand
표 6.1.
  

bmc:
  address:
  credentialsName:
  disableCertificateVerification:
Copy to Clipboard Toggle word wrap

image:
  url:
  checksum:
  checksumType:
  format:
Copy to Clipboard Toggle word wrap

raid:
  hardwareRAIDVolumes:
  softwareRAIDVolumes:
Copy to Clipboard Toggle word wrap

참고

spec:
   raid:
     hardwareRAIDVolume: []
Copy to Clipboard Toggle word wrap

rootDeviceHints:
  deviceName:
  hctl:
  model:
  vendor:
  serialNumber:
  minSizeGigabytes:
  wwn:
  wwnWithExtension:
  wwnVendorExtension:
  rotational:
Copy to Clipboard Toggle word wrap

6.5.2.2.

Expand
표 6.2.
  

hardware:
  cpu
    arch:
    model:
    clockMegahertz:
    flags:
    count:
Copy to Clipboard Toggle word wrap

hardware:
  firmware:
Copy to Clipboard Toggle word wrap

hardware:
  nics:
  - ip:
    name:
    mac:
    speedGbps:
    vlans:
    vlanId:
    pxe:
Copy to Clipboard Toggle word wrap

hardware:
  ramMebibytes:
Copy to Clipboard Toggle word wrap

hardware:
  storage:
  - name:
    rotational:
    sizeBytes:
    serialNumber:
Copy to Clipboard Toggle word wrap

hardware:
  systemVendor:
    manufacturer:
    productName:
    serialNumber:
Copy to Clipboard Toggle word wrap

provisioning:
  state:
  id:
  image:
  raid:
  firmware:
  rootDeviceHints:
Copy to Clipboard Toggle word wrap

6.5.3.

  1. $ oc get bmh -n openshift-machine-api -o yaml
    Copy to Clipboard Toggle word wrap
    참고

  2. $ oc get bmh -n openshift-machine-api
    Copy to Clipboard Toggle word wrap
  3. $ oc get bmh <host_name> -n openshift-machine-api -o yaml
    Copy to Clipboard Toggle word wrap

    apiVersion: metal3.io/v1alpha1
    kind: BareMetalHost
    metadata:
      creationTimestamp: "2022-06-16T10:48:33Z"
      finalizers:
      - baremetalhost.metal3.io
      generation: 2
      name: openshift-worker-0
      namespace: openshift-machine-api
      resourceVersion: "30099"
      uid: 1513ae9b-e092-409d-be1b-ad08edeb1271
    spec:
      automatedCleaningMode: metadata
      bmc:
        address: redfish://10.46.61.19:443/redfish/v1/Systems/1
        credentialsName: openshift-worker-0-bmc-secret
        disableCertificateVerification: true
      bootMACAddress: 48:df:37:c7:f7:b0
      bootMode: UEFI
      consumerRef:
        apiVersion: machine.openshift.io/v1beta1
        kind: Machine
        name: ocp-edge-958fk-worker-0-nrfcg
        namespace: openshift-machine-api
      customDeploy:
        method: install_coreos
      online: true
      rootDeviceHints:
        deviceName: /dev/disk/by-id/scsi-<serial_number>
      userData:
        name: worker-user-data-managed
        namespace: openshift-machine-api
    status:
      errorCount: 0
      errorMessage: ""
      goodCredentials:
        credentials:
          name: openshift-worker-0-bmc-secret
          namespace: openshift-machine-api
        credentialsVersion: "16120"
      hardware:
        cpu:
          arch: x86_64
          clockMegahertz: 2300
          count: 64
          flags:
          - 3dnowprefetch
          - abm
          - acpi
          - adx
          - aes
          model: Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz
        firmware:
          bios:
            date: 10/26/2020
            vendor: HPE
            version: U30
        hostname: openshift-worker-0
        nics:
        - mac: 48:df:37:c7:f7:b3
          model: 0x8086 0x1572
          name: ens1f3
        ramMebibytes: 262144
        storage:
        - hctl: "0:0:0:0"
          model: VK000960GWTTB
          name: /dev/disk/by-id/scsi-<serial_number>
          sizeBytes: 960197124096
          type: SSD
          vendor: ATA
        systemVendor:
          manufacturer: HPE
          productName: ProLiant DL380 Gen10 (868703-B21)
          serialNumber: CZ200606M3
      lastUpdated: "2022-06-16T11:41:42Z"
      operationalStatus: OK
      poweredOn: true
      provisioning:
        ID: 217baa14-cfcf-4196-b764-744e184a3413
        bootMode: UEFI
        customDeploy:
          method: install_coreos
        image:
          url: ""
        raid:
          hardwareRAIDVolumes: null
          softwareRAIDVolumes: []
        rootDeviceHints:
          deviceName: /dev/disk/by-id/scsi-<serial_number>
        state: provisioned
      triedCredentials:
        credentials:
          name: openshift-worker-0-bmc-secret
          namespace: openshift-machine-api
        credentialsVersion: "16120"
    Copy to Clipboard Toggle word wrap

6.5.4.

  1. $ oc get bmh -n openshift-machine-api
    Copy to Clipboard Toggle word wrap
  2. $ oc annotate baremetalhost <node_name> -n openshift-machine-api 'baremetalhost.metal3.io/detached=true' 
    1
    Copy to Clipboard Toggle word wrap
    1
  3. $ oc edit bmh <node_name> -n openshift-machine-api
    Copy to Clipboard Toggle word wrap
  4. $ oc annotate baremetalhost <node_name> -n openshift-machine-api 'baremetalhost.metal3.io/detached'-
    Copy to Clipboard Toggle word wrap

6.5.5.

주의

6.5.6.

  1. apiVersion: metal3.io/v1alpha1
    kind: DataImage
    metadata:
      name: <node_name> 
    1
    
    spec:
      url: "http://dataimage.example.com/non-bootable.iso" 
    2
    Copy to Clipboard Toggle word wrap
    1
    2
  2. $ vim <node_name>-dataimage.yaml
    Copy to Clipboard Toggle word wrap
  3. $ oc apply -f <node_name>-dataimage.yaml -n <node_namespace> 
    1
    Copy to Clipboard Toggle word wrap
    1
  4. 참고

  5. $ oc get dataimage <node_name> -n openshift-machine-api -o yaml
    Copy to Clipboard Toggle word wrap

    apiVersion: v1
    items:
    - apiVersion: metal3.io/v1alpha1
      kind: DataImage
      metadata:
        annotations:
          kubectl.kubernetes.io/last-applied-configuration: |
            {"apiVersion":"metal3.io/v1alpha1","kind":"DataImage","metadata":{"annotations":{},"name":"bmh-node-1","namespace":"openshift-machine-api"},"spec":{"url":"http://dataimage.example.com/non-bootable.iso"}}
        creationTimestamp: "2024-06-10T12:00:00Z"
        finalizers:
        - dataimage.metal3.io
        generation: 1
        name: bmh-node-1
        namespace: openshift-machine-api
        ownerReferences:
        - apiVersion: metal3.io/v1alpha1
          blockOwnerDeletion: true
          controller: true
          kind: BareMetalHost
          name: bmh-node-1
          uid: 046cdf8e-0e97-485a-8866-e62d20e0f0b3
        resourceVersion: "21695581"
        uid: c5718f50-44b6-4a22-a6b7-71197e4b7b69
      spec:
        url: http://dataimage.example.com/non-bootable.iso
      status:
        attachedImage:
          url: http://dataimage.example.com/non-bootable.iso
        error:
          count: 0
          message: ""
        lastReconciled: "2024-06-10T12:05:00Z"
    Copy to Clipboard Toggle word wrap

6.5.7.

6.5.7.1.

spec:
  settings:
    ProcTurboMode: Disabled
1
Copy to Clipboard Toggle word wrap

1
참고

6.5.7.2.

Expand
표 6.3.
  
status:
  conditions:
  - lastTransitionTime:
    message:
    observedGeneration:
    reason:
    status:
    type:
Copy to Clipboard Toggle word wrap

status:
  schema:
    name:
    namespace:
    lastUpdated:
Copy to Clipboard Toggle word wrap

status:
  settings:
Copy to Clipboard Toggle word wrap

6.5.8.

  1. $ oc get hfs -n openshift-machine-api -o yaml
    Copy to Clipboard Toggle word wrap
    참고

  2. $ oc get hfs -n openshift-machine-api
    Copy to Clipboard Toggle word wrap
  3. $ oc get hfs <host_name> -n openshift-machine-api -o yaml
    Copy to Clipboard Toggle word wrap

6.5.9.

중요

  1. $ oc get hfs -n openshift-machine-api
    Copy to Clipboard Toggle word wrap
  2. $ oc edit hfs <host_name> -n openshift-machine-api
    Copy to Clipboard Toggle word wrap

  3. spec:
      settings:
        name: value 
    1
    Copy to Clipboard Toggle word wrap

    1
  4.  $ oc get bmh <host_name> -n openshift-machine name
    Copy to Clipboard Toggle word wrap

  5. $ oc annotate machine <machine_name> machine.openshift.io/delete-machine=true -n openshift-machine-api
    Copy to Clipboard Toggle word wrap

  6. $ oc get nodes
    Copy to Clipboard Toggle word wrap
  7. $ oc get machinesets -n openshift-machine-api
    Copy to Clipboard Toggle word wrap
  8. $ oc scale machineset <machineset_name> -n openshift-machine-api --replicas=<n-1>
    Copy to Clipboard Toggle word wrap

  9. $ oc scale machineset <machineset_name> -n openshift-machine-api --replicas=<n>
    Copy to Clipboard Toggle word wrap

6.5.10.

  1. $ oc get hfs -n openshift-machine-api
    Copy to Clipboard Toggle word wrap
  2. $ oc describe hfs <host_name> -n openshift-machine-api
    Copy to Clipboard Toggle word wrap

    Events:
      Type    Reason            Age    From                                    Message
      ----    ------            ----   ----                                    -------
      Normal  ValidationFailed  2m49s  metal3-hostfirmwaresettings-controller  Invalid BIOS setting: Setting ProcTurboMode is invalid, unknown enumeration value - Foo
    Copy to Clipboard Toggle word wrap

    중요

6.5.11.

Expand
표 6.4.
  
<BIOS_setting_name>
  attribute_type:
  allowable_values:
  lower_bound:
  upper_bound:
  min_length:
  max_length:
  read_only:
  unique:
Copy to Clipboard Toggle word wrap

6.5.12.

  1. $ oc get firmwareschema -n openshift-machine-api
    Copy to Clipboard Toggle word wrap
  2. $ oc get firmwareschema <instance_name> -n openshift-machine-api -o yaml
    Copy to Clipboard Toggle word wrap

6.5.13.

6.5.13.1.

Expand
표 6.5.
  
updates:
  component:
  url:
Copy to Clipboard Toggle word wrap

6.5.13.2.

Expand
표 6.6.
  
components:
  component:
  initialVersion:
  currentVersion:
  lastVersionFlashed:
  updatedAt:
Copy to Clipboard Toggle word wrap

updates:
  component:
  url:
Copy to Clipboard Toggle word wrap

6.5.14.

  1. $ oc get hostfirmwarecomponents -n openshift-machine-api -o yaml
    Copy to Clipboard Toggle word wrap
  2. $ oc get hostfirmwarecomponents -n openshift-machine-api
    Copy to Clipboard Toggle word wrap
  3. $ oc get hostfirmwarecomponents <host_name> -n openshift-machine-api -o yaml
    Copy to Clipboard Toggle word wrap

    ---
    apiVersion: metal3.io/v1alpha1
    kind: HostFirmwareComponents
    metadata:
      creationTimestamp: 2024-04-25T20:32:06Z"
      generation: 1
      name: ostest-master-2
      namespace: openshift-machine-api
      ownerReferences:
      - apiVersion: metal3.io/v1alpha1
        blockOwnerDeletion: true
        controller: true
        kind: BareMetalHost
        name: ostest-master-2
        uid: 16022566-7850-4dc8-9e7d-f216211d4195
      resourceVersion: "2437"
      uid: 2038d63f-afc0-4413-8ffe-2f8e098d1f6c
    spec:
      updates: []
    status:
      components:
      - component: bios
        currentVersion: 1.0.0
        initialVersion: 1.0.0
      - component: bmc
        currentVersion: "1.00"
        initialVersion: "1.00"
      conditions:
      - lastTransitionTime: "2024-04-25T20:32:06Z"
        message: ""
        observedGeneration: 1
        reason: OK
        status: "True"
        type: Valid
      - lastTransitionTime: "2024-04-25T20:32:06Z"
        message: ""
        observedGeneration: 1
        reason: OK
        status: "False"
        type: ChangeDetected
      lastUpdated: "2024-04-25T20:32:06Z"
      updates: []
    Copy to Clipboard Toggle word wrap

6.5.15.

  1. $ oc get hostfirmwarecomponents -n openshift-machine-api -o yaml
    Copy to Clipboard Toggle word wrap
  2. $ oc edit <host_name> hostfirmwarecomponents -n openshift-machine-api 
    1
    Copy to Clipboard Toggle word wrap
    1

    ---
    apiVersion: metal3.io/v1alpha1
    kind: HostFirmwareComponents
    metadata:
      creationTimestamp: 2024-04-25T20:32:06Z"
      generation: 1
      name: ostest-master-2
      namespace: openshift-machine-api
      ownerReferences:
      - apiVersion: metal3.io/v1alpha1
        blockOwnerDeletion: true
        controller: true
        kind: BareMetalHost
        name: ostest-master-2
        uid: 16022566-7850-4dc8-9e7d-f216211d4195
      resourceVersion: "2437"
      uid: 2038d63f-afc0-4413-8ffe-2f8e098d1f6c
    spec:
      updates:
        - name: bios 
    1
    
          url: https://myurl.with.firmware.for.bios 
    2
    
        - name: bmc 
    3
    
          url: https://myurl.with.firmware.for.bmc 
    4
    
    status:
      components:
      - component: bios
        currentVersion: 1.0.0
        initialVersion: 1.0.0
      - component: bmc
        currentVersion: "1.00"
        initialVersion: "1.00"
      conditions:
      - lastTransitionTime: "2024-04-25T20:32:06Z"
        message: ""
        observedGeneration: 1
        reason: OK
        status: "True"
        type: Valid
      - lastTransitionTime: "2024-04-25T20:32:06Z"
        message: ""
        observedGeneration: 1
        reason: OK
        status: "False"
        type: ChangeDetected
      lastUpdated: "2024-04-25T20:32:06Z"
    Copy to Clipboard Toggle word wrap

    1
    2
    3
    4
  3. $ oc get bmh <host_name> -n openshift-machine name 
    1
    Copy to Clipboard Toggle word wrap
    1
  4. $ oc annotate machine <machine_name> machine.openshift.io/delete-machine=true -n openshift-machine-api 
    1
    Copy to Clipboard Toggle word wrap
    1
  5. $ oc get nodes
    Copy to Clipboard Toggle word wrap
  6. $ oc get machinesets -n openshift-machine-api
    Copy to Clipboard Toggle word wrap
  7. $ oc scale machineset <machineset_name> -n openshift-machine-api --replicas=<n-1> 
    1
    Copy to Clipboard Toggle word wrap
    1
  8. $ oc scale machineset <machineset_name> -n openshift-machine-api --replicas=<n> 
    1
    Copy to Clipboard Toggle word wrap
    1

7장.

참고

7.1.

  1. $ curl -s https://mirror.openshift.com/pub/openshift-v4/clients/ocp/$VERSION/openshift-client-linux-$VERSION.tar.gz | tar zxvf - oc
    Copy to Clipboard Toggle word wrap
    $ sudo cp oc /usr/local/bin
    Copy to Clipboard Toggle word wrap
  2. $ echo -ne "root" | base64
    Copy to Clipboard Toggle word wrap
    $ echo -ne "password" | base64
    Copy to Clipboard Toggle word wrap
  3. $ vim bmh.yaml
    Copy to Clipboard Toggle word wrap
    • ---
      apiVersion: v1 
      1
      
      kind: Secret
      metadata:
       name: openshift-worker-<num>-network-config-secret 
      2
      
       namespace: openshift-machine-api
      type: Opaque
      stringData:
       nmstate: | 
      3
      
        interfaces: 
      4
      
        - name: <nic1_name> 
      5
      
          type: ethernet
          state: up
          ipv4:
            address:
            - ip: <ip_address> 
      6
      
              prefix-length: 24
            enabled: true
        dns-resolver:
          config:
            server:
            - <dns_ip_address> 
      7
      
        routes:
          config:
          - destination: 0.0.0.0/0
            next-hop-address: <next_hop_ip_address> 
      8
      
            next-hop-interface: <next_hop_nic1_name> 
      9
      
      ---
      apiVersion: v1
      kind: Secret
      metadata:
        name: openshift-worker-<num>-bmc-secret 
      10
      
        namespace: openshift-machine-api
      type: Opaque
      data:
        username: <base64_of_uid> 
      11
      
        password: <base64_of_pwd> 
      12
      
      ---
      apiVersion: metal3.io/v1alpha1
      kind: BareMetalHost
      metadata:
        name: openshift-worker-<num> 
      13
      
        namespace: openshift-machine-api
      spec:
        online: True
        bootMACAddress: <nic1_mac_address> 
      14
      
        bmc:
          address: <protocol>://<bmc_url> 
      15
      
          credentialsName: openshift-worker-<num>-bmc-secret 
      16
      
          disableCertificateVerification: True 
      17
      
          username: <bmc_username> 
      18
      
          password: <bmc_password> 
      19
      
        rootDeviceHints:
          deviceName: <root_device_hint> 
      20
      
        preprovisioningNetworkDataName: openshift-worker-<num>-network-config-secret 
      21
      Copy to Clipboard Toggle word wrap
      1
      2 10 13 16
      3
      4
      ---
         interfaces:
         - name: <nic_name>
           type: ethernet
           state: up
           ipv4:
             enabled: false
           ipv6:
             enabled: false
      Copy to Clipboard Toggle word wrap
      5 6 7 8 9
      11 12
      14
      15
      17
      18 19
      20
      21
    • ---
      apiVersion: v1
      kind: Secret
      metadata:
        name: openshift-worker-<num>-bmc-secret 
      1
      
        namespace: openshift-machine-api
      type: Opaque
      data:
        username: <base64_of_uid> 
      2
      
        password: <base64_of_pwd> 
      3
      
      ---
      apiVersion: metal3.io/v1alpha1
      kind: BareMetalHost
      metadata:
        name: openshift-worker-<num> 
      4
      
        namespace: openshift-machine-api
      spec:
        online: True
        bootMACAddress: <nic1_mac_address> 
      5
      
        bmc:
          address: <protocol>://<bmc_url> 
      6
      
          credentialsName: openshift-worker-<num>-bmc-secret 
      7
      
          disableCertificateVerification: True 
      8
      
          username: <bmc_username> 
      9
      
          password: <bmc_password> 
      10
      
        rootDeviceHints:
          deviceName: <root_device_hint> 
      11
      
        preprovisioningNetworkDataName: openshift-worker-<num>-network-config-secret 
      12
      Copy to Clipboard Toggle word wrap
      1 4 7
      2 3
      5
      6
      8
      9 10
      11
      12
    참고

  4. $ oc -n openshift-machine-api create -f bmh.yaml
    Copy to Clipboard Toggle word wrap

    secret/openshift-worker-<num>-network-config-secret created
    secret/openshift-worker-<num>-bmc-secret created
    baremetalhost.metal3.io/openshift-worker-<num> created
    Copy to Clipboard Toggle word wrap

  5. $ oc -n openshift-machine-api get bmh openshift-worker-<num>
    Copy to Clipboard Toggle word wrap

    NAME                    STATE       CONSUMER   ONLINE   ERROR
    openshift-worker-<num>  available              true
    Copy to Clipboard Toggle word wrap

    참고

7.2.

중요

  • 중요

  1. $ oc get clusteroperator baremetal
    Copy to Clipboard Toggle word wrap

    NAME        VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
    baremetal   4.17   True        False         False      3d15h
    Copy to Clipboard Toggle word wrap

  2. $ oc delete bmh -n openshift-machine-api <host_name>
    $ oc delete machine -n openshift-machine-api <machine_name>
    Copy to Clipboard Toggle word wrap

  3. $ cat <<EOF | oc apply -f -
    apiVersion: v1
    kind: Secret
    metadata:
      name: control-plane-<num>-bmc-secret 
    1
    
      namespace: openshift-machine-api
    data:
      username: <base64_of_uid> 
    2
    
      password: <base64_of_pwd> 
    3
    
    type: Opaque
    ---
    apiVersion: metal3.io/v1alpha1
    kind: BareMetalHost
    metadata:
      name: control-plane-<num> 
    4
    
      namespace: openshift-machine-api
    spec:
      automatedCleaningMode: disabled
      bmc:
        address: <protocol>://<bmc_ip> 
    5
    
        credentialsName: control-plane-<num>-bmc-secret 
    6
    
      bootMACAddress: <NIC1_mac_address> 
    7
    
      bootMode: UEFI
      externallyProvisioned: false
      online: true
    EOF
    Copy to Clipboard Toggle word wrap
    1 4 6
    2
    3
    5
    7

  4. $ oc get bmh -n openshift-machine-api
    Copy to Clipboard Toggle word wrap

    NAME                          STATE                    CONSUMER                   ONLINE   ERROR   AGE
    control-plane-1.example.com   available                control-plane-1            true             1h10m
    control-plane-2.example.com   externally provisioned   control-plane-2            true             4h53m
    control-plane-3.example.com   externally provisioned   control-plane-3            true             4h53m
    compute-1.example.com         provisioned              compute-1-ktmmx            true             4h53m
    compute-1.example.com         provisioned              compute-2-l2zmb            true             4h53m
    Copy to Clipboard Toggle word wrap

  5. $ cat <<EOF | oc apply -f -
    apiVersion: machine.openshift.io/v1beta1
    kind: Machine
    metadata:
      annotations:
        metal3.io/BareMetalHost: openshift-machine-api/control-plane-<num> 
    1
    
      labels:
        machine.openshift.io/cluster-api-cluster: control-plane-<num> 
    2
    
        machine.openshift.io/cluster-api-machine-role: master
        machine.openshift.io/cluster-api-machine-type: master
      name: control-plane-<num> 
    3
    
      namespace: openshift-machine-api
    spec:
      metadata: {}
      providerSpec:
        value:
          apiVersion: baremetal.cluster.k8s.io/v1alpha1
          customDeploy:
            method: install_coreos
          hostSelector: {}
          image:
            checksum: ""
            url: ""
          kind: BareMetalMachineProviderSpec
          metadata:
            creationTimestamp: null
          userData:
            name: master-user-data-managed
    EOF
    Copy to Clipboard Toggle word wrap
    1 2 3
  6. $ oc get bmh -A
    Copy to Clipboard Toggle word wrap

    NAME                          STATE                    CONSUMER                   ONLINE   ERROR   AGE
    control-plane-1.example.com   provisioned              control-plane-1            true             2h53m
    control-plane-2.example.com   externally provisioned   control-plane-2            true             5h53m
    control-plane-3.example.com   externally provisioned   control-plane-3            true             5h53m
    compute-1.example.com         provisioned              compute-1-ktmmx            true             5h53m
    compute-2.example.com         provisioned              compute-2-l2zmb            true             5h53m
    Copy to Clipboard Toggle word wrap

  7. $ oc get nodes
    Copy to Clipboard Toggle word wrap

    NAME                           STATUS      ROLES     AGE   VERSION
    control-plane-1.example.com    available   master    4m2s  v1.30.3
    control-plane-2.example.com    available   master    141m  v1.30.3
    control-plane-3.example.com    available   master    141m  v1.30.3
    compute-1.example.com          available   worker    87m   v1.30.3
    compute-2.example.com          available   worker    87m   v1.30.3
    Copy to Clipboard Toggle word wrap

    참고

7.3.

  1. oc edit provisioning
    Copy to Clipboard Toggle word wrap
      apiVersion: metal3.io/v1alpha1
      kind: Provisioning
      metadata:
        creationTimestamp: "2021-08-05T18:51:50Z"
        finalizers:
        - provisioning.metal3.io
        generation: 8
        name: provisioning-configuration
        resourceVersion: "551591"
        uid: f76e956f-24c6-4361-aa5b-feaf72c5b526
      spec:
        provisioningDHCPRange: 172.22.0.10,172.22.0.254
        provisioningIP: 172.22.0.3
        provisioningInterface: enp1s0
        provisioningNetwork: Managed
        provisioningNetworkCIDR: 172.22.0.0/24
        virtualMediaViaExternalNetwork: true 
    1
    
      status:
        generations:
        - group: apps
          hash: ""
          lastGeneration: 7
          name: metal3
          namespace: openshift-machine-api
          resource: deployments
        - group: apps
          hash: ""
          lastGeneration: 1
          name: metal3-image-cache
          namespace: openshift-machine-api
          resource: daemonsets
        observedGeneration: 8
        readyReplicas: 0
    Copy to Clipboard Toggle word wrap
    1
  2. oc edit machineset
    Copy to Clipboard Toggle word wrap
      apiVersion: machine.openshift.io/v1beta1
      kind: MachineSet
      metadata:
        creationTimestamp: "2021-08-05T18:51:52Z"
        generation: 11
        labels:
          machine.openshift.io/cluster-api-cluster: ostest-hwmdt
          machine.openshift.io/cluster-api-machine-role: worker
          machine.openshift.io/cluster-api-machine-type: worker
        name: ostest-hwmdt-worker-0
        namespace: openshift-machine-api
        resourceVersion: "551513"
        uid: fad1c6e0-b9da-4d4a-8d73-286f78788931
      spec:
        replicas: 2
        selector:
          matchLabels:
            machine.openshift.io/cluster-api-cluster: ostest-hwmdt
            machine.openshift.io/cluster-api-machineset: ostest-hwmdt-worker-0
        template:
          metadata:
            labels:
              machine.openshift.io/cluster-api-cluster: ostest-hwmdt
              machine.openshift.io/cluster-api-machine-role: worker
              machine.openshift.io/cluster-api-machine-type: worker
              machine.openshift.io/cluster-api-machineset: ostest-hwmdt-worker-0
          spec:
            metadata: {}
            providerSpec:
              value:
                apiVersion: baremetal.cluster.k8s.io/v1alpha1
                hostSelector: {}
                image:
                  checksum: http:/172.22.0.3:6181/images/rhcos-<version>.<architecture>.qcow2.<md5sum> 
    1
    
                  url: http://172.22.0.3:6181/images/rhcos-<version>.<architecture>.qcow2 
    2
    
                kind: BareMetalMachineProviderSpec
                metadata:
                  creationTimestamp: null
                userData:
                  name: worker-user-data
      status:
        availableReplicas: 2
        fullyLabeledReplicas: 2
        observedGeneration: 11
        readyReplicas: 2
        replicas: 2
    Copy to Clipboard Toggle word wrap
    1
    2

7.4.

  1. $ oc get bmh -n openshift-machine-api
    Copy to Clipboard Toggle word wrap

    NAME                 STATUS   PROVISIONING STATUS      CONSUMER
    openshift-master-0   OK       externally provisioned   openshift-zpwpq-master-0
    openshift-master-1   OK       externally provisioned   openshift-zpwpq-master-1
    openshift-master-2   OK       externally provisioned   openshift-zpwpq-master-2
    openshift-worker-0   OK       provisioned              openshift-zpwpq-worker-0-lv84n
    openshift-worker-1   OK       provisioned              openshift-zpwpq-worker-0-zd8lm
    openshift-worker-2   error    registering
    Copy to Clipboard Toggle word wrap

  2. $ oc get -n openshift-machine-api bmh <bare_metal_host_name> -o yaml
    Copy to Clipboard Toggle word wrap

    ...
    status:
      errorCount: 12
      errorMessage: MAC address b4:96:91:1d:7c:20 conflicts with existing node openshift-worker-1
      errorType: registration error
    ...
    Copy to Clipboard Toggle word wrap

7.5.

  1. $  oc -n openshift-machine-api get bmh openshift-worker-<num>
    Copy to Clipboard Toggle word wrap

    NAME              STATE     ONLINE ERROR  AGE
    openshift-worker  available true          34h
    Copy to Clipboard Toggle word wrap
  2. $ oc get nodes
    Copy to Clipboard Toggle word wrap
    NAME                                                STATUS   ROLES           AGE     VERSION
    openshift-master-1.openshift.example.com            Ready    master          30h     v1.30.3
    openshift-master-2.openshift.example.com            Ready    master          30h     v1.30.3
    openshift-master-3.openshift.example.com            Ready    master          30h     v1.30.3
    openshift-worker-0.openshift.example.com            Ready    worker          30h     v1.30.3
    openshift-worker-1.openshift.example.com            Ready    worker          30h     v1.30.3
    Copy to Clipboard Toggle word wrap
  3. $ oc get machinesets -n openshift-machine-api
    Copy to Clipboard Toggle word wrap
    NAME                                DESIRED   CURRENT   READY   AVAILABLE   AGE
    ...
    openshift-worker-0.example.com      1         1         1       1           55m
    openshift-worker-1.example.com      1         1         1       1           55m
    Copy to Clipboard Toggle word wrap
  4. $ oc scale --replicas=<num> machineset <machineset> -n openshift-machine-api
    Copy to Clipboard Toggle word wrap

  5. $ oc -n openshift-machine-api get bmh openshift-worker-<num>
    Copy to Clipboard Toggle word wrap

    NAME                    STATE             CONSUMER                          ONLINE   ERROR
    openshift-worker-<num>  provisioning      openshift-worker-<num>-65tjz      true
    Copy to Clipboard Toggle word wrap

    NAME                    STATE             CONSUMER                          ONLINE   ERROR
    openshift-worker-<num>  provisioned       openshift-worker-<num>-65tjz      true
    Copy to Clipboard Toggle word wrap
  6. $ oc get nodes
    Copy to Clipboard Toggle word wrap
    NAME                                          STATUS   ROLES   AGE     VERSION
    openshift-master-1.openshift.example.com      Ready    master  30h     v1.30.3
    openshift-master-2.openshift.example.com      Ready    master  30h     v1.30.3
    openshift-master-3.openshift.example.com      Ready    master  30h     v1.30.3
    openshift-worker-0.openshift.example.com      Ready    worker  30h     v1.30.3
    openshift-worker-1.openshift.example.com      Ready    worker  30h     v1.30.3
    openshift-worker-<num>.openshift.example.com  Ready    worker  3m27s   v1.30.3
    Copy to Clipboard Toggle word wrap

    $ ssh openshift-worker-<num>
    Copy to Clipboard Toggle word wrap
    [kni@openshift-worker-<num>]$ journalctl -fu kubelet
    Copy to Clipboard Toggle word wrap

Legal Notice

Copyright © 2025 Red Hat

OpenShift documentation is licensed under the Apache License 2.0 (https://www.apache.org/licenses/LICENSE-2.0).

Modified versions must remove all Red Hat trademarks.

Portions adapted from https://github.com/kubernetes-incubator/service-catalog/ with modifications by Red Hat.

Red Hat, Red Hat Enterprise Linux, the Red Hat logo, the Shadowman logo, JBoss, OpenShift, Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries.

Linux® is the registered trademark of Linus Torvalds in the United States and other countries.

Java® is a registered trademark of Oracle and/or its affiliates.

XFS® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries.

MySQL® is a registered trademark of MySQL AB in the United States, the European Union and other countries.

Node.js® is an official trademark of Joyent. Red Hat Software Collections is not formally related to or endorsed by the official Joyent Node.js open source or commercial project.

The OpenStack® Word Mark and OpenStack logo are either registered trademarks/service marks or trademarks/service marks of the OpenStack Foundation, in the United States and other countries and are used with the OpenStack Foundation’s permission. We are not affiliated with, endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.

All other trademarks are the property of their respective owners.

맨 위로 이동
Red Hat logoGithubredditYoutubeTwitter

자세한 정보

평가판, 구매 및 판매

커뮤니티

Red Hat 문서 정보

Red Hat을 사용하는 고객은 신뢰할 수 있는 콘텐츠가 포함된 제품과 서비스를 통해 혁신하고 목표를 달성할 수 있습니다. 최신 업데이트를 확인하세요.

보다 포괄적 수용을 위한 오픈 소스 용어 교체

Red Hat은 코드, 문서, 웹 속성에서 문제가 있는 언어를 교체하기 위해 최선을 다하고 있습니다. 자세한 내용은 다음을 참조하세요.Red Hat 블로그.

Red Hat 소개

Red Hat은 기업이 핵심 데이터 센터에서 네트워크 에지에 이르기까지 플랫폼과 환경 전반에서 더 쉽게 작업할 수 있도록 강화된 솔루션을 제공합니다.

Theme

© 2025 Red Hat