Installing OpenShift Container Platform with the Assisted Installer


Assisted Installer for OpenShift Container Platform 2026

User Guide

Red Hat Customer Content Services

Abstract

Information about the Assisted Installer and its usage

Preface

Providing feedback on the Assisted Installer documentation

You can give feedback or report an error in our documentation by creating a Jira issue. You must have a Red Hat Jira account.

Procedure

  1. Log in to Jira.
  2. Click Create.
  3. Set the following fields as specified:

    • Space: Hybrid Cloud Infrastructure Documentation
    • Work type: Bug
    • Component: Assisted Installer SaaS
  4. Complete the Summary, Description, Reporter, and Priority fields.
  5. Click Create to submit the form.

    The form creates an issue in the Red Hat Hybrid Cloud Infrastructure (HCIDOCS) Jira project.

Chapter 1. About the Assisted Installer

You can install OpenShift Container Platform on on-premise hardware or on-premise virtual machines by using the Assisted Installer.

1.1. Using the Assisted Installer

The Assisted Installer for Red Hat OpenShift Container Platform is a user-friendly installation solution offered on the Red Hat Hybrid Cloud Console. The Assisted Installer supports various deployment platforms with a focus on bare metal, Nutanix, vSphere, and Oracle Cloud Infrastructure. The Assisted Installer also supports various CPU architectures, including x86_64, s390x (IBM Z®), arm64, and ppc64le (IBM Power®).

You can install OpenShift Container Platform on premises in a connected environment, with an optional HTTP/S proxy, for the following platforms:

  • Highly available OpenShift Container Platform or single-node OpenShift cluster
  • OpenShift Container Platform on bare metal or vSphere with full platform integration, or other virtualization platforms without integration
  • Optionally, OpenShift Virtualization and Red Hat OpenShift Data Foundation

For access, see the following links:

1.2. Features

The Assisted Installer provides installation functionality as a service. This software-as-a-service (SaaS) approach has the following features:

Web interface
No bootstrap node
  • You do not need a bootstrap node because the bootstrapping process runs on a node within the cluster.
Streamlined installation workflow
  • You do not need in-depth knowledge of OpenShift Container Platform to deploy a cluster. The Assisted Installer provides reasonable default configurations.
  • You do not need to run the OpenShift Container Platform installer locally.
  • You have access to the latest Assisted Installer for the latest tested z-stream releases.
Advanced networking options
  • The Assisted Installer supports IPv4 and dual stack networking with OVN only, NMState-based static IP addressing, and an HTTP/S proxy.
  • OVN is the default Container Network Interface (CNI) for OpenShift Container Platform 4.12 and later.
  • SDN is supported up to OpenShift Container Platform 4.14. SDN supports IPv4 only.
Preinstallation validation
  • Before installing, the Assisted Installer checks the following configurations:

    • Network connectivity
    • Network bandwidth
    • Connectivity to the registry
    • Upstream DNS resolution of the domain name
    • Time synchronization between cluster nodes
    • Cluster node hardware
    • Installation configuration parameters
REST API
  • You can automate the installation process by using the Assisted Installer REST API.

1.3. Host topology

The OpenShift Container Platform architecture allows you to select a standard Kubernetes role for each of the discovered hosts. These roles define the function of the host within the cluster.

1.3.1. Supported host roles

During the installation process, you can select a role for a host or configure the Assisted Installer to assign it for you.

The host must meet the minimum requirements for the role you selected. You can find the hardware requirements by referring to the Prerequisites section of this document or using the preflight requirement API.

If you do not select a role, the system selects one for you. You can change the role at any time before installation starts.

Each host can have any of the following roles:

  • Control plane
  • Compute
  • Arbiter
  • Auto-assign
1.3.1.1. Control plane (master) role

The control plane nodes run the services that are required to control the cluster, including the API server. The control plane schedules workloads, maintains cluster state, and ensures stability.

1.3.1.2. Compute (worker) role

The compute nodes are responsible for executing workloads for cluster users. Compute nodes advertise their capacity, so that the control plane scheduler can identify suitable compute nodes for running pods and containers.

1.3.1.3. Arbiter role

Arbiter nodes are a more cost-effective alternative to control plane nodes. They function similarly but run only the essential components required to maintain the etcd quorum and prevent a split-brain condition. Because they do not host the full control plane or any workloads, arbiter nodes can use less powerful hardware.

The Assisted Installer provides arbiter nodes for Two-Node OpenShift with Arbiter (TNA) clusters. Support for Two-Node OpenShift with Arbiter clusters begins with OpenShift Container Platform version 4.19 and later. For more details, see "Two-Node OpenShift with Arbiter (TNA) resource requirements" in Additional resources.

To install a Two-Node OpenShift with Arbiter cluster, assign the arbiter or auto-assign role to at least one of the nodes, and set the control plane node count for the cluster to 2.

Important

Two-Node OpenShift with Arbiter (TNA) is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.

1.3.1.4. Auto-assign role

The Assisted Installer sets each host to an auto-assign role by default. Auto-assign allows the Assisted Installer to automatically determine whether the host should function as a control plane, arbiter, or compute (worker) role, based on detected hardware and network latency.

To determine the most suitable role, the Assisted Installer evaluates each host’s memory, CPU, disk space, and network performance. It assigns an internal suggested_role value to each host, which drives the auto-assignment process when auto-assigned is enabled. Pre-installation validations ensure the resulting role allocation is valid.

The logic for auto-assigning roles is as follows:

  1. Sort hosts by capability

    The Assisted Installer sorts the hosts by their hardware capabilities, from weakest to strongest. All hosts must meet the minimum requirements.

  2. Assign control plane roles

    The Assisted Installer assigns the control plane role to the weakest hosts first, until it reaches the number of control plane nodes specified by the control_plane_count field. A host is assigned a control plane role only if passes the necessary control plane role validations. For details on specifying the control plane count, see Additional resources.

  3. Assign arbiter role

    The Assisted Installer assigns the arbiter role to a host only when the following conditions are met:

    • The control plane count is 2.
    • The host meets the minimum hardware requirements for the cluster.
    • One of the following applies:

      • The cluster already contains two control plane nodes, either manually assigned or auto-assigned; or
      • The host does not meet the minimum hardware requirements for a control plane node.
  4. Handle GPU hosts

    By default, the Assisted Installer designates non-GPU hosts as control plane nodes and GPU hosts as worker nodes.

    The Assisted Installer designates a GPU host as a control plane node in either of the following scenarios:

    • When no other available hosts meet the minimum requirements.
    • When the number of required control plane nodes exceeds the number of available non-GPU hosts.
  5. Assign remaining hosts

    The Assisted Installer designates all remaining hosts as worker (compute) nodes. This approach ensures that the Assisted Installer prioritizes the most capable hosts for worker roles, while still maintaining the necessary number of valid control plane and arbiter nodes.

To assign a role to a host using the web console or the API, or to troubleshoot pre-installation validation errors for hosts with an auto-assign role, see Additional resources.

1.3.2. Control plane count

The control plane count is the number of control plane (master) nodes in the cluster. Using a higher number of control plane nodes boosts fault tolerance and availability, minimizing downtime during failures. The number of control plane nodes that the Assisted Installer supports varies according to OpenShift Container Platform version:

  • All versions of OpenShift Container Platform support one or three control plane nodes, where one control plane node is a Single-node OpenShift cluster.
  • From OpenShift Container Platform version 4.18 and later, the Assisted Installer also supports four or five control plane nodes on a bare metal or user-managed networking platform with an x86_64 architecture. An implementation can support any number of compute nodes.
  • From OpenShift Container Platform version 4.19 and later, the Assisted Installer also supports two control plane nodes for a Two-Node OpenShift with Arbiter (TNA) cluster topology. A cluster with only two control plane nodes must have at least one host with an arbiter role. For details, see "Supported host roles" in Additional resources.
Important

Two-Node OpenShift with Arbiter (TNA) is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.

To specify the required number of control plane nodes for your cluster in either the web console or API, see Additional resources.

1.3.3. Control plane workload scheduling

For smaller clusters, scheduling workloads to run on control plane nodes improves efficiency and maximizes resource usage. You can enable this option during installation setup or as a postinstallation step.

Use the following guidelines to determine when to use this feature:

  • Single-node OpenShift clusters, Two-Node OpenShift with Arbiter clusters, or clusters with up to one worker (compute) node: The system schedules workloads on control plane nodes by default. This setting cannot be changed.
  • Clusters of between two to seven worker nodes: This configuration supports the manual scheduling of workloads on both control plane (master) and compute (worker) nodes.
  • Clusters with more than seven worker nodes: Scheduling workloads on control plane nodes is not recommended.

Schedulable control plane nodes have the role Control plane, Worker.

Important

When you configure control plane nodes to be schedulable for workloads, an additional subscription is required for each control plane node that function as a compute (worker) node.

For guidance on configuring control plane nodes as schedulable in the Assisted Installer during installation, and for post-installation steps in OpenShift Container Platform, see Additional resources.

1.4. API support policy

Assisted Installer APIs are supported for a minimum of three months from the announcement of deprecation.

1.4.1. API deprecation notice

The following table presents the deprecated and modified APIs in the Assisted Installer.

  • assisted_service API

    Expand
    Affected modelsDescription of change
    • cluster
    • cluster-create-params

    The high_availability_mode field is deprecated starting from April 2025, and is planned to be removed in three months. Red Hat will provide bug fixes and support for this feature during the current release lifecycle, but this feature will no longer receive enhancements and will be removed.

    The alternative is to use control_plane_count instead. This change enables support for clusters with 4 or 5 control plane nodes, in addition to the previously supported configurations with 1 or 3 control plane nodes. The Assisted Installer supports 4 or 5 control plane nodes starting from OpenShift Container Platform version 4.18 and later.

Chapter 2. Prerequisites

The Assisted Installer validates the following prerequisites to ensure successful installation. If you use a firewall, you must configure it so that the Assisted Installer can access the resources it requires to function.

2.1. Supported CPU architectures

The Assisted Installer is supported on the following CPU architectures:

  • x86_64
  • arm64
  • ppc64le (IBM Power®)
  • s390x (IBM Z®)

2.2. Supported drive types

Install Red Hat OpenShift Container Platform with the Assisted Installer using the appropriate drive types.

Supported drive types

This table shows the installation drive types supported for the different OpenShift Container Platform versions and CPU architectures.

Expand
Drive typesRHOCP VersionSupported CPU ArchitecturesComments

HDD

All

All

A hard disk drive.

SSD

All

All

An SSD or NVMe drive.

Multipath

All

All

A Linux multipath device that can aggregate paths for various protocols. Using multipath enhances availablity and performance. Currently, the Assisted Installer supports multipathing for Fibre Channel and iSCSI protocals.

FC (Fibre Channel)

All

s390x, x86_64

Indicates a single path Fibre Channel (FC) drive. For a multipath Fibre Channel configuration, see 'Multipath' in this table.

iSCSI

4.15 and later

x86_64

  • You can install a cluster on a single or multipath iSCSI boot device.
  • The Assisted Installer supports iSCSI boot volumes through iPXE boot.
  • A minimal ISO image is mandatory on iSCSI boot volumes. Using a full ISO image will result in an error.
  • iSCSI boot requires two machine network interfaces; one for the iSCSI traffic and the other for the OpenShift Container Platform cluster installation.
  • A static IP address is not supported when using iSCSI boot volumes.

RAID

4.14 and later

All

A software RAID drive. The RAID should be configured via BIOS/UEFI. If this option is unavailable, you can configure OpenShift Container Platform to mirror the drives. For details, see Encrypting and mirroring disks during installation.

ECK

All

s390x

IBM drive.

ECKD (ESE)

All

s390x

IBM drive.

FBA

All

s390x

IBM drive.

Unsupported drive types

This table shows the installation drive types that are not supported.

Expand
Drive typesComments

Unknown

The system could not detect the drive type.

FDD

A floppy disk drive.

ODD

An optical disk drive (e.g., CD-ROM).

Virtual

A loopback device.

LVM

A Linux Logical Volume Management drive.

2.3. Resource requirements

This section describes the resource requirements for different clusters and installation options.

The multicluster engine for Kubernetes requires additional resources.

If you deploy the multicluster engine with storage, such as OpenShift Data Foundation or LVM Storage, you must also assign additional resources to each node.

2.3.1. Multi-node cluster resource requirements

The resource requirements of a multi-node (high-availability) cluster depend on the installation options.

Description
A standard OpenShift Container Platform cluster configuration consists of three to five control plane nodes and two or more worker nodes. This configuration ensures full high availability for control plane services.
Multi-node cluster basic installation
  • Control plane nodes:

    • 4 CPU cores
    • 16 GB RAM
    • 100 GB storage

      Note

      The disks must be reasonably fast, with an etcd wal_fsync_duration_seconds p99 duration that is less than 10 ms. For more information, see the Red Hat Knowledgebase solution How to Use 'fio' to Check Etcd Disk Performance in OCP.

  • Compute nodes:

    • 2 CPU cores
    • 8 GB RAM
    • 100 GB storage
Multi-node cluster + multicluster engine
  • Additional 4 CPU cores
  • Additional 16 GB RAM

    Note

    If you deploy multicluster engine without OpenShift Data Foundation, no storage is configured. You configure the storage after the installation.

Multi-node cluster + multicluster engine + OpenShift Data Foundation
  • Additional 75 GB storage

The resource requirements of a Two-Node OpenShift with Arbiter (TNA) cluster depend on the installation options.

Description

A Two-Node OpenShift with Arbiter (TNA) cluster is a compact, cost-effective OpenShift Container Platform topology. It consists of two control plane nodes and a lightweight arbiter node. The arbiter node stores the full etcd data, maintaining an etcd quorum and preventing split brain. It does not run the additional control plane components kube-apiserver and kube-controller-manager, nor does it run workloads. For details, see Overview of etcd.

To install a Two-Node OpenShift with Arbiter cluster, assign an arbiter role to at least one of the nodes and set the control plane node count for the cluster to 2. Although OpenShift Container Platform does not currently impose a limit on the number of arbiter nodes, the typical deployment includes only one to minimize the use of hardware resources.

Following installation, you can add additional arbiter nodes to a Two-Node OpenShift with Arbiter cluster but not to a standard multi-node cluster. It is also not possible to convert between a Two-Node OpenShift with Arbiter and standard topology.

Support for a Two-Node OpenShift with Arbiter cluster begins with OpenShift Container Platform version 4.19 and later. This configuration is available only for bare-metal installations.

Important

Two-Node OpenShift with Arbiter (TNA) is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.

Two-Node OpenShift with Arbiter basic installation
  • Control plane nodes:

    • 4 CPU cores
    • 16 GB RAM
    • 100 GB storage

      Note

      The disks must be reasonably fast, with an etcd wal_fsync_duration_seconds p99 duration that is less than 10 ms. For more information, see the Red Hat Knowledgebase solution How to Use 'fio' to Check Etcd Disk Performance in OCP.

  • Arbiter node:

    • 2 CPU cores
    • 8 GB RAM
    • 50 GB storage
Two-Node OpenShift with Arbiter + multicluster engine
  • Additional 4 CPU cores
  • Additional 16 GB RAM

    Note

    If you deploy multicluster engine without OpenShift Data Foundation, no storage is configured. You configure the storage after the installation.

Two-Node OpenShift with Arbiter + multicluster engine + OpenShift Data Foundation
  • Additional 75 GB storage

The resource requirements for single-node OpenShift depend on the installation options.

Description
A single-node OpenShift cluster is an OpenShift Container Platform deployment that runs entirely on a single node. Single-node OpenShift includes the control plane and worker functionality on one physical or virtual machine.
Single-node OpenShift basic installation
  • 8 CPU cores
  • 16 GB RAM
  • 100 GB storage
Single-node OpenShift + multicluster engine
  • Additional 8 CPU cores
  • Additional 32 GB RAM

    Note

    If you deploy multicluster engine without OpenShift Data Foundation, LVM Storage is enabled.

Single-node OpenShift + multicluster engine + OpenShift Data Foundation
  • Additional 95 GB storage

2.4. Networking requirements

For hosts of type VMware, set clusterSet disk.EnableUUID to TRUE, even when the platform is not vSphere.

2.4.1. General networking requirements

The network must meet the following requirements:

  • You have configured either dynamic (DHCP) or static IP addressing.
  • You have selected the correct route configuration for your IP addressing method:

    • For dynamic IP addressing, ensure that you have configured your network routes dynamically via DHCP.
    • For static IP addressing, ensure that you have configured the network routes manually via the static networking configurations.
    Important

    You cannot combine dynamic IP addresses with static route configurations. When the Assisted Installer receives a dynamic IP address (with a /128 prefix), it specifically looks for network routes that were also configured dynamically, such as those advertised via Router Advertisement (RA). If a network route is configured manually (with a /64 prefix, for example), the Assisted Installer ignores it.

  • You have opened port 6443 to allow the API URL access to the cluster using the oc CLI tool when outside the firewall.
  • You have opened port 22624 in all firewalls. The Machine Config Operator (MCO) and new worker nodes use port 22624 to get the ignition data from the cluster API.
  • You have opened port 443 to allow console access outside the firewall. Port 443 is also used for all ingress traffic.
  • You have configured DNS to connect to the cluster API or ingress endpoints from outside the cluster.
  • Optional: You have created a DNS Pointer record (PTR) for each node in the cluster if using static IP addressing.
Note

You must create a DNS PTR record to boot with a preset hostname if the hostname will not come from another source (/etc/hosts or DHCP). Otherwise, the Assisted Installer’s automatic node renaming feature will rename the nodes to their network interface MAC address.

2.4.2. External DNS

Installing a multi-node cluster with user-managed networking requires external DNS records. The Assisted Installer does not require external DNS records to complete the installation of multi-node clusters with cluster-managed networking or single-node OpenShift clusters. Configure the external DNS records after installation to connect to the cluster from an external source.

External DNS requires the following records:

  • A/AAAA record for api.<cluster_name>.<base_domain>.
  • A/AAAA record for api-int.<cluster_name>.<base_domain>.
  • A/AAAA record with a wildcard for *.apps.<cluster_name>.<base_domain>.
  • A/AAAA record for each node in the cluster.
Important
  • Do not create a wildcard, such as *.<cluster_name>.<base_domain>, or the installation will not proceed.
  • A/AAAA record settings at top-level domain registrars can take significant time to update. Ensure the newly created DNS Records are resolving before installation to prevent installation delays.
  • For DNS record examples, see Example DNS configuration.

The OpenShift Container Platform cluster’s network must also meet the following requirements:

  • Connectivity between all cluster nodes
  • Connectivity for each node to the internet
  • Access to an NTP server for time synchronization between the cluster nodes

The following DNS configuration provides A and PTR record configuration examples that meet the DNS requirements for deploying OpenShift Container Platform using the Assisted Installer. The examples are not meant to recommend one DNS solution over another.

In the examples, the cluster name is ocp4 and the base domain is example.com.

2.4.2.1. Example DNS A record configuration

The following example DNS zone database is a BIND zone file that shows sample A records for name resolution in a cluster installed using the Assisted Installer:

$TTL 1W
@	IN	SOA	ns1.example.com.	root (
			2019070700	; serial
			3H		; refresh (3 hours)
			30M		; retry (30 minutes)
			2W		; expiry (2 weeks)
			1W )		; minimum (1 week)
	IN	NS	ns1.example.com.
	IN	MX 10	smtp.example.com.
;
;
ns1.example.com.		IN	A	192.168.1.1
smtp.example.com.		IN	A	192.168.1.5
;
helper.example.com.		IN	A	192.168.1.5
;
api.ocp4.example.com.		IN	A	192.168.1.5
api-int.ocp4.example.com.	IN	A	192.168.1.5
;
*.apps.ocp4.example.com.	IN	A	192.168.1.5
;
control-plane0.ocp4.example.com.	IN	A	192.168.1.97
control-plane1.ocp4.example.com.	IN	A	192.168.1.98
control-plane2.ocp4.example.com.	IN	A	192.168.1.99
;
worker0.ocp4.example.com.	IN	A	192.168.1.11
worker1.ocp4.example.com.	IN	A	192.168.1.7
;
;EOF

where:

  • api.ocp4.example.com. provides name resolution for the Kubernetes API. The record refers to the IP address of the API load balancer.
  • api-int.ocp4.example.com. provides name resolution for the Kubernetes API. The record refers to the IP address of the API load balancer and is used for internal cluster communications.
  • *.apps.ocp4.example.com. provides name resolution for the wildcard routes. The record refers to the IP address of the application ingress load balancer. The application ingress load balancer targets the machines that run the Ingress Controller pods. The Ingress Controller pods run on the worker machines by default.

    Note

    In the example, the same load balancer is used for the Kubernetes API and application ingress traffic. In production scenarios, you can deploy the API and application ingress load balancers separately so that you can scale the load balancer infrastructure for each in isolation.

  • control-plane0.ocp4.example.com. and adjacent records provide name resolution for the control plane machines.
  • worker0.ocp4.example.com. and adjacent record provide name resolution for the worker machines.
2.4.2.2. Example DNS PTR record configuration

The following example DNS zone database for reverse records is a BIND zone file that shows sample PTR records for reverse name resolution in a cluster installed using the Assisted Installer:

$TTL 1W
@	IN	SOA	ns1.example.com.	root (
			2019070700	; serial
			3H		; refresh (3 hours)
			30M		; retry (30 minutes)
			2W		; expiry (2 weeks)
			1W )		; minimum (1 week)
	IN	NS	ns1.example.com.
;
5.1.168.192.in-addr.arpa.	IN	PTR	api.ocp4.example.com.
5.1.168.192.in-addr.arpa.	IN	PTR	api-int.ocp4.example.com.
;
97.1.168.192.in-addr.arpa.	IN	PTR	control-plane0.ocp4.example.com.
98.1.168.192.in-addr.arpa.	IN	PTR	control-plane1.ocp4.example.com.
99.1.168.192.in-addr.arpa.	IN	PTR	control-plane2.ocp4.example.com.
;
11.1.168.192.in-addr.arpa.	IN	PTR	worker0.ocp4.example.com.
7.1.168.192.in-addr.arpa.	IN	PTR	worker1.ocp4.example.com.
;
;EOF

where:

  • api.ocp4.example.com provides reverse DNS resolution for the Kubernetes API. The PTR record refers to the record name of the API load balancer.
  • api-int.ocp4.example.com provides reverse DNS resolution for the Kubernetes API. The PTR record refers to the record name of the API load balancer and is used for internal cluster communications.
  • control-plane0.ocp4.example.com and adjacent records provide reverse DNS resolution for the control plane machines.
  • worker0.ocp4.example.com and adjacent record provide reverse DNS resolution for the worker machines.
Note

A PTR record is not required for the OpenShift Container Platform application wildcard.

2.4.3. Networking requirements for IBM Z

In IBM Z® environments, advanced networking technologies like Open Systems Adapter (OSA), HiperSockets, and Remote Direct Memory Access (RDMA) over Converged Ethernet (RoCE) require specific configurations that deviate from the standard settings used in Assisted Installer deployments. These overrides are necessary to accommodate their unique requirements and ensure a successful and efficient deployment on IBM Z®.

The following table lists the network devices that are supported for the network configuration override functionality:

Expand
Network devicez/VMKVMLPAR ClassicLPAR Dynamic Partition Manager (DPM)

OSA virtual switch

Supported

Not applicable

Not applicable

Not applicable

Direct attached OSA

Supported

Only through a Linux bridge

Supported

Supported

RDMA over Converged Ethernet (RoCE)

Supported

Only through a Linux bridge

Supported

Supported

HiperSockets

Supported

Only through a Linux bridge

Supported

Supported

Linux bridge

Not applicable

Supported

Not applicable

Not applicable

2.4.3.1. Configuring network overrides in IBM Z

You can specify a static IP address on IBM Z® machines that uses Logical Partition (LPAR) and z/VM. This is specially useful when the network devices do not have a static MAC address assigned to them.

Procedure

  1. If you have an existing .parm file, edit it to include the following entry:

    ai.ip_cfg_override=1

    This parameter allows the file to add the network settings to the CoreOS installer.

    The following is an example of the .parm file:

    rd.neednet=1 cio_ignore=all,!condev
    console=ttysclp0
    coreos.live.rootfs_url=<coreos_url>
    ip=<ip>::<gateway>:<netmask>:<hostname>::none nameserver=<dns>
    rd.znet=qeth,<network_adaptor_range>,layer2=1
    rd.<disk_type>=<adapter>
    rd.zfcp=<adapter>,<wwpn>,<lun> random.trust_cpu=on
    zfcp.allow_lun_scan=0
    ai.ip_cfg_override=1
    ignition.firstboot ignition.platform.id=metal
    random.trust_cpu=on

    Replace the parameters as follows:

    • For the coreos.live.rootfs_url artifact, specify the matching rootfs artifact for the kernel and initramfs that you are booting. Only HTTP and HTTPS protocols are supported.
    • Regarding rd.<disk_type>, for installations on direct access storage devices (DASD) type disks, use rd. to specify the DASD where Red Hat Enterprise Linux (RHEL) is to be installed. For installations on Fibre Channel Protocol (FCP) disks, use rd.zfcp=<adapter>,<wwpn>,<lun> to specify the FCP disk where RHEL is to be installed.
    • For rd.zfcp, specify values for adapter, wwpn, and lun as in the following example: rd.zfcp=0.0.8002,0x500507630400d1e3,0x4000404600000000.
    • Regarding ai.ip_cfg_override=1, specify this parameter when using an OSA network adapter or HiperSockets.

      Note

      The override parameter overrides the host’s network configuration settings.

2.5. Preflight validations

The Assisted Installer ensures the cluster meets the prerequisites before installation, because it eliminates complex postinstallation troubleshooting, thereby saving significant amounts of time and effort. Before installing software on the nodes, the Assisted Installer conducts the following validations:

  • Ensures network connectivity
  • Ensures sufficient network bandwidth
  • Ensures connectivity to the registry
  • Ensures that any upstream DNS can resolve the required domain name
  • Ensures time synchronization between cluster nodes
  • Verifies that the cluster nodes meet the minimum hardware requirements
  • Validates the installation configuration parameters

If the Assisted Installer does not successfully validate the foregoing requirements, installation will not proceed.

You can customize the OpenShift Container Platform deployment by selecting one or more Operators or Operator bundles during the installation.

3.1. Customizing with Operators

The Assisted Installer Operators are used to package, deploy, and manage services and applications in OpenShift Container Platform. Through the Assisted Installer, you can install the following Operators, categorized according to their functionality.

Important

Before starting the installation, familiarize yourself with the Assisted Installer Operators, including their prerequisites and limitations.

The additional requirements apply to each Operator individually. If you select more than one Operator, or if the Assisted Installer automatically selects an Operator due to dependencies, the total required resources is the sum of the resource requirements for each Operator.

If you require advanced options, install the Operators after you have installed the cluster.

3.1.1. Storage Operators

The Storage category contains the following Operators:

  • Local Storage Operator
  • Logical Volume Manager Storage Operator
  • OpenShift Data Foundation Operator
  • OpenShift API for Data Protection (OADP) Operator
3.1.1.1. Installing the Local Storage Operator

The Local Storage Operator (LSO) enables the provisioning of persistent storage through local volumes. Local persistent volumes provide access to local storage devices, such as drives or partitions, by using the standard persistent volume claim interface.

You can perform the following actions using Local Storage Operator (LSO):

  • Assign the storage devices to the storage classes without modifying the device configuration.
  • Statically provision PVs and storage classes by configuring the LocalVolume custom resource (CR).
  • Create workloads and PVCs while being aware of the underlying storage topology.

Selecting the OpenShift Virtualization Operator, either independently or as part of the Virtualization bundle, automatically activates Local Storage Operator (LSO) in the background.

Note

While installing the Local Storage Operator (LSO) through the OCP OperatorHub, you can manually enable cluster monitoring for the Operator. Because the Assisted Installer does not include this setting, it automatically enables cluster monitoring for the Operator during installation, without the option of disabling it.

Prerequisites

You can use LVM Storage to dynamically provision block storage on a limited resources cluster.

Note

While installing the Logical Volume Manager Storage Operator through the OCP OperatorHub, you can manually enable cluster monitoring for the Operator. Because the Assisted Installer does not include this setting, it automatically enables cluster monitoring for the Operator during installation, without the option of disabling it.

Prerequisites

  • Requires at least 1 non-boot drive per host.
  • Requires 100 MiB of additional RAM.
  • Requires 1 additional CPU core for each non-boot drive.

You can use OpenShift Data Foundation for file, block, and object storage. This storage option is recommended for all OpenShift Container Platform clusters. OpenShift Data Foundation requires a separate subscription.

Note

While installing the Red Hat OpenShift Data Foundation Operator through the OCP OperatorHub, you can manually enable cluster monitoring for the Operator. Because the Assisted Installer does not include this setting, it automatically enables cluster monitoring for the Operator during installation, without the option of disabling it.

Prerequisites

  • There are at least 3 compute (workers) nodes, each with 19 additional GiB of memory and 8 additional CPU cores.
  • There are at least 2 drives per compute node. For each drive, there is an additional 5 GB of RAM.
  • You comply to the additional requirements specified here: Planning your deployment.
Note

You cannot install the OpenShift Data Foundation Operator on Oracle third-party platforms such as Oracle® Cloud Infrastructure or Oracle® Compute Cloud@Customer.

The OpenShift API for Data Protection (OADP) product safeguards customer applications on OpenShift Container Platform. It offers comprehensive disaster recovery protection, covering OpenShift Container Platform applications, application-related cluster resources, persistent volumes, and internal images. OADP is also capable of backing up both containerized applications and virtual machines (VMs). OADP does not serve as a disaster recovery solution for etcd or OpenShift Operators.

Using the Assisted Installer, you can install the OADP Operator either separately through the API or as part of the Virtualization Operator bundle in the web console. For more information about the use of this Operator in OpenShift Container Platform, see "Additional resources".

Prerequisites

3.1.2. Virtualization Operators

The Virtualization category contains the following Operators:

  • OpenShift Virtualization Operator
  • Migration Toolkit for Virtualization Operator
  • OpenShift sandboxed containers Operator

You can deploy OpenShift Virtualization to perform the following tasks:

  • Create and manage Linux and Windows virtual machines (VMs).
  • Run pod and VM workloads alongside each other in a cluster.
  • Connect to VMs through a variety of consoles and CLI tools.
  • Import and clone existing VMs.
  • Manage network interface controllers and storage drives attached to VMs.
  • Live migrate VMs between nodes.

The OpenShift Virtualization Operator requires backend storage and might automatically activate a storage Operator in the background, according to the following criteria:

  • None - If the CPU architecture is ARM64, no storage Operator is activated.
  • LVM Storage - For single-node OpenShift clusters on any other CPU architecture deploying OpenShift Container Platform 4.12 or higher.
  • Local Storage Operator (LSO) - For all other deployments.

Prerequisites

  • Requires enabled CPU virtualization support in the firmware on all nodes.
  • Requires an additional 360 MiB of memory and 2 CPU cores for each compute (worker) node.
  • Requires an additional 150 MiB of memory and 4 CPU cores for each control plane node.
  • Requires Red Hat OpenShift Data Foundation (recommended for creating additional on-premise clusters), Logical Volume Manager Storage, or another persistent storage service.

    Important

    Deploying OpenShift Virtualization without Red Hat OpenShift Data Foundation results in the following scenarios:

    • Multi-node cluster: No storage is configured. You must configure storage after the OpenShift Data Foundation configuration.
    • Single-node OpenShift: Logical Volume Manager Storage (LVM Storage) is installed.

    You must review the prerequisites to ensure that your environment has sufficient additional resources for OpenShift Virtualization.

  • OpenShift Virtualization is not supported on the following platforms: Nutanix, vSphere.
  • OpenShift Virtualization is not compatible with the following CPU architectures: S390X, PPC64LE.
  • OpenShift Virtualization is supported from OpenShift Container Platform 4:14 and later.

The Migration Toolkit for Virtualization Operator allows you to migrate virtual machines at scale to a local or remote Red Hat OpenShift Virtualization cluster. You can perform the migration from any of the following source providers:

  • VMware vSphere
  • Red Hat Virtualization (RHV)
  • Red Hat OpenShift Virtualization
  • OpenStack

When you select the Migration Toolkit for Virtualization Operator, the Assisted Installer automatically activates the OpenShift Virtualization Operator. For a Single-node OpenShift installation, the Assisted Installer also activates the LVM Storage Operator.

You can install the Migration Toolkit for Virtualization Operator on OpenShift Container Platform using the Assisted Installer, either independently or as part of the OpenShift Virtualization Operator bundle.

Prerequisites

  • Requires OpenShift Container Platform version 4.14 or later.
  • Requires an x86_64 CPU architecture.
  • Requires an additional 1024 MiB of memory and 1 CPU core for each control plane node and worker node.
  • Requires the additional resources specified for the OpenShift Virtualization Operator, installed together with OpenShift Virtualization. For details, see the prerequisites in the OpenShift Virtualization Operator section.

Next steps

After completing the installation, the Migration menu appears in the navigation pane of the Red Hat OpenShift web console.

The Migration menu provides access to the Migration Toolkit for Virtualization. Use the toolkit to create and execute a migration plan with the relevant source and destination providers.

For details, see either of the following chapters in the Migration Toolkit for Virtualization Guide:

The OpenShift sandboxed containers Operator provides an additional virtual machine (VM) isolation layer for pods, which manages the installation, configuration, and updating of sandboxed containers runtime (Kata containers) on Red Hat OpenShift clusters. You can install the sandboxed containers runtime in an Red Hat OpenShift cluster by using the Assisted Installer.

Prerequisites

The required functionality for the OpenShift Container Platform is supported by two main components:

  • OpenShift Container Platform: Use OpenShift Container Platform version 4.17 or later to install OpenShift sandboxed containers on an Red Hat OpenShift cluster using the Assisted Installer. To learn more about the requirements for OpenShift sandboxed containers, see OpenShift sandboxed containers.
  • Kata runtime: This includes Red Hat Enterprise Linux CoreOS (RHCOS) and updates with every OpenShift Container Platform release. The Operator depends on the features that come with the RHCOS host and the environment that it runs in.

    Note

    You must install Red Hat Enterprise Linux CoreOS (RHCOS) on the worker nodes. RHEL nodes are not supported.

3.1.3. Artificial Intelligence (AI) Operators

The AI category contains the following Operators:

  • Red Hat® OpenShift® Artificial Intelligence (AI) Operator
  • AMD GPU Operator
  • NVIDIA GPU Operator

Red Hat® OpenShift® Artificial Intelligence (AI) is a flexible, scalable artificial intelligence (AI) and machine learning (ML) platform that enables enterprises to create and deliver AI-enabled applications at scale across hybrid cloud environments. Red Hat® OpenShift® AI enables the following functionality:

  • Data acquisition and preparation.
  • Model training and fine-tuning.
  • Model serving and model monitoring.
  • Hardware acceleration.

The OpenShift AI Operator enables you to install Red Hat® OpenShift® AI on your OpenShift Container Platform cluster. From OpenShift Container Platform version 4.17 and later, you can use the Assisted Installer to deploy the OpenShift AI Operator to your cluster during the installation.

You can install the OpenShift Artificial Intelligence (AI) Operator either separately or as part of the OpenShift AI Operator bundle.

Important

The integration of the OpenShift AI Operator into the Assisted Installer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.

Prerequisites

  • The prerequisites for installing the OpenShift AI Operator separately are as follows:

    • You are installing OpenShift Container Platform version 4.17 or later.
    • You meet the following miminum requirements for the OpenShift AI Operator:

      • Requires at least 2 compute (worker) nodes, each with 32 additional GiB of memory and 8 additional CPU cores.
      • Requires at least 1 supported GPU. Both AMD and NVIDIA GPUs are supported.
    • You meet the additional minimum requirements specified for the dependent Red Hat OpenShift Data Foundation Operator.
    • You meet the additional requirements specified here: Requirements for OpenShift AI.
  • See the additional prerequisites for the OpenShift AI Operator bundle, if you are installing the Operator as part of the bundle.
Note

You cannot install the OpenShift AI Operator on Oracle third-party platforms such as Oracle® Cloud Infrastructure or Oracle® Compute Cloud@Customer.

3.1.3.2. Installing the AMD GPU Operator

The Advanced Micro Devices (AMD) Graphics Processing Unit (GPU) Operator simplifies the deployment and management of AMD Instinct™ GPUs within a Red Hat OpenShift Container Platform cluster. The hardware acceleration capabilities of the Operator automate several key tasks, making it easier to create artificial intelligence and machine learning (AI/ML) applications. Accelerating specific areas of GPU functions can minimize CPU processing and memory usage, improving overall application speed, memory consumption, and bandwidth restrictions.

In the Assisted Installer, you can install the AMD GPU Operator separately or as part of the OpenShift AI Operator bundle. Selecting the AMD GPU Operator automatically activates the Kernel Module Management Operator.

Note

While installing the AMD GPU Operator through the OCP OperatorHub, you can manually enable cluster monitoring for the Operator. Because the Assisted Installer does not include this setting, it automatically enables cluster monitoring for the Operator during installation, without the option of disabling it.

Important

The integration of the AMD GPU Operator into the Assisted Installer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.

Prerequisites

  • Requires at least 1 supported AMD GPU.
  • See the additional prerequisites for the OpenShift AI Operator bundle if you are installing the Operator as part of the bundle.
3.1.3.3. Installing the NVIDIA GPU Operator

The NVIDIA GPU Operator uses the operator framework within Kubernetes to automate the management of all NVIDIA software components needed to provision Graphical Processing Units (GPUs).

Some of these software components are as follows:

  • NVIDIA drivers to enable Compute Unified Device Architecture (CUDA).
  • The Kubernetes device plugin for GPUs.
  • The NVIDIA Container Toolkit.
  • Automatic node labelling using GPU Feature Discovery (GFD)
  • GPU monitoring through the Data Center GPU Manager (DCGM).

In OpenShift Container Platform, the Operator provides a consistent, automated, and cloud-native way to leverage the power of NVIDIA GPUs for artificial intelligence, machine learning, high-performance computing, and other GPU-accelerated workloads.

You can install the NVIDIA GPU Operator either separately or as part of the OpenShift AI Operator bundle. Selecting the NVIDIA GPU Operator automatically activates the Node Feature Discovery Operator.

Important

The integration of the NVIDIA GPU Operator into the Assisted Installer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.

Prerequisites

  • Requires at least 1 supported NVIDIA GPU.
  • See the additional prerequisites for the OpenShift AI Operator bundle if you are installing the Operator as part of the bundle.

3.1.4. Network Operators

The Network category contains the following Operators:

  • Kubernetes NMState Operator
  • OpenShift Service Mesh Operator
  • MetalLB Operator

NMState is a declarative NetworkManager API designed for configuring network settings using YAML or JSON-based instructions. The Kubernetes NMState Operator allows you to configure network interface types, DNS, and routing on the cluster nodes using NMState.

You can install the Kubernetes NMState Operator on OpenShift Container Platform using the Assisted Installer, either separately or as part of the OpenShift Virtualization Operator bundle. Installing the Kubernetes NMState Operator with the Assisted Installer automatically creates a kubernetes-nmstate instance, which deploys the NMState State Controller as a daemon set across all of the cluster nodes. The daemons on the cluster nodes periodically report on the state of each node’s network interfaces to the API server.

Prerequisites

  • Supports OpenShift Container Platform 4.12 or later.
  • Requires an x86_64 CPU architecture.
  • Cannot be installed on the Nutanix and Oracle Cloud Infrastructure platforms.

Red Hat OpenShift Service Mesh addresses a variety of problems in a microservice architecture by creating a centralized point of control in an application. It adds a transparent layer on existing distributed applications without requiring any changes to the application code.

Microservice architectures split the work of enterprise applications into modular services, which can make scaling and maintenance easier. However, as an enterprise application built on a microservice architecture grows in size and complexity, it becomes difficult to understand and manage. Service Mesh can address those architecture problems by capturing or intercepting traffic between services and can modify, redirect, or create new requests to other services.

Service Mesh provides an easy way to create a network of deployed services that provides discovery, load balancing, service-to-service authentication, failure recovery, metrics, and monitoring. A service mesh also provides more complex operational functionality, including A/B testing, canary releases, access control, and end-to-end authentication

Red Hat OpenShift Service Mesh requires the use of the Red Hat OpenShift Service Mesh Operator which allows you to connect, secure, control, and observe the microservices that comprise your applications. You can also install other Operators to enhance your service mesh experience. Service mesh is based on the open source Istio project.

Using the Assisted Installer, you can install the OpenShift Service Mesh Operator either separately through the API or as part of the OpenShift AI Operator bundle in the web console.

Important

The integration of the OpenShift Service Mesh Operator into the Assisted Installer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.

Prerequisites

3.1.4.3. Installing the MetalLB Operator

You can install the MetalLB Operator to enable the use of LoadBalancer services in environments that do not have a built-in cloud load balancer, such as bare-metal clusters.

When you create a LoadBalancer service, MetalLB assigns an external IP address from a predefined pool. MetalLB advertises this IP address on the host network, making the service reachable from outside the cluster. When external traffic enters your OpenShift Container Platform cluster through the MetalLB LoadBalancer service, the return traffic to the client has the external IP address of the load balancer as the source IP.

Using the Assisted Installer, you can install the MetalLB Operator either separately through the API or as part of the Virtualization Operator bundle in the web console. For more information about the use of this Operator in OpenShift Container Platform, see "Additional resources".

Prerequisites

3.1.5. Remediation Operators

The Remediation category contains the following Operators:

  • Node Health Check Operator
  • Fence Agents Remediation Operator
  • Self Node Remediation Operator
3.1.5.1. Installing the Node Health Check Operator

The Node Health Check Operator monitors node conditions based on a defined set of criteria to assess their health status. When detecting an issue, the Operator delegates remediation tasks to the appropriate remediation provider to remediate the unhealthy nodes. The Assisted Installer supports the following remediation providers:

Using the Assisted Installer, you can install the Node Health Check Operator either separately through the API or as part of the Virtualization Operator bundle in the web console.

Prerequisites

You can use Fence Agents Remediation Operator to automatically recover unhealthy nodes on environments with a traditional API end-point. When a node in the OpenShift Container Platform cluster becomes unhealthy or unresponsive, the Fence Agents Remediation Operator utilizes an external set of fencing agents to isolate it from the rest of the cluster. A fencing agent then resets the unhealthy node in an attempt to resolve transient hardware or software issues. Before or during the reboot process, the Fence Agents Remediation Operator safely moves workloads (pods) running on the unhealthy node to other healthy nodes in the cluster.

Using the Assisted Installer, you can install the Fence Agents Remediation Operator either separately through the API or as part of the Virtualization Operator bundle in the web console.

Prerequisites

Next steps

  • Create the FenceAgentsRemediationTemplate custom resource to define the required fencing agents and remediation parameters. For details, see Configuring the Fence Agents Remediation Operator.
  • Configure the NodeHealthCheck custom resource by either replacing the default SelfNodeRemediation provider with FenceAgentsRemediation or by adding FenceAgentsRemediation as an additional remediation provider.

The Self Node Remediation Operator automatically reboots unhealthy nodes. This remediation strategy minimizes downtime for stateful applications and ReadWriteOnce (RWO) volumes, and restores compute capacity if transient failures occur.

You can use the Self Node Remediation Operator as a remediation provider for the Node Health Check Operator. Currently, it is only possible to install the Self Node Remediation Operator through the API.

Important

The integration of the Self Node Remediation Operator into the Assisted Installer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.

3.1.6. Security and Access Control Operators

The Security & Access Control category contains the following Operators:

  • Authorino Operator
  • Kernel Module Management Operator
3.1.6.1. Installing the Authorino Operator

The Authorino Operator provides an easy way to install Authorino, providing configurability options at the time of installation.

Authorino is a Kubernetes-native, external authorization service designed to secure APIs and applications. It intercepts requests to services and determines whether to allow or deny access based on configured authentication and authorization policies. Authorino provides a centralized and declarative way to manage access control for your Kubernetes-based applications without requiring code changes.

Using the Assisted Installer, you can install the Authorino Operator either separately through the API or as part of the OpenShift AI Operator bundle in the web console.

Important

The integration of the Authorino Operator into the Assisted Installer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.

Prerequisites

The Kernel Module Management (KMM) Operator manages, builds, signs, and deploys out-of-tree kernel modules and device plugins on OpenShift Container Platform clusters.

KMM adds a new Module CRD which describes an out-of-tree kernel module and its associated device plugin. You can use Module resources to configure how to load the module, define ModuleLoader images for kernel versions, and include instructions for building and signing modules for specific kernel versions.

KMM is designed to accommodate multiple kernel versions at once for any kernel module, allowing for seamless node upgrades and reduced application downtime.

You can install the Kernel Module Management Operator either independently or as part of the OpenShift AI Operator bundle.

Important

The integration of the Kernel Module Management Operator into the Assisted Installer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.

Prerequisites

  • If you are installing the Operator as part of the OpenShift AI Operator bundle, see the bundle prerequisites.
  • If you are installing the Operator separately, there are no additional prerequisites.

The CI/CD & Dev Productivity category contains the following Operators:

  • OpenShift Pipelines Operator
  • OpenShift Serverless Operator

Red Hat OpenShift Pipelines is a cloud-native, continuous integration and continuous delivery (CI/CD) solution based on Kubernetes resources. It uses Tekton building blocks to automate deployments across multiple platforms by abstracting away the underlying implementation details. Tekton introduces various standard custom resource definitions (CRDs) for defining CI/CD pipelines that are portable across Kubernetes distributions.

The Red Hat OpenShift Pipelines Operator handles the installation and management of OpenShift Pipelines. The Operator supports the following use cases:

  • Continuous Integration (CI) - Automating code compilation, testing, and static analysis.
  • Continuous Delivery/Deployment (CD) - Automating the deployment of applications to various environments (development, staging, production).
  • Microservices Development - Supporting decentralized teams working on microservice-based architectures.
  • Building Container Images - Efficiently building and pushing container images to registries.
  • Orchestrating Complex Workflows - Defining multi-step processes for building, testing, and deploying applications across different platforms.

Using the Assisted Installer, you can install the OpenShift Pipelines Operator either separately through the API or as part of the OpenShift AI Operator bundle in the web console.

Important

The integration of the OpenShift Pipelines Operator into the Assisted Installer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.

Prerequisites

The Red Hat OpenShift Serverless Operator enables you to install and use the following components on your OpenShift Container Platform cluster:

  • Knative Serving - Deploys and automatically scales stateless, containerized applications according to demand. It simplifies code deployment, and handles web requests and background processes.
  • Knative Eventing - Provides the building blocks for an event-driven architecture on Kubernetes. It enables loose coupling between services by allowing them to communicate asynchronously through events, rather than through direct calls.
  • Knative Broker for Apache Kafka - This is a specific implementation of a Knative Broker. It provides a robust, scalable, and high-performance mechanism for routing events within Knative Eventing, in environments where Apache Kafka is the preferred message broker.

The OpenShift Serverless Operator manages Knative custom resource definitions (CRDs) for your cluster and enables you to configure them without directly modifying individual config maps for each component.

Using the Assisted Installer, you can install the OpenShift Serverless Operator either separately through the API or as part of the OpenShift AI Operator bundle in the web console.

Important

The integration of the OpenShift Serverless Operator into the Assisted Installer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.

Prerequisites

3.1.8. Platform Operations and Lifecycle Operators

The Platform Operations & Lifecycle category contains the following Operators:

  • Multicluster engine Operator
  • Node Maintenance Operator
  • Cluster Observability Operator
  • OpenShift Loki Operator
  • OpenShift Logging Operator

You can deploy the multicluster engine for Kubernetes to perform the following tasks in a large, multi-cluster environment:

  • Provision and manage additional Kubernetes clusters from your initial cluster.
  • Use hosted control planes to reduce management costs and optimize cluster deployment by decoupling the control and data planes.
  • Use GitOps Zero Touch Provisioning to manage remote edge sites at scale.

You can deploy the multicluster engine with OpenShift Data Foundation on all OpenShift Container Platform clusters.

Prerequisites

  • Requires an additional 16384 MiB of memory and 4 CPU cores for each compute (worker) node.
  • Requires an additional 16384 MiB of memory and 4 CPU cores for each control plane node.
  • Requires OpenShift Data Foundation (recommended for creating additional on-premise clusters), LVM Storage, or another persistent storage service.

    Important

    Deploying multicluster engine without OpenShift Data Foundation results in the following scenarios:

    • Multi-node cluster: No storage is configured. You must configure storage after the installation process.
    • Single-node OpenShift: LVM Storage is installed.

    You must review the prerequisites to ensure that your environment has enough additional resources for the multicluster engine.

3.1.8.2. Installing the Node Maintenance Operator

The Node Maintenance Operator facilitates planned maintenance by placing nodes into maintenance mode.

The Node Maintenance Operator watches for new or deleted NodeMaintenance custom resources (CRs). When it detects a new NodeMaintenance CR, it prevents new workloads from being scheduled on that node, and cordons off the node from the rest of the cluster. The Operator then evicts all pods that can be evicted from the node. When the administrator deletes the NodeMaintenance CR associated with the node, maintenance ends and the Operator makes the node available for new workloads.

Using the Assisted Installer, you can install the Node Maintenance Operator either separately through the API or as part of the Virtualization Operator bundle in the web console.

Prerequisites

The Cluster Observability Operator (COO) is an optional component of the OpenShift Container Platform designed for creating and managing highly customizable monitoring stacks. It enables cluster administrators to automate configuration and management of monitoring needs extensively, offering a more tailored and detailed view of each namespace compared to the default OpenShift Container Platform monitoring system.

The Cluster Observability Operator deploys the following monitoring components:

  • Prometheus - A highly available Prometheus instance capable of sending metrics to an external endpoint by using remote write.
  • Thanos Querier (optional) - Enables querying of Prometheus instances from a central location.
  • Alertmanager (optional) - Provides alert configuration capabilities for different services.
  • UI plugins (optional) - Enhances the observability capabilities with plugins for monitoring, logging, distributed tracing and troubleshooting.
  • Korrel8r (optional) - Provides observability signal correlation, powered by the open source Korrel8r project.

Using the Assisted Installer, you can install the Cluster Observability Operator Operator either separately through the API or as part of the Virtualization Operator bundle in the web console. Install the Cluster Observability Operator Operator together with OpenShift Logging and OpenShift Loki Operators, to support Red Hat OpenShift Virtualization Engine deployments. For more information about the use of this Operator in OpenShift Container Platform, see "Additional resources".

Note

While installing the Cluster Observability Operator Operator through the OCP OperatorHub, you can manually enable cluster monitoring for the Operator. Because the Assisted Installer does not include this setting, it automatically enables cluster monitoring for the Operator during installation, without the option of disabling it.

Prerequisites

3.1.8.4. Installing the OpenShift Loki Operator

The OpenShift Loki Operator provides a log aggregation system designed to store and query logs from all applications and infrastructure components. The Operator implements this functionality through the LokiStack custom resource (CR).

The LokiStack CR manages Loki, which is a scalable, highly-available, multi-tenant log aggregation system. The resource also includes a web proxy with OpenShift Container Platform authentication, which enforces multi-tenancy and facilitates the saving and indexing of data in Loki log stores. You can configure Loki as the backend to store all collected flows with a maximal level of detail.

With the Assisted Installer, you can install the OpenShift Loki Operator either separately through the API or as part of the Virtualization Operator bundle in the web console. Install the OpenShift Loki Operator together with the Cluster Observability Operator and OpenShift Logging Operators, to support Red Hat OpenShift Virtualization Engine deployments.

Prerequisites

3.1.8.5. Installing the OpenShift Logging Operator

The OpenShift Logging Operator installs and manages the entire logging stack for your cluster. It works automatically with the OpenShift Container Platform security settings, ensuring that the system collects and stores logs without compromising multi-tenant isolation.

The Operator deploys collector agents, such as Vector or Fluentd, as pods on every node. These collectors use specific cluster roles to securely gather data from three distinct sources:

  • collect-audit-logs: Gathers Kubernetes and OpenShift API logs and security-related events.
  • collect-application-logs: Gathers logs from user-created projects and containers.
  • collect-infrastructure-logs: Gathers logs from the platform itself, including nodes and control plane services.

Once collected, the Operator manages the pipelines that filter and route this data. You can send your logs to internal storage such as Loki or Elasticsearch, or forward them to an external logging service. Throughout this process, the Operator ensures that all data handling follows the cluster’s built-in security and access control policies.

Using the Assisted Installer, you can install the OpenShift Logging Operator either separately through the API or as part of the Virtualization Operator bundle in the web console. Install the OpenShift Logging Operator together with the Cluster Observability Operator and OpenShift Loki Operators, to support Red Hat OpenShift Virtualization Engine deployments.

Prerequisites

3.1.9. Scheduling Operators

The Scheduling category contains the following Operators:

  • Node Feature Discovery (NFD) Operator
  • Kube Descheduler Operator
  • NUMA Resources Operator

The Node Feature Discovery (NFD) Operator automates the deployment and management of the Node Feature Discovery (NFD) add-on. The Node Feature Discovery add-on detects the configurations and hardware features of each node in an OpenShift Container Platform cluster. The add-on labels each node with hardware-specific information such as vendor, kernel configuration, or operating system version, making the cluster aware of the underlying hardware and software capabilities of the nodes.

With the Node Feature Discovery (NFD) Operator, administrators can easily gather information about the nodes to use for scheduling, resource management, and more by controlling the life-cycle of Node Feature Discovery (NFD).

You can install the Node Feature Discovery Operator separately or as part of the OpenShift AI Operator bundle.

Note

While installing the Node Feature Discovery Operator through the OCP OperatorHub, you can manually enable cluster monitoring for the Operator. Because the Assisted Installer does not include this setting, it automatically enables cluster monitoring for the Operator during installation, without the option of disabling it.

Important

The integration of the Node Feature Discovery Operator into the Assisted Installer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.

Prerequisites

  • If you are installing the Operator as part of the OpenShift AI Operator bundle, see the bundle prerequisites.
  • If you are installing the Operator separately, there are no additional prerequisites.
3.1.9.2. Installing the Kube Descheduler Operator

The Kube Descheduler Operator is a Kubernetes Operator that automates the deployment, configuration, and management of the Kubernetes Descheduler within a cluster. You can use the Kube Descheduler Operator to evict pods (workloads) based on specific strategies, so that the pods can be rescheduled onto more appropriate nodes.

You can benefit from descheduling running pods in situations such as the following:

  • Nodes are underutilized or overutilized.
  • Pods and node affinity requirements, such as taints or labels, have changed and the original scheduling decisions are no longer appropriate for certain nodes.
  • Node failure requires pods to be moved.
  • New nodes are added to clusters.
  • Pods have been restarted excessively.

Using the Assisted Installer, you can install the Kube Descheduler Operator either separately through the API or as part of the Virtualization Operator bundle in the web console.

Note

While installing the Kube Descheduler Operator through the OCP OperatorHub, you can manually enable cluster monitoring for the Operator. Because the Assisted Installer does not include this setting, it automatically enables cluster monitoring for the Operator during installation, without the option of disabling it.

Prerequisites

3.1.9.3. Installing the NUMA Resources Operator

Non-Uniform Memory Access (NUMA) is a compute platform architecture that allows different CPUs to access different regions of memory at different speeds. NUMA resource topology refers to the locations of CPUs, memory, and PCI devices relative to each other in the compute node. Colocated resources are said to be in the same NUMA zone. For high-performance applications, the cluster needs to process pod workloads in a single NUMA zone.

The NUMA Resources Operator allows you to schedule high-performance workloads in the same NUMA zone. It deploys a node resources exporting agent that reports on available cluster node NUMA resources, and a secondary scheduler that manages the workloads.

Using the Assisted Installer, you can install the NUMA Resources Operator either separately through the API or as part of the Virtualization Operator bundle in the web console. For more information about the use of this Operator in OpenShift Container Platform, see Additional resources.

Note

While installing the NUMA Resources Operator through the OCP OperatorHub, you can manually enable cluster monitoring for the Operator. Because the Assisted Installer does not include this setting, it automatically enables cluster monitoring for the Operator during installation, without the option of disabling it.

Prerequisites

Next steps

Create the NUMAResourcesOperator custom resource and deploy the NUMA-aware secondary pod scheduler. For details, see Scheduling NUMA-aware workloads (RHOCP).

3.2. Customizing with Operator Bundles

An Operator bundle is a recommended packaging format that combines related Operators to deliver a comprehensive set of capabilities. By selecting a bundle, administrators can extend functionality beyond that of a single Operator.

This approach makes the Assisted Installer more opinionated, offering an optimized platform for each selected bundle. It reduces the adoption barrier and minimizes the expertise required for customers to quickly access essential features. Additionally, it establishes a single, well-tested, and widely recognized path for platform deployments.

Meanwhile, individual Operators remain independent and free of unnecessary dependencies, ensuring a lightweight and flexible solution for small or specialized deployments, such as on single-node OpenShift.

When an administrator specifies an Operator bundle, the Assisted Installer automatically provisions the associated Operators included in the bundle. These Operators are predefined and cannot be deselected, ensuring consistency. Administrators can modify the selection after the installation has completed.

Virtualization lets you create multiple simulated environments or resources from a single, physical hardware system. The Virtualization Operator bundle provides a recommended and proven path for virtualization platform deployments, minimizing obstacles. The solution supports the addition of nodes and Day-2 administrative operations.

The Virtualization Operator bundle prompts the Assisted Installer to install the following Operators together:

Prerequisites

  • You are installing OpenShift Container Platform version 4.14 or later.
  • There is enabled CPU virtualization support in BIOS (Intel-VT/AMD-V) on all nodes.
  • Each control plane (master) node has an an additional 1024 MiB of memory and 3 CPU cores.
  • Each compute (worker) node has an additional 1024 MiB of memory and 5 CPU cores.
  • You have included the additional resources required to support the selected storage Operator.
  • You are installing a cluster of three or more nodes. The Virtualization Operator bundle is not available on single-node OpenShift.

3.2.2. Installing the OpenShift AI Operator bundle

The OpenShift AI Operator bundle enables the training, serving, monitoring, and management of Artificial Intelligence (AI) and Machine Learning (ML) models and applications. It simplifies the deployment of AI and ML components on your OpenShift cluster.

The OpenShift AI Operator bundle bundle prompts the Assisted Installer to install the following Operators together:

Important

The OpenShift AI Operator bundle is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.

Prerequisites

  • The installation of the NVIDIA GPU, AMD GPU, and Kernel Module Management Operators depends on the Graphics Processing Unit (GPU) detected on your hosts following host discovery.

After you meet the cluster nodes and network requirements, you can begin installing the cluster.

4.1. Preinstallation considerations

Before installing OpenShift Container Platform with the Assisted Installer, you must consider the following configuration choices:

  • Which base domain to use
  • Which OpenShift Container Platform product version to install
  • Whether to install a full cluster or single-node OpenShift
  • Whether to use a DHCP server or a static network configuration
  • Whether to use IPv4 or dual-stack networking
  • Whether to install OpenShift Virtualization
  • Whether to install Red Hat OpenShift Data Foundation
  • Whether to install multicluster engine for Kubernetes
  • Whether to integrate with the platform when installing on vSphere or Nutanix
  • Whether to install a multi-architecture compute cluster

4.2. Setting the cluster details

To create a cluster with the Assisted Installer web user interface, use the following procedure.

Procedure

  1. Log in to the Red Hat Hybrid Cloud Console.
  2. On the Red Hat OpenShift tile, click OpenShift.
  3. On the Red Hat OpenShift Container Platform tile, click Create cluster.
  4. Click the Datacenter tab.
  5. Under Assisted Installer, click Create cluster.
  6. Enable the I’m installing on a disconnected/air-gapped/secured environment toggle to deploy an OpenShift Container Platform cluster by using a self-contained ISO image. The image includes a simplified user interface together with all necessary artifacts. This method supports installations in both connected and disconnected environments without requiring an external or local registry. It currently addresses Red Hat OpenShift Virtualization Engine use cases.

    This installation method is outside the scope of this guide. For details and installation instructions, see Installing a cluster without an external registry in the OpenShift Container Platform documentation.

    Important

    Installing a Red Hat OpenShift Virtualization Engine cluster using this method is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

    For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.

  7. Enter a name for the cluster in the Cluster name field.
  8. Enter a base domain for the cluster in the Base domain field. All subdomains for the cluster will use this base domain.

    Note

    The base domain must be a valid DNS name. You must not have a wildcard domain set up for the base domain.

  9. From the OpenShift version dropdown list, select the version that you want to install and click Select. By default, the dropdown lists the latest OpenShift version. If you need an older version that is not displayed, click Show all available versions at the bottom of the list, and use the search box to find it.

    Important
    • For a multi-architecture compute cluster installation, select OpenShift Container Platform 4.12 or later, and use the -multi option. For instructions on installing a multi-architecture compute cluster, see Installing multi-architecture compute clusters.
    • For IBM Power® and IBM Z® platforms, only OpenShift Container Platform 4.13 and later is supported.
    • If you are booting from an iSCSI drive, select OpenShift Container Platform version 4.15 or later.
  10. Optional: The Assisted Installer defaults to using x86_64 CPU architecture. If you are installing OpenShift Container Platform on a different architecture, select the architecture to use. Valid values are arm64, ppc64le, and s390x. Remember that some features are not available with arm64, ppc64le, and s390x CPU architectures.

    Important

    For a multi-architecture compute cluster installation, you can use x86_64 or 64-bit ARM CPU architecture for the control plane nodes. Automatic conversion from x86_64 to 64-bit ARM is only supported on Amazon Web Services (AWS). For instructions on installing a multi-architecture compute cluster, see Installing multi-architecture compute clusters.

  11. Optional: The Assisted Installer already has the pull secret associated to your account. If you want to use a different pull secret, select Edit pull secret.
  12. Optional: If you are installing OpenShift Container Platform on a third-party platform, select the platform from the Integrate with external partner platforms list. Valid values are Nutanix, vSphere or Oracle Cloud Infrastructure. The Assisted Installer defaults to having no platform integration.

    Note
    • The Assisted Installer supports Oracle Cloud Infrastructure (OCI) integration from OpenShift Container Platform 4.14 and later.
    • For details on external partner integrations, see Additional resources.
  13. From the Number of control plane nodes field, optionally change the default value of three control plane nodes for your installation. The possible options are 1 (single-node OpenShift), 2, 3, 4, or 5.

    Important

    The Assisted Installer supports the following control plane node configurations:

    • 4 or 5 control plane nodes from OpenShift Container Platform 4.18 and later, on a bare metal or user-managed networking platform with an x86_64 CPU architecture.
    • 2 control plane nodes from OpenShift Container Platform 4.19 and later, for a Two-Node OpenShift with Arbiter cluster topology. If the number of control plane nodes for a cluster is 2, then it must have at least one additional arbiter host.
    • Currently, single-node OpenShift is not supported on IBM Power® platform.

    For details, see Control plane count.

  14. Optional: Select Include custom manifests if you have at least one custom manifest to include in the installation. A custom manifest has additional configurations not currently supported in the Assisted Installer. Selecting the checkbox adds the Custom manifests step to the wizard, where you upload the manifests.

    Important
    • If you are installing OpenShift Container Platform on the Oracle Cloud Infrastructure (OCI) third-party platform, it is mandatory to add the custom manifests provided by Oracle.
    • If you have already added custom manifests, clearing the Include custom manifests checkbox automatically deletes them all. You must confirm the deletion.
  15. Optional: The Assisted Installer defaults to DHCP networking. If you are using a static IP configuration, bridges or bonds for the cluster nodes instead of DHCP reservations, select Static IP, bridges, and bonds. Selecting the checkbox adds the Static network configurations step to the wizard. For details, see Configuring static networks.

    Important

    A static IP configuration is not supported in the following scenarios:

    • OpenShift Container Platform installations on Oracle Cloud Infrastructure.
    • OpenShift Container Platform installations on iSCSI boot volumes.
  16. Optional: If you want to enable encryption of the installation disks, under Enable encryption of installation disks you can select one of the following:

    • For single-node OpenShift, select Control plane node, worker.
    • For multi-node clusters, select Control plane nodes to encrypt the control plane node installation disks. Select Workers to encrypt worker node installation disks. Select Arbiter to encrypt the arbiter node installation disks.

      Important

      You cannot change the base domain, the single-node OpenShift checkbox, the CPU architecture, the host’s network configuration, or the disk-encryption after installation begins.

4.3. Configuring static networks

The Assisted Installer supports the following static network configurations:

  • Network-wide configurations - These settings apply to all hosts across the cluster and include configurations such as networking stack type, VLANs, and DNS.
  • Host-specific configurations - These settings enable the configuration of individual host interfaces, and support features such as bonds, bridges, and VLANs for each host.

You define these settings in the NMState YAML file, which uses the NMState library for declarative network management. The Static networking step of the Assisted Installer web console provides two options for configuring the NMState YAML file:

  • Form view - The form view provides fields for entering configuration parameters and supports basic network configurations. The network-wide configurations and host-specific configurations are two separate steps. The form creates the YAML file in the background when saved.
  • YAML view - In the YAML view, you can upload an NMState YAML file that you created locally. This method supports advanced configurations, and includes both network-wide and host-specific settings within a single file.
Important
  • For installations on IBM Z® with z/VM, ensure that the z/VM nodes and vSwitches are properly configured for static networks and NMState. Also, the z/VM nodes must have a fixed MAC address assigned as the pool MAC addresses might cause issues with NMState. For more information about NMState, see NMState Declarative Network API.
  • A static IP configuration is not supported in the following scenarios:

    • OpenShift Container Platform installations on Oracle Cloud Infrastructure.
    • OpenShift Container Platform installations on iSCSI boot volumes.

Select the form view for basic configurations that use a single network interface.

Note

To add new hosts that will use the new or edited configurations, you will need to regenerate the Discovery ISO in the 'Host discovery' step and boot your new hosts from it.

Prerequisites

  • You have selected the Static IP, bridges and bonds option under Hosts' network configuration on the Cluster details page. Selecting this option adds the Static network configurations step to the wizard.

Procedure

  1. Go to the Static network configurations page.
  2. From the Configure via options, select Form view.
  3. Enter the network-wide configurations:

    1. Select the Networking stack type. Valid options are IPv4 and Dual stack.

      Important
      • IPv6 is not currently supported for single stack.
      • IPv4 networking with SDN is supported up to OpenShift Container Platform 4.14.
      • IPv4 and dual-stack IPv4/IPv6 networking with OVN only are supported from OpenShift Container Platform 4.15 and later.
    2. If the cluster hosts are on a shared VLAN, select the Use VLAN checkbox and enter the VLAN ID.
    3. Enter the network-wide IP addresses. If you selected Dual stack networking, you must enter both IPv4 and IPv6 addresses.

      1. Enter the DNS server IP address.
      2. Enter the cluster subnet’s IP address range in CIDR notation.
      3. Enter the default gateway IP address.
  4. Click Next.
  5. Enter the host-specific configurations:

    1. Enter the IP address and the MAC address for the host.
    2. Optional: You can use bonds to combine network interfaces for increased bandwidth and ensure redundancy. Creating a bond with a static IP address aggregates two network interfaces per host.

      1. Select the Use bond checkbox.
      2. From the Bond type dropdown list, select the bond type. The default bond type is Active-Backup (1). For a description of the bond types, see Bonding modes.
      3. Enter the MAC address for Port 1.
      4. Enter the MAC address for Port 2.
      5. Enter the IP address for the bond.
  6. Click Add another host configuration to configure additional hosts.
  7. Click Next.

If you are configuring multiple interfaces or other advanced networking features, select the YAML view to enter the network state for each host that uses NMState syntax. For more information about NMState, see NMState Declarative Network API.

Prerequisites

  • You have selected the Static IP, bridges and bonds option under Hosts' network configuration on the Cluster details page. Selecting this option adds the Static network configurations step to the wizard.

Procedure

  1. Go to the Static network configurations page.
  2. From the Configure via options, select YAML view.
  3. Upload, drag and drop, or copy and paste a YAML file containing NMState into the editor for network configurations. For guidance, see NMState configuration examples.
  4. In the MAC to interface name mapping fields, enter the MAC address and interface name for each host interface used in your network configuration. For guidance, see MAC to interface mapping.
  5. Select the Copy the YAML content checkbox to copy the YAML content between multiple hosts.
  6. Click Add another host configuration to configure additional hosts.
  7. Click Next.

4.4. Installing Operators and Operator bundles

You can customize your deployment by selecting Operators and Operator bundles during the installation. If you require advanced options, add the Operators or bundles after you have installed the cluster.

When installing Operators and Operator bundles through the web console, the following conditions apply:

  • Some Operators are only available as part of a bundle. You cannot install them individually.
  • Some Operators are available either individually or as part of a bundle. If you install them as part of a bundle, you can only remove them by deselecting the bundle.
  • Some Operators are only available for installing individually. They are not included in either of the bundles.

This step is optional.

4.4.1. Installing standalone Operators

You can select more than one standalone Operator and add Operator bundles as needed. Operators that appear greyed out are only available for installation as part of a bundle.

For instructions on installing Operator bundles, see Installing Operator bundles.

Prerequisites

Procedure

  1. On the Operators page, expand the Single Operators arrow to display the full list of Operators. You can use the Find Bundles or Operators field to filter the list and display matching items.
  2. Select from the following Operators:

    Expand
    CategoryOperatorComments

    Storage

    Logical Volume Manager Storage

     

    OpenShift Data Foundation

     

    Virtualization

    OpenShift Virtualization

    The OpenShift Virtualization Operator requires backend storage and might automatically activate a storage Operator in the background, according to the following criteria:

    • None - If the CPU architecture is ARM64, no storage Operator is activated.
    • LVM Storage - For single-node OpenShift clusters on any other CPU architecture deploying OpenShift Container Platform 4.12 or higher.
    • Local Storage Operator (LSO) - For all other deployments.

    Migration Toolkit for Virtualization

    Selecting the Migration Toolkit for Virtualization Operator automatically activates the OpenShift Virtualization Operator. For a Single-node OpenShift installation, the Assisted Installer also activates the LVM Storage Operator.

    OpenShift sandboxed containers

     

    Artificial Intelligence (AI)

    OpenShift AI

    Developer Preview feature.

    AMD GPU

    • Developer Preview feature.
    • Selecting the AMD GPU Operator automatically activates the Kernel Module Management Operator.

    NVIDIA GPU

    • Developer Preview feature.
    • Selecting the NVIDIA GPU Operator automatically activates the Node Feature Discovery Operator.

    Network

    NMState

    Currently, you cannot install the Kubernetes NMState Operator on the Nutanix or Oracle Cloud Infrastructure (OCI) third-party platforms.

    Security & Access Control

    Kernel Module Management

    Developer Preview feature.

    Platform Operations & Lifecycle

    Multicluster engine

    You can deploy the multicluster engine with OpenShift Data Foundation on all OpenShift Container Platform clusters. Deploying the multicluster engine without OpenShift Data Foundation results in the following storage configurations:

    • Multi-node cluster: No storage is configured. You must configure storage after the installation.
    • Single-node OpenShift: LVM Storage is installed.

    Cluster Observability

    Selecting the Cluster Observability Operator automatically activates the OpenShift Loki and OpenShift Logging Operators.

    Scheduling

    Node Feature Discovery

    Developer Preview feature.

    Important

    The integration of the AMD GPU, Kernel Module Management, Node Feature Discovery, NVIDIA GPU, and OpenShift AI Operators into the Assisted Installer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.

  3. To install the Self Node Remediation Operator, see Installing Operators by using the API. It is not currently possible to install this Operator using the web console.
  4. Click Next.

4.4.2. Installing Operator bundles

You can select more than one Operator bundle together with additional Operators as needed.

For instructions on installing individual Operators, see Installing Operators.

Prerequisites

Procedure

  1. On the Operators page, select from the following Operator bundles:

    • Virtualization - Contains the following Operators:

      • OpenShift Virtualization
      • Kube Descheduler
      • Node Maintenance
      • Migration Toolkit for Virtualization
      • Kubernetes NMState
      • Fence Agents Remediation
      • Node Health Check
      • Local Storage Operator (LSO)
      • Cluster Observability
      • Loki
      • OpenShift Logging
      • MetalLB
      • NUMA Resources
      • OADP
    • OpenShift AI - Contains the following Operators:

      • Kubernetes Authorino
      • OpenShift Data Foundation
      • OpenShift AI
      • AMD GPU
      • Node Feature Discovery
      • NVIDIA GPU
      • OpenShift Pipelines
      • OpenShift Service Mesh
      • OpenShift Serverless
      • Kernel Module Management
    Important

    The OpenShift AI Operator bundle is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.

  2. Expand the Single Operators arrow to view the Operators selected for the bundle. You can use the Find Bundles or Operators field to filter the list and display matching items.
  3. Click Next.

4.5. Adding hosts to the cluster

You must add one or more hosts to the cluster. Adding a host to the cluster involves generating a discovery ISO. The discovery ISO runs Red Hat Enterprise Linux CoreOS (RHCOS) in-memory with an agent.

Important
  • If you are installing the IBM Z® architecture, use the following table to identify the image file type:

    Expand
    ArchitectureBoot methodImage type

    Logical Partition-Classic

    iPXE

    Full image file: Download a self-contained ISO image

    Logical Partition-Data Protection Manager

    ISO or iPXE

    Minimal image file: Download an ISO image that fetches content when booting up

  • ISO images are not supported for installations on IBM Z (s390x) with z/VM or logical partitioning (LPAR) nodes; use the "Booting hosts with iPXE" procedure. ISO images and iPXE are supported for installations on RHEL KVM.

Perform the following procedure for each host on the cluster.

Procedure

  1. In the Host discovery step, click the Add hosts button and select the provisioning type.

    1. Select Minimal image file: Provision with virtual media to download a smaller image that will fetch the data needed to boot. The nodes must have virtual media capability. This is the recommended method for x86_64 and arm64 architectures.

      Important

      This option is mandatory in the following scenarios:

      • If you are installing OpenShift Container Platform on Oracle Cloud Infrastructure.
      • If you are installing OpenShift Container Platform on iSCSI boot volumes.
    2. Select Full image file: Provision with physical media to download the larger full image. This is the recommended method for the ppc64le architecture and for the s390x architecture when installing with RHEL KVM.
    3. Select iPXE: Provision from your network server to boot the hosts using iPXE. This is the recommended method on IBM Z® with z/VM nodes and LPAR (both static and DPM). ISO boot is the recommended method on the RHEL KVM installation.

      Note

      If you are installing OpenShift Container Platform on RHEL KVM, in some circumstances, the VMs on the KVM host are not rebooted on first boot and need to be restarted manually.

  2. Optional: Activate the Run workloads on control plane nodes switch to schedule workloads to run on control plane nodes, in addition to the default worker nodes.

    Note

    This option is available for clusters of five or more nodes. For clusters of under five nodes, the system runs workloads on the control plane nodes only, by default. For more details, see Control plane workload scheduling.

  3. Optional: If the cluster hosts require the use of a proxy, select Configure cluster-wide proxy settings. Enter the username, password, required domains or IP addresses, and port for the HTTP and HTTPS URLs of the proxy server. If the cluster hosts are behind a firewall, allow the nodes to access the required domains or IP addresses through the firewall. For details, see Configuring your firewall for OpenShift Container Platform.

    Note

    The proxy username and password must be URL-encoded.

  4. Optional: Add an SSH public key so that you can connect to the cluster nodes as the core user. Having a login to the cluster nodes can provide you with debugging information during the installation.

    Important

    Do not skip this procedure in production environments, where disaster recovery and debugging is required.

    1. If you do not have an existing SSH key pair on your local machine, follow the steps in Generating a key pair for cluster node SSH access.
    2. In the SSH public key field, click Browse to upload the id_rsa.pub file containing the SSH public key. Alternatively, drag and drop the file into the field from the file manager. To see the file in the file manager, select Show hidden files in the menu.
  5. Optional: If the cluster hosts are in a network with a re-encrypting man-in-the-middle (MITM) proxy, or if the cluster needs to trust certificates for other purposes such as container image registries, select Configure cluster-wide trusted certificates. Add additional certificates in X.509 format.
  6. Configure the discovery image if needed. For details, see Configuring the discovery image.
  7. Optional: If you are installing on a platform and want to integrate with the platform, select Integrate with your virtualization platform. You must boot all hosts and ensure they appear in the host inventory. All the hosts must be on the same platform. For more details, see the relevant platform in Additional resources.
  8. Click Generate Discovery ISO or Generate Script File.
  9. Download the discovery ISO or iPXE script.
  10. Boot the host(s) with the discovery image or iPXE script. For details, see Booting hosts with the discovery image.

4.6. Configuring hosts

After booting the hosts with the discovery ISO, the hosts will appear in a table on the Host Discovery page. You can optionally configure the hostname and role for each host. You can also delete a host if necessary.

Procedure

  1. Go to the Host Discovery tab.
  2. In multi-host clusters, you can select a role for each host after host discovery:

    1. In the Role column, expand the Auto-assign arrow for the host.
    2. Choose one of the following options:

      • Auto-assign - Automatically determines whether the host is a control plane, worker or arbiter node. This is the default setting.
      • Control plane node - Assigns the control plane (master) role to the host, allowing the host to manage and coordinate the cluster.
      • Worker - Assigns the compute (worker) role to the host, enabling the host to run application workloads.
      • Arbiter - Assigns the arbiter role to a host, providing a cost-effective solution for components that require a quorum.

      The minimum hardware requirements for control plane nodes exceed that of worker nodes. If you assign a role to a host, ensure that you assign the control plane role to hosts that meet the minimum hardware requirements.

      For more details about the different host roles, see Supported host roles.

  3. Click the Status link to view hardware, network, and operator validations for the host.
  4. Optionally select or deselect all hosts by clicking the <number> selected checkbox or alternatively by selecting Select all or Select none from the dropdown list.
  5. Optionally change one or more hostnames:

    1. To rename a single host, from the Options (⋮) menu for the host, select Change hostname. If necessary, enter a new name for the host and click Change. You must ensure that each host has a valid and unique hostname.
    2. To rename multiple selected hosts, from the Actions list, select Change hostname. In the Change Hostname dialog, type the new name and include {{n}} to make each hostname unique. Then click Change.

      Note

      You can see the new names appearing in the Preview pane as you type. The name will be identical for all selected hosts, with the exception of a single-digit increment per host.

  6. Optionally remove one or more hosts:

    1. To remove a single host, from the Options (⋮) menu for the host, select Remove host. Click Remove host to confirm the deletion.
    2. To remove multiple selected hosts at the same time, from the Actions list, select Remove. Click Remove hosts to confirm the deletion.
    Note

    In a regular deployment, a cluster can have three or more hosts, and at least three of these must be control plane nodes. If you delete a host that is also a control plane node, or if there are only two hosts, you will get a message saying that the system is not ready. To restore a host, you must reboot it from the discovery ISO.

  7. From the Options (⋮) menu for the host, optionally select View host events. The events in the list are presented chronologically.
  8. Click the arrow to the left of a host name to expand the host details.

    Once all cluster hosts appear with a status of Ready, proceed to the next step.

4.7. Configuring storage disks

Each of the hosts retrieved during host discovery can have multiple storage disks. The storage disks are listed for the host on the Storage page of the Assisted Installer wizard.

You can optionally modify the default configurations for each disk.

Important
  • Starting from OpenShift Container Platform 4.14, you can configure nodes with Intel® Virtual RAID on CPU (VROC) to manage NVMe RAIDs. For details, see Configuring an Intel® Virtual RAID on CPU (VROC) data volume.
  • Starting from OpenShift Container Platform 4.15, you can install a cluster on a single or multipath iSCSI boot device using the Assisted Installer.

4.7.1. Changing the installation disk

The Assisted Installer randomly assigns an installation disk by default. If there are multiple storage disks for a host, you can select a different disk to be the installation disk. This automatically unassigns the previous disk.

Note

Red Hat Enterprise Linux CoreOS (RHCOS) supports multipathing over Fibre Channel on the installation disk, allowing stronger resilience to hardware failure to achieve higher host availability. Multipathing is enabled by default in the agent ISO image, with an /etc/multipath.conf configuration. For details, see Modifying the DM Multipath configuration file.

Procedure

  1. Navigate to the Storage page of the wizard.
  2. Expand a host to display the associated storage disks.
  3. Select Installation disk from the Role list.

    Note

    Multipath devices are automatically discovered and listed in the host’s inventory. To assign a multipath Fibre Channel disk as the installation disk, choose a disk with Drive type set to Multipath, rather than to FC which indicates a single path.

  4. When all storage disks return to Ready status, proceed to the next step.

4.7.2. Disabling disk formatting

The Assisted Installer marks all bootable disks for formatting during the installation process by default, regardless of whether or not they have been defined as the installation disk. Formatting causes data loss.

You can choose to disable the formatting of a specific disk. Perform this with caution, as bootable disks can interfere with the installation process, mainly in terms of boot order.

You cannot disable formatting for the installation disk.

Procedure

  1. Navigate to the Storage page of the wizard.
  2. Expand a host to display the associated storage disks.
  3. Clear Format for a disk.
  4. When all storage disks return to Ready status, proceed to the next step.

4.8. Configuring networking

Before installing OpenShift Container Platform, you must configure the cluster network.

Procedure

  1. In the Networking step, select one of the following network management types if it is not already selected for you:

    • Cluster-Managed Networking: Selecting cluster-managed networking means that the Assisted Installer will configure a standard network topology. This configuration includes an integrated load balancer and virtual routing for managing the API and Ingress VIP addresses. For details, see Cluster-managed networking.

      Note
      • Currently, cluster-managed networking is not supported on IBM Z® and IBM Power®.
      • Cluster-managed networking is not supported on single-node OpenShift.
    • User-Managed Networking: Selecting user-managed networking deploys OpenShift Container Platform with a non-standard network topology. Select user-managed networking if you want to deploy with an external load balancer and DNS, or if you intend to deploy the cluster nodes across many distinct subnets. For details, see User-managed networking.

      Note

      Oracle Cloud Infrastructure (OCI) is available for OpenShift Container Platform 4.14 with a user-managed networking configuration only.

    Important

    The Assisted Installer supports a third network management type called Cluster-Managed Networking with a User-Managed Load Balancer. This network management type provides automated cluster networking with an external load balancer. Currently you can configure this network management type through the API only. For details, see Installing cluster-managed networking with a user-managed load balancer.

  2. For cluster-managed networking, configure the following settings:

    1. Select your Networking stack type:

      • IPv4: Select this type when your hosts are only using IPv4.
      • Dual-stack: Select dual-stack when your hosts use IPv4 with IPv6, with either stack set as primary.

        Note
        • IPv6 is not currently supported for single stack.
        • IPv4 networking with SDN is supported up to OpenShift Container Platform 4.14.
        • IPv4 and dual-stack IPv4/IPv6 networking with OVN only are supported in OpenShift Container Platform 4.15 and later.
        Important

        Support for IPv6 as the primary stack in a dual-stack configuration is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

        For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.

    2. Define the Machine network. You can use the default network or select a subnet from the dropdown list.

      For a dual-stack implementation, specify both a primary and a secondary machine network, ensuring that one is IPv4 and the other IPv6. When you select IPv6 as the primary network, the secondary network dropdown list shows only IPv4 options, and vice versa.

      Important

      For iSCSI boot volumes, the hosts connect over two machine networks: one designated for the OpenShift Container Platform installation and the other for iSCSI traffic. Ensure you select the OpenShift Container Platform network from the dropdown list. The iSCSI host IP address should not be on the machine network. Choosing the iSCSI network will result in an Insufficient status for the host in the Networking step.

    3. Define an API virtual IP. An API virtual IP provides an endpoint for all users to interact with and configure the platform.

      For a dual-stack implementation, define the API virtual IP for both the primary and secondary stack. The IP family (IPv4 or IPv6) must match the corresponding machine network family.

    4. Define an Ingress virtual IP. An Ingress virtual IP provides an endpoint for application traffic flowing from outside the cluster.

      For a dual-stack implementation, define the Ingress virtual IP for both the primary and secondary stack. The IP family (IPv4 or IPv6) must match the corresponding machine network family.

  3. Configure the advanced networking properties:

    • For an IPv4 implementation, optionally select Use advanced networking to configure the following properties:

      • Cluster network CIDR: Define an IP address block to assign pod IP addresses.
      • Cluster network host prefix: Define a subnet prefix length to assign to each node.
      • Service network CIDR: Define an IP address block to assign service IP addresses.
    • For a dual-stack implementation, these fields are mandatory and populated automatically for both primary and secondary stacks, according to the machine networks selected. You can change these values as required.
  4. Optional: Select Host SSH Public Key for troubleshooting after installation to connect to hosts using a public SSH key for troubleshooting after installation.
  5. Wait for the validations to complete, and then click Next. The Review and create page presents a summary of your configurations in the Networking section.

4.9. Adding manifests and patches

You can upload custom manifests and patches for system manifests in the Assisted Installer web console. You can also replace and remove these files.

For information about adding and modifying custom manifests by using the Assisted Installer API, see Additional resources.

This section provides an overview of custom manifests and system manifest patches, including formatting considerations and the required naming conventions for uploading the files.

Follow these guidelines to ensure that the files you upload comply with the system requirements.

4.9.1.1. Custom manifests

A custom manifest is a JSON or YAML file that contains advanced configurations not currently supported in the Assisted Installer user interface. You can create a custom manifest or use one provided by a third party.

You can upload a custom manifest from your file system to either the openshift folder or the manifests folder. The number of custom manifest files permitted is unlimited.

You can upload only one file at a time. However, each uploaded YAML file can contain multiple custom manifests. Uploading a multi-document YAML manifest is faster than adding the YAML files individually.

For a file containing a single custom manifest, accepted file extensions include .yaml, .yml, or .json. For a file containing multiple custom manifests, accepted file types include .yaml or .yml.

Single custom manifest example:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: 99-openshift-machineconfig-master-kargs
spec:
  kernelArguments:
    - loglevel=7

Multiple custom manifest example:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: 99-openshift-machineconfig-master-kargs
spec:
  kernelArguments:
    - loglevel=7
---
apiVersion: machineconfiguration.openshift.io/v2
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 98-openshift-machineconfig-worker-kargs
spec:
  kernelArguments:
    - loglevel=5
Note

When you install OpenShift Container Platform on the Oracle Cloud Infrastructure (OCI) external platform, you must add the custom manifests provided by Oracle. For additional external partner integrations such as vSphere or Nutanix, this step is optional.

4.9.1.2. Patches for system manifests

A manifest patch file conforms to the syntax of a YAML patch. Its purpose is to modify a system manifest that is automatically created by the Assisted Installer during installation preparation. Manifest patches are used to adjust onfigurations, manage updates, or apply changes in a structured and automated way. This approach ensures consistency and helps avoid errors when altering complex YAML documents.

The yaml-patch package is an implementation of JavaScript Object Notation (JSON) Patch, directly transposed to YAML. The general syntax of a system manifest YAML patch is the following:

- op: <add | remove | replace | move | copy | test>
  from: <source-path>
  path: <target-path>
  value: <any-yaml-structure>

where:

  • op specifies the operation, which can be one of the following: add, remove, replace, move, copy, or test. See JavaScript Object Notation (JSON) Patch for an explanation of each operation.
  • from specifies the source path and is used with the move and copy operations.
  • path specifies the target path and is always required.
  • value accepts any valid YAML structure and is used with the add, replace and test operations.
4.9.1.2.2. Creating a manifest patch

When creating a new patch for a system manifest, use the following naming convention: <file to be patched>.patch_<suffix>. The name ensures that the correct manifest is overwritten, and the suffix allows for the application of many patches to the same manifest.

For example, if the original file has the name 50-masters-chrony-configuration.yaml, call the new patch 50-masters-chrony-configuration.yaml.patch_1_apply-chrony-dhcp or similar.

Procedure

  1. Verify that the Assisted Installer has automatically added the original YAML file to the cluster manifests at the start of the installation, as in the example below:

    apiVersion: machineconfiguration.openshift.io/v1
    kind: MachineConfig
    metadata:
      labels:
        machineconfiguration.openshift.io/role: worker
      name: 50-masters-chrony-configuration
    spec:
      config:
        ignition:
          version: 3.1.0
        storage:
          files:
          - contents:
              source: data:text/plain;charset=utf-8;base64,CnBvb2wgMC5yaGVsLnBvb2wubnRwLm9yZyBpYnVyc3QKZHJpZnRmaWxlIC92YXIvbGliL2Nocm9ueS9kcmlmdAptYWtlc3RlcCAxLjAgMwpydGNzeW5jCmxvZ2RpciAvdmFyL2xvZy9jaHJvbnkKc2VydmVyIHN0YXRpYy4xOTAuMTExLjE2MS41LmNsaWVudHMueW91ci1zZXJ2ZXIuZGUgaWJ1cnN0CnNlcnZlciAyMDguNjcuNzIuNDMgaWJ1cnN0CnNlcnZlciBoMTM1LTEzNC0xMTEtMTIyLm1kc253aS5icm9hZGJhbmQuZHluYW1pYy50ZHMubmV0IGlidXJzdApzZXJ2ZXIgdGltZS5yaWNoaWVtY2ludG9zaC5jb20gaWJ1cnN0CnNlcnZlciBhcm0xLm1heGhvc3QuaW8gaWJ1cnN0CnNlcnZlciBzcGlkZXkucmVsbGltLmNvbSBpYnVyc3QKc2VydmVyIHVzLWVhc3QtMi5jbGVhcm5ldC5wdyBpYnVyc3QKc2VydmVyIDEwOC42MS43My4yNDMgaWJ1cnN0CnNlcnZlciBudHAxLm50cC0wMDEucHJvZC5pYWQyLmRjLnJlZGhhdC5jb20gaWJ1cnN0CnNlcnZlciBjbG9jay5yZWRoYXQuY29tIGlidXJzdApzZXJ2ZXIgaXAyMjkuaXAtNTEtODEtMjI2LnVzIGlidXJzdApzZXJ2ZXIgNjYuMjA1LjI0OS4yOCBpYnVyc3QKc2VydmVyIDEwLjExLjE2MC4yMzggaWJ1cnN0CnNlcnZlciBjLTczLTE5My02Mi01NC5oc2QxLndhLmNvbWNhc3QubmV0IGlidXJzdApzZXJ2ZXIgbnRwMi53aWt0ZWwuY29tIGlidXJzdA==
            mode: 420
            path: /etc/chrony.conf
            overwrite: true

    where:

    • Directory name is OpenShift.
    • Filename is 50-masters-chrony-configuration.yaml.
  2. To patch this YAML file with different content, generate a new base64 representation of the content and create a patch file:

    1. Generate base64 file content for /etc/chrony.conf:

      $ cat << EOF | base64 --wrap 0of
      driftfile /var/lib/chrony/drift
      makestep 1.0 3
      rtcsync
      logdir /var/log/chrony
      sourcedir /run/chrony-dhcp
      EOF
      echo
      ZHJpZnRmaWxlIC92YXIvbGliL2Nocm9ueS9kcmlmdAptYWtlc3RlcCAxLjAgMwpydGNzeW5jCmxvZ2RpciAvdmFyL2xvZy9jaHJvbnkKc291cmNlZGlyIC9ydW4vY2hyb255LWRoY3AK
    2. Create a patch file using this base64 string:

       ---
      - op: replace
        path: /spec/config/storage/files/0/contents
        value: data:text/plain;charset=utf-8;base64,ZHJpZnRmaWxlIC92YXIvbGliL2Nocm9ueS9kcmlmdAptYWtlc3RlcCAxLjAgMwpydGNzeW5jCmxvZ2RpciAvdmFyL2xvZy9jaHJvbnkKc291cmNlZGlyIC9ydW4vY2hyb255LWRoY3AK

      where:

      • Directory name is OpenShift.
      • Filename is 50-masters-chrony-configuration.yaml.patch_1_apply-chrony-dhcp
  3. Submit the patch file through the Assisted Installer web console or API.

When uploading a custom manifest or patch, enter the filename and select a destination folder. The filename must be unique across both folders; you cannot use the same file name in both folders.

Prerequisites

  • You have saved a custom manifest file to a local directory using an appropriate file name and extension.

Procedure

  1. On the Cluster details page of the wizard, select the Include custom manifests checkbox.
  2. On the Custom manifest page, in the folder field, select the Assisted Installer folder where you want to save the manifest or patch.

    Note

    You can upload a file to either the openShift or manifest folder. For a manifest patch, the system will look in both folders for the target file that it needs to patch.

  3. In the Filename field, enter a name for the manifest file, including the extension:

    • For custom manifests, examples include manifest1.json or multiple1.yaml.
    • For manifest patches, an example is 50-masters-chrony-configuration.yaml.patch_1_apply-chrony-dhcp.
  4. Under Content, click the Upload icon or Browse button to upload a file. Alternatively, drag the file into the Content field from your file system.
  5. To upload another file, click Add another manifest and repeat the process. This saves any previously uploaded files.
  6. Click Next to save all files and proceed to the Review and create page. Custom manifests displays a list of the uploaded custom manifests and patches.

You can rename uploaded custom manifest or patch files, and save custom manifest files to a different folder. Additionally, you can copy the contents of an existing file, or download it to the folder specified in your Chrome download settings.

It is not possible to edit the content of an uploaded manifest or patch file. Instead, you can overwrite the existing file.

Prerequisites

  • You have uploaded at least one custom manifest or patch file.

Procedure

  1. To change the location of a custom manifest file, select a different folder from the Folder list.
  2. To change the file name, type the new name for the manifest or patch in the File name field. Patch files should respect the patch naming conventions discussed earlier in this section.
  3. To overwrite a manifest or patch file, save a new file with the same file name in either the openshift or manifest folder.

    Note

    The system will automatically detect and replace the original file, regardless of which folder it is in.

  4. To download a manifest or patch to your file system, click the Download icon.
  5. To copy a manifest or patch, click the Copy to clipboard icon.
  6. To apply the changes, click either Add another manifest or Next.

You can remove uploaded custom manifests or patches before installation in one of two ways:

  • Removing a single custom manifest or patch.
  • Removing all manifests and patches at the same time.
Warning

Once you have removed a manifest or patch file you cannot undo the action. The workaround is to upload the file again.

4.9.4.1. Removing all custom manifests and patches

You can remove all custom manifests and patches at the same time. This also hides the Custom manifest page.

Prerequisites

  • You have uploaded at least one custom manifest or patch file.

Procedure

  1. Browse to the Cluster details page of the wizard.
  2. Clear the Include custom manifests checkbox.
  3. In the Remove custom manifests dialog box, click Remove.

You can delete one file at a time. This option does not allow deletion of the last remaining manifest or patch.

Prerequisites

  • You have uploaded at least two custom manifest or patch files.

Procedure

  1. Browse to the Custom manifests page.
  2. Hover over the manifest name to display the Delete (minus) icon.
  3. Click the icon and then click Delete in the dialog box.

4.10. Installing the cluster

After completing the configuration and ensuring all the nodes are Ready, begin the installation. The installation process takes time, and you can monitor progress in the Assisted Installer web console. Nodes will reboot during the installation, and initialize after installation.

Prerequisites

  • Before installing the cluster, ensure the cluster and each host have passed the preinstallation validations. For details, see Preinstallation validations.

Procedure

  1. In the Review and create step, click Install cluster.
  2. Click the link in the Status column of the Host Inventory list to see the installation status of a particular host.

4.11. Completing the installation

After the cluster is installed and initialized, the Assisted Installer indicates that the installation is finished. The Assisted Installer provides the console URL, the kubeadmin username and password, and the kubeconfig file. Additionally, the Assisted Installer provides cluster details including the OpenShift Container Platform version, base domain, CPU architecture, API and Ingress IP addresses, and the cluster and service network IP addresses.

Prerequisites

  • You have installed the oc CLI tool.

Procedure

  1. Make a copy of the kubeadmin username and password.
  2. Download the kubeconfig file and copy it to the auth directory under your working directory:

    $ mkdir -p <working_directory>/auth
    $ cp kubeconfig <working_directory>/auth
    Note

    The kubeconfig file is available for download for 20 days after completing the installation.

  3. Add the kubeconfig file to your environment:

    $ export KUBECONFIG=<your working directory>/auth/kubeconfig
  4. Login with the oc CLI tool:

    $ oc login -u kubeadmin -p <password>

    Replace <password> with the password of the kubeadmin user.

  5. Click the web console URL or click Launch OpenShift Console to open the console.
  6. Enter the kubeadmin username and password. Follow the instructions in the OpenShift Container Platform console to configure an identity provider and configure alert receivers.
  7. Add a bookmark of the OpenShift Container Platform console.
  8. Complete any postinstallation platform integration steps.

After you meet the cluster nodes and network requirements, you can begin installing the cluster by using the Assisted Installer API. To use the API, you must perform the following procedures:

  • Set up the API authentication.
  • Configure the pull secret.
  • Register a new cluster definition.
  • Create an infrastructure environment for the cluster.

Once you perform these steps, you can modify the cluster definition, create discovery ISOs, add hosts to the cluster, and install the cluster.

5.1. Generating the offline token

Download the offline token from the Assisted Installer web console. You will use the offline token to set the API token.

Prerequisites

Procedure

  1. In the menu, click Downloads.
  2. In the Tokens section under OpenShift Cluster Manager API Token, click View API Token.
  3. Click Load Token.

    Important

    Disable pop-up blockers.

  4. In the Your API token section, copy the offline token.
  5. In your terminal, set the offline token to the OFFLINE_TOKEN variable:

    $ export OFFLINE_TOKEN=<copied_token>
    Tip

    To make the offline token permanent, add it to your profile.

  6. (Optional) Confirm the OFFLINE_TOKEN variable definition.

    $ echo ${OFFLINE_TOKEN}

5.2. Authenticating with the REST API

API calls require authentication with the API token. Assuming you use API_TOKEN as a variable name, add -H "Authorization: Bearer ${API_TOKEN}" to API calls to authenticate with the REST API.

Note

The API token expires after 15 minutes.

Prerequisites

  • You have generated the OFFLINE_TOKEN variable.

Procedure

  1. On the command line terminal, set the API_TOKEN variable using the OFFLINE_TOKEN to validate the user.

    $ export API_TOKEN=$( \
      curl \
      --silent \
      --header "Accept: application/json" \
      --header "Content-Type: application/x-www-form-urlencoded" \
      --data-urlencode "grant_type=refresh_token" \
      --data-urlencode "client_id=cloud-services" \
      --data-urlencode "refresh_token=${OFFLINE_TOKEN}" \
      "https://sso.redhat.com/auth/realms/redhat-external/protocol/openid-connect/token" \
      | jq --raw-output ".access_token" \
    )
  2. Confirm the API_TOKEN variable definition:

    $ echo ${API_TOKEN}
  3. Create a script in your path for one of the token generating methods. For example:

    $ vim ~/.local/bin/refresh-token
    export API_TOKEN=$( \
      curl \
      --silent \
      --header "Accept: application/json" \
      --header "Content-Type: application/x-www-form-urlencoded" \
      --data-urlencode "grant_type=refresh_token" \
      --data-urlencode "client_id=cloud-services" \
      --data-urlencode "refresh_token=${OFFLINE_TOKEN}" \
      "https://sso.redhat.com/auth/realms/redhat-external/protocol/openid-connect/token" \
      | jq --raw-output ".access_token" \
    )

    Then, save the file.

  4. Change the file mode to make it executable:

    $ chmod +x ~/.local/bin/refresh-token
  5. Refresh the API token:

    $ source refresh-token
  6. Verify that you can access the API by running the following command:

    $ curl -s https://api.openshift.com/api/assisted-install/v2/component-versions -H "Authorization: Bearer ${API_TOKEN}" | jq

    Result

    {
      "release_tag": "v2.11.3",
      "versions": {
        "assisted-installer": "registry.redhat.io/rhai-tech-preview/assisted-installer-rhel8:v1.0.0-211",
        "assisted-installer-controller": "registry.redhat.io/rhai-tech-preview/assisted-installer-reporter-rhel8:v1.0.0-266",
        "assisted-installer-service": "quay.io/app-sre/assisted-service:78d113a",
        "discovery-agent": "registry.redhat.io/rhai-tech-preview/assisted-installer-agent-rhel8:v1.0.0-195"
      }
    }

5.3. Configuring the pull secret

Many of the Assisted Installer API calls require the pull secret. Download the pull secret to a file so that you can reference it in API calls. The pull secret is a JSON object that will be included as a value within the request’s JSON object. The pull secret JSON must be formatted to escape the quotes.

For example, convert the pull secret JSON from its original format here:

{"auths":{"cloud.openshift.com": ...

To the following format with escaped quotes for use in an API request:

{\"auths\":{\"cloud.openshift.com\": ...

Procedure

  1. In the menu, click OpenShift.
  2. In the submenu, click Downloads.
  3. In the Tokens section under Pull secret, click Download.
  4. To use the pull secret from a shell variable, execute the following command:

    $ export PULL_SECRET=$(cat ~/Downloads/pull-secret.txt | jq -R .)
  5. To slurp the pull secret file using jq, reference it in the pull_secret variable, piping the value to tojson to ensure that it is properly formatted as escaped JSON. For example:

    $ curl https://api.openshift.com/api/assisted-install/v2/clusters \
    -H "Authorization: Bearer ${API_TOKEN}" \
    -H "Content-Type: application/json" \
    -d "$(jq --null-input \
            --slurpfile pull_secret ~/Downloads/pull-secret.txt '
        {
            "name": "testcluster",
            "control_plane_count": "3",
            "openshift_version": "4.11",
            "pull_secret": $pull_secret[0] | tojson,
            "base_dns_domain": "example.com"
        }
    ')"

    where:

    • --slurpfile pull_secret ~/Downloads/pull-secret.txt slurps the pull secret file.
    • "pull_secret": $pull_secret[0] | tojson formats the pull secret to escaped JSON format.
  6. Confirm the PULL_SECRET variable definition:

    $ echo ${PULL_SECRET}

5.4. Generating the SSH public key

During the installation of OpenShift Container Platform, you can optionally provide an SSH public key to the installation program. This is useful for initiating an SSH connection to a remote node when troubleshooting an installation error.

If you do not have an existing SSH key pair on your local machine to use for the authentication, create one now.

For more information, see Generating a key pair for cluster node SSH access.

Prerequisites

  • Generate the OFFLINE_TOKEN and API_TOKEN variables.

Procedure

  1. From the root user in your terminal, get the SSH public key:

    $ cat /root/.ssh/id_rsa.pub
  2. Set the SSH public key to the CLUSTER_SSHKEY variable:

    $ CLUSTER_SSHKEY=<downloaded_ssh_key>
  3. Confirm the CLUSTER_SSHKEY variable definition:

    $ echo ${CLUSTER_SSHKEY}

5.5. Registering a new cluster

To register a new cluster definition with the API, use the /v2/clusters endpoint.

The following parameters are mandatory:

  • name
  • openshift-version
  • pull_secret
  • cpu_architecture

See the cluster-create-params model in the API viewer for details on the fields you can set when registering a new cluster.

This document does not cover every endpoint of the Assisted Installer API, but you can review all of the endpoints in the API viewer or the swagger.yaml file.

Prerequisites

  • You have generated a valid API_TOKEN. Tokens expire every 15 minutes.
  • You have downloaded the pull secret.
  • Optional: You have assigned the pull secret to the $PULL_SECRET variable.

Procedure

  1. Refresh the API token:

    $ source refresh-token
  2. Register a new cluster by using one of the following methods:

    • Reference the pull secret file in the request:

      $ curl -s -X POST https://api.openshift.com/api/assisted-install/v2/clusters \
        -H "Authorization: Bearer ${API_TOKEN}" \
        -H "Content-Type: application/json" \
        -d "$(jq --null-input \
            --slurpfile pull_secret ~/Downloads/pull-secret.txt ' \
        { \
            "name": "testcluster", \
            "openshift_version": "<version>", \
            "control_plane_count": "<number>", \
            "cpu_architecture" : "<architecture_name>", \
            "base_dns_domain": "example.com", \
            "pull_secret": $pull_secret[0] | tojson \
        } \
        ')" | jq '.id'
    • Write the configuration to a JSON file and then reference it in the request:

      $ cat << EOF > cluster.json
      {
        "name": "testcluster",
        "openshift_version": "<version>",
        "control_plane_count": "<number>",
        "base_dns_domain": "example.com",
        "network_type": "examplenetwork",
        "cluster_network_cidr":"11.111.1.0/14"
        "cluster_network_host_prefix": 11,
        "service_network_cidr": "111.11.1.0/16",
        "api_vips":[{"ip": ""}],
        "ingress_vips": [{"ip": ""}],
        "vip_dhcp_allocation": false,
        "additional_ntp_source": "clock.redhat.com,clock2.redhat.com",
        "ssh_public_key": "$CLUSTER_SSHKEY",
        "pull_secret": $PULL_SECRET
      }
      EOF
      $ curl -s -X POST "https://api.openshift.com/api/assisted-install/v2/clusters" \
        -d @./cluster.json \
        -H "Content-Type: application/json" \
        -H "Authorization: Bearer $API_TOKEN" \
        | jq '.id'
  3. Set the values as follows:

    • Replace <version> with the OpenShift Container Platform version:

      • To install the latest OpenShift version, use the x.y format, such as 4.16 for version 4.16.10. To install a specific OpenShift version, use the x.y.z format, such as 4.16.3.
      • To install a multi-architecture compute cluster, add the -multi extension, such as 4.16-multi for the latest version or 4.16.3-multi for a specific version.
      • If you are booting from an iSCSI drive, enter OpenShift Container Platform version 4.15 or later.
    • Replace <number> with the number of control planes in your cluster. Possible values are as follows:

      • 1 for a single-node OpenShift cluster. Currently, single-node OpenShift is not supported on IBM Power® platform.
      • 2 or more for a Two-Node OpenShift with Arbiter cluster. This option is supported from OpenShift Container Platform 4.19 and later on a Two-Node OpenShift with Arbiter cluster topology. A cluster with 2 control plane nodes must have at least one additional arbiter host. For details, see Two-Node OpenShift with Arbiter resource requirements.
      • 3, 4, or 5 for a multi-node OpenShift Container Platform cluster. If control_plane_count is omitted, the default setting is 3. The Assisted Installer supports 4 or 5 control plane nodes from OpenShift Container Platform 4.18 and later, on a bare metal or user-managed networking platform with an x86_64 CPU architecture. For details, see Control plane count.
    • If you are referencing the pull secret, replace <architecture_name> with the CPU architecture. Options include x86_64, arm64, ppc64le, s390x, or multi. Specify multi for a multi-architecture compute cluster.

      Note

      The control_plane_count field replaces the high_availability_mode field, which is deprecated. For details, see API deprecation notice.

  4. Assign the returned cluster_id to the CLUSTER_ID variable and export it:

    $ export CLUSTER_ID=<cluster_id>
    Note

    If you close your terminal session, you need to export the CLUSTER_ID variable again in a new terminal session.

  5. Check the status of the new cluster:

    $ curl -s -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID" \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer $API_TOKEN" \
      | jq

    Once you register a new cluster definition, create the infrastructure environment for the cluster.

    Note

    You cannot see the cluster configuration settings in the Assisted Installer user interface until you create the infrastructure environment.

5.5.1. Installing Operators

You can customize your deployment by adding Operators to the cluster during installation. You can install one or more Operators individually or add a group of Operators that form a bundle. If you require advanced options, add the Operators after you have installed the cluster.

This step is optional.

5.5.1.1. Installing standalone Operators

Before selecting Operators for installation, you can verify which Operators are available in the Assisted Installer. You can also check whether an Operator is supported for a specific OCP version, CPU architecture, or platform.

You set the required Operator definitions by using the POST method for the assisted-service/v2/clusters/{cluster_id} endpoint and by setting the olm_operators parameter.

The Assisted Installer allows you to install the following Operators:

Expand
CategoryOperatorAPI valueComments

Storage

 

Local Storage Operator (LSO)

lso

Also part of the OpenShift AI Operator bundle and Virtualization bundle.

 

Logical Volume Manager Storage

lvm

 
 

OpenShift Data Foundation

odf

Also part of the OpenShift AI Operator bundle.

 

OpenShift API for Data Protection

oadp

Also part of the Virtualization bundle.

Virtualization

 

OpenShift Virtualization

cnv

This Operator requires backend storage and might automatically activate a storage Operator in the background, according to the following criteria:

  • None - If the CPU architecture is ARM64, no storage Operator is activated.
  • LVM Storage - For single-node OpenShift clusters on any other CPU architecture deploying OpenShift Container Platform 4.12 or higher.
  • Local Storage Operator - For all other deployments.

Currently, OpenShift Virtualization is not supported on IBM Z® and IBM Power®. This Operator is also part of the Virtualization bundle.

 

Migration Toolkit for Virtualization

mtv

  • Installing this Operator automatically activates the OpenShift Virtualization Operator. For a Single-node OpenShift installation, the Assisted Installer also activates the LVM Storage Operator.
  • This Operator is also part of the Virtualization bundle.
 

OpenShift sandboxed containers

osc

 

Artificial Intelligence (AI)

 

OpenShift AI

openshift-ai

Developer Preview feature.

 

AMD GPU

amd-gpu

  • Developer Preview feature.
  • Installing this Operator automatically activates the Kernel Module Management Operator.
 

NVIDIA GPU

nvidia-gpu

  • Developer Preview feature.
  • Installing this Operator automatically activates the Node Feature Discovery Operator.
  • Also part of the OpenShift AI Operator bundle.

Network

 

Kubernetes NMState

nmstate

  • Currently, you cannot install this Operator on the Nutanix or Oracle Cloud Infrastructure (OCI) third-party platforms.
  • Also part of the Virtualization bundle.
 

OpenShift Service Mesh

servicemesh

  • Developer Preview feature.
  • Also part of the OpenShift AI Operator bundle.
 

MetalLB

metallb

  • Developer Preview feature.
  • Also part of the Virtualization Operator bundle.

Remediation

 

Node Health Check

node-healthcheck

Also part of the Virtualization bundle.

 

Fence Agents Remediation

fence-agents-remediation

Also part of the Virtualization bundle.

 

Self Node Remediation

self-node-remediation

You can only install this Operator through the API.

Security & Access Control

 

Authorino

authorino

  • Developer Preview feature.
  • Also part of the OpenShift AI Operator bundle.
 

Kernel Module Management

kmm

Developer Preview feature.

CI/CD & Dev Productivity

 

OpenShift Pipelines

pipelines

  • Developer Preview feature.
  • Also part of the OpenShift AI Operator bundle.
 

OpenShift Serverless

(serverless)

  • Developer Preview feature.
  • Also part of the OpenShift AI Operator bundle.

Platform Operations & Lifecycle

 

Multicluster engine

mce

Deploying the multicluster engine without OpenShift Data Foundation results in the following storage configurations:

  • Multi-node cluster: No storage is configured. You must configure storage after the installation.
  • Single-node OpenShift: LVM Storage is installed.
 

Node Maintenance

node-maintenance

Also part of the Virtualization Operator bundle.

 

Cluster Observability Operator

cluster-observability

  • Install with OpenShift Loki and OpenShift Logging Operators.
  • Also part of the Virtualization Operator bundle.
 

OpenShift Loki

loki

  • Install with {coo} and OpenShift Logging Operators.
  • Also part of the Virtualization Operator bundle.
 

OpenShift Logging

openshift-logging

  • Install with {coo} and OpenShift Loki Operators.
  • Also part of the Virtualization Operator bundle.

Scheduling

 

Node Feature Discovery

node-feature-discovery

Developer Preview feature.

 

Kube Descheduler

kube-descheduler

Also part of the Virtualization Operator bundle.

 

NUMA Resources

numaresources

Also part of the Virtualization Operator bundle.

Important

The integration of some of these Operators into the Assisted Installer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.

For instructions on installing Operator bundles, see Installing bundle Operators.

Prerequisites

Procedure

  1. Optional: Verify which Operators are available in the Assisted Installer, by running the following command:

    $ curl -s "https://api.openshift.com/api/assisted-install/v2/supported-operators" -H "Authorization: Bearer ${API_TOKEN}" | jq .
  2. Optional: Verify whether an Operator is supported for a specific OpenShift Container Platform version by running the following command:

    $ curl -s "https://api.openshift.com/api/assisted-install/v2/support-levels/features?openshift_version=<version-number> -H "Authorization: Bearer ${API_TOKEN}" | jq .features.<OPERATOR-NAME>

    This command accepts the following optional parameters:

    • cpu_architecture: Possible values are x86_64, aarch64, arm64,ppc64le,s390x, or multi.
    • platform_type: Possible values are baremetal, none, nutanix, vsphere, or external.
    • external_platform_name: A third-party platform such as oci, vSphere, nutanix, or ovirt.
    • feature_ids: An array representing specific features. Currently, the only value is "SNO".

      Example input:

      $ curl -s "https://api.openshift.com/api/assisted-install/v2/support-levels/features?openshift_version=4.13&cpu_architecture=x86_64&platform_type=baremetal&external_platform_name=oci&feature_ids=SNO" -H "Authorization: Bearer ${API_TOKEN}" | jq .features.OSC

      Specify the name of the Operator in upper case, for example, .NODE-FEATURE-DISCOVERY for Node Feature Discovery or .OSC for OpenShift sandboxed containers.

      Example output:

      "supported"

      Possible output statuses are supported, dev-preview, tech-preview, and unavailable.

  3. Optional: Remove the .features suffix from the previous command to retrieve a list of supported features for the specified parameters.

    Example input:

    $ curl -X GET "http://api.openshift.com/api/assisted-install/v2/support-levels/features/?openshift_version=4.18.1&cpu_architecture=x86_64&platform_type=none" -s | jq
  4. Optional: Add the /detailed parameter to the previous command to retrieve a detailed response for the specified parameters. When a feature is not supported, the output includes the reason. Operator dependencies appear under dependencies.

    Example input:

    $ curl -X GET "http://api.openshift.com/api/assisted-install/v2/support-levels/features/detailed?openshift_version=4.18.1&cpu_architecture=x86_64&platform_type=none" -s | jq
  5. Specify the Operators to install by running the following command:

    $ curl -s -X POST https://api.openshift.com/api/assisted-install/v2/clusters \
    -H "Authorization: Bearer ${API_TOKEN}" \
    -H "Content-Type: application/json" \
    -d "$(jq --null-input \
       --slurpfile pull_secret ~/Downloads/pull-secret.txt '
    {
       "name": "testcluster",
       "openshift_version": "4.15",
       "cpu_architecture" : "x86_64",
       "base_dns_domain": "example.com",
       "olm_operators": [
      { "name": "mce" }
      ,
      { "name": "odf" }
      ,
     { "name": "amd-gpu" }
        ]
       "pull_secret": $pull_secret[0] | tojson
    }
    ')" | jq '.id'

    In the olm_operators array, list the Operators in the bundle that you want to install, for example cnv for OpenShift Virtualization or mce for multicluster engine. Installing an Operator automatically activates any dependent Operators.

5.5.1.2. Installing Operator bundles

The Assisted Installer supports the following Operator bundles through the web console:

  • Virtualization Operator bundle
  • OpenShift AI Operator bundle

Although the API does not provide an endpoint for installing Operator bundles, you can identify the Operators included in a bundle and install those Operators together.

Important

The OpenShift AI Operator bundle is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.

Prerequisites

Procedure

  1. Optional: Check which Operator bundles are available in the Assisted Installer by running the following command:

    $ curl -s "https://api.openshift.com/api/assisted-install/v2/operators/bundles" -H "Authorization: Bearer ${API_TOKEN}" | jq .

    This command accepts the following optional parameters:

    • openshift_version: The OpenShift Container Platform version number.
    • cpu_architecture: Possible values are x86_64, aarch64, arm64,ppc64le,s390x, or multi.
    • platform_type: Possible values are baremetal, none, nutanix, vsphere, or external.
    • external_platform_name: A third-party platform such as oci, vSphere, nutanix, or ovirt.
    • feature_ids: An array representing specific features. Currently, the only value is "SNO", which supports the OpenShift AI Operator bundle, but not the Virtualization bundle.

    Example input:

    $ curl -X GET "http://127.0.0.1:8090/api/assisted-install/v2/operators/bundles?openshift_version=4.18.1&cpu_architecture=x86_64&platform_type=none&feature_ids=SNO" -s | jq .

    Example output:

    [
      {
        "description": "Train, serve, monitor and manage AI/ML models and applications using GPUs.",
        "id": "openshift-ai",
        "operators": [
          "lvm",
          "amd-gpu",
          "node-feature-discovery",
          "nvidia-gpu",
          "openshift-ai"
        ],
        "title": "OpenShift AI"
      }
    ]
  2. Optional: Check which Operators belong to a specific bundle by running the following command:

    $ curl -s "https://api.openshift.com/api/assisted-install/v2/operators/bundles/<bundle-name>" -H "Authorization: Bearer ${API_TOKEN}" | jq .

    For <bundle-name>, specify virtualization or openshift-ai.

    Example input:

    $ curl -s "https://api.openshift.com/api/assisted-install/v2/operators/bundles/virtualization" -H "Authorization: Bearer ${API_TOKEN}" | jq .

    Example output:

    {
      "description": "Run virtual machines alongside containers on one platform.",
      "id": "virtualization",
      "operators": [
        "kube-descheduler",
        "node-maintenance",
        "mtv",
        "nmstate",
        "fence-agents-remediation",
        "cnv",
        "node-healthcheck",
        "cluster-observability"
        "metallb"
        "numaresources"
        "oadp"
        "loki"
        "openshift-logging"
      ],
      "title": "Virtualization"
    }
  3. Install the Operators associated with the bundle by running the command displayed in the following example:

    $ curl -s -X POST https://api.openshift.com/api/assisted-install/v2/clusters \
    -H "Authorization: Bearer ${API_TOKEN}" \
    -H "Content-Type: application/json" \
    -d "$(jq --null-input \
       --slurpfile pull_secret ~/Downloads/pull-secret.txt '
    {
       "name": "testcluster",
       "openshift_version": "4.15",
       "cpu_architecture" : "x86_64",
       "base_dns_domain": "example.com",
       "olm_operators": [
      { "name": "node-healthcheck" }
    
      { "name": "fence-agents-remediation" }
    
      { "name": "kube-descheduler" }
    
      { "name": "mtv" }
    
      { "name": "nmstate" }
    
      { "name": "node-maintenance" }
    
      { "name": "cnv" }
    
      { "name": "cluster-observability" }
    
      { "name": "metallb" }
    
      { "name": "numaresources" }
    
      { "name": "oadp" }
        ]
         "pull_secret": $pull_secret[0] | tojson
    }
    ')" | jq '.id'

    where:

    • The olm_operators array specifies the Operators in the bundle you are installing. The example lists the Operators for the Virtualization bundle.
    • In the Virtualization bundle, specifying cnv automatically installs lso in the background.
    • In the OpenShift AI Operator bundle:

      • Specifying nvidia-gpu automatically installs node-feature-discovery.
      • Specifying amd-gpu automatically installs kmm.

Use the schedulable_masters attribute to enable workloads to run on control plane nodes.

Prerequisites

  • You have generated a valid API_TOKEN. Tokens expire every 15 minutes.
  • You have created a $PULL_SECRET variable.
  • You are installing OpenShift Container Platform 4.14 or later.

Procedure

  1. Follow the instructions for installing Assisted Installer using the Assisted Installer API.
  2. When you reach the step for registering a new cluster, set the schedulable_masters value to true, as follows:

    $ curl https://api.openshift.com/api/assisted-install/v2/clusters/${CLUSTER_ID} \
    -X PATCH \
    -H "Authorization: Bearer ${API_TOKEN}" \
    -H "Content-Type: application/json" \
    -d '
    {
      "schedulable_masters": true
    }
    ' | jq

5.5.3. Configuring the network management type

You can install the following network management types through the Assisted Installer:

  • Cluster-managed networking
  • User-managed networking
  • Cluster-managed networking with a user-managed load balancer

This step is optional. If you do not define a network management type, the Assisted Installer applies cluster-managed networking by default to all highly available clusters. For single-node OpenShift, the Assisted Installer applies user-managed networking by default.

You define the network management type by adding the user_managed_networking and load_balancer attributes to the cluster definition, as in the following example:

$ curl -s -X POST https://api.openshift.com/api/assisted-install/v2/clusters \
   -H "Authorization: Bearer ${API_TOKEN}" \
   -H "Content-Type: application/json" \
   -d "$(jq --null-input \
      --slurpfile pull_secret ~/Downloads/pull-secret.txt '
   {
      "name": "testcluster",
      "openshift_version": "4.18",
      "cpu_architecture" : "x86_64",
      "base_dns_domain": "example.com",
      "user_managed_networking": "false",
      "load_balancer": {
         "type": "cluster-managed"
      }
   "pull_secret": $pull_secret[0] | tojson
   }
   ')" | jq '.id'

where:

  • user_managed_networking is either true or false.
  • load_balancer can have the type user-managed or cluster-managed.

You can review the user_managed_networking and load_balancer valid values in the swagger.yaml file. For details, see Additional resources.

5.5.3.1. Installing cluster-managed networking

Selecting cluster-managed networking means that the Assisted Installer will configure a standard network topology. This configuration includes an integrated load balancer and virtual routing for managing the API and Ingress VIP addresses. For details, see Cluster-managed networking.

Prerequisites

  • You are installing an OpenShift Container Platform cluster of three or more control plane nodes.

    Note

    Currently, cluster-managed networking is not supported on IBM Z® and IBM Power®.

Procedure

  • To define cluster-managed networking, add the following attributes and values to your cluster definition:

    "user_managed_networking": false,
    "load_balancer": {
            "type": "cluster-managed"
    }

    Where the load_balancer attribute is optional. If omitted for this configuration, the type is automatically set to user-managed for single-node OpenShift or to cluster-managed for all other implementations.

5.5.3.2. Installing user-managed networking

Selecting user-managed networking deploys OpenShift Container Platform with a non-standard network topology. Select user-managed networking if you want to deploy a cluster with an external load balancer and DNS, or if you intend to deploy the cluster nodes across many distinct subnets.

For details, see Cluster-managed networking.

The Assisted Installer lets you deploy more than one external load balancer for user-managed networking.

Note

Oracle Cloud Infrastructure (OCI) is available for OpenShift Container Platform 4.14 with a user-managed networking configuration only.

Procedure

  • To define user-managed networking, add the following attributes to your cluster definition:

    "user_managed_networking": true,
    Note

    The load_balancer attribute is not required when user-managed networking is set to true, because you will be provisioning your own load balancer.

When you enable user-managed networking, the following network validations change:

  • The L3 connectivity check (ICMP) replaces the L2 check (ARP).
  • The maximum transmission unit (MTU) validation verifies the MTU value for all interfaces and not only for the machine network.

Cluster-managed networking with a user-managed load balancer is a hybrid network management type designed for scenarios that require automated cluster networking with external control over load balancing. This approach enables users to provide one or more external load balancers (for example, an API load balancer and an Ingress load balancer), while retaining the bare-metal features installed in cluster-managed networking.

For details, see Cluster-managed networking with a user-managed load balancer.

Important

Cluster-managed networking with a user-managed load balancer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.

Use the Assisted Installer API to deploy cluster-managed networking with a user-managed load balancer on a bare-metal or vSphere platform.

Prerequisites

  • You are installing OpenShift Container Platform version 4.16 or higher.
  • You are installing on a bare-metal or vSphere platform.
  • You are using IPv4 single-stack networking.
  • You are installing an OpenShift Container Platform cluster of three or more control plane nodes.
  • For a vSphere platform installation, you meet the additional requirements specified in vSphere installation requirements.

Procedure

  1. Configure the load balancer to be accessible from all hosts and have access to the following services:

    • OpenShift Machine Config Operator (MCO) - On control plane nodes.
    • OpenShift API - On control plane nodes.
    • Ingress Controller - On compute (worker) nodes.

    For details, see Configuring a user-managed load balancer (steps 1 and 2).

  2. Configure the DNS records for your cluster to target the front-end IP addresses of the user-managed load balancer. You must update records to your DNS server for the cluster API and applications over the load balancer:

    • Configure the DNS record to make your primary API accessible:

      <load_balancer_ip_address> <record_name> api.<cluster_name>.<base_domain>
    • Configure the DNS record to route external traffic to your applications via an ingress controller:

      <load_balancer_ip_address> <record_name> apps.<cluster_name>.<base_domain>
    • For vSphere only, configure the DNS record to support internal API access within your network:

      <load_balancer_ip_address> <record_name> api-int.<cluster_name>.<base_domain>

    For details, see Configuring a user-managed load balancer (step 3).

  3. Add the following configurations to the Assisted Installer API cluster definitions:

    1. Set the user_managed_networking and load_balancer fields to the following values:

      "user_managed_networking": false,
      "load_balancer": {
              "type": "user-managed"
      }

      For details, see Changing the network management type.

    2. Specify the Ingress and API VIPs. These should correspond to the load balancer IP address:

          "ingress_vips": [
              {
                  "cluster_id": "<cluster-id>",
                  "ip": "<load-balancer-ip>"
              }
          ],
          "api_vips": [
              {
                  "cluster_id": "<cluster-id>",
                  "ip": "<load-balancer-ip>"
              }
          ]
    3. Specify a list of machine networks to ensure the following:

      • Each node has at least one network interface card (NIC) with an IP address in at least one machine network.
      • The load balancer IP, which is also the API VIP and Ingress VIP, is included in at least one of the machine networks.

        Example:

         "machine_networks": [
            {
                "cidr": "<hosts-cidr-1>",
                "cluster_id": "<cluster-id>"
            },
            {
                "cidr": "<hosts-cidr-2>",
                "cluster_id": "<cluster-id>"
            },
            {
                "cidr": "<load-balancer-cidr>",
                "cluster_id": "<cluster-id>"
            }
        ]

      For more details, see Machine network.

When you enable this network management type, the following network validations change:

  • The L3 connectivity check (ICMP) replaces the L2 check (ARP).
  • The maximum transmission unit (MTU) validation verifies the MTU value for all interfaces and not only for the machine network.

5.6. Modifying a cluster

To modify a cluster definition with the API, use the /v2/clusters/{cluster_id} endpoint. Modifying a cluster resource is a common operation for adding settings such as changing the network type or enabling user-managed networking. See the v2-cluster-update-params model in the API viewer for details on the fields you can set when modifying a cluster definition.

You can add or remove Operators from a cluster resource that has already been registered.

Note

To create partitions on nodes, see Configuring storage on nodes in the OpenShift Container Platform documentation.

Prerequisites

  • You have created a new cluster resource.

Procedure

  1. Refresh the API token:

    $ source refresh-token
  2. Modify the cluster. For example, change the SSH key:

    $ curl https://api.openshift.com/api/assisted-install/v2/clusters/${CLUSTER_ID} \
    -X PATCH \
    -H "Authorization: Bearer ${API_TOKEN}" \
    -H "Content-Type: application/json" \
    -d '
    {
        "ssh_public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZrD4LMkAEeoU2vShhF8VM+cCZtVRgB7tqtsMxms2q3TOJZAgfuqReKYWm+OLOZTD+DO3Hn1pah/mU3u7uJfTUg4wEX0Le8zBu9xJVym0BVmSFkzHfIJVTn6SfZ81NqcalisGWkpmkKXVCdnVAX6RsbHfpGKk9YPQarmRCn5KzkelJK4hrSWpBPjdzkFXaIpf64JBZtew9XVYA3QeXkIcFuq7NBuUH9BonroPEmIXNOa41PUP1IWq3mERNgzHZiuU8Ks/pFuU5HCMvv4qbTOIhiig7vidImHPpqYT/TCkuVi5w0ZZgkkBeLnxWxH0ldrfzgFBYAxnpTU8Ih/4VhG538Ix1hxPaM6cXds2ic71mBbtbSrk+zjtNPaeYk1O7UpcCw4jjHspU/rVV/DY51D5gSiiuaFPBMucnYPgUxy4FMBFfGrmGLIzTKiLzcz0DiSz1jBeTQOX++1nz+KDLBD8CPdi5k4dq7lLkapRk85qdEvgaG5RlHMSPSS3wDrQ51fD8= user@hostname"
    }
    ' | jq

5.6.1. Modifying Operators by using the API

You can add or remove Operators from a cluster resource that has already been registered as part of a previous installation. This is only possible before you start the OpenShift Container Platform installation.

You modify the required Operator definition by using the PATCH method for the assisted-service/v2/clusters/{cluster_id} endpoint and by setting the olm_operators parameter.

For an overview of each Operator that you intend to install, together with its prerequisites and dependencies, see Customizing your installation using Operators and Operator Bundles.

Prerequisites

  • You have refreshed the API token.
  • You have exported the CLUSTER_ID as an environment variable.

Procedure

  • Run the following command to modify the Operators:

    $ curl https://api.openshift.com/api/assisted-install/v2/clusters/${CLUSTER_ID} \
    -X PATCH \
    -H "Authorization: Bearer ${API_TOKEN}" \
    -H "Content-Type: application/json" \
    -d '
    {
        "olm_operators": [{"name": "mce"}, {"name": "cnv"}],
    }
    ' | jq '.id'
    • In the olm_operators array, enter the complete list of Operators to install, rather than only the differences. The example above includes mce for multicluster engine and cnv for OpenShift Virtualization. For a full list of Operators and their API values, see Installing Operators.
    • To remove all Operators, specify an empty array as follows: "olm_operators": [].
{
  <various cluster properties>,
  "monitored_operators": [
    {
      "cluster_id": "b5259f97-be09-430e-b5eb-d78420ee509a",
      "name": "console",
      "operator_type": "builtin",
      "status_updated_at": "0001-01-01T00:00:00.000Z",
      "timeout_seconds": 3600
    },
    {
      "cluster_id": "b5259f97-be09-430e-b5eb-d78420ee509a",
      "name": "cvo",
      "operator_type": "builtin",
      "status_updated_at": "0001-01-01T00:00:00.000Z",
      "timeout_seconds": 3600
    },
    {
      "cluster_id": "b5259f97-be09-430e-b5eb-d78420ee509a",
      "name": "mce",
      "namespace": "multicluster-engine",
      "operator_type": "olm",
      "status_updated_at": "0001-01-01T00:00:00.000Z",
      "subscription_name": "multicluster-engine",
      "timeout_seconds": 3600
    },
    {
      "cluster_id": "b5259f97-be09-430e-b5eb-d78420ee509a",
      "name": "cnv",
      "namespace": "openshift-cnv",
      "operator_type": "olm",
      "status_updated_at": "0001-01-01T00:00:00.000Z",
      "subscription_name": "hco-operatorhub",
      "timeout_seconds": 3600
    },
    {
      "cluster_id": "b5259f97-be09-430e-b5eb-d78420ee509a",
      "name": "lvm",
      "namespace": "openshift-local-storage",
      "operator_type": "olm",
      "status_updated_at": "0001-01-01T00:00:00.000Z",
      "subscription_name": "local-storage-operator",
      "timeout_seconds": 4200
    }
  ],
  <more cluster properties>
Note

The output is the description of the new cluster state. The monitored_operators property in the output contains Operators of two types:

  • "operator_type": "builtin": Operators of this type are an integral part of OpenShift Container Platform.
  • "operator_type": "olm": Operators of this type are added manually by a user or automatically, as a dependency. In this example, the LVM Storage Operator is added automatically as a dependency of OpenShift Virtualization.

When you install an OpenShift Container Platform cluster on a bare-metal platform, the Machine API (MAPI) included with the platform can interfere with nodes managed externally by the Cluster API (CAPM3). This occurs because both the Machine API and the Cluster API continuously overwrite the providerID attribute, creating a reconciliation loop.

  • The Machine API configures providerID as follows:

    providerID: baremetalhost:///openshift-machine-api/dev-master-0/96ca2aa2-a42c-4312-bc58-b0280655e722
  • The Cluster ID configures providerID as follows:

    providerID: metal3://test-capi/bmh-vm-01/test-sno-vnr4f

Because the Cluster API manages nodes externally, the workload cluster does not require Machine API for node lifecycle management. However, it still requires the networking components provided by the bare-metal platform, such as keepalived and CoreDNS.

To prevent a conflict, install OpenShift Container Platform on a bare-metal platform with the Machine API disabled. This is possible from OpenShift Container Platform 4.19 and later.

Note

You can disable the Machine API at install time only. After the installation is complete, this option is no longer available.

Prerequisites

Procedure

  1. Refresh the API token by running the following command:

    $ source refresh-token
  2. Patch the installation configuration for the bare-metal cluster by running the command in the example below. Update the list of capabilities as needed for your installation, but exclude MachineAPI to disable the Machine API.

    For a list of supported capabilities, see capabilities in the OpenShift Container Platform documentation.

    $ curl https://api.openshift.com/api/assisted-install/v2/clusters/${CLUSTER_ID}/install-config \
    -X PATCH \
    -H "Authorization: Bearer ${API_TOKEN}" \
    -H "Content-Type: application/json" \
    -d '
    {
        "capabilities": {
            "baselineCapabilitySet": "None",
            "additionalEnabledCapabilities": [
                "openshift-samples",
                "marketplace",
                "Console",
                "baremetal",
                "Insights",
                "Storage",
                "NodeTuning",
                "CSISnapshot",
                "OperatorLifecycleManager",
                "Ingress"
            ]
        }
    }
    ' | jq

    After applying this patch, the Assisted Installer generates an installation configuration that supports a bare-metal installation without deploying the Machine API.

5.7. Registering a new infrastructure environment

Once you register a new cluster definition with the Assisted Installer API, create an infrastructure environment using the v2/infra-envs endpoint. Registering a new infrastructure environment requires the following settings:

  • name
  • pull_secret
  • cpu_architecture

See the infra-env-create-params model in the API viewer for details on the fields you can set when registering a new infrastructure environment. You can modify an infrastructure environment after you create it. As a best practice, consider including the cluster_id when creating a new infrastructure environment. The cluster_id will associate the infrastructure environment with a cluster definition. When creating the new infrastructure environment, the Assisted Installer will also generate a discovery ISO.

Prerequisites

  • You have generated a valid API_TOKEN. Tokens expire every 15 minutes.
  • You have downloaded the pull secret.
  • Optional: You have registered a new cluster definition and exported the cluster_id.

Procedure

  1. Refresh the API token:

    $ source refresh-token
  2. Register a new infrastructure environment. Provide a name, preferably something including the cluster name. This example provides the cluster ID to associate the infrastructure environment with the cluster resource. The following example specifies the image_type. You can specify either full-iso or minimal-iso. The default value is minimal-iso.

    1. Optional: You can register a new infrastructure environment by slurping the pull secret file in the request:

      $ curl https://api.openshift.com/api/assisted-install/v2/infra-envs \
      -H "Authorization: Bearer ${API_TOKEN}" \
      -H "Content-Type: application/json" \
      -d "$(jq --null-input \
        --slurpfile pull_secret ~/Downloads/pull-secret.txt \
        --arg cluster_id ${CLUSTER_ID} '
          {
            "name": "testcluster-infra-env",
            "image_type":"full-iso",
            "cluster_id": $cluster_id,
            "cpu_architecture" : "<architecture_name>",
            "pull_secret": $pull_secret[0] | tojson
          }
      ')" | jq '.id'

      Replace <architecture_name> with one of the following CPU architectures: x86_64, arm64, ppc64le, s390x, or multi.

    2. Optional: You can register a new infrastructure environment by writing the configuration to a JSON file and then referencing it in the request:

      $ cat << EOF > infra-envs.json
      {
       "name": "testcluster",
       "pull_secret": $PULL_SECRET,
       "proxy": {
          "http_proxy": "",
          "https_proxy": "",
          "no_proxy": ""
        },
        "ssh_authorized_key": "$CLUSTER_SSHKEY",
        "image_type": "full-iso",
        "cluster_id": "${CLUSTER_ID}",
        "openshift_version": "4.11"
      }
      EOF
      $ curl -s -X POST "https://api.openshift.com/api/assisted-install/v2/infra-envs"
       -d @./infra-envs.json
       -H "Content-Type: application/json"
       -H "Authorization: Bearer $API_TOKEN"
       | jq '.id'
  3. Assign the returned id to the INFRA_ENV_ID variable and export it:

    $ export INFRA_ENV_ID=<id>
    Note

    Once you create an infrastructure environment and associate it to a cluster definition via the cluster_id, you can see the cluster settings in the Assisted Installer web user interface. If you close your terminal session, you need to re-export the id in a new terminal session.

5.8. Modifying an infrastructure environment

You can modify an infrastructure environment using the /v2/infra-envs/{infra_env_id} endpoint. Modifying an infrastructure environment is a common operation for adding settings such as networking, SSH keys, or ignition configuration overrides.

See the infra-env-update-params model in the API viewer for details on the fields you can set when modifying an infrastructure environment. When modifying the new infrastructure environment, the Assisted Installer will also re-generate the discovery ISO.

Prerequisites

  • You have created a new infrastructure environment.

Procedure

  1. Refresh the API token:

    $ source refresh-token
  2. Modify the infrastructure environment:

    $ curl https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID} \
    -X PATCH \
    -H "Authorization: Bearer ${API_TOKEN}" \
    -H "Content-Type: application/json" \
    -d "$(jq --null-input \
      --slurpfile pull_secret ~/Downloads/pull-secret.txt '
        {
          "image_type":"minimal-iso",
          "pull_secret": $pull_secret[0] | tojson
        }
    ')" | jq

5.8.1. Adding kernel arguments

Providing kernel arguments to the Red Hat Enterprise Linux CoreOS (RHCOS) kernel via the Assisted Installer means passing specific parameters or options to the kernel at boot time, particularly when you cannot customize the kernel parameters of the discovery ISO. Kernel parameters can control various aspects of the kernel’s behavior and the operating system’s configuration, affecting hardware interaction, system performance, and functionality. Kernel arguments are used to customize or inform the node’s RHCOS kernel about the hardware configuration, debugging preferences, system services, and other low-level settings.

The RHCOS installer kargs modify command supports the append, delete, and replace options.

You can modify an infrastructure environment using the /v2/infra-envs/{infra_env_id} endpoint. When modifying the new infrastructure environment, the Assisted Installer will also re-generate the discovery ISO.

Procedure

  1. Refresh the API token:

    $ source refresh-token
  2. Modify the kernel arguments:

    $ curl https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID} \
    -X PATCH \
    -H "Authorization: Bearer ${API_TOKEN}" \
    -H "Content-Type: application/json" \
    -d "$(jq --null-input \
      --slurpfile pull_secret ~/Downloads/pull-secret.txt '
        {
          "kernel_arguments": [{ "operation": "append", "value": "<karg>=<value>" }],
          "image_type":"minimal-iso",
          "pull_secret": $pull_secret[0] | tojson
        }
    ')" | jq
    • Replace <karg> with the kernel argument and <value> with the kernel argument value. For example: rd.net.timeout.carrier=60.
    • You can specify multiple kernel arguments by adding a JSON object for each kernel argument.

5.9. Applying a static network configuration

You can optionally apply a static network configuration by using the Assisted Installer API.

Important

A static IP configuration is not supported in the following scenarios:

  • OpenShift Container Platform installations on Oracle Cloud Infrastructure.
  • OpenShift Container Platform installations on iSCSI boot volumes.

Prerequisites

  • You have created an infrastructure environment using the API or have created a cluster using the web console.
  • You have your infrastructure environment ID exported in your shell as $INFRA_ENV_ID.
  • You have credentials to use when accessing the API and have exported a token as $API_TOKEN in your shell.

Procedure

  1. Create YAML files with a static network configuration available as server-a.yaml and server-b.yaml. For guidance, see NMState configuration examples.
  2. Optional: Get the MAC address of the host by running the following command:

    $ ip addr

    You will need this information for the MAC to interface name mapping in the next step.

  3. Create a temporary file /tmp/request-body.txt with the API request and the MAC to interface mapping, as in the example below:

    $ jq -n --arg NMSTATE_YAML1 "$(cat server-a.yaml)" --arg NMSTATE_YAML2 "$(cat server-b.yaml)" \
    '{
      "static_network_config": [
        {
          "network_yaml": $NMSTATE_YAML1,
          "mac_interface_map": [{"mac_address": "02:00:00:2c:23:a5", "logical_nic_name": "eth0"}, {"mac_address": "02:00:00:68:73:dc", "logical_nic_name": "eth1"}]
        },
        {
          "network_yaml": $NMSTATE_YAML2,
          "mac_interface_map": [{"mac_address": "02:00:00:9f:85:eb", "logical_nic_name": "eth1"}, {"mac_address": "02:00:00:c8:be:9b", "logical_nic_name": "eth0"}]
         }
      ]
    }' >> /tmp/request-body.txt

    For more details, see MAC to interface mapping.

  4. Refresh the API token:

    $ source refresh-token
  5. Send the request to the Assisted Service API endpoint:

    $ curl -H "Content-Type: application/json" \
    -X PATCH -d @/tmp/request-body.txt \
    -H "Authorization: Bearer ${API_TOKEN}" \
    https://api.openshift.com/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID

5.10. Adding hosts

After configuring the cluster resource and infrastructure environment, download the discovery ISO image. You can choose from two images:

  • Full ISO image: Use the full ISO image when booting must be self-contained. The image includes everything needed to boot and start the Assisted Installer agent. The ISO image is about 1GB in size. This is the recommended method for the s390x architecture when installing with RHEL KVM.
  • Minimal ISO image: Use the minimal ISO image when the virtual media connection has limited bandwidth. This is the default setting. The image includes only what the agent requires to boot a host with networking. The majority of the content is downloaded upon boot. The ISO image is about 100MB in size.

    This option is mandatory in the following scenarios:

    • If you are installing OpenShift Container Platform on Oracle Cloud Infrastructure.
    • If you are installing OpenShift Container Platform on iSCSI boot volumes.
Note

Currently, ISO images are supported on IBM Z® (s390x) with KVM, iPXE with z/VM, and LPAR (both static and DPM). For details, see Booting hosts using iPXE.

You can boot hosts with the discovery image using three methods. For details, see Booting hosts with the discovery image.

Prerequisites

  • You have created a cluster.
  • You have created an infrastructure environment.
  • You have completed the configuration.
  • If the cluster hosts require the use of a proxy, select Configure cluster-wide proxy settings. Enter the username, password, required domains or IP addresses, and port for the HTTP and HTTPS URLs of the proxy server. If the cluster hosts are behind a firewall, allow the nodes to access the required domains or IP addresses through the firewall. See Configuring your firewall for OpenShift Container Platform for more information.

    Note

    The proxy username and password must be URL-encoded.

  • You have selected an image type or will use the default minimal-iso.

Procedure

  1. Configure the discovery image if needed. For details, see Configuring the discovery image.
  2. Refresh the API token:

    $ source refresh-token
  3. Get the download URL:

    $ curl -H "Authorization: Bearer ${API_TOKEN}" \
    https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/downloads/image-url

    Example output:

    {
      "expires_at": "2024-02-07T20:20:23.000Z",
      "url": "https://api.openshift.com/api/assisted-images/bytoken/<TOKEN>/<OCP_VERSION>/<CPU_ARCHITECTURE>/<FULL_OR_MINIMAL_IMAGE>.iso"
    }
  4. Download the discovery image:

    $ wget -O discovery.iso <url>

    Replace <url> with the download URL from the previous step.

  5. Boot the host(s) with the discovery image.
  6. Assign a role to host(s).

5.10.1. Selecting a role

You can select a role for the host by using the /v2/infra-envs/{infra_env_id}/hosts/{host_id} endpoint. A host can have one of the following roles:

  • master - Assigns the control plane role to the host, allowing the host to manage and coordinate the cluster.
  • arbiter - Assigns the arbiter role to the host, providing a cost-effective solution for components that require a quorum.
  • worker - Assigns the compute role to the host, enabling the host to run application workloads.
  • auto-assign - Automatically determines whether the host is a master, worker, or arbiter. This is the default setting.

Use this procedure to assign a role to the host. If the host_role setting is omitted, the host defaults to auto-assign.

Prerequisites

  • You have added hosts to the cluster.

Procedure

  1. Refresh the API token:

    $ source refresh-token
  2. Get the host IDs:

    $ curl -s -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID" \
    --header "Content-Type: application/json" \
      -H "Authorization: Bearer $API_TOKEN" \
    | jq '.host_networks[].host_ids'

    Example output:

    [
      "1062663e-7989-8b2d-7fbb-e6f4d5bb28e5"
    ]
  3. Add the host_role setting:

    $ curl https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/hosts/<host_id> \
    -X PATCH \
    -H "Authorization: Bearer ${API_TOKEN}" \
    -H "Content-Type: application/json" \
    -d '
        {
          "host_role":"worker"
        }
    ' | jq
    • Replace <host_id> with the ID of the host.
    • For host_role, enter "master", `"arbiter", or "worker". For details, see Supported host roles.

5.10.2. Defining storage disk configuration

Each host retrieved during host discovery can have multiple storage disks. You can optionally change the default configurations for each disk.

Important
  • Starting from OpenShift Container Platform 4.14, you can configure nodes with Intel® Virtual RAID on CPU (VROC) to manage NVMe RAIDs. For details, see Additional resources.
  • Starting from OpenShift Container Platform 4.15, you can install a cluster on a single or multipath iSCSI boot device using the Assisted Installer.
5.10.2.1. Viewing the storage disks

You can view the hosts in your cluster, and the disks on each host. You can then perform actions on a specific disk.

Prerequisites

  • Configure the cluster and discover the hosts.

Procedure

  1. Refresh the API token:

    $ source refresh-token
  2. Get the host IDs for the cluster:

    $ curl -s "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID" \
      -H "Authorization: Bearer $API_TOKEN" \
    | jq '.host_networks[].host_ids'

    Example output:

    "1022623e-7689-8b2d-7fbd-e6f4d5bb28e5"
    Note

    This is the ID of a single host. Multiple host IDs are separated by commas.

  3. Get the disks for a specific host:

    $ curl https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/hosts/<host_id> \
    -H "Authorization: Bearer ${API_TOKEN}" \
    | jq '.inventory | fromjson | .disks'

    Replace <host_id> with the ID of the relevant host.

    Example output:

      [
      {
        "by_id": "/dev/disk/by-id/wwn-0x6c81f660f98afb002d3adc1a1460a506",
        "by_path": "/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:0:0",
        "drive_type": "HDD",
        "has_uuid": true,
        "hctl": "1:2:0:0",
        "id": "/dev/disk/by-id/wwn-0x6c81f660f98afb002d3adc1a1460a506",
        "installation_eligibility": {
          "eligible": true,
          "not_eligible_reasons": null
        },
        "model": "PERC_H710P",
        "name": "sda",
        "path": "/dev/sda",
        "serial": "0006a560141adc3a2d00fb8af960f681",
        "size_bytes": 6595056500736,
        "vendor": "DELL",
        "wwn": "0x6c81f660f98afb002d3adc1a1460a506"
      }
    ]
    Note

    This is the output for one disk. It has the disk_id and installation_eligibility properties for the disk.

5.10.2.2. Changing the installation disk

The Assisted Installer randomly assigns an installation disk by default. If there are multiple storage disks for a host, you can select a different disk to be the installation disk. This automatically unassigns the previous disk.

You can select any disk whose installation_eligibility property is eligible: true to be the installation disk.

Note

Red Hat Enterprise Linux CoreOS (RHCOS) supports multipathing over Fibre Channel on the installation disk, allowing stronger resilience to hardware failure to achieve higher host availability. Multipathing is enabled by default in the agent ISO image, with an /etc/multipath.conf configuration. For details, see Modifying the DM Multipath configuration file.

Prerequisites

  • Configure the cluster and discover the hosts.

Procedure

  1. Get the host and storage disk IDs. For details, see Viewing the storage disks.
  2. Optional: Identify the current installation disk:

    $ curl https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/hosts/<host_id> \
    -H "Authorization: Bearer ${API_TOKEN}" \
    | jq '.installation_disk_id'

    Replace <host_id> with the ID of the relevant host.

  3. Assign a new installation disk:

    Note

    Multipath devices are automatically discovered and listed in the host’s inventory. To assign a multipath Fibre Channel disk as the installation disk, choose a disk with "drive_type" set to "Multipath", rather than to "FC" which indicates a single path.

    $ curl https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/hosts/<host_id> \
    -X PATCH \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer ${API_TOKEN}" \
    
    {
      "disks_selected_config": [
        {
          "id": "<disk_id>",
          "role": "install"
        }
      ]
    }
    • Replace <host_id> with the ID of the host.
    • Replace <disk_id> with the ID of the new installation disk.
5.10.2.3. Disabling disk formatting

The Assisted Installer marks all bootable disks for formatting during the installation process by default, regardless of whether or not they have been defined as the installation disk. Formatting causes data loss.

You can choose to disable the formatting of a specific disk. Disable formatting with caution, as bootable disks can interfere with the installation process, specifically the boot order.

You cannot disable formatting for the installation disk.

Prerequisites

  • Configure the cluster and discover the hosts.

Procedure

  1. Get the host and storage disk IDs. For details, see Viewing the storage disks.
  2. Run the following command:

    $ curl https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/hosts/<host_id> \
    -X PATCH \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer ${API_TOKEN}" \
    
    {
     "disks_skip_formatting": [
       {
         "disk_id": "<disk_id>",
         "skip_formatting": true
       }
     ]
    }
    • Replace <host_id> with the ID of the host.
    • Replace <disk_id> with the ID of the disk. If there is more than one disk, separate the IDs with a comma.
  3. To re-enable formatting, change the skip_formatting value back to false.

5.11. Modifying hosts

After adding hosts, modify the hosts as needed. The most common modifications are to the host_name and the host_role parameters.

You can modify a host by using the /v2/infra-envs/{infra_env_id}/hosts/{host_id} endpoint. See the host-update-params model in the API viewer for details on the fields you can set when modifying a host.

A host might be one of the following roles:

  • master - Assigns the control plane role to the host, allowing the host to manage and coordinate the cluster.
  • arbiter - Assigns the arbiter role to the host, providing a cost-effective solution for components that require a quorum.
  • worker - Assigns the compute role to the host, enabling the host to run application workloads.
  • auto-assign - Automatically determines whether the host is a master, worker, or `arbiter' node.

Use the following procedure to set the host’s role. If the host_role setting is omitted, the host defaults to auto-assign.

Prerequisites

  • You have added hosts to the cluster.

Procedure

  1. Refresh the API token:

    $ source refresh-token
  2. Get the host IDs:

    $ curl -s -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID" \
    --header "Content-Type: application/json" \
      -H "Authorization: Bearer $API_TOKEN" \
    | jq '.host_networks[].host_ids'
  3. Modify the host settings by using the example below:

    $ curl https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/hosts/<host_id> \
    -X PATCH \
    -H "Authorization: Bearer ${API_TOKEN}" \
    -H "Content-Type: application/json" \
    -d '
        {
          "host_role":"worker"
          "host_name" : "worker-1"
        }
    ' | jq

    Replace <host_id> with the ID of the host.

5.12. Adding manifests and patches

The Assisted Installer supports the following manifest types:

  • Custom manifests - A custom manifest is a JSON or YAML file that contains advanced configurations not currently supported in the Assisted Installer. You can create a custom manifest or use one provided by a third party.
  • Manifest patches - A manifest patch is used to adjust configurations, manage updates, or apply changes in a structured and automated way. Its purpose is to update a system manifest that is automatically created by the Assisted Installer during installation preparation.

You can get and add custom manifests and manifest patches by using the Assisted Installer API.

5.12.1. Getting manifests

You can get a list of manifests stored in the installation configuration to aid you when adding, removing, replacing, or patching manifests. To get a manifest with the API, use the /v2/clusters/$CLUSTER_ID/manifests endpoint.

Prerequisites

  • You have registered a new cluster definition and exported the cluster_id to the $CLUSTER_ID BASH variable.
  • You have generated a valid API_TOKEN. Tokens expire every 15 minutes.

Procedure

  1. Refresh the API token:

    $ source refresh-token
  2. Get the list of manifests by running the following command:

    $ curl "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID/manifests?include_system_generated=true" \
        -H "Authorization: Bearer $API_TOKEN" \
        | jq

    Example output:

    [
      {
        "file_name": "manifest.json",
        "folder": "manifests",
        "manifest_source": "user"
      }
    ]

    If there are no manifests, you receive an HTTP 404 "Object not found" error.

    Example output:

    {
      "code": "404",
      "href": "",
      "id": 404,
      "kind": "Error",
      "reason": "Object Not Found"
    }
  3. Get a specific manifest by running the following command:

    $ curl -X 'GET' \
      'http://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_I/manifests/files?folder=<folder_name>&file_name=<file_name>' \
      -H "Authorization: Bearer $API_TOKEN" \
      -H 'accept: application/octet-stream'

    If the file does not exist, you receive an HTTP 404 "Folder/file doesn’t exist in cluster" error.

    Example output:

    {
      "code": "404",
      "href": "",
      "id": 404,
      "kind": "Error",
      "reason": "Cluster manifest <folder_name>/<file_name>.json doesn't exist in cluster $CLUSTER_ID"
    }
  4. Delete a specific manifest by running the following command:

    curl -X 'DELETE' \
      'http://api.openshift.com/api/assisted-install/v2/clusters/c07b16ec-7eed-45d1-9ccd-88d2e86c4087/manifests?folder=manifests&file_name=manifest.json' \
      -H "Authorization: Bearer $API_TOKEN" \
      -H 'accept: application/json'

5.12.2. Adding custom manifests

A custom manifest is a JSON or YAML file that contains advanced configurations not currently supported in the Assisted Installer user interface. You can create a custom manifest or use one provided by a third party. To create a custom manifest with the API, use the /v2/clusters/$CLUSTER_ID/manifests endpoint.

You can upload a base64-encoded custom manifest to either the openshift folder or the manifests folder with the Assisted Installer API. There is no limit to the number of custom manifests permitted.

You can only upload one base64-encoded JSON manifest at a time. However, each uploaded base64-encoded YAML file can contain multiple custom manifests. Uploading a multi-document YAML manifest is faster than adding the YAML files individually.

For a file containing a single custom manifest, accepted file extensions include .yaml, .yml, or .json.

{
    "apiVersion": "machineconfiguration.openshift.io/v1",
    "kind": "MachineConfig",
    "metadata": {
        "labels": {
            "machineconfiguration.openshift.io/role": "primary"
        },
        "name": "10_primary_storage_config"
    },
    "spec": {
        "config": {
            "ignition": {
                "version": "3.2.0"
            },
            "storage": {
                "disks": [
                    {
                        "device": "</dev/xxyN>",
                        "partitions": [
                            {
                                "label": "recovery",
                                "startMiB": 32768,
                                "sizeMiB": 16384
                            }
                        ]
                    }
                ],
                "filesystems": [
                    {
                        "device": "/dev/disk/by-partlabel/recovery",
                        "label": "recovery",
                        "format": "xfs"
                    }
                ]
            }
        }
    }
}

For a file containing multiple custom manifests, accepted file types include .yaml or .yml.

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: 99-openshift-machineconfig-master-kargs
spec:
  kernelArguments:
    - loglevel=7
---
apiVersion: machineconfiguration.openshift.io/v2
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 98-openshift-machineconfig-worker-kargs
spec:
  kernelArguments:
    - loglevel=5
Note
  • When you install OpenShift Container Platform on the Oracle Cloud Infrastructure (OCI) external platform, you must add the custom manifests provided by Oracle. For additional external partner integrations such as vSphere or Nutanix, this step is optional.
  • For more information about custom manifests, see Preparing custom manifests and manifest patches.

Prerequisites

  • You have generated a valid API_TOKEN. Tokens expire every 15 minutes.
  • You have registered a new cluster definition and exported the cluster_id to the $CLUSTER_ID BASH variable.

Procedure

  1. Create a custom manifest file.
  2. Save the custom manifest file using the appropriate extension for the file format.
  3. Refresh the API token:

    $ source refresh-token
  4. Add the custom manifest to the cluster by executing the following command:

    $ curl -X POST "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID/manifests" \
        -H "Authorization: Bearer $API_TOKEN" \
        -H "Content-Type: application/json" \
        -d '{
                "file_name":"manifest.json",
                "folder":"manifests",
                "content":"'"$(base64 -w 0 ~/manifest.json)"'"
        }' | jq

    Replace manifest.json with the name of your manifest file. The second instance of manifest.json is the path to the file. Ensure the path is correct.

    Example output:

    {
      "file_name": "manifest.json",
      "folder": "manifests"
    }
    Note

    The base64 -w 0 command base64-encodes the manifest as a string and omits carriage returns. Encoding with carriage returns will generate an exception.

  5. Verify that the Assisted Installer added the manifest:

    $ curl -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID/manifests/files?folder=manifests&file_name=manifest.json" -H "Authorization: Bearer $API_TOKEN"

    Replace manifest.json with the name of your manifest file.

5.12.3. Adding manifest patches

A manifest patch modifies a system manifest that is automatically created by the Assisted Installer before installation. You can use a manifest patch to change the folder name, file name, or contents of the generated manifest.

Patches must follow the syntax of a YAML patch and use the .yml.patch or yaml.patch extension. The extension can include an additional suffix, consisting of an underscore followed by alphanumeric characters, for example .yml.patch_ocp_manifests or .yaml.patch_ocp_manifests.

For details, see Preparing custom manifests and manifest patches.

Prerequisites

  • You have registered a new cluster definition and exported the cluster_id to the $CLUSTER_ID BASH variable.
  • You have generated a valid API_TOKEN. Tokens expire every 15 minutes.

Procedure

  1. Refresh the API token by running the following command:

    $ source refresh-token
  2. Create a patch file, such as ~/mypatch.yml.patch, with this content:

    - op: replace
      path: /status/infrastructureTopology
      value: HighlyAvailable

    Example:

    $ cat <<EOF > ~/mypatch.yml.patch
    - op: replace
      path: /status/infrastructureTopology
      value: HighlyAvailable
    EOF
  3. Run the following command to upload the patch file, setting file_name to the name of the file you created:

    $ curl -X POST "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID/manifests" \
        -H "Authorization: Bearer $API_TOKEN" \
        -H "Content-Type: application/json" \
        -d '{
                "file_name":"mypatch.yml.patch",
                "folder":"manifests",
                "content":"'"$(base64 -w 0 ~/mypatch.yml.patch)"'"
        }' | jq

5.13. Installing the cluster

Once the cluster hosts past validation, you can install the cluster.

Prerequisites

  • You have created a cluster and infrastructure environment.
  • You have added hosts to the infrastructure environment.
  • You have ensured that the cluster and hosts have passed validation. For details, see Preinstallation validations.

Procedure

  1. Refresh the API token:

    $ source refresh-token
  2. Install the cluster:

    $ curl -H "Authorization: Bearer $API_TOKEN" \
    -X POST \
    https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID/actions/install | jq
  3. Complete any postinstallation platform integration steps.

Chapter 6. Enabling disk encryption

You can enable encryption of installation disks by using either the TPM v2 or Tang encryption modes.

Note

In some situations, when you enable TPM disk encryption in the firmware for a bare-metal host and then boot it from an ISO that you generate with the Assisted Installer, the cluster deployment can get stuck. This can happen if there are left-over TPM encryption keys from an earlier installation on the host. For more information, see "Bug BZ#2011634" in Additional resources. If you experience this problem, contact Red Hat Support.

6.1. Enabling TPM v2 encryption

You can enable Trusted Platform Module (TPM) v2 encryption from the Assisted Installer web console or API.

Prerequisites

  • Ensure that TPM v2 encryption is enabled in the BIOS firmware on each host. Most Dell systems require this. Check the manual for your computer. The Assisted Installer will also validate that TPM is enabled in the firmware. See the disk-encryption model in the Assisted Installer API for additional details.
  • Verify that a TPM v2 encryption chip is installed on each node and enabled in the firmware.

Procedure

  1. Optional: Using the web console, in the Cluster details step, enable the encryption of installation disks for any of the following nodes: control plane nodes, workers, or arbiter.
  2. Optional: Using the API, include the following settings in the "Modifying hosts" procedure to enable TPM v2 encryption:

    1. Set the disk_encryption.enable_on setting to one of the following:

      • "none"
      • "all"
      • "masters"
      • "arbiters"
      • "workers"
      • "masters,arbiters"
      • "masters,workers"
      • "arbiters,workers"
      • "masters,arbiters,workers"
    2. Set the disk_encryption.mode setting to tpmv2.

      Example

      $ curl https://api.openshift.com/api/assisted-install/v2/clusters/${CLUSTER_ID} \
      -X PATCH \
      -H "Authorization: Bearer ${API_TOKEN}" \
      -H "Content-Type: application/json" \
      -d '
      {
        "disk_encryption": {
          "enable_on": "none",
          "mode": "tpmv2"
        }
      }
      ' | jq

6.2. Enabling Tang encryption

You can enable Tang encryption from the Assisted Installer web console or API.

Prerequisites

  • You have access to a Red Hat Enterprise Linux (RHEL) 8 machine that you can use to generate a thumbprint of the Tang exchange key.

Procedure

  1. Set up a Tang server or access an existing one. See Network-bound disk encryption for instructions. You can set multiple Tang servers, but the Assisted Installer must be able to connect to all of them during installation.
  2. On the Tang server, retrieve the thumbprint for the Tang server using tang-show-keys:

    $ tang-show-keys <port>

    Optional: Replace <port> with the port number. The default port number is 80.

    Example thumbprint

    1gYTN_LpU9ZMB35yn5IbADY5OQ0

  3. Optional: Retrieve the thumbprint for the Tang server using jose.

    1. Ensure jose is installed on the Tang server:

      $ sudo dnf install jose
    2. On the Tang server, retrieve the thumbprint using jose:

      $ sudo jose jwk thp -i /var/db/tang/<public_key>.jwk

      Replace <public_key> with the public exchange key for the Tang server.

      Example thumbprint

      1gYTN_LpU9ZMB35yn5IbADY5OQ0

  4. Optional: Using the web console, in the Cluster details step, enable the encryption of installation disks for any of the following nodes: control plane nodes, workers, or arbiter. You will be required to enter URLs and thumbprints for the Tang servers.
  5. Optional: Using the API, include the following settings in the "Modifying hosts" procedure to enable Tang encryption:

    1. Set the disk_encryption.enable_on setting to one of the following:

      • "none"
      • "all"
      • "masters"
      • "arbiters"
      • "workers"
      • "masters,arbiters"
      • "masters,workers"
      • "arbiters,workers"
      • "masters,arbiters,workers"
    2. Set the disk_encryption.mode setting to tang.
    3. Set disk_encyrption.tang_servers to provide the URL and thumbprint details about one or more Tang servers. Within the tang_servers value, comment out the quotes within the object(s).

      Example

      $ curl https://api.openshift.com/api/assisted-install/v2/clusters/${CLUSTER_ID} \
      -X PATCH \
      -H "Authorization: Bearer ${API_TOKEN}" \
      -H "Content-Type: application/json" \
      -d '
      {
        "disk_encryption": {
          "enable_on": "all",
          "mode": "tang",
          "tang_servers": "[{\"url\":\"http://tang.example.com:7500\",\"thumbprint\":\"PLjNyRdGw03zlRoGjQYMahSZGu9\"},{\"url\":\"http://tang2.example.com:7500\",\"thumbprint\":\"XYjNyRdGw03zlRoGjQYMahSZGu3\"}]"
        }
      }
      ' | jq

Chapter 7. Configuring the discovery image

The Assisted Installer uses a preconfigured discovery image to run an agent that performs hardware and network validations before installing OpenShift Container Platform. You can customize this discovery image by using a utility called Ignition. Changes to the discovery image are not saved and apply only during the current validation run.

7.1. Creating an Ignition configuration file

Ignition is a low-level system configuration utility, which is part of the temporary initial root filesystem, the initramfs. When Ignition runs on the first boot, it finds configuration data in the Ignition configuration file and applies it to the host before switch_root is called to pivot to the host’s root filesystem.

Ignition uses a JSON configuration specification file to represent the set of changes that occur on the first boot.

Important

Ignition versions newer than 3.2 are not supported, and will raise an error.

Procedure

  1. Create an Ignition file and specify the configuration specification version:

    $ vim ~/ignition.conf
    {
      "ignition": { "version": "3.1.0" }
    }
  2. Add configuration data to the Ignition file. For example, add a password to the core user.

    1. Generate a password hash:

      $ openssl passwd -6
    2. Add the generated password hash to the core user:

      {
        "ignition": { "version": "3.1.0" },
        "passwd": {
          "users": [
            {
              "name": "core",
              "passwordHash": "$6$spam$M5LGSMGyVD.9XOboxcwrsnwNdF4irpJdAWy.1Ry55syyUiUssIzIAHaOrUHr2zg6ruD8YNBPW9kW0H8EnKXyc1"
            }
          ]
        }
      }
  3. Save the Ignition file and export it to the IGNITION_FILE variable:

    $ export IGNITION_FILE=~/ignition.conf

7.2. Modifying the discovery image with Ignition

After you create an Ignition configuration file, you can modify the discovery image by patching the infrastructure environment through the Assisted Installer API.

Prerequisites

  • If you used the web console to create the cluster, you have set up the API authentication.
  • You have an infrastructure environment and you have exported the infrastructure environment id to the INFRA_ENV_ID variable.
  • You have a valid Ignition file and have exported the file name as $IGNITION_FILE.

Procedure

  1. Create an ignition_config_override JSON object and redirect it to a file:

    $ jq -n \
      --arg IGNITION "$(jq -c . $IGNITION_FILE)" \
      '{ignition_config_override: $IGNITION}' \
      > discovery_ignition.json
  2. Refresh the API token:

    $ source refresh-token
  3. Patch the infrastructure environment:

    $ curl \
      --header "Authorization: Bearer $API_TOKEN" \
      --header "Content-Type: application/json" \
      -XPATCH \
      -d @discovery_ignition.json \
      https://api.openshift.com/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID | jq

    The ignition_config_override object references the Ignition file.

  4. Download the updated discovery image.

Chapter 8. Booting hosts with the discovery image

The Assisted Installer uses an initial image to run an agent that performs hardware and network validations before attempting to install OpenShift Container Platform. You can boot hosts with the discovery image by using a USB drive, Redfish virtual media, or iPXE.

8.1. Creating an ISO image on a USB drive

You can install the Assisted Installer agent by using a USB drive that contains the discovery ISO image. Starting the host with the USB drive prepares the host for the software installation.

Procedure

  1. On the administration host, insert a USB drive into a USB port.
  2. Copy the ISO image to the USB drive, for example:

    # dd if=<path_to_iso> of=<path_to_usb> status=progress
    • Replace <path_to_iso> with the relative path to the downloaded discovery ISO file, for example, discovery.iso.
    • Replace <path_to_usb> with the location of the connected USB drive, for example, /dev/sdb.

    After you copy the ISO to the USB drive, you can use the USB drive to install the Assisted Installer agent on the cluster host.

8.2. Booting with a USB drive

To register nodes with the Assisted Installer using a bootable USB drive, use the following procedure.

Procedure

  1. Insert the RHCOS discovery ISO USB drive into the target host.
  2. Configure the boot drive order in the server firmware settings to boot from the attached discovery ISO, and then reboot the server.
  3. Wait for the host to boot up.

    • For web console installations, on the administration host, return to the browser. Wait for the host to appear in the list of discovered hosts.
    • For API installations, refresh the token, check the enabled host count, and gather the host IDs:

      $ source refresh-token
      $ curl -s -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID" \
      --header "Content-Type: application/json" \
        -H "Authorization: Bearer $API_TOKEN" \
      | jq '.enabled_host_count'
      $ curl -s -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID" \
      --header "Content-Type: application/json" \
        -H "Authorization: Bearer $API_TOKEN" \
      | jq '.host_networks[].host_ids'

      The command returns output similar to the following example:

      [
        "1062663e-7989-8b2d-7fbb-e6f4d5bb28e5"
      ]

You can provision hosts in your network by using ISOs that you install using the Redfish Baseboard Management Controller (BMC) API.

Prerequisites

  • Download the installation Red Hat Enterprise Linux CoreOS (RHCOS) ISO.

Procedure

  1. Copy the ISO file to an HTTP server accessible in your network.
  2. Boot the host from the hosted ISO file, for example:

    1. Call the redfish API to set the hosted ISO as the VirtualMedia boot media by running the following command:

      $ curl -k -u <bmc_username>:<bmc_password> \
      -d '{"Image":"<hosted_iso_file>", "Inserted": true}' \
      -H "Content-Type: application/json" \
      -X POST <host_bmc_address>/redfish/v1/Managers/iDRAC.Embedded.1/VirtualMedia/CD/Actions/VirtualMedia.InsertMedia
      • Replace <bmc_username>:<bmc_password> with the username and password for the target host BMC.
      • Replace <hosted_iso_file> with the URL for the hosted installation ISO, for example: https://example.com/rhcos-live-minimal.iso. The ISO must be accessible from the target host machine.
      • Replace <host_bmc_address> with the BMC IP address of the target host machine.
    2. Set the host to boot from the VirtualMedia device by running the following command:

      $ curl -k -u <bmc_username>:<bmc_password> \
      -X PATCH -H 'Content-Type: application/json' \
      -d '{"Boot": {"BootSourceOverrideTarget": "Cd", "BootSourceOverrideMode": "UEFI", "BootSourceOverrideEnabled": "Once"}}' \
      <host_bmc_address>/redfish/v1/Systems/System.Embedded.1
    3. Reboot the host:

      $ curl -k -u <bmc_username>:<bmc_password> \
      -d '{"ResetType": "ForceRestart"}' \
      -H 'Content-type: application/json' \
      -X POST <host_bmc_address>/redfish/v1/Systems/System.Embedded.1/Actions/ComputerSystem.Reset
    4. Optional: If the host is powered off, you can boot it using the {"ResetType": "On"} switch. Run the following command:

      $ curl -k -u <bmc_username>:<bmc_password> \
      -d '{"ResetType": "On"}' -H 'Content-type: application/json' \
      -X POST <host_bmc_address>/redfish/v1/Systems/System.Embedded.1/Actions/ComputerSystem.Reset

8.4. Booting hosts using iPXE

The Assisted Installer provides an iPXE script including all of the artifacts needed to boot the discovery image for an infrastructure environment. Due to the limitations of the current HTTPS implementation of iPXE, the recommendation is to download and expose the needed artifacts in an HTTP server. Currently, even if iPXE supports HTTPS protocol, the supported algorithms are old and not recommended.

The full list of supported ciphers is in https://ipxe.org/crypto.

Prerequisites

  • You have created an infrastructure environment by using the API or you have created a cluster by using the web console.
  • You have your infrastructure environment ID exported in your shell as $INFRA_ENV_ID.
  • You have credentials to use when accessing the API and have exported a token as $API_TOKEN in your shell.
Note

If you configure iPXE by using the web console, the $INFRA_ENV_ID and $API_TOKEN variables are preset.

  • You have an HTTP server to host the images.
Note

IBM Power® only supports PXE, which has the following requirements:

  • GRUB2 installed at /var/lib/tftpboot
  • DHCP and TFTP for PXE

Procedure

  1. Download the iPXE script directly from the web console, or get the iPXE script from the Assisted Installer by running the following command:

    $ curl \
      --silent \
      --header "Authorization: Bearer $API_TOKEN" \
      https://api.openshift.com/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID/downloads/files?file_name=ipxe-script > ipxe-script

    The following example shows the iPXE script format:

    #!ipxe
    initrd --name initrd http://api.openshift.com/api/assisted-images/images/<infra_env_id>/pxe-initrd?arch=x86_64&image_token=<token_string>&version=4.10
    kernel http://api.openshift.com/api/assisted-images/boot-artifacts/kernel?arch=x86_64&version=4.10 initrd=initrd coreos.live.rootfs_url=http://api.openshift.com/api/assisted-images/boot-artifacts/rootfs?arch=x86_64&version=4.10 random.trust_cpu=on rd.luks.options=discard ignition.firstboot ignition.platform.id=metal console=tty1 console=ttyS1,115200n8 coreos.inst.persistent-kargs="console=tty1 console=ttyS1,115200n8"
    boot
  2. Download the required artifacts by extracting URLs from the ipxe-script:

    1. Download the initial RAM disk by running the following command:

      $ awk '/^initrd /{print $NF}' ipxe-script \
      | xargs curl -o initrd.img -L
    2. Download the Linux kernel by running the following command:

      $ awk '/^kernel /{print $2}' ipxe-script | xargs curl -o kernel -L
    3. Download the root filesystem by running the following command:

      $ grep ^kernel ipxe_script | xargs -n1 | grep ^coreos.live.rootfs_url | cut -d = -f 2,3,4 | xargs curl -o rootfs.img -L
  3. Change the URLs to the different artifacts in the ipxe-script to match your local HTTP server. For example:

    #!ipxe
    set webserver http://192.168.0.1
    initrd --name initrd $webserver/initrd.img
    kernel $webserver/kernel initrd=initrd coreos.live.rootfs_url=$webserver/rootfs.img random.trust_cpu=on rd.luks.options=discard ignition.firstboot ignition.platform.id=metal console=tty1 console=ttyS1,115200n8 coreos.inst.persistent-kargs="console=tty1 console=ttyS1,115200n8"
    boot
  4. Optional: When installing with RHEL KVM on IBM Z® you must boot the host by specifying additional kernel arguments:

    random.trust_cpu=on rd.luks.options=discard ignition.firstboot ignition.platform.id=metal console=tty1 console=ttyS1,115200n8 coreos.inst.persistent-kargs="console=tty1 console=ttyS1,115200n8
    Note

    When you install with iPXE on RHEL KVM, the VMs on the VM host might not start on the first boot. You must start them manually.

  5. Optional: When installing on IBM Power® you must download the initramfs, kernel, and root images as follows:

    1. Copy the initrd.img and kernel.img images to the /var/lib/tftpboot/rhcos PXE directory.
    2. Copy the rootfs.img to the /var/www/html/install HTTPD directory.
    3. Add the following entry to the /var/lib/tftpboot/boot/grub2/grub.cfg directory:

      if [ ${net_default_mac} == fa:1d:67:35:13:20 ]; then
      default=0
      fallback=1
      timeout=1
      menuentry "CoreOS (BIOS)" {
      echo "Loading kernel"
      linux "/rhcos/kernel.img" ip=dhcp rd.neednet=1 ignition.platform.id=metal ignition.firstboot coreos.live.rootfs_url=http://9.114.98.8:8000/install/rootfs.img
      echo "Loading initrd"
      initrd "/rhcos/initrd.img"
      }
      fi

Chapter 9. Preinstallation validations

Before starting an installation, the Assisted Installer runs a series of checks to verify that the cluster configuration meets all requirements. You can review validation results and resolve any issues before proceeding with installation.

9.1. Overview of preinstallation validations

The Assisted Installer aims to make cluster installation as simple, efficient, and error-free as possible. It performs validation checks on the configuration and the gathered telemetry before starting an installation.

The Assisted Installer uses the information provided before installation, such as control plane topology, network configuration and hostnames. It will also use real time telemetry from the hosts you are attempting to install.

When a host boots the discovery ISO, an agent will start on the host. The agent will send information about the state of the host to the Assisted Installer.

The Assisted Installer uses all of this information to compute real-time preinstallation validations.

9.1.1. Host and cluster validations

The Assisted Installer performs validations at both the host and cluster levels.

Host
Host validations ensure that the configuration of a given host is valid for installation.
Cluster
Cluster validations ensure that the configuration of the whole cluster is valid for installation.

9.1.2. Blocking and non-blocking validations

All validations are either blocking or non-blocking to the installation.

Blocking
A blocking validation prevents progress of the installation, meaning that you need to resolve the issue and pass the blocking validation before you can proceed.
Non-blocking
A non-blocking validation is a warning and will tell you of things that might cause you a problem.

9.2. Host validations

You can retrieve and review validation results for each discovered host.

Run the following procedure to retrieve host validation results by using the REST API.

If you use the web console, many of these validations will not show up by name. To get a list of validations consistent with the labels, use the following procedure.

Prerequisites

  • You have installed the jq utility.
  • You have created an Infrastructure Environment by using the API or have created a cluster by using the web console.
  • You have hosts booted with the discovery ISO
  • You have your Cluster ID exported in your shell as CLUSTER_ID.
  • You have credentials to use when accessing the API and have exported a token as API_TOKEN in your shell.

Procedure

  1. Refresh the API token:

    $ source refresh-token
  2. Get all validations for all hosts:

    $ curl \
      --silent \
      --header "Authorization: Bearer $API_TOKEN" \
      https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID/hosts \
      | jq -r .[].validations_info \
      | jq 'map(.[])'
  3. Get non-passing validations for all hosts:

    $ curl \
      --silent \
      --header "Authorization: Bearer $API_TOKEN" \
      https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID/hosts \
      | jq -r .[].validations_info \
      | jq 'map(.[]) | map(select(.status=="failure" or .status=="pending")) | select(length>0)'

9.2.2. Host validations in detail

The Assisted Installer performs the following host validations.

Expand
ParameterValidation typeDescription

connected

non-blocking

Checks that the host has recently communicated with the Assisted Installer.

has-inventory

non-blocking

Checks that the Assisted Installer received the inventory from the host.

has-min-cpu-cores

non-blocking

Checks that the number of CPU cores meets the minimum requirements.

has-min-memory

non-blocking

Checks that the amount of memory meets the minimum requirements.

has-min-valid-disks

non-blocking

Checks that at least one available disk meets the eligibility criteria.

has-cpu-cores-for-role

blocking

Checks that the number of cores meets the minimum requirements for the host role.

has-memory-for-role

blocking

Checks that the amount of memory meets the minimum requirements for the host role.

ignition-downloadable

blocking

For Day 2 hosts, checks that the host can download ignition configuration from the Day 1 cluster.

belongs-to-majority-group

blocking

The majority group is the largest full-mesh connectivity group on the cluster, where all members can communicate with all other members. [role="_abstract"] This validation checks that hosts in a multi-node, Day 1 cluster are in the majority group.

valid-platform-network-settings

blocking

Checks that the platform is valid for the network settings.

ntp-synced

non-blocking

Checks if an NTP server has been successfully used to synchronize time on the host.

container-images-available

non-blocking

Checks if container images have been successfully pulled from the image registry.

sufficient-installation-disk-speed

blocking

Checks that disk speed metrics from an earlier installation meet requirements, if they exist.

sufficient-network-latency-requirement-for-role

blocking

Checks that the average network latency between hosts in the cluster meets the requirements.

sufficient-packet-loss-requirement-for-role

blocking

Checks that the network packet loss between hosts in the cluster meets the requirements.

has-default-route

blocking

Checks that the host has a default route configured.

api-domain-name-resolved-correctly

blocking

For a multi node cluster with user managed networking. Checks that the host is able to resolve the API domain name for the cluster.

api-int-domain-name-resolved-correctly

blocking

For a multi node cluster with user managed networking. Checks that the host is able to resolve the internal API domain name for the cluster.

apps-domain-name-resolved-correctly

blocking

For a multi node cluster with user managed networking. Checks that the host is able to resolve the internal apps domain name for the cluster.

compatible-with-cluster-platform

non-blocking

Checks that the host is compatible with the cluster platform

dns-wildcard-not-configured

blocking

Checks that the wildcard DNS *.<cluster_name>.<base_domain> is not configured, because this causes known problems for OpenShift

disk-encryption-requirements-satisfied

non-blocking

Checks that the type of host and disk encryption configured meet the requirements.

non-overlapping-subnets

blocking

Checks that this host does not have any overlapping subnets.

hostname-unique

blocking

Checks that the hostname is unique in the cluster.

hostname-valid

blocking

Checks the validity of the hostname, meaning that it matches the general form of hostnames and is not forbidden.

  • The hostname must have 63 characters or less.
  • The hostname must start and end with a lowercase alphanumeric character.
  • The hostname must have only lowercase alphanumeric characters, dashes, and periods.

belongs-to-machine-cidr

blocking

Checks that the host IP is in the address range of the machine CIDR.

lso-requirements-satisfied

blocking

Validates that the host meets the requirements of the Local Storage Operator.

odf-requirements-satisfied

blocking

Validates that the host meets the requirements of the OpenShift Data Foundation Operator.

  • Each host running ODF workloads (control plane nodes in compact mode, compute nodes in standard mode) requires an eligible disk. This is a disk with at least 25GB that is not the installation disk and is of type SSD or HDD.
  • All hosts must have manually assigned roles.

cnv-requirements-satisfied

blocking

Validates that the host meets the requirements of Container Native Virtualization.

  • The BIOS of the host must have CPU virtualization enabled.
  • Host must have enough CPU cores and RAM available for Container Native Virtualization.
  • Will validate the Host Path Provisioner if necessary.

lvm-requirements-satisfied

blocking

Validates that the host meets the requirements of the Logical Volume Manager Storage Operator.

  • Host has at least one additional empty disk, not partitioned and not formatted.

vsphere-disk-uuid-enabled

non-blocking

Verifies that each valid disk sets disk.EnableUUID to TRUE. In vSphere this will result in each disk having a UUID.

compatible-agent

blocking

Checks that the discovery agent version is compatible with the agent docker image version.

no-skip-installation-disk

blocking

Checks that installation disk is not skipping disk formatting.

no-skip-missing-disk

blocking

Checks that all disks marked to skip formatting are in the inventory. A disk ID can change on reboot, and this validation prevents issues caused by that.

media-connected

blocking

Checks the connection of the installation media to the host.

machine-cidr-defined

non-blocking

Checks that the machine network definition exists for the cluster.

id-platform-network-settings

blocking

Checks that the platform is compatible with the network settings. Some platforms are only permitted when installing Single Node Openshift or when using User Managed Networking.

mtu-valid

non-blocking

Checks the maximum transmission unit (MTU) of hosts and networking devices in the cluster environment to identify compatibility issues. For details, see Changing the MTU for the cluster network.

9.3. Cluster validations

You can retrieve and review validation results for the cluster.

Run the following procedure to retrieve cluster validation results by using the REST API.

If you use the web console, many of these validations will not show up by name. To obtain a list of validations consistent with the labels, use the following procedure.

Prerequisites

  • You have installed the jq utility.
  • You have created an Infrastructure Environment by using the API or have created a cluster by using the web console.
  • You have your Cluster ID exported in your shell as CLUSTER_ID.
  • You have credentials to use when accessing the API and have exported a token as API_TOKEN in your shell.

Procedure

  1. Refresh the API token:

    $ source refresh-token
  2. Get all cluster validations:

    $ curl \
      --silent \
      --header "Authorization: Bearer $API_TOKEN" \
      https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID \
      | jq -r .validations_info \
      | jq 'map(.[])'
  3. Get non-passing cluster validations:

    $ curl \
      --silent \
      --header "Authorization: Bearer $API_TOKEN" \
      https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID \
      | jq -r .validations_info \
      | jq '. | map(.[] | select(.status=="failure" or .status=="pending")) | select(length>0)'

9.3.2. Cluster validations in detail

The Assisted Installer performs the following cluster validations.

Expand
ParameterValidation typeDescription

machine-cidr-defined

non-blocking

Checks that the machine network definition exists for the cluster.

cluster-cidr-defined

non-blocking

Checks that the cluster network definition exists for the cluster.

service-cidr-defined

non-blocking

Checks that the service network definition exists for the cluster.

no-cidrs-overlapping

blocking

Checks that the defined networks do not overlap.

networks-same-address-families

blocking

Checks that the defined networks share the same address families (valid address families are IPv4, IPv6)

network-prefix-valid

blocking

Checks the cluster network prefix to ensure that it is valid and allows enough address space for all hosts.

machine-cidr-equals-to-calculated-cidr

blocking

For a non user managed networking cluster. Checks that apiVIPs or ingressVIPs are members of the machine CIDR if they exist.

api-vips-defined

non-blocking

For a non user managed networking cluster. Checks that apiVIPs exist.

api-vips-valid

blocking

For a non user managed networking cluster. Checks if the apiVIPs belong to the machine CIDR and are not in use.

ingress-vips-defined

blocking

For a non user managed networking cluster. Checks that ingressVIPs exist.

ingress-vips-valid

non-blocking

For a non user managed networking cluster. Checks if the ingressVIPs belong to the machine CIDR and are not in use.

all-hosts-are-ready-to-install

blocking

Checks that all hosts in the cluster are in the "ready to install" status.

sufficient-masters-count

blocking [role="_abstract"]

  • For a multi-node OpenShift Container Platform installation, checks that the current number of hosts in the cluster designated either manually or automatically to be control plane (master) nodes equals the number that the user defined for the cluster as the control_plane_count value.
  • For a single-node OpenShift installation, checks that there is exactly one control plane (master) node and no compute (worker) nodes.

dns-domain-defined

non-blocking

Checks that the base DNS domain exists for the cluster.

pull-secret-set

non-blocking

Checks that the pull secret exists. Does not check that the pull secret is valid or authorized.

ntp-server-configured

blocking

Checks that each of the host clocks are no more than 4 minutes out of sync with each other.

lso-requirements-satisfied

blocking

Validates that the cluster meets the requirements of the Local Storage Operator.

odf-requirements-satisfied

blocking

Validates that the cluster meets the requirements of the OpenShift Data Foundation Operator.

  • The cluster has either at least three control plane (master) nodes and no compute (worker) nodes at all (compact mode), or at least three control plane (master) nodes and at least three compute (worker) nodes (standard mode).
  • Each host running ODF workloads (control plane nodes in compact mode, compute nodes in standard mode) requires a non-installation disk of type SSD` or HDD and with at least 25GB of storage.
  • All hosts must have manually assigned roles.

cnv-requirements-satisfied

blocking

Validates that the cluster meets the requirements of Container Native Virtualization.

  • The CPU architecture for the cluster is x86

lvm-requirements-satisfied

blocking

Validates that the cluster meets the requirements of the Logical Volume Manager Storage Operator.

  • The cluster must be single node.
  • The cluster must be running Openshift >= 4.11.0.

network-type-valid

blocking

Checks the validity of the network type if it exists.

  • The network type must be OpenshiftSDN (OpenShift Container Platform 4.14 or earlier) or OVNKubernetes.
  • OpenshiftSDN does not support IPv6 or Single Node Openshift. OpenshiftSDN is not supported for OpenShift Container Platform 4.15 and later releases.
  • OVNKubernetes does not support VIP DHCP allocation.

Chapter 10. Network configuration

The following sections describe the basics of network configuration with the Assisted Installer.

10.1. Cluster networking requirements

OpenShift Container Platform uses the network types and addresses listed in the following table.

Expand
TypeDNSDescription

clusterNetwork

 

The IP address pools from which pod IP addresses are allocated.

serviceNetwork

 

The IP address pool for services.

machineNetwork

 

The IP address blocks for machines that form the cluster. For a dual-stack configuration, specify both a primary and a secondary machineNetwork. The first network listed determines the primary stack, which can be either IPv4 or IPv6.

apiVIPs

api.<clustername.clusterdomain>

The VIPs to use for API communication. You must provide this setting or preconfigure the address in the DNS so that the default name resolves correctly. For dual-stack networking, specify one IPv4 address and one IPv6 address, with the primary address listed first.

ingressVIPs

*.apps.<clustername.clusterdomain>

The VIPs to use for ingress traffic. For dual-stack networking, specify one IPv4 address and one IPv6 address, with the primary address listed first.

Currently, the Assisted Service can deploy OpenShift Container Platform clusters by using one of the following configurations:

  • Single-stack IPv4. IPv6 is not currently supported in a single-stack configuration.
  • Dual-stack IPv4 and IPv6, with either address family as primary.

Open Virtual Network (OVN) is the default Container Network Interface (CNI) in OpenShift Container Platform 4.12 and later releases. Software-Defined Networking (SDN) is supported up to OpenShift Container Platform 4.14 only.

Important

Support for IPv6 as the primary stack in a dual-stack configuration is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.

10.1.1. Networking Limitations

Cluster networking has the following limitations.

Software Defined Networking (SDN)
  • The SDN controller is not supported with single-node OpenShift.
  • The SDN controller does not support dual-stack networking.
  • The SDN controller is not supported for OpenShift Container Platform 4.15 and later releases. For more information, see Deprecation of the OpenShift SDN network plugin in the OpenShift Container Platform release notes.
OVN-Kubernetes
For more information, see About the OVN-Kubernetes network plugin.

10.1.2. Cluster network

The cluster network is a network from which every pod deployed in the cluster gets its IP address. Given that the workload might live across many nodes forming the cluster, it is important for the network provider to be able to easily find an individual node based on the pod’s IP address. To do this, clusterNetwork.cidr is further split into subnets of the size defined in clusterNetwork.hostPrefix.

The host prefix specifies a length of the subnet assigned to each individual node in the cluster. An example of how a cluster might assign addresses for the multi-node cluster:

  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23

Creating a 3-node cluster by using this snippet might create the following network topology:

  • Pods scheduled in node #1 get IPs from 10.128.0.0/23
  • Pods scheduled in node #2 get IPs from 10.128.2.0/23
  • Pods scheduled in node #3 get IPs from 10.128.4.0/23

Explaining OVN-Kubernetes internals is out of scope for this document. However, the pattern described earlier provides a way to route Pod-to-Pod traffic between different nodes without keeping a big list of mapping between Pods and their corresponding nodes.

10.1.3. Machine network

Machine networks are IP networks that connect all the cluster nodes within OpenShift Container Platform.

The Assisted Installer supports a single machine network for most cluster installations. In such cases, the Assisted Installer automatically determines the appropriate machine network based on the API and Ingress virtual IPs (VIPs) that you specify.

The Assisted Installer supports two machine networks in the following scenarios:

  • For dual stack configurations, the Assisted Installer automatically allocates two machine networks, based on the IPv4 and IPv6 subnets and the API and Ingress VIPs that you specify.
  • For iSCSI boot volumes, the hosts are automatically connected over two machine networks: one designated for the OpenShift Container Platform installation and the other for iSCSI traffic. During the installation process, ensure that you select the OpenShift Container Platform network. Using the iSCSI network will result in an error for the host.

The Assisted Installer supports multiple machine networks for the "cluster-managed networking with a user-managed load balancer" network management type. When installing this network management type, you must manually define the machine networks in the API cluster definitions, with the following conditions:

  • Each node must have at least one network interface in at least one machine network.
  • The load balancer IPs (VIPs) should be included in at least one of the machine networks.

Currently, you can install cluster-managed networking with a user-managed load balancer using the Assisted Installer API only.

Depending on whether you are deploying single-node OpenShift or a multi-node OpenShift Container Platform cluster, different values are mandatory. The following table explains this in more detail.

Expand
ParameterSingle-node OpenShiftMulti-node cluster with DHCP modeMulti-node cluster without DHCP mode

clusterNetwork

Required

Required

Required

serviceNetwork

Required

Required

Required

machineNetwork

Auto-assign possible (*)

Auto-assign possible (*)

Auto-assign possible (*)

apiVIP

Forbidden

Forbidden

Required

apiVIPs

Forbidden

Forbidden

Required in 4.12 and later releases

ingressVIP

Forbidden

Forbidden

Required

ingressVIPs

Forbidden

Forbidden

Required in 4.12 and later releases

(*) Auto assignment of the machine network CIDR happens if there is only a single host network. Otherwise you need to specify it explicitly.

10.1.5. Air-gapped environments

The workflow for deploying a cluster without Internet access has some prerequisites, which are out of scope of this document. You can consult Zero Touch Provisioning the hard way Git repository for some insights.

10.2. Network management types

The Assisted Installer supports the following network management types.

For details on changing the network management type, see either of the following sections:

10.2.1. Cluster-managed networking

Cluster-managed networking is the default option for deploying OpenShift Container Platform clusters. It minimizes user intervention by automatically provisioning and managing key network components.

The main characteristics of cluster-managed networking are the following:

  • Integrates automated load balancing and virtual routing for managing the Virtual IP (VIP) addresses to ensure redundancy.
  • Automatically supports an extensive internal DNS (CoreDNS) for service discovery.
  • Hosts all control plane nodes within a single, contiguous subnet, simplifying routing and connectivity within the cluster.
  • Supports the installation of platform-specific features such as the Bare Metal Operator for bare metal.
  • Available for clusters with three or more control plane nodes; not available for single-node OpenShift.

You can configure cluster-managed networking both the web console or API. If you do not define a network management type, the Assisted Installer applies cluster-managed networking automatically for highly available clusters.

10.2.2. User-managed networking

User-managed networking allows customers with custom or non-standard network topologies to deploy OpenShift Container Platform clusters. It provides control and flexibility, allowing you to integrate OpenShift Container Platform with existing and complex network infrastructures.

The main characteristics of user-managed networking are the following:

  • Allows users to configure one or more external load balancers for handling API and Ingress IP addresses.
  • Enables control plane nodes to span multiple subnets.
  • Can be deployed on both single-node OpenShift and high-availability clusters.

You can configure user-managed networking in both the Assisted Installer web console or API.

Cluster-managed networking with a user-managed load balancer is a hybrid network management type designed for scenarios that require automated cluster networking with external control over load balancing.

This approach combines elements from both cluster-managed and user-managed networking. The main characteristics of this network management type are as follows:

  • Allows users to configure one or more external load balancers for handling API and Ingress IP addresses.
  • Automatically supports an extensive internal DNS (CoreDNS) for service discovery.
  • Enables control plane nodes to span multiple subnets.
  • Supports the installation of platform specific features such as the Bare Metal Operator for bare metal.
  • Provides high fault tolerance and disaster recovery for the control plane nodes.

The Assisted Installer supports cluster-managed networking with a user-managed load balancer on a bare-metal or vSphere platform. Currently you can configure this network management type through the API only.

Important

Cluster-managed networking with a user-managed load balancer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.

10.3. VIP DHCP allocation

The VIP DHCP allocation is a feature allowing users to skip the requirement of manually providing virtual IPs for API and Ingress by leveraging the ability of a service to automatically assign those IP addresses from the DHCP server.

If you enable the VIP DHCP allocation feature, the service will not use the api_vips and ingress_vips defined in the cluster configuration. Instead, it will request IP addresses from the DHCP server on the machine network and use the assigned VIPs accordingly.

Important

VIP DHCP allocation is currently limited to the OpenShift Container Platform SDN network type. SDN is not supported from OpenShift Container Platform version 4.15 and later. Therefore, support for VIP DHCP allocation is also ending from OpenShift Container Platform 4.15 and later.

10.3.1. Enabling VIP DHCP allocation

You can enable automatic VIP allocation through DHCP.

This is not an OpenShift Container Platform feature and it has been implemented in the Assisted Service to make the configuration easier. For a more detailed explanation of the syntax for the VIP addresses, see Installer-provisioned infrastructure for a bare-metal installation.

Procedure

  1. Follow the instructions for registering a new cluster by using the API. For details, see Registering a new cluster.
  2. Add the following payload settings to the cluster configuration:

    1. Set vip_dhcp_allocation to true.
    2. Set network_type to OpenShiftSDN.
    3. Include your network configurations for cluster_networks, service_networks, and machine_networks.

    Example payload to enable auto-allocation:

    $ cat << EOF > payload.json
    
    {
      "vip_dhcp_allocation": true,
      "network_type": "OpenShiftSDN",
      "user_managed_networking": false,
      "cluster_networks": [
        {
          "cidr": "10.128.0.0/14",
          "host_prefix": 23
        }
      ],
      "service_networks": [
        {
          "cidr": "172.30.0.0/16"
        }
      ],
      "machine_networks": [
        {
          "cidr": "192.168.127.0/24"
        }
      ]
    }
    
    EOF
  3. Submit the payload to the Assisted Service API to apply the configuration, by running the following command:

    $ curl -s -X POST "https://api.openshift.com/api/assisted-install/v2/clusters/<cluster-id>" \
      -d @./payload.json \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer $API_TOKEN" \
      | jq '.id'

10.3.2. Disabling VIP DHCP allocation

If you want to manually control your VIP assignments, you can disable VIP DHCP allocation.

Procedure

  1. Follow the instructions for registering a new cluster by using the API. For details, see Registering a new cluster.
  2. Add the following payload settings to the cluster configuration:

    1. Set vip_dhcp_allocation to false.
    2. Specify the IP addresses for api_vips and ingress_vips. You can take these IPs from your machine_networks configuration.
    3. Set network_type to OVNKubernetes, OpenShiftSDN, or another supported SDN type, if applicable.
    4. Include your network configurations for cluster_networks and service_networks.

    Example payload to disable autoallocation:

    $ cat << EOF > payload.json
    
    {
      "api_vips": [
        {
            "ip": "192.168.127.100"
        }
      ],
      "ingress_vips": [
        {
            "ip": "192.168.127.101"
        }
      ],
      "vip_dhcp_allocation": false,
      "network_type": "OVNKubernetes",
      "user_managed_networking": false,
      "cluster_networks": [
        {
          "cidr": "10.128.0.0/14",
          "host_prefix": 23
        }
      ],
      "service_networks": [
        {
          "cidr": "172.30.0.0/16"
        }
      ]
    }
    
    EOF
  3. Submit the payload to the Assisted Service API to apply the configuration, by running the following command:

    $ curl -s -X POST "https://api.openshift.com/api/assisted-install/v2/clusters/<cluster-id>" \
      -d @./payload.json \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer $API_TOKEN" \
      | jq '.id'

10.4. Static network configuration

You can define static network configurations for each host through either the Assisted Installer web console or API. The Assisted Installer applies the settings to the Discovery ISO when you create a new ISO or update an existing one.

When using the API or the YAML view in the web console, create one or more NMState YAML files and map each host MAC address to its corresponding network interface name.

The Form view in the web console does not require these steps.

10.4.1. NMState configuration examples

The NMState YAML file specifies the required static network configuration for the host, including interface details, IP addresses, routes, and DNS settings. The Assisted Installer replaces the logical interface names (for example, eth0) with the actual names during host discovery.

The following examples show NMState YAML configurations that you can copy and adapt. For more examples, see the NMState documentation.

For details on applying the static networking configurations in the Assisted Installer, see Configuring static networks (web console) or Applying a static network configuration (API).

Standard NMState configuration example

This example shows a standard static network configuration with a default route and DNS server.

dns-resolver:
  config:
    server:
    - 192.168.126.1
interfaces:
- ipv4:
    address:
    - ip: 192.168.126.30
      prefix-length: 24
    dhcp: false
    enabled: true
  name: eth0
  state: up
  type: ethernet
- ipv4:
    address:
    - ip: 192.168.141.30
      prefix-length: 24
    dhcp: false
    enabled: true
  name: eth1
  state: up
  type: ethernet
routes:
  config:
  - destination: 0.0.0.0/0
    next-hop-address: 192.168.126.1
    next-hop-interface: eth0
    table-id: 254
Tagged VLAN example

Replace the relevant section of the standard YAML as follows to define a tagged VLAN interface on top of a physical network interface (NIC).

Important

This example and the next one show part of the YAML file only and are not meant for use as-is. Using them incorrectly can cause your machines to lose network connectivity.

interfaces:
  - ipv4:
      address:
      - ip: 192.168.143.15
        prefix-length: 24
      dhcp: false
      enabled: true
    ipv6:
      enabled: false
    name: eth0.404
    state: up
    type: vlan
    vlan:
      base-iface: eth0
      id: 404
      reorder-headers: true
Network bond example

Replace the relevant section of the standard YAML as follows to configure a network bond for redundancy by using the active-backup mode.

interfaces:
- ipv4:
    address:
    - ip: 192.168.138.15
      prefix-length: 24
    dhcp: false
    enabled: true
    ipv6:
      enabled: false
    link-aggregation:
      mode: active-backup
      options:
        miimon: "140"
      port:
      - eth0
      - eth1
    name: bond0
    state: up
    type: bond

10.4.2. MAC-to-NIC mapping examples

Each host requires a mapping between its MAC addresses and corresponding network interface cards (NICs). This mapping serves two main purposes:

  • To identify the correct node on which to apply YAML file.
  • To replace logical/temporary interface names, such as eth0 or ens3, in cases where the YAML file does not already use physical network interface names or identifier: mac-address.

You define the MAC-to-NIC mapping configurations in the NMState YAML file when using the Assisted Installer API for the installation.

If you are using the YAML view of the web console for the installation, this mapping is not required. Instead, you specify the mapping manually in the MAC to interface name mapping fields. For details, see Configuring static networks using YAML view (web console).

Example of MAC interface mapping with logical interface names

In this example, the mapping identifies the node and replaces the temporary interface name.

  • YAML file:

    dns-resolver:
      config:
        server:
        - 192.168.126.1
    interfaces:
    - ipv4:
        address:
        - ip: 192.168.126.30
          prefix-length: 24
        dhcp: false
        enabled: true
      name: eth0
      state: up
      type: ethernet
    - ipv4:
        address:
        - ip: 192.168.141.30
          prefix-length: 24
        dhcp: false
        enabled: true
      name: eth1
      state: up
      type: ethernet
    routes:
      config:
      - destination: 0.0.0.0/0
        next-hop-address: 192.168.126.1
        next-hop-interface: eth0
        table-id: 254
  • MAC mapping:

    mac_interface_map: [
        {
          mac_address: 02:00:00:2c:23:a5,
          logical_nic_name: eth0
        },
        {
          mac_address: 02:00:00:68:73:dc,
          logical_nic_name: eth1
        }
    ]
Example of MAC interface mapping with identifier: mac-address interface names

In this example, the NMState YAML configuration contains identifier: mac-address. This means the mapping only needs to specify a single MAC address to identify one node.

  • YAML file:

    dns-resolver:
      config:
        server:
        - 192.168.126.1
    interfaces:
      - name: eth0
        type: ethernet
        state: up
        identifier: mac-address
        mac-address: 1e:bd:23:e9:fb:94
        ipv4:
          enabled: true
          dhcp: true
        ipv6:
          enabled: true
          dhcp: true
          autoconf: true
    routes:
      config:
      - destination: 0.0.0.0/0
        next-hop-address: 192.168.126.1
        next-hop-interface: eth0
        table-id: 254
  • MAC mapping:

    mac_interface_map: [
        {
          mac_address: 1e:bd:23:e9:fb:95,
          logical_nic_name: test
        },
    ]

10.5. Converting to dual-stack networking

A dual-stack configuration enables clusters to host pods across both IPv4 and IPv6 subnets.

You configure dual-stack by specifying both IPv4 and IPv6 network address families in the configuration file. The order in which you list the IPv4 and IPv6 values determines the primary and secondary stack. The order must remain consistent across all networking parameters, including the machine network, cluster network, service network, API VIP, and Ingress VIP.

In the examples, listing the IPv4 network first makes it the primary stack, with IPv6 as the secondary stack. To set IPv6 as the primary stack, reverse the IPv4 and IPv6 values.

Important

Support for IPv6 as the primary stack in a dual-stack configuration is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.

Before starting, ensure that you are familiar with Converting to IPv4/IPv6 dual stack networking.

Example payload for a single-node OpenShift cluster
{
  "network_type": "OVNKubernetes",
  "user_managed_networking": false,
  "cluster_networks": [
    {
      "cidr": "10.128.0.0/14",
      "host_prefix": 23
    },
    {
      "cidr": "fd01::/48",
      "host_prefix": 64
    }
  ],
  "service_networks": [
    {"cidr": "172.30.0.0/16"}, {"cidr": "fd02::/112"}
  ],
  "machine_networks": [
    {"cidr": "192.168.127.0/24"},{"cidr": "1001:db8::/120"}
  ]
}
Example payload for a multi-node OpenShift cluster
{
  "vip_dhcp_allocation": false,
  "network_type": "OVNKubernetes",
  "user_managed_networking": false,
  "api_vips": [
     {
        "ip": "192.168.127.100"
     },
     {
        "ip": "2001:0db8:85a3:0000:0000:8a2e:0370:7334"
     }
  ],
  "ingress_vips": [
     {
        "ip": "192.168.127.101"
     },
     {
        "ip": "2001:0db8:85a3:0000:0000:8a2e:0370:7335"
     }
  ],
  "cluster_networks": [
    {
      "cidr": "10.128.0.0/14",
      "host_prefix": 23
    },
    {
      "cidr": "fd01::/48",
      "host_prefix": 64
    }
  ],
  "service_networks": [
    {"cidr": "172.30.0.0/16"}, {"cidr": "fd02::/112"}
  ],
  "machine_networks": [
    {"cidr": "192.168.127.0/24"},{"cidr": "1001:db8::/120"}
  ]
}

Chapter 11. Expanding the cluster

You can expand a cluster installed with the Assisted Installer by adding hosts through the user interface or the API.

11.1. Checking for multi-architecture support

You must check that your cluster can support multiple architectures before you add a node with a different architecture.

Procedure

  1. Log in to the cluster using the CLI.
  2. Check that your cluster uses the architecture payload by running the following command:

    $ oc adm release info -o json | jq .metadata.metadata

Verification

  • If you see the following output, your cluster supports multiple architectures:

    {
      "release.openshift.io/architecture": "multi"
    }

A cluster with an x86_64 or arm64 control plane can support worker nodes that have two different CPU architectures. Multi-architecture clusters combine the strengths of each architecture and support a variety of workloads.

For example, you can add arm64, IBM Power® (ppc64le), or IBM Z® (s390x) worker nodes to an existing OpenShift Container Platform cluster with an x86_64.

The main steps of the installation are as follows:

  1. Create and register a multi-architecture compute cluster.
  2. Create an x86_64 or arm64 infrastructure environment, download the ISO discovery image for the environment, and add the control plane. An arm64 infrastructure environment is available for Amazon Web Services (AWS) and Google Cloud (GC) only.
  3. Create an arm64, ppc64le, or s390x infrastructure environment, download the ISO discovery images for arm64, ppc64le, or s390x, and add the worker nodes.

For the supported platforms for each OpenShift Container Platform version, see About clusters with multi-architecture compute machines. Use the appropriate platforms for the version you are installing.

Procedure

  1. Start the procedure for installing OpenShift Container Platform using the API. For details, see Installing with the Assisted Installer API.
  2. When you reach the Registering a new cluster step of the installation, register the cluster as a multi-architecture compute cluster:

    $ curl -s -X POST https://api.openshift.com/api/assisted-install/v2/clusters \
    -H "Authorization: Bearer ${API_TOKEN}" \
    -H "Content-Type: application/json" \
    -d "$(jq --null-input \
       --slurpfile pull_secret ~/Downloads/pull-secret.txt '
    {
       "name": "testcluster",
       "openshift_version": "<version-number>-multi",
       "cpu_architecture" : "multi"
       "control_plane_count": "<number>"
       "base_dns_domain": "example.com",
       "pull_secret": $pull_secret[0] | tojson
    }
    ')" | jq '.id'
    • Use the multi- option for openshift-version; for example, "4.20-multi".
    • Set cpu_architecture to "multi".
    • Set control_plane_count to "3", "4", or "5". The option of 4 or 5 control plane nodes is available from OpenShift Container Platform 4.18 and later. Single-node OpenShift is not supported for a multi-architecture compute cluster. The control_plane_count field replaces high_availability_mode, which is deprecated.
  3. When you reach the Registering a new infrastructure environment step of the installation, set cpu_architecture to "x86_64":

    $ curl https://api.openshift.com/api/assisted-install/v2/infra-envs \
    -H "Authorization: Bearer ${API_TOKEN}" \
    -H "Content-Type: application/json" \
    -d "$(jq --null-input \
     --slurpfile pull_secret ~/Downloads/pull-secret.txt \
     --arg cluster_id ${CLUSTER_ID} '
       {
         "name": "testcluster-infra-env",
         "image_type":"full-iso",
         "cluster_id": $cluster_id,
         "cpu_architecture" : "x86_64"
         "pull_secret": $pull_secret[0] | tojson
       }
    ')" | jq '.id'
  4. When you reach the Adding hosts step of the installation, set host_role to "master":

    Note

    For details, see Supported host roles.

    $ curl https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/hosts/<host_id> \
    -X PATCH \
    -H "Authorization: Bearer ${API_TOKEN}" \
    -H "Content-Type: application/json" \
    -d '
       {
         "host_role":"master"
       }
    ' | jq
  5. Download the discovery image for the x86_64 architecture.
  6. Boot the x86_64 architecture hosts using the generated discovery image.
  7. Start the installation and wait for the cluster to be fully installed.
  8. Repeat the Registering a new infrastructure environment step of the installation. This time, set cpu_architecture to one of the following: "ppc64le" (for IBM Power®), "s390x" (for IBM Z®), or "arm64". For example:

    $ curl -s -X POST https://api.openshift.com/api/assisted-install/v2/clusters \
    -H "Authorization: Bearer ${API_TOKEN}" \
    -H "Content-Type: application/json" \
    -d "$(jq --null-input \
       --slurpfile pull_secret ~/Downloads/pull-secret.txt '
    {
       "name": "testcluster",
       "openshift_version": "4.12",
       "cpu_architecture" : "arm64"
       "control_plane_count": "3"
       "base_dns_domain": "example.com",
       "pull_secret": $pull_secret[0] | tojson
    }
    ')" | jq '.id'
  9. Repeat the Adding hosts step of the installation. This time, set host_role to "worker":

    Note

    For details, see Supported host roles.

    $ curl https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/hosts/<host_id> \
    -X PATCH \
    -H "Authorization: Bearer ${API_TOKEN}" \
    -H "Content-Type: application/json" \
    -d '
       {
         "host_role":"worker"
       }
    ' | jq
  10. Download the discovery image for the arm64, ppc64le or s390x architecture.
  11. Boot the architecture hosts using the generated discovery image.
  12. Start the installation and wait for the cluster to be fully installed.

To verify the installation, view the arm64, ppc64le, or s390x worker nodes in the cluster by running the following command:

$ oc get nodes -o wide

11.3. Adding hosts with the web console

You can add hosts to clusters that were created using the Assisted Installer.

Note

When you add new worker nodes to an existing cluster on Day 2 using the Assisted Installer, these nodes do not become part of an existing machine set. As a result, the Machine API will not automatically scale or replace them. To remove these nodes, you must do so manually, rather than through the machine set.

For details on removing nodes in OpenShift Container Platform, see Manually scaling a compute machine set for nodes managed as part of a machine set and Deleting a machine for un-managed nodes that are not part of a machine set.

Prerequisites

  • Ensure that you are installing OpenShift Container Platform version 4.11 and later.
  • Ensure that the new node shares the same subnet as the Day 1 network. The subnet is specified in the machineNetwork field of the install-config.yaml file. This requirement applies to cluster-managed networks such as bare metal or vSphere, and not to user-managed networks.

Procedure

  1. Log in to OpenShift Cluster Manager and click the cluster that you want to expand.
  2. Click Add hosts and download the discovery ISO for the new host, adding an SSH public key and configuring cluster-wide proxy settings as needed.
  3. Optional: Modify ignition files as needed.
  4. Boot the target host using the discovery ISO, and wait for the host to be discovered in the console.
  5. Select the host role. It can be either a worker or a control plane host.
  6. Start the installation.
  7. As the installation proceeds, the installation generates pending certificate signing requests (CSRs) for the host. When prompted, approve the pending CSRs to complete the installation.

    When the host is successfully installed, it is listed as a host in the cluster web console.

    Important

    New hosts will be encrypted using the same method as the original cluster.

11.4. Adding hosts with the API

You can add hosts to clusters using the Assisted Installer REST API.

Note

When you add new worker nodes to an existing cluster on Day 2 using the Assisted Installer, these nodes do not become part of an existing machine set. As a result, the Machine API will not automatically scale or replace them. To remove these nodes, you must do so manually, rather than through the machine set.

For details on removing nodes in OpenShift Container Platform, see Manually scaling a compute machine set for nodes managed as part of a machine set and Deleting a machine for un-managed nodes that are not part of a machine set.

Prerequisites

  • Install the Red Hat OpenShift Cluster Manager CLI (ocm).
  • Log in to Red Hat OpenShift Cluster Manager as a user with cluster creation privileges.
  • Install jq.
  • Ensure that all the required DNS records exist for the cluster that you want to expand.
  • Ensure that you are installing OpenShift Container Platform version 4.11 and later.
  • Ensure that the new node shares the same subnet as the Day 1 network. The subnet is specified in the machineNetwork field of the install-config.yaml file. This requirement applies to cluster-managed networks such as bare metal or vSphere, and not to user-managed networks.

Procedure

  1. Authenticate against the Assisted Installer REST API and generate an API token for your session. The generated token is valid for 15 minutes only.
  2. Set the $API_URL variable by running the following command:

    $ export API_URL=<api_url>

    Replace <api_url> with the Assisted Installer API URL, for example, https://api.openshift.com.

  3. Import the cluster by running the following commands:

    1. Set the $CLUSTER_ID variable:

      1. Log in to the cluster and run the following command:

        $ export CLUSTER_ID=$(oc get clusterversion -o jsonpath='{.items[].spec.clusterID}')
      2. Display the $CLUSTER_ID variable output:

        $ echo ${CLUSTER_ID}
    2. Set the $CLUSTER_REQUEST variable that is used to import the cluster:

      $ export CLUSTER_REQUEST=$(jq --null-input --arg openshift_cluster_id "$CLUSTER_ID" \
      '{
        "api_vip_dnsname": "<api_vip>",
        "openshift_cluster_id": "<cluster_id>",
        "name": "<openshift_cluster_name>"
      }')
      • Replace <api_vip> with the hostname for the cluster’s API server. This can be the DNS domain for the API server or the IP address of the single node which the host can reach. For example, api.compute-1.example.com.
      • Replace <cluster_id> with the $CLUSTER_ID output from the previous substep.
      • Replace <openshift_cluster_name> with the plain text name for the cluster. The cluster name should match the cluster name that was set during the Day 1 cluster installation.
    3. Import the cluster and set the $CLUSTER_ID variable. Run the following command:

      $ CLUSTER_ID=$(curl "$API_URL/api/assisted-install/v2/clusters/import" -H "Authorization: Bearer ${API_TOKEN}" -H 'accept: application/json' -H 'Content-Type: application/json' \
        -d "$CLUSTER_REQUEST" | tee /dev/stderr | jq -r '.id')
  4. Generate the InfraEnv resource for the cluster and set the $INFRA_ENV_ID variable by running the following commands:

    1. Download the pull secret file from Red Hat OpenShift Cluster Manager at console.redhat.com.
    2. Set the $INFRA_ENV_REQUEST variable:

      export INFRA_ENV_REQUEST=$(jq --null-input \
          --slurpfile pull_secret <path_to_pull_secret_file> \//
          --arg ssh_pub_key "$(cat <path_to_ssh_pub_key>)" \//
          --arg cluster_id "$CLUSTER_ID" '{
        "name": "<infraenv_name>",
        "pull_secret": $pull_secret[0] | tojson,
        "cluster_id": $cluster_id,
        "ssh_authorized_key": $ssh_pub_key,
        "image_type": "<iso_image_type>",
        "cpu_architecture": "<architecture_name>"
      }')
      • Replace <path_to_pull_secret_file> with the path to the local file containing the downloaded pull secret from Red Hat OpenShift Cluster Manager at console.redhat.com.
      • Replace <path_to_ssh_pub_key> with the path to the public SSH key required to access the host. If you do not set this value, you cannot access the host while in discovery mode.
      • Replace <infraenv_name> with the plain text name for the InfraEnv resource.
      • Replace <iso_image_type> with the ISO image type, either full-iso or minimal-iso.
    3. Post the $INFRA_ENV_REQUEST to the /v2/infra-envs API and set the $INFRA_ENV_ID variable:

      $ INFRA_ENV_ID=$(curl "$API_URL/api/assisted-install/v2/infra-envs" -H "Authorization: Bearer ${API_TOKEN}" -H 'accept: application/json' -H 'Content-Type: application/json' -d "$INFRA_ENV_REQUEST" | tee /dev/stderr | jq -r '.id')
  5. Get the URL of the discovery iPXE or ISO for the cluster host by running the following command:

    $ curl -s "$API_URL/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID" -H "Authorization: Bearer ${API_TOKEN}" | jq -r '.download_url'

    Example output

    https://api.openshift.com/api/assisted-images/images/41b91e72-c33e-42ee-b80f-b5c5bbf6431a?arch=x86_64&image_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NTYwMjYzNzEsInN1YiI6IjQxYjkxZTcyLWMzM2UtNDJlZS1iODBmLWI1YzViYmY2NDMxYSJ9.1EX_VGaMNejMhrAvVRBS7PDPIQtbOOc8LtG8OukE1a4&type=minimal-iso&version=4.12

  6. Do one of the following depending on your installation:

    • Download the ISO:

      $ curl -L -s '<iso_url>' --output rhcos-live-minimal.iso

      Replace <iso_url> with the URL for the ISO from the previous step.

    • Download the iPXE boot artifacts:

      $ curl -H "Authorization: Bearer ${API_TOKEN}" -s "$API_URL/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID/downloads/files?file_name=ipxe-script" >> discovery.txt
  7. Boot the new worker host from the downloaded rhcos-live-minimal.iso or the iPXE boot artifacts.
  8. Get the list of hosts in the cluster that are not installed. Keep running the following command until the new host shows up:

    $ curl -s "$API_URL/api/assisted-install/v2/clusters/$CLUSTER_ID" -H "Authorization: Bearer ${API_TOKEN}" | jq -r '.hosts[] | select(.status != "installed").id'

    Example output

    2294ba03-c264-4f11-ac08-2f1bb2f8c296

  9. Set the $HOST_ID variable for the new host, for example:

    $ HOST_ID=<host_id>

    Replace <host_id> with the host ID from the previous step.

  10. Check that the host is ready to install by running the following command:

    Note

    Ensure that you copy the entire command including the complete jq expression.

    $ curl -s $API_URL/api/assisted-install/v2/clusters/$CLUSTER_ID -H "Authorization: Bearer ${API_TOKEN}" | jq '
    def host_name($host):
        if (.suggested_hostname // "") == "" then
            if (.inventory // "") == "" then
                "Unknown hostname, please wait"
            else
                .inventory | fromjson | .hostname
            end
        else
            .suggested_hostname
        end;
    
    def is_notable($validation):
        ["failure", "pending", "error"] | any(. == $validation.status);
    
    def notable_validations($validations_info):
        [
            $validations_info // "{}"
            | fromjson
            | to_entries[].value[]
            | select(is_notable(.))
        ];
    
    {
        "Hosts validations": {
            "Hosts": [
                .hosts[]
                | select(.status != "installed")
                | {
                    "id": .id,
                    "name": host_name(.),
                    "status": .status,
                    "notable_validations": notable_validations(.validations_info)
                }
            ]
        },
        "Cluster validations info": {
            "notable_validations": notable_validations(.validations_info)
        }
    }
    ' -r

    Example output

    {
      "Hosts validations": {
        "Hosts": [
          {
            "id": "97ec378c-3568-460c-bc22-df54534ff08f",
            "name": "localhost.localdomain",
            "status": "insufficient",
            "notable_validations": [
              {
                "id": "ntp-synced",
                "status": "failure",
                "message": "Host couldn't synchronize with any NTP server"
              },
              {
                "id": "api-domain-name-resolved-correctly",
                "status": "error",
                "message": "Parse error for domain name resolutions result"
              },
              {
                "id": "api-int-domain-name-resolved-correctly",
                "status": "error",
                "message": "Parse error for domain name resolutions result"
              },
              {
                "id": "apps-domain-name-resolved-correctly",
                "status": "error",
                "message": "Parse error for domain name resolutions result"
              }
            ]
          }
        ]
      },
      "Cluster validations info": {
        "notable_validations": []
      }
    }

  11. When the previous command shows that the host is ready, start the installation using the /v2/infra-envs/{infra_env_id}/hosts/{host_id}/actions/install API by running the following command:

    $ curl -X POST -s "$API_URL/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID/hosts/$HOST_ID/actions/install"  -H "Authorization: Bearer ${API_TOKEN}"
  12. As the installation proceeds, the installation generates pending certificate signing requests (CSRs) for the host.

    Important

    You must approve the CSRs to complete the installation.

    Keep running the following API call to monitor the cluster installation:

    $ curl -s "$API_URL/api/assisted-install/v2/clusters/$CLUSTER_ID" -H "Authorization: Bearer ${API_TOKEN}" | jq '{
        "Cluster day-2 hosts":
            [
                .hosts[]
                | select(.status != "installed")
                | {id, requested_hostname, status, status_info, progress, status_updated_at, updated_at, infra_env_id, cluster_id, created_at}
            ]
    }'

    Example output

    {
      "Cluster day-2 hosts": [
        {
          "id": "a1c52dde-3432-4f59-b2ae-0a530c851480",
          "requested_hostname": "control-plane-1",
          "status": "added-to-existing-cluster",
          "status_info": "Host has rebooted and no further updates will be posted. Please check console for progress and to possibly approve pending CSRs",
          "progress": {
            "current_stage": "Done",
            "installation_percentage": 100,
            "stage_started_at": "2022-07-08T10:56:20.476Z",
            "stage_updated_at": "2022-07-08T10:56:20.476Z"
          },
          "status_updated_at": "2022-07-08T10:56:20.476Z",
          "updated_at": "2022-07-08T10:57:15.306369Z",
          "infra_env_id": "b74ec0c3-d5b5-4717-a866-5b6854791bd3",
          "cluster_id": "8f721322-419d-4eed-aa5b-61b50ea586ae",
          "created_at": "2022-07-06T22:54:57.161614Z"
        }
      ]
    }

  13. Optional: Run the following command to see all the events for the cluster:

    $ curl -s "$API_URL/api/assisted-install/v2/events?cluster_id=$CLUSTER_ID" -H "Authorization: Bearer ${API_TOKEN}" | jq -c '.[] | {severity, message, event_time, host_id}'

    Example output

    {"severity":"info","message":"Host compute-0: updated status from insufficient to known (Host is ready to be installed)","event_time":"2022-07-08T11:21:46.346Z","host_id":"9d7b3b44-1125-4ad0-9b14-76550087b445"}
    {"severity":"info","message":"Host compute-0: updated status from known to installing (Installation is in progress)","event_time":"2022-07-08T11:28:28.647Z","host_id":"9d7b3b44-1125-4ad0-9b14-76550087b445"}
    {"severity":"info","message":"Host compute-0: updated status from installing to installing-in-progress (Starting installation)","event_time":"2022-07-08T11:28:52.068Z","host_id":"9d7b3b44-1125-4ad0-9b14-76550087b445"}
    {"severity":"info","message":"Uploaded logs for host compute-0 cluster 8f721322-419d-4eed-aa5b-61b50ea586ae","event_time":"2022-07-08T11:29:47.802Z","host_id":"9d7b3b44-1125-4ad0-9b14-76550087b445"}
    {"severity":"info","message":"Host compute-0: updated status from installing-in-progress to added-to-existing-cluster (Host has rebooted and no further updates will be posted. Please check console for progress and to possibly approve pending CSRs)","event_time":"2022-07-08T11:29:48.259Z","host_id":"9d7b3b44-1125-4ad0-9b14-76550087b445"}
    {"severity":"info","message":"Host: compute-0, reached installation stage Rebooting","event_time":"2022-07-08T11:29:48.261Z","host_id":"9d7b3b44-1125-4ad0-9b14-76550087b445"}

  14. Log in to the cluster and approve the pending CSRs to complete the installation.

Verification

  • Check that the new host was successfully added to the cluster with a status of Ready:

    $ oc get nodes

    Example output

    NAME                           STATUS   ROLES           AGE   VERSION
    control-plane-1.example.com    Ready    master,worker   56m   v1.25.0
    compute-1.example.com          Ready    worker          11m   v1.25.0

You can replace a control plane (master) node in a healthy OpenShift Container Platform cluster that has three to five control plane nodes, by adding a new control plane node and removing an existing control plane node.

If the cluster is unhealthy, you must peform additional operations before you can manage the control plane nodes. See "Replacing a control plane node in an unhealthy cluster" for more information.

11.5.1. Adding a new control plane node

Add the new control plane node, and verify that it is healthy. In the example below, the new node is node-5.

Prerequisites

  • You are using OpenShift Container Platform 4.11 or later.
  • You have installed a healthy cluster with at least three control plane nodes.
  • You have created a single control plane node to be added to the cluster for Day 2.

Procedure

  1. Retrieve pending Certificate Signing Requests (CSRs) for the new Day 2 control plane node:

    $ oc get csr | grep Pending

    Example output

    csr-5sd59   8m19s   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   <none>              Pending
    csr-xzqts   10s     kubernetes.io/kubelet-serving                 system:node:node-5                                                   <none>              Pending

  2. Approve all pending CSRs for the new node (node-5 in this example):

    $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve
    Important

    You must approve the CSRs to complete the installation.

  3. Confirm that the new control plane node is in Ready status:

    $ oc get nodes

    Example output

    NAME       STATUS   ROLES    AGE     VERSION
    node-0   Ready    master   4h42m   v1.24.0+3882f8f
    node-1   Ready    master   4h27m   v1.24.0+3882f8f
    node-2   Ready    master   4h43m   v1.24.0+3882f8f
    node-3   Ready    worker   4h29m   v1.24.0+3882f8f
    node-4   Ready    worker   4h30m   v1.24.0+3882f8f
    node-5   Ready    master   105s    v1.24.0+3882f8f

    Note

    The etcd operator requires a Machine custom resource (CR) that references the new node when the cluster runs with a Machine API. The machine API is automatically activated when the cluster has three or more control plane nodes.

  4. Create the BareMetalHost and Machine CRs and link them to the new control plane’s Node CR.

    1. Create the BareMetalHost CR with a unique .metadata.name value:

      apiVersion: metal3.io/v1alpha1
      kind: BareMetalHost
      metadata:
        name: node-5
        namespace: openshift-machine-api
      spec:
        automatedCleaningMode: metadata
        bootMACAddress: 00:00:00:00:00:02
        bootMode: UEFI
        customDeploy:
          method: install_coreos
        externallyProvisioned: true
        online: true
        userData:
          name: master-user-data-managed
          namespace: openshift-machine-api
    2. Apply the BareMetalHost CR:

      $ oc apply -f <filename>

      Replace <filename> with the name of the BareMetalHost CR.

    3. Create the Machine CR using the unique .metadata.name value:

      apiVersion: machine.openshift.io/v1beta1
      kind: Machine
      metadata:
        annotations:
          machine.openshift.io/instance-state: externally provisioned
          metal3.io/BareMetalHost: openshift-machine-api/node-5
        finalizers:
        - machine.machine.openshift.io
        labels:
          machine.openshift.io/cluster-api-cluster: <cluster_name>
          machine.openshift.io/cluster-api-machine-role: master
          machine.openshift.io/cluster-api-machine-type: master
        name: node-5
        namespace: openshift-machine-api
      spec:
        metadata: {}
        providerSpec:
          value:
            apiVersion: baremetal.cluster.k8s.io/v1alpha1
            customDeploy:
              method: install_coreos
            hostSelector: {}
            image:
              checksum: ""
              url: ""
            kind: BareMetalMachineProviderSpec
            metadata:
              creationTimestamp: null
            userData:
              name: master-user-data-managed

      Replace <cluster_name> with the name of the specific cluster, for example, test-day2-1-6qv96.

      To get the cluster name, run the following command:

      $ oc get infrastructure cluster -o=jsonpath='{.status.infrastructureName}{"\n"}'
    4. Apply the Machine CR:

      $ oc apply -f <filename>

      Replace <filename> with the name of the Machine CR.

    5. Link BareMetalHost, Machine, and Node by running the link-machine-and-node.sh script:

      1. Copy the link-machine-and-node.sh script below to a local machine:

        #!/bin/bash
        
        # Credit goes to
        # https://bugzilla.redhat.com/show_bug.cgi?id=1801238.
        # This script will link Machine object
        # and Node object. This is needed
        # in order to have IP address of
        # the Node present in the status of the Machine.
        
        set -e
        
        machine="$1"
        node="$2"
        
        if [ -z "$machine" ] || [ -z "$node" ]; then
            echo "Usage: $0 MACHINE NODE"
            exit 1
        fi
        
        node_name=$(echo "${node}" | cut -f2 -d':')
        
        oc proxy &
        proxy_pid=$!
        function kill_proxy {
            kill $proxy_pid
        }
        trap kill_proxy EXIT SIGINT
        
        HOST_PROXY_API_PATH="http://localhost:8001/apis/metal3.io/v1alpha1/namespaces/openshift-machine-api/baremetalhosts"
        
        function print_nics() {
            local ips
            local eob
            declare -a ips
        
            readarray -t ips < <(echo "${1}" \
                                 | jq '.[] | select(. | .type == "InternalIP") | .address' \
                                 | sed 's/"//g')
        
            eob=','
            for (( i=0; i<${#ips[@]}; i++ )); do
                if [ $((i+1)) -eq ${#ips[@]} ]; then
                    eob=""
                fi
                cat <<- EOF
                  {
                    "ip": "${ips[$i]}",
                    "mac": "00:00:00:00:00:00",
                    "model": "unknown",
                    "speedGbps": 10,
                    "vlanId": 0,
                    "pxe": true,
                    "name": "eth1"
                  }${eob}
        EOF
            done
        }
        
        function wait_for_json() {
            local name
            local url
            local curl_opts
            local timeout
        
            local start_time
            local curr_time
            local time_diff
        
            name="$1"
            url="$2"
            timeout="$3"
            shift 3
            curl_opts="$@"
            echo -n "Waiting for $name to respond"
            start_time=$(date +%s)
            until curl -g -X GET "$url" "${curl_opts[@]}" 2> /dev/null | jq '.' 2> /dev/null > /dev/null; do
                echo -n "."
                curr_time=$(date +%s)
                time_diff=$((curr_time - start_time))
                if [[ $time_diff -gt $timeout ]]; then
                    printf '\nTimed out waiting for %s' "${name}"
                    return 1
                fi
                sleep 5
            done
            echo " Success!"
            return 0
        }
        wait_for_json oc_proxy "${HOST_PROXY_API_PATH}" 10 -H "Accept: application/json" -H "Content-Type: application/json"
        
        addresses=$(oc get node -n openshift-machine-api "${node_name}" -o json | jq -c '.status.addresses')
        
        machine_data=$(oc get machines.machine.openshift.io -n openshift-machine-api -o json "${machine}")
        host=$(echo "$machine_data" | jq '.metadata.annotations["metal3.io/BareMetalHost"]' | cut -f2 -d/ | sed 's/"//g')
        
        if [ -z "$host" ]; then
            echo "Machine $machine is not linked to a host yet." 1>&2
            exit 1
        fi
        
        # The address structure on the host doesn't match the node, so extract
        # the values we want into separate variables so we can build the patch
        # we need.
        hostname=$(echo "${addresses}" | jq '.[] | select(. | .type == "Hostname") | .address' | sed 's/"//g')
        
        set +e
        read -r -d '' host_patch << EOF
        {
          "status": {
            "hardware": {
              "hostname": "${hostname}",
              "nics": [
        $(print_nics "${addresses}")
              ],
              "systemVendor": {
                "manufacturer": "Red Hat",
                "productName": "product name",
                "serialNumber": ""
              },
              "firmware": {
                "bios": {
                  "date": "04/01/2014",
                  "vendor": "SeaBIOS",
                  "version": "1.11.0-2.el7"
                }
              },
              "ramMebibytes": 0,
              "storage": [],
              "cpu": {
                "arch": "x86_64",
                "model": "Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz",
                "clockMegahertz": 2199.998,
                "count": 4,
                "flags": []
              }
            }
          }
        }
        EOF
        set -e
        
        echo "PATCHING HOST"
        echo "${host_patch}" | jq .
        
        curl -s \
             -X PATCH \
             "${HOST_PROXY_API_PATH}/${host}/status" \
             -H "Content-type: application/merge-patch+json" \
             -d "${host_patch}"
        
        oc get baremetalhost -n openshift-machine-api -o yaml "${host}"
      2. Make the script executable:

        $ chmod +x link-machine-and-node.sh
      3. Run the script:

        $ bash link-machine-and-node.sh node-5 node-5
        Note

        The first node-5 instance represents the machine, and the second represents the node.

  5. Confirm members of etcd by executing into one of the pre-existing control plane nodes:

    1. Open a remote shell session to the control plane node:

      $ oc rsh -n openshift-etcd etcd-node-0
    2. List etcd members:

      # etcdctl member list -w table

      Example output

      +--------+---------+--------+--------------+--------------+---------+
      |   ID   |  STATUS |  NAME  |  PEER ADDRS  | CLIENT ADDRS | LEARNER |
      +--------+---------+--------+--------------+--------------+---------+
      |76ae1d00| started |node-0  |192.168.111.24|192.168.111.24|  false  |
      |2c18942f| started |node-1  |192.168.111.26|192.168.111.26|  false  |
      |61e2a860| started |node-2  |192.168.111.25|192.168.111.25|  false  |
      |ead5f280| started |node-5  |192.168.111.28|192.168.111.28|  false  |
      +--------+---------+--------+--------------+--------------+---------+

  6. Monitor the etcd operator configuration process until completion:

    $ oc get clusteroperator etcd

    Example output (upon completion)

    NAME   VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
    etcd   4.11.5    True        False         False      5h54m

  7. Confirm etcd health by running the following commands:

    1. Open a remote shell session to the control plane node:

      $ oc rsh -n openshift-etcd etcd-node-0
    2. Check endpoint health:

      # etcdctl endpoint health

      Example output

      192.168.111.24 is healthy: committed proposal: took = 10.383651ms
      192.168.111.26 is healthy: committed proposal: took = 11.297561ms
      192.168.111.25 is healthy: committed proposal: took = 13.892416ms
      192.168.111.28 is healthy: committed proposal: took = 11.870755ms

  8. Verify that all nodes are ready:

    $ oc get nodes

    Example output

    NAME      STATUS   ROLES    AGE     VERSION
    node-0    Ready    master   6h20m   v1.24.0+3882f8f
    node-1    Ready    master   6h20m   v1.24.0+3882f8f
    node-2    Ready    master   6h4m    v1.24.0+3882f8f
    node-3    Ready    worker   6h7m    v1.24.0+3882f8f
    node-4    Ready    worker   6h7m    v1.24.0+3882f8f
    node-5    Ready    master   99m     v1.24.0+3882f8f

  9. Verify that the cluster Operators are all available:

    $ oc get ClusterOperators

    Example output

    NAME                                      VERSION AVAILABLE PROGRESSING DEGRADED SINCE MSG
    authentication                            4.11.5  True      False       False    5h57m
    baremetal                                 4.11.5  True      False       False    6h19m
    cloud-controller-manager                  4.11.5  True      False       False    6h20m
    cloud-credential                          4.11.5  True      False       False    6h23m
    cluster-autoscaler                        4.11.5  True      False       False    6h18m
    config-operator                           4.11.5  True      False       False    6h19m
    console                                   4.11.5  True      False       False    6h4m
    csi-snapshot-controller                   4.11.5  True      False       False    6h19m
    dns                                       4.11.5  True      False       False    6h18m
    etcd                                      4.11.5  True      False       False    6h17m
    image-registry                            4.11.5  True      False       False    6h7m
    ingress                                   4.11.5  True      False       False    6h6m
    insights                                  4.11.5  True      False       False    6h12m
    kube-apiserver                            4.11.5  True      False       False    6h16m
    kube-controller-manager                   4.11.5  True      False       False    6h16m
    kube-scheduler                            4.11.5  True      False       False    6h16m
    kube-storage-version-migrator             4.11.5  True      False       False    6h19m
    machine-api                               4.11.5  True      False       False    6h15m
    machine-approver                          4.11.5  True      False       False    6h19m
    machine-config                            4.11.5  True      False       False    6h18m
    marketplace                               4.11.5  True      False       False    6h18m
    monitoring                                4.11.5  True      False       False    6h4m
    network                                   4.11.5  True      False       False    6h20m
    node-tuning                               4.11.5  True      False       False    6h18m
    openshift-apiserver                       4.11.5  True      False       False    6h8m
    openshift-controller-manager              4.11.5  True      False       False    6h7m
    openshift-samples                         4.11.5  True      False       False    6h12m
    operator-lifecycle-manager                4.11.5  True      False       False    6h18m
    operator-lifecycle-manager-catalog        4.11.5  True      False       False    6h19m
    operator-lifecycle-manager-pkgsvr         4.11.5  True      False       False    6h12m
    service-ca                                4.11.5  True      False       False    6h19m
    storage                                   4.11.5  True      False       False    6h19m

  10. Verify that the cluster version is correct:

    $ oc get ClusterVersion

    Example output

    NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
    version   4.11.5    True        False         5h57m   Cluster version is 4.11.5

11.5.2. Removing the existing control plane node

Remove the control plane node that you are replacing. This is node-0 in the example below.

Prerequisites

  • You have added a new healthy control plane node.

Procedure

  1. Delete the BareMetalHost CR of the pre-existing control plane node:

    $ oc delete bmh -n openshift-machine-api node-0
  2. Confirm that the machine is unhealthy:

    $ oc get machine -A

    Example output

    NAMESPACE               NAME     PHASE     AGE
    openshift-machine-api   node-0   Failed    20h
    openshift-machine-api   node-1   Running   20h
    openshift-machine-api   node-2   Running   20h
    openshift-machine-api   node-3   Running   19h
    openshift-machine-api   node-4   Running   19h
    openshift-machine-api   node-5   Running   14h

  3. Delete the Machine CR:

    $ oc delete machine -n openshift-machine-api node-0
    machine.machine.openshift.io "node-0" deleted
  4. Confirm removal of the Node CR:

    $ oc get nodes

    Example output

    NAME      STATUS   ROLES    AGE   VERSION
    node-1    Ready    master   20h   v1.24.0+3882f8f
    node-2    Ready    master   19h   v1.24.0+3882f8f
    node-3    Ready    worker   19h   v1.24.0+3882f8f
    node-4    Ready    worker   19h   v1.24.0+3882f8f
    node-5    Ready    master   15h   v1.24.0+3882f8f

  5. Check etcd-operator logs to confirm status of the etcd cluster:

    $ oc logs -n openshift-etcd-operator etcd-operator-8668df65d-lvpjf

    Example output

    E0927 07:53:10.597523       1 base_controller.go:272] ClusterMemberRemovalController reconciliation failed: cannot remove member: 192.168.111.23 because it is reported as healthy but it doesn't have a machine nor a node resource

  6. Remove the physical machine to allow the etcd operator to reconcile the cluster members:

    1. Open a remote shell session to the control plane node:

      $ oc rsh -n openshift-etcd etcd-node-1
    2. Monitor the progress of etcd operator reconciliation by checking members and endpoint health:

      # etcdctl member list -w table; etcdctl endpoint health

      Example output

      +--------+---------+--------+--------------+--------------+---------+
      |   ID   |  STATUS |  NAME  |  PEER ADDRS  | CLIENT ADDRS | LEARNER |
      +--------+---------+--------+--------------+--------------+---------+
      |2c18942f| started | node-1 |192.168.111.26|192.168.111.26|  false  |
      |61e2a860| started | node-2 |192.168.111.25|192.168.111.25|  false  |
      |ead4f280| started | node-5 |192.168.111.28|192.168.111.28|  false  |
      +--------+---------+--------+--------------+--------------+---------+
      192.168.111.26 is healthy: committed proposal: took = 10.458132ms
      192.168.111.25 is healthy: committed proposal: took = 11.047349ms
      192.168.111.28 is healthy: committed proposal: took = 11.414402ms

You can replace an unhealthy control plane (master) node in an OpenShift Container Platform cluster that has three to five control plane nodes, by removing the unhealthy control plane node and adding a new one.

For details on replacing a control plane node in a healthy cluster, see "Replacing a control plane node in a healthy cluster".

f the cluster is unhealthy, you must peform additional operations before you can manage the control plane nodes. See "Replacing a control plane node in an unhealthy cluster" for more information.

11.6.1. Removing an unhealthy control plane node

Remove the unhealthy control plane node from the cluster. This is node-0 in the example below.

Prerequisites

  • You have installed a cluster with at least three control plane nodes.
  • At least one of the control plane nodes is not ready.

Procedure

  1. Check the node status to confirm that a control plane node is not ready:

    $ oc get nodes

    Example output

    NAME      STATUS      ROLES    AGE   VERSION
    node-0    NotReady    master   20h   v1.24.0+3882f8f
    node-1    Ready       master   20h   v1.24.0+3882f8f
    node-2    Ready       master   20h   v1.24.0+3882f8f
    node-3    Ready       worker   20h   v1.24.0+3882f8f
    node-4    Ready       worker   20h   v1.24.0+3882f8f

  2. Confirm in the etcd-operator logs that the cluster is unhealthy:

    $ oc logs -n openshift-etcd-operator deployment/etcd-operator

    Example output

    E0927 08:24:23.983733       1 base_controller.go:272] DefragController reconciliation failed: cluster is unhealthy: 2 of 3 members are available, node-0 is unhealthy

  3. Confirm the etcd members by running the following commands:

    1. Open a remote shell session to the control plane node:

      $ oc rsh -n openshift-etcd node-1
    2. List the etcdctl members:

      # etcdctl member list -w table

      Example output

      +--------+---------+--------+--------------+--------------+---------+
      |   ID   | STATUS  |  NAME  |  PEER ADDRS  | CLIENT ADDRS | LEARNER |
      +--------+---------+--------+--------------+--------------+---------+
      |61e2a860| started | node-0 |192.168.111.25|192.168.111.25|  false  |
      |2c18942f| started | node-1 |192.168.111.26|192.168.111.26|  false  |
      |ead4f280| started | node-2 |192.168.111.28|192.168.111.28|  false  |
      +--------+---------+--------+--------------+--------------+---------+

  4. Confirm that etcdctl endpoint health reports an unhealthy member of the cluster:

    # etcdctl endpoint health

    Example output

    {"level":"warn","ts":"2022-09-27T08:25:35.953Z","logger":"client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000680380/192.168.111.25","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing dial tcp 192.168.111.25: connect: no route to host\""}
    192.168.111.28 is healthy: committed proposal: took = 12.465641ms
    192.168.111.26 is healthy: committed proposal: took = 12.297059ms
    192.168.111.25 is unhealthy: failed to commit proposal: context deadline exceeded
    Error: unhealthy cluster

  5. Remove the unhealthy control plane by deleting the Machine custom resource (CR):

    $ oc delete machine -n openshift-machine-api node-0
    Note

    The Machine and Node CRs might not be deleted because they are protected by finalizers. If this occurs, you must delete the Machine CR manually by removing all finalizers.

  6. Verify in the etcd-operator logs whether the unhealthy machine has been removed:

    $ oc logs -n openshift-etcd-operator deployment/etcd-operator

    Example output

    I0927 08:58:41.249222       1 machinedeletionhooks.go:135] skip removing the deletion hook from machine node-0 since its member is still present with any of: [{InternalIP } {InternalIP 192.168.111.25}]

  7. If you see that removal has been skipped, as in the above log example, manually remove the unhealthy etcdctl member:

    1. Open a remote shell session to the control plane node:

      $ oc rsh -n openshift-etcd node-1
    2. List the etcdctl members:

      # etcdctl member list -w table

      Example output

      +--------+---------+--------+--------------+--------------+---------+
      |   ID   |  STATUS |  NAME  |  PEER ADDRS  | CLIENT ADDRS | LEARNER |
      +--------+---------+--------+--------------+--------------+---------+
      |61e2a860| started | node-0 |192.168.111.25|192.168.111.25|  false  |
      |2c18942f| started | node-1 |192.168.111.26|192.168.111.26|  false  |
      |ead4f280| started | node-2 |192.168.111.28|192.168.111.28|  false  |
      +--------+---------+--------+--------------+--------------+---------+

    3. Confirm that etcdctl endpoint health reports an unhealthy member of the cluster:

      # etcdctl endpoint health

      Example output

      {"level":"warn","ts":"2022-09-27T10:31:07.227Z","logger":"client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0000d6e00/192.168.111.25","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing dial tcp 192.168.111.25: connect: no route to host\""}
      192.168.111.28 is healthy: committed proposal: took = 13.038278ms
      192.168.111.26 is healthy: committed proposal: took = 12.950355ms
      192.168.111.25 is unhealthy: failed to commit proposal: context deadline exceeded
      Error: unhealthy cluster

    4. Remove the unhealthy etcdctl member from the cluster:

      # etcdctl member remove 61e2a86084aafa62

      Example output

      Member 61e2a86084aafa62 removed from cluster 6881c977b97990d7

    5. Verify that the unhealthy etcdctl member was removed by running the following command:

      # etcdctl member list -w table

      Example output

      +----------+---------+--------+--------------+--------------+-------+
      |    ID    | STATUS  |  NAME  |  PEER ADDRS  | CLIENT ADDRS |LEARNER|
      +----------+---------+--------+--------------+--------------+-------+
      | 2c18942f | started | node-1 |192.168.111.26|192.168.111.26| false |
      | ead4f280 | started | node-2 |192.168.111.28|192.168.111.28| false |
      +----------+---------+--------+--------------+--------------+-------+

11.6.2. Adding a new control plane node

Add a new control plane node to replace the unhealthy node that you removed. In the example below, the new node is node-5.

Prerequisites

Procedure

  1. Retrieve pending Certificate Signing Requests (CSRs) for the new Day 2 control plane node:

    $ oc get csr | grep Pending

    Example output

    csr-5sd59   8m19s   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   <none>              Pending
    csr-xzqts   10s     kubernetes.io/kubelet-serving                 system:node:node-5                                                   <none>              Pending

  2. Approve all pending CSRs for the new node (node-5 in this example):

    $ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve
    Note

    You must approve the CSRs to complete the installation.

  3. Confirm that the control plane node is in Ready status:

    $ oc get nodes

    Example output

    NAME      STATUS    ROLES     AGE     VERSION
    node-1    Ready     master    20h     v1.24.0+3882f8f
    node-2    Ready     master    20h     v1.24.0+3882f8f
    node-3    Ready     worker    20h     v1.24.0+3882f8f
    node-4    Ready     worker    20h     v1.24.0+3882f8f
    node-5    Ready     master    2m52s   v1.24.0+3882f8f

    The etcd operator requires a Machine CR referencing the new node when the cluster runs with a Machine API. The machine API is automatically activated when the cluster has three control plane nodes.

  4. Create the BareMetalHost and Machine CRs and link them to the new control plane’s Node CR.

    Important

    Boot-it-yourself will not create BareMetalHost and Machine CRs, so you must create them. Failure to create the BareMetalHost and Machine CRs will generate errors in the etcd operator.

    1. Create the BareMetalHost CR with a unique .metadata.name value:

      apiVersion: metal3.io/v1alpha1
      kind: BareMetalHost
      metadata:
        name: node-5
        namespace: openshift-machine-api
      spec:
        automatedCleaningMode: metadata
        bootMACAddress: 00:00:00:00:00:02
        bootMode: UEFI
        customDeploy:
          method: install_coreos
        externallyProvisioned: true
        online: true
        userData:
          name: master-user-data-managed
          namespace: openshift-machine-api
    2. Apply the BareMetalHost CR:

      $ oc apply -f <filename>

      Replace <filename> with the name of the BareMetalHost CR.

    3. Create the Machine CR using the unique .metadata.name value:

      apiVersion: machine.openshift.io/v1beta1
      kind: Machine
      metadata:
        annotations:
          machine.openshift.io/instance-state: externally provisioned
          metal3.io/BareMetalHost: openshift-machine-api/node-5
        finalizers:
        - machine.machine.openshift.io
        labels:
          machine.openshift.io/cluster-api-cluster: test-day2-1-6qv96
          machine.openshift.io/cluster-api-machine-role: master
          machine.openshift.io/cluster-api-machine-type: master
        name: node-5
        namespace: openshift-machine-api
      spec:
        metadata: {}
        providerSpec:
          value:
            apiVersion: baremetal.cluster.k8s.io/v1alpha1
            customDeploy:
              method: install_coreos
            hostSelector: {}
            image:
              checksum: ""
              url: ""
            kind: BareMetalMachineProviderSpec
            metadata:
              creationTimestamp: null
            userData:
             name: master-user-data-managed
    4. Apply the Machine CR:

      $ oc apply -f <filename>

      Replace <filename> with the name of the Machine CR.

    5. Link BareMetalHost, Machine, and Node by running the link-machine-and-node.sh script:

      1. Copy the link-machine-and-node.sh script below to a local machine:

        #!/bin/bash
        
        # Credit goes to
        # https://bugzilla.redhat.com/show_bug.cgi?id=1801238.
        # This script will link Machine object
        # and Node object. This is needed
        # in order to have IP address of
        # the Node present in the status of the Machine.
        
        set -e
        
        machine="$1"
        node="$2"
        
        if [ -z "$machine" ] || [ -z "$node" ]; then
            echo "Usage: $0 MACHINE NODE"
            exit 1
        fi
        
        node_name=$(echo "${node}" | cut -f2 -d':')
        
        oc proxy &
        proxy_pid=$!
        function kill_proxy {
            kill $proxy_pid
        }
        trap kill_proxy EXIT SIGINT
        
        HOST_PROXY_API_PATH="http://localhost:8001/apis/metal3.io/v1alpha1/namespaces/openshift-machine-api/baremetalhosts"
        
        function print_nics() {
            local ips
            local eob
            declare -a ips
        
            readarray -t ips < <(echo "${1}" \
                                 | jq '.[] | select(. | .type == "InternalIP") | .address' \
                                 | sed 's/"//g')
        
            eob=','
            for (( i=0; i<${#ips[@]}; i++ )); do
                if [ $((i+1)) -eq ${#ips[@]} ]; then
                    eob=""
                fi
                cat <<- EOF
                  {
                    "ip": "${ips[$i]}",
                    "mac": "00:00:00:00:00:00",
                    "model": "unknown",
                    "speedGbps": 10,
                    "vlanId": 0,
                    "pxe": true,
                    "name": "eth1"
                  }${eob}
        EOF
            done
        }
        
        function wait_for_json() {
            local name
            local url
            local curl_opts
            local timeout
        
            local start_time
            local curr_time
            local time_diff
        
            name="$1"
            url="$2"
            timeout="$3"
            shift 3
            curl_opts="$@"
            echo -n "Waiting for $name to respond"
            start_time=$(date +%s)
            until curl -g -X GET "$url" "${curl_opts[@]}" 2> /dev/null | jq '.' 2> /dev/null > /dev/null; do
                echo -n "."
                curr_time=$(date +%s)
                time_diff=$((curr_time - start_time))
                if [[ $time_diff -gt $timeout ]]; then
                    printf '\nTimed out waiting for %s' "${name}"
                    return 1
                fi
                sleep 5
            done
            echo " Success!"
            return 0
        }
        wait_for_json oc_proxy "${HOST_PROXY_API_PATH}" 10 -H "Accept: application/json" -H "Content-Type: application/json"
        
        addresses=$(oc get node -n openshift-machine-api "${node_name}" -o json | jq -c '.status.addresses')
        
        machine_data=$(oc get machines.machine.openshift.io -n openshift-machine-api -o json "${machine}")
        host=$(echo "$machine_data" | jq '.metadata.annotations["metal3.io/BareMetalHost"]' | cut -f2 -d/ | sed 's/"//g')
        
        if [ -z "$host" ]; then
            echo "Machine $machine is not linked to a host yet." 1>&2
            exit 1
        fi
        
        # The address structure on the host doesn't match the node, so extract
        # the values we want into separate variables so we can build the patch
        # we need.
        hostname=$(echo "${addresses}" | jq '.[] | select(. | .type == "Hostname") | .address' | sed 's/"//g')
        
        set +e
        read -r -d '' host_patch << EOF
        {
          "status": {
            "hardware": {
              "hostname": "${hostname}",
              "nics": [
        $(print_nics "${addresses}")
              ],
              "systemVendor": {
                "manufacturer": "Red Hat",
                "productName": "product name",
                "serialNumber": ""
              },
              "firmware": {
                "bios": {
                  "date": "04/01/2014",
                  "vendor": "SeaBIOS",
                  "version": "1.11.0-2.el7"
                }
              },
              "ramMebibytes": 0,
              "storage": [],
              "cpu": {
                "arch": "x86_64",
                "model": "Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz",
                "clockMegahertz": 2199.998,
                "count": 4,
                "flags": []
              }
            }
          }
        }
        EOF
        set -e
        
        echo "PATCHING HOST"
        echo "${host_patch}" | jq .
        
        curl -s \
             -X PATCH \
             "${HOST_PROXY_API_PATH}/${host}/status" \
             -H "Content-type: application/merge-patch+json" \
             -d "${host_patch}"
        
        oc get baremetalhost -n openshift-machine-api -o yaml "${host}"
      2. Make the script executable:

        $ chmod +x link-machine-and-node.sh
      3. Run the script:

        $ bash link-machine-and-node.sh node-5 node-5
        Note

        The first node-5 instance represents the machine, and the second represents the node.

  5. Confirm members of etcd by running the following commands:

    1. Open a remote shell session to the control plane node:

      $ oc rsh -n openshift-etcd node-1
    2. List the etcdctl members:

      # etcdctl member list -w table

      Example output

      +---------+-------+--------+--------------+--------------+-------+
      |   ID    | STATUS|  NAME  |   PEER ADDRS | CLIENT ADDRS |LEARNER|
      +---------+-------+--------+--------------+--------------+-------+
      | 2c18942f|started| node-1 |192.168.111.26|192.168.111.26| false |
      | ead4f280|started| node-2 |192.168.111.28|192.168.111.28| false |
      | 79153c5a|started| node-5 |192.168.111.29|192.168.111.29| false |
      +---------+-------+--------+--------------+--------------+-------+

  6. Monitor the etcd operator configuration process until completion:

    $ oc get clusteroperator etcd

    Example output (upon completion)

    NAME   VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
    etcd   4.11.5    True        False         False      22h

  7. Confirm etcdctl health by running the following commands:

    1. Open a remote shell session to the control plane node:

      $ oc rsh -n openshift-etcd node-1
    2. Check endpoint health:

      # etcdctl endpoint health

      Example output

      192.168.111.26 is healthy: committed proposal: took = 9.105375ms
      192.168.111.28 is healthy: committed proposal: took = 9.15205ms
      192.168.111.29 is healthy: committed proposal: took = 10.277577ms

  8. Confirm the health of the nodes:

    $ oc get Nodes

    Example output

    NAME     STATUS   ROLES    AGE   VERSION
    node-1   Ready    master   20h   v1.24.0+3882f8f
    node-2   Ready    master   20h   v1.24.0+3882f8f
    node-3   Ready    worker   20h   v1.24.0+3882f8f
    node-4   Ready    worker   20h   v1.24.0+3882f8f
    node-5   Ready    master   40m   v1.24.0+3882f8f

  9. Verify that the cluster Operators are all available:

    $ oc get ClusterOperators

    Example output

    NAME                               VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
    authentication                     4.11.5    True        False         False      150m
    baremetal                          4.11.5    True        False         False      22h
    cloud-controller-manager           4.11.5    True        False         False      22h
    cloud-credential                   4.11.5    True        False         False      22h
    cluster-autoscaler                 4.11.5    True        False         False      22h
    config-operator                    4.11.5    True        False         False      22h
    console                            4.11.5    True        False         False      145m
    csi-snapshot-controller            4.11.5    True        False         False      22h
    dns                                4.11.5    True        False         False      22h
    etcd                               4.11.5    True        False         False      22h
    image-registry                     4.11.5    True        False         False      22h
    ingress                            4.11.5    True        False         False      22h
    insights                           4.11.5    True        False         False      22h
    kube-apiserver                     4.11.5    True        False         False      22h
    kube-controller-manager            4.11.5    True        False         False      22h
    kube-scheduler                     4.11.5    True        False         False      22h
    kube-storage-version-migrator      4.11.5    True        False         False      148m
    machine-api                        4.11.5    True        False         False      22h
    machine-approver                   4.11.5    True        False         False      22h
    machine-config                     4.11.5    True        False         False      110m
    marketplace                        4.11.5    True        False         False      22h
    monitoring                         4.11.5    True        False         False      22h
    network                            4.11.5    True        False         False      22h
    node-tuning                        4.11.5    True        False         False      22h
    openshift-apiserver                4.11.5    True        False         False      163m
    openshift-controller-manager       4.11.5    True        False         False      22h
    openshift-samples                  4.11.5    True        False         False      22h
    operator-lifecycle-manager         4.11.5    True        False         False      22h
    operator-lifecycle-manager-catalog 4.11.5    True        False         False      22h
    operator-lifecycle-manager-pkgsvr  4.11.5    True        False         False      22h
    service-ca                         4.11.5    True        False         False      22h
    storage                            4.11.5    True        False         False      22h

  10. Verify that the cluster version is correct:

    $ oc get ClusterVersion

    Example output

    NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
    version   4.11.5    True        False         22h     Cluster version is 4.11.5

Chapter 12. Installing on Nutanix

If you install OpenShift Container Platform on Nutanix, the Assisted Installer can integrate the OpenShift Container Platform cluster with the Nutanix platform, which exposes the Machine API to Nutanix and enables autoscaling and the dynamic provisioning of storage containers with the Nutanix Container Storage Interface (CSI).

Important

To deploy an OpenShift Container Platform cluster and maintain its daily operation, you need access to a Nutanix account with the necessary environment requirements. For details, see "Environment requirements" in Additional resources.

12.1. Adding hosts on Nutanix with the UI

To add hosts on Nutanix with the user interface (UI), generate the minimal discovery image ISO from the Assisted Installer. This downloads a smaller image that will fetch the data needed to boot a host with networking and is the default setting. The majority of the content downloads upon boot. The ISO image is about 100MB in size.

After this is complete, you must create an image for the Nutanix platform and create the Nutanix virtual machines.

Prerequisites

  • You have created a cluster profile in the Assisted Installer UI.
  • You have a Nutanix cluster environment set up, and made a note of the cluster name and subnet name.

Procedure

  1. In the Cluster details page, select Nutanix from the Integrate with external partner platforms dropdown list. The Include custom manifest checkbox is optional.
  2. In the Host discovery page, click the Add hosts button.
  3. Optional: Add an SSH public key so that you can connect to the Nutanix VMs as the core user. Having a login to the cluster hosts can provide you with debugging information during the installation.

    1. If you do not have an existing SSH key pair on your local machine, follow the steps in Generating a key pair for cluster node SSH access.
    2. In the SSH public key field, click Browse to upload the id_rsa.pub file containing the SSH public key or drag and drop the file into the field from the file manager. To see the file in the file manager, select Show hidden files in the menu.
  4. Select the required provisioning type.

    Note

    Minimal image file: Provision with virtual media downloads a smaller image that will fetch the data needed to boot.

  5. In Networking, select Cluster-managed networking. Nutanix does not support User-managed networking.

    1. Optional: If the cluster hosts require the use of a proxy, select Configure cluster-wide proxy settings. Enter the username, password, required domains or IP addresses, and port for the HTTP and HTTPS URLs of the proxy server. If the cluster hosts are behind a firewall, allow the nodes to access the required domains or IP addresses through the firewall. See Configuring your firewall for OpenShift Container Platform for more information.

      Note

      The proxy username and password must be URL-encoded.

    2. Optional: Configure the discovery image if you want to boot it with an ignition file. For details, see Configuring the discovery image.
  6. Click Generate Discovery ISO.
  7. Copy the Discovery ISO URL.
  8. In the Nutanix Prism UI, follow the directions to upload the discovery image from the Assisted Installer.
  9. In the Nutanix Prism UI, create the control plane (master) VMs through Prism Central.

    1. Enter the Name. For example, control-plane or master.
    2. Enter the Number of VMs. This should be 3, 4, or 5 for the control plane.
    3. Ensure the remaining settings meet the minimum requirements for control plane hosts.
  10. In the Nutanix Prism UI, create the worker VMs through Prism Central.

    1. Enter the Name. For example, worker.
    2. Enter the Number of VMs. You should create at least 2 worker nodes.
    3. Ensure the remaining settings meet the minimum requirements for worker hosts.
  11. Return to the Assisted Installer user interface and wait until the Assisted Installer discovers the hosts and each of them have a Ready status.
  12. Continue with the installation procedure.

12.2. Adding hosts on Nutanix with the API

To add hosts on Nutanix with the API, generate the discovery image ISO from the Assisted Installer. Use the minimal discovery image ISO. This is the default setting. The image includes only what is required to boot a host with networking. The majority of the content is downloaded upon boot. The ISO image is about 100MB in size.

Once this is complete, you must create an image for the Nutanix platform and create the Nutanix virtual machines.

Prerequisites

  • You have set up the Assisted Installer API authentication.
  • You have created an Assisted Installer cluster profile.
  • You have created an Assisted Installer infrastructure environment.
  • You have your infrastructure environment ID exported in your shell as $INFRA_ENV_ID.
  • You have completed the Assisted Installer cluster configuration.
  • You have a Nutanix cluster environment set up, and made a note of the cluster name and subnet name.

Procedure

  1. Configure the discovery image if you want it to boot with an ignition file.
  2. Create a Nutanix cluster configuration file to hold the environment variables:

    $ touch ~/nutanix-cluster-env.sh
    $ chmod +x ~/nutanix-cluster-env.sh

    If you have to start a new terminal session, you can reload the environment variables easily. For example:

    $ source ~/nutanix-cluster-env.sh
  3. Assign the Nutanix cluster’s name to the NTX_CLUSTER_NAME environment variable in the configuration file:

    $ cat << EOF >> ~/nutanix-cluster-env.sh
    export NTX_CLUSTER_NAME=<cluster_name>
    EOF

    Replace <cluster_name> with the name of the Nutanix cluster.

  4. Assign the Nutanix cluster’s subnet name to the NTX_SUBNET_NAME environment variable in the configuration file:

    $ cat << EOF >> ~/nutanix-cluster-env.sh
    export NTX_SUBNET_NAME=<subnet_name>
    EOF

    Replace <subnet_name> with the name of the Nutanix cluster’s subnet.

  5. Refresh the API token:

    $ source refresh-token
  6. Get the download URL:

    $ curl -H "Authorization: Bearer ${API_TOKEN}" \
    https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/downloads/image-url
  7. Create the Nutanix image configuration file:

    $ cat << EOF > create-image.json
    {
      "spec": {
        "name": "ocp_ai_discovery_image.iso",
        "description": "ocp_ai_discovery_image.iso",
        "resources": {
          "architecture": "X86_64",
          "image_type": "ISO_IMAGE",
          "source_uri": "<image_url>",
          "source_options": {
            "allow_insecure_connection": true
          }
        }
      },
      "metadata": {
        "spec_version": 3,
        "kind": "image"
      }
    }
    EOF

    Replace <image_url> with the image URL downloaded from the previous step.

  8. Create the Nutanix image:

    $ curl  -k -u <user>:'<password>' -X 'POST' \
    'https://<domain-or-ip>:<port>/api/nutanix/v3/images \
      -H 'accept: application/json' \
      -H 'Content-Type: application/json' \
      -d @./create-image.json | jq '.metadata.uuid'
    • Replace <user> with the Nutanix user name.
    • Replace '<password>' with the Nutanix password.
    • Replace <domain-or-ip> with the domain name or IP address of the Nutanix plaform.
    • Replace <port> with the port for the Nutanix server. The port defaults to 9440.
  9. Assign the returned UUID to the NTX_IMAGE_UUID environment variable in the configuration file:

    $ cat << EOF >> ~/nutanix-cluster-env.sh
    export NTX_IMAGE_UUID=<uuid>
    EOF
  10. Get the Nutanix cluster UUID:

    $ curl -k -u <user>:'<password>' -X 'POST' \
      'https://<domain-or-ip>:<port>/api/nutanix/v3/clusters/list' \
      -H 'accept: application/json' \
      -H 'Content-Type: application/json' \
      -d '{
      "kind": "cluster"
    }'  | jq '.entities[] | select(.spec.name=="<nutanix_cluster_name>") | .metadata.uuid'
    • Replace <user> with the Nutanix user name.
    • Replace '<password>' with the Nutanix password.
    • Replace <domain-or-ip> with the domain name or IP address of the Nutanix platform.
    • Replace <port> with the port for the Nutanix server. The port defaults to 9440.
    • Replace <nutanix_cluster_name> with the name of the Nutanix cluster.
  11. Assign the returned Nutanix cluster UUID to the NTX_CLUSTER_UUID environment variable in the configuration file:

    $ cat << EOF >> ~/nutanix-cluster-env.sh
    export NTX_CLUSTER_UUID=<uuid>
    EOF

    Replace <uuid> with the returned UUID of the Nutanix cluster.

  12. Get the Nutanix cluster’s subnet UUID:

    $ curl -k -u <user>:'<password>' -X 'POST' \
      'https://<domain-or-ip>:<port>/api/nutanix/v3/subnets/list' \
      -H 'accept: application/json' \
      -H 'Content-Type: application/json' \
      -d '{
      "kind": "subnet",
      "filter": "name==<subnet_name>"
    }' | jq '.entities[].metadata.uuid'
    • Replace <user> with the Nutanix user name.
    • Replace '<password>' with the Nutanix password.
    • Replace <domain-or-ip> with the domain name or IP address of the Nutanix platform.
    • Replace <port> with the port for the Nutanix server. The port defaults to 9440.
    • Replace <subnet_name> with the name of the cluster’s subnet.
  13. Assign the returned Nutanix subnet UUID to the NTX_CLUSTER_UUID environment variable in the configuration file:

    $ cat << EOF >> ~/nutanix-cluster-env.sh
    export NTX_SUBNET_UUID=<uuid>
    EOF

    Replace <uuid> with the returned UUID of the cluster subnet.

  14. Ensure the Nutanix environment variables are set:

    $ source ~/nutanix-cluster-env.sh
  15. Create a VM configuration file for each Nutanix host. Create three to five control plane (master) VMs and at least two worker VMs. For example:

    $ touch create-master-0.json
    $ cat << EOF > create-master-0.json
    {
       "spec": {
          "name": "<host_name>",
          "resources": {
             "power_state": "ON",
             "num_vcpus_per_socket": 1,
             "num_sockets": 16,
             "memory_size_mib": 32768,
             "disk_list": [
                {
                   "disk_size_mib": 122880,
                   "device_properties": {
                      "device_type": "DISK"
                   }
                },
                {
                   "device_properties": {
                      "device_type": "CDROM"
                   },
                   "data_source_reference": {
                     "kind": "image",
                     "uuid": "$NTX_IMAGE_UUID"
                  }
                }
             ],
             "nic_list": [
                {
                   "nic_type": "NORMAL_NIC",
                   "is_connected": true,
                   "ip_endpoint_list": [
                      {
                         "ip_type": "DHCP"
                      }
                   ],
                   "subnet_reference": {
                      "kind": "subnet",
                      "name": "$NTX_SUBNET_NAME",
                      "uuid": "$NTX_SUBNET_UUID"
                   }
                }
             ],
             "guest_tools": {
                "nutanix_guest_tools": {
                   "state": "ENABLED",
                   "iso_mount_state": "MOUNTED"
                }
             }
          },
          "cluster_reference": {
             "kind": "cluster",
             "name": "$NTX_CLUSTER_NAME",
             "uuid": "$NTX_CLUSTER_UUID"
          }
       },
       "api_version": "3.1.0",
       "metadata": {
          "kind": "vm"
       }
    }
    EOF

    Replace <host_name> with the name of the host.

  16. Boot each Nutanix virtual machine:

    $ curl -k -u <user>:'<password>' -X 'POST' \
      'https://<domain-or-ip>:<port>/api/nutanix/v3/vms' \
      -H 'accept: application/json' \
      -H 'Content-Type: application/json' \
      -d @./<vm_config_file_name> | jq '.metadata.uuid'
    • Replace <user> with the Nutanix user name.
    • Replace '<password>' with the Nutanix password.
    • Replace <domain-or-ip> with the domain name or IP address of the Nutanix platform.
    • Replace <port> with the port for the Nutanix server. The port defaults to 9440.
    • Replace <vm_config_file_name> with the name of the VM configuration file.
  17. Assign the returned VM UUID to a unique environment variable in the configuration file:

    $ cat << EOF >> ~/nutanix-cluster-env.sh
    export NTX_MASTER_0_UUID=<uuid>
    EOF

    Replace <uuid> with the returned UUID of the VM.

    Note

    The environment variable must have a unique name for each VM.

  18. Wait until the Assisted Installer has discovered each VM and they have passed validation.

    $ curl -s -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID"
    --header "Content-Type: application/json"
    -H "Authorization: Bearer $API_TOKEN"
    | jq '.enabled_host_count'
  19. Modify the cluster definition to enable integration with Nutanix:

    $ curl https://api.openshift.com/api/assisted-install/v2/clusters/${CLUSTER_ID} \
    -X PATCH \
    -H "Authorization: Bearer ${API_TOKEN}" \
    -H "Content-Type: application/json" \
    -d '
    {
        "platform_type":"nutanix"
    }
    ' | jq
  20. Continue with the installation procedure.

12.3. Nutanix postinstallation configuration

Complete and validate the OpenShift Container Platform integration with the Nutanix cloud provider.

Note

By default, the installation program downloads and installs the Red Hat Enterprise Linux CoreOS (RHCOS) image. If Prism Central does not have internet access, you can host the RHCOS image on any HTTP server and point the installation program to the image or you can use Prism Central to upload the image manually.

After installing OpenShift Container Platform on the Nutanix platform by using the Assisted Installer, update the following Nutanix configuration settings manually.

Prerequisites

  • You have your Nutanix Prism Element username.
  • You have your Nutanix Prism Element password.
  • You have your Nutanix Prism storage container.
  • The Assisted Installer has finished installing the cluster successfully.
  • You have connected the cluster to the Red Hat console.redhat.com.
  • You have access to the Red Hat OpenShift Container Platform command line interface.

Procedure

  1. In the OpenShift Container Platform command line interface, update the Nutanix cluster configuration settings:

    $ oc patch infrastructure/cluster --type=merge --patch-file=/dev/stdin <<-EOF
    {
      "spec": {
        "platformSpec": {
          "nutanix": {
            "prismCentral": {
              "address": "<prismcentral_address>",
              "port": <prismcentral_port>
            },
            "prismElements": [
              {
                "endpoint": {
                  "address": "<prismelement_address>",
                  "port": <prismelement_port>
                },
                "name": "<prismelement_clustername>"
              }
            ]
          },
          "type": "Nutanix"
        }
      }
    }
    EOF
    • Replace <prismcentral_address> with the Nutanix Prism Central address.
    • Replace <prismcentral_port> with the Nutanix Prism Central port.
    • Replace <prismelement_address> with the Nutanix Prism Element address.
    • Replace <prismelement_port> with the Nutanix Prism Element port.
    • Replace <prismelement_clustername> with the Nutanix Prism Element cluster name.

    Example output:

    infrastructure.config.openshift.io/cluster patched

    For additional details, see Creating a compute machine set on Nutanix.

    Note

    Optional: You can define prism category key and value pairs. These category key-value pairs must exist in Prism Central. You can define the key-value pairs in separate categories for compute nodes, control plane nodes, or all nodes.

  2. Create the Nutanix secret:

    $ cat <<EOF | oc create -f -
    apiVersion: v1
    kind: Secret
    metadata:
       name: nutanix-credentials
       namespace: openshift-machine-api
    type: Opaque
    stringData:
      credentials: |
    [{"type":"basic_auth","data":{"prismCentral":{"username":"${<prismcentral_username>}","password":"${<prismcentral_password>}"},"prismElements":null}}]
    EOF
    • Replace <prismcentral_username> with the Nutanix Prism Central username.
    • Replace <prismcentral_password> with the Nutanix Prism Central password.

    Example output:

    secret/nutanix-credentials created
  3. When installing OpenShift Container Platform version 4.13 or later, update the Nutanix cloud provider configuration:

    1. Get the Nutanix cloud provider configuration YAML file:

      $ oc get cm cloud-provider-config -o yaml -n openshift-config > cloud-provider-config-backup.yaml
    2. Create a backup of the configuration file:

      $ cp cloud-provider-config_backup.yaml cloud-provider-config.yaml
    3. Open the configuration YAML file:

      $ vi cloud-provider-config.yaml
    4. Edit the configuration YAML file as follows:

      kind: ConfigMap
      apiVersion: v1
      metadata:
        name: cloud-provider-config
        namespace: openshift-config
      data:
        config: |
          {
          	"prismCentral": {
          		"address": "<prismcentral_address>",
          		"port":<prismcentral_port>,
          		"credentialRef": {
          		   "kind": "Secret",
          		   "name": "nutanix-credentials",
          		   "namespace": "openshift-cloud-controller-manager"
          		}
          	},
          	"topologyDiscovery": {
          		"type": "Prism",
          		"topologyCategories": null
          	},
          	"enableCustomLabeling": true
          }
    5. Apply the configuration updates:

      $ oc apply -f cloud-provider-config.yaml

      Example output:

      Warning: resource configmaps/cloud-provider-config is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically.
      
      configmap/cloud-provider-config configured

12.3.2. Creating the Nutanix CSI Operator group

Create an Operator group for the Nutanix CSI Operator.

For a description of operator groups and related concepts, see Understanding Operators.

Prerequisites

  • You have updated the Nutanix configuration settings.

Procedure

  1. Open the Nutanix CSI Operator Group YAML file:

    $ vi openshift-cluster-csi-drivers-operator-group.yaml
  2. Edit the YAML file as follows:

    apiVersion: operators.coreos.com/v1
    kind: OperatorGroup
    metadata:
      generateName: openshift-cluster-csi-drivers
      namespace: openshift-cluster-csi-drivers
    spec:
      targetNamespaces:
      - openshift-cluster-csi-drivers
      upgradeStrategy: Default
  3. Create the Operator Group:

    $ oc create -f openshift-cluster-csi-drivers-operator-group.yaml

    Example output:

    operatorgroup.operators.coreos.com/openshift-cluster-csi-driversjw9cd created

12.3.3. Installing the Nutanix CSI Operator

The Nutanix Container Storage Interface (CSI) Operator for Kubernetes deploys and manages the Nutanix CSI Driver.

For instructions on performing this step through the OpenShift Container Platform web console, see the Installing the Operator section of the Nutanix CSI Operator documentation.

Prerequisites

  • You have created the Nutanix CSI Operator group.

Procedure

  1. Get the parameter values for the Nutanix CSI Operator YAML file:

    1. Check that the Nutanix CSI Operator exists:

      $ oc get packagemanifests | grep nutanix

      Example output:

      nutanixcsioperator   Certified Operators   129m
    2. Assign the default channel for the Operator to a BASH variable:

      $ DEFAULT_CHANNEL=$(oc get packagemanifests nutanixcsioperator -o jsonpath={.status.defaultChannel})
    3. Assign the starting cluster service version (CSV) for the Operator to a BASH variable:

      $ STARTING_CSV=$(oc get packagemanifests nutanixcsioperator -o jsonpath=\{.status.channels[*].currentCSV\})
    4. Assign the catalog source for the subscription to a BASH variable:

      $ CATALOG_SOURCE=$(oc get packagemanifests nutanixcsioperator -o jsonpath=\{.status.catalogSource\})
    5. Assign the Nutanix CSI Operator source namespace to a BASH variable:

      $ SOURCE_NAMESPACE=$(oc get packagemanifests nutanixcsioperator -o jsonpath=\{.status.catalogSourceNamespace\})
  2. Create the Nutanix CSI Operator YAML file using the BASH variables:

    $ cat << EOF > nutanixcsioperator.yaml
    apiVersion: operators.coreos.com/v1alpha1
    kind: Subscription
    metadata:
      name: nutanixcsioperator
      namespace: openshift-cluster-csi-drivers
    spec:
      channel: $DEFAULT_CHANNEL
      installPlanApproval: Automatic
      name: nutanixcsioperator
      source: $CATALOG_SOURCE
      sourceNamespace: $SOURCE_NAMESPACE
      startingCSV: $STARTING_CSV
    EOF
  3. Create the CSI Nutanix Operator:

    $ oc apply -f nutanixcsioperator.yaml

    Example output:

    subscription.operators.coreos.com/nutanixcsioperator created
  4. Run the following command until the Operator subscription state changes to AtLatestKnown. This indicates that the Operator subscription has been created, and might take some time.

    $ oc get subscription nutanixcsioperator -n openshift-cluster-csi-drivers -o 'jsonpath={..status.state}'

12.3.4. Deploying the Nutanix CSI storage driver

The Nutanix Container Storage Interface (CSI) Driver for Kubernetes provides scalable and persistent storage for stateful applications.

For instructions on performing this step through the OpenShift Container Platform web console, see the Installing the CSI Driver using the Operator section of the Nutanix CSI Operator documentation.

Prerequisites

  • You have installed the Nutanix CSI Operator.

Procedure

  1. Create a NutanixCsiStorage resource to deploy the driver:

    $ cat <<EOF | oc create -f -
    apiVersion: crd.nutanix.com/v1alpha1
    kind: NutanixCsiStorage
    metadata:
      name: nutanixcsistorage
      namespace: openshift-cluster-csi-drivers
    spec: {}
    EOF

    Example output:

    snutanixcsistorage.crd.nutanix.com/nutanixcsistorage created
  2. Create a Nutanix secret YAML file for the CSI storage driver:

    $ cat <<EOF | oc create -f -
    apiVersion: v1
    kind: Secret
    metadata:
      name: ntnx-secret
      namespace: openshift-cluster-csi-drivers
    stringData:
      # prism-element-ip:prism-port:admin:password
      key: <prismelement_address:prismelement_port:prismcentral_username:prismcentral_password>
    EOF

    Replace the key parameters with actual values while keeping the same format.

    Example output:

    secret/nutanix-secret created

Verify that you can create a storage class and a bound persistent volume claim.

Prerequisites

  • You have deployed the Nutanix CSI storage driver.

Procedure

  1. Verify that you can create a storage class:

    $ cat <<EOF | oc create -f -
    kind: StorageClass
    apiVersion: storage.k8s.io/v1
    metadata:
      name: nutanix-volume
      annotations:
        storageclass.kubernetes.io/is-default-class: 'true'
    provisioner: csi.nutanix.com
    parameters:
      csi.storage.k8s.io/fstype: ext4
      csi.storage.k8s.io/provisioner-secret-namespace: openshift-cluster-csi-drivers
      csi.storage.k8s.io/provisioner-secret-name: ntnx-secret
      storageContainer: <nutanix_storage_container>
      csi.storage.k8s.io/controller-expand-secret-name: ntnx-secret
      csi.storage.k8s.io/node-publish-secret-namespace: openshift-cluster-csi-drivers
      storageType: NutanixVolumes
      csi.storage.k8s.io/node-publish-secret-name: ntnx-secret
      csi.storage.k8s.io/controller-expand-secret-namespace: openshift-cluster-csi-drivers
    reclaimPolicy: Delete
    allowVolumeExpansion: true
    volumeBindingMode: Immediate
    EOF
    Note

    Take <nutanix_storage_container> from the Nutanix configuration; for example, SelfServiceContainer.

    Example output:

    storageclass.storage.k8s.io/nutanix-volume created
  2. Verify that you can create the Nutanix persistent volume claim (PVC):

    1. Create the persistent volume claim (PVC):

      $ cat <<EOF | oc create -f -
      kind: PersistentVolumeClaim
      apiVersion: v1
      metadata:
        name: nutanix-volume-pvc
        namespace: openshift-cluster-csi-drivers
        annotations:
          volume.beta.kubernetes.io/storage-provisioner: csi.nutanix.com
          volume.kubernetes.io/storage-provisioner: csi.nutanix.com
        finalizers:
          - kubernetes.io/pvc-protection
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        storageClassName: nutanix-volume
        volumeMode: Filesystem
      EOF

      Example output:

      persistentvolumeclaim/nutanix-volume-pvc created
    2. Validate that the persistent volume claim (PVC) status is Bound:

      $ oc get pvc -n openshift-cluster-csi-drivers

      Example output:

      NAME                 STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS     AGE
      nutanix-volume-pvc   Bound                                        nutanix-volume   52s

Chapter 13. Installing on vSphere

The Assisted Installer integrates the OpenShift Container Platform cluster with the vSphere platform, which exposes the Machine API to vSphere and enables auto-scaling.

General support for the following Broadcom products ends on October 2nd, 2025:

  • VMware vSphere 7.x
  • VMware vCenter 7.x
  • VMware Cloud Foundation (VCF) 4.x

This change follows Broadcom’s earlier End of General Support (EoGS) announcement, which extended support for these products up to this date.

For details, see the following references:

13.2. Adding hosts on vSphere

You can add hosts to the Assisted Installer cluster using the online vSphere client or the govc vSphere CLI tool. The following procedure demonstrates adding hosts with the govc CLI tool. To use the online vSphere Client, see the documentation for vSphere.

To add hosts on vSphere with the vSphere govc CLI, generate the discovery image ISO from the Assisted Installer. The minimal discovery image ISO is the default setting. This image includes only what is required to boot a host with networking. The majority of the content is downloaded upon boot. The ISO image is about 100MB in size.

After this is complete, you must create an image for the vSphere platform and create the vSphere virtual machines.

Prerequisites

  • You are using vSphere 8.0 or higher.
  • You have the vSphere govc CLI tool installed and configured.
  • You have set clusterSet disk.EnableUUID to TRUE in vSphere.
  • You have created a cluster in the Assisted Installer web console, or
  • You have created an Assisted Installer cluster profile and infrastructure environment with the API.
  • You have exported your infrastructure environment ID in your shell as $INFRA_ENV_ID.

Procedure

  1. Configure the discovery image if you want it to boot with an ignition file.
  2. In Cluster details, select vSphere from the Integrate with external partner platforms dropdown list. The Include custom manifest checkbox is optional.
  3. In Host discovery, click the Add hosts button and select the provisioning type.
  4. Add an SSH public key so that you can connect to the vSphere VMs as the core user. Having a login to the cluster hosts can provide you with debugging information during the installation.

    1. If you do not have an existing SSH key pair on your local machine, follow the steps in Generating a key pair for cluster node SSH access.
    2. In the SSH public key field, click Browse to upload the id_rsa.pub file containing the SSH public key. Alternatively, drag and drop the file into the field from the file manager. To see the file in the file manager, select Show hidden files in the menu.
  5. Select the required discovery image ISO.

    Note

    Minimal image file: Provision with virtual media downloads a smaller image that will fetch the data needed to boot.

  6. In Networking, select Cluster-managed networking or User-managed networking:

    1. Optional: If the cluster hosts require the use of a proxy, select Configure cluster-wide proxy settings. Enter the username, password, required domains or IP addresses, and port for the HTTP and HTTPS URLs of the proxy server. If the cluster hosts are behind a firewall, allow the nodes to access the required domains or IP addresses through the firewall. See Configuring your firewall for OpenShift Container Platform for more information.

      Note

      The proxy username and password must be URL-encoded.

    2. Optional: If the cluster hosts are in a network with a re-encrypting man-in-the-middle (MITM) proxy or the cluster needs to trust certificates for other purposes such as container image registries, select Configure cluster-wide trusted certificates and add the additional certificates.
    3. Optional: Configure the discovery image if you want to boot it with an ignition file. For details, see Additional Resources.
  7. Click Generate Discovery ISO.
  8. Copy the Discovery ISO URL.
  9. Download the discovery ISO:

    $ wget - O vsphere-discovery-image.iso <discovery_url>

    Replace <discovery_url> with the Discovery ISO URL from the preceding step.

  10. On the command line, power off and delete any preexisting virtual machines:

    $ for VM in $(/usr/local/bin/govc ls /<datacenter>/vm/<folder_name>)
    do
     	/usr/local/bin/govc vm.power -off $VM
     	/usr/local/bin/govc vm.destroy $VM
    done
    • Replace <datacenter> with the name of the data center.
    • Replace <folder_name> with the name of the VM inventory folder.
  11. Remove preexisting ISO images from the data store, if there are any:

    $ govc datastore.rm -ds <iso_datastore> <image>
    • Replace <iso_datastore> with the name of the data store.
    • Replace <image> with the name of the ISO image.
  12. Upload the Assisted Installer discovery ISO:

    $ govc datastore.upload -ds <iso_datastore>  vsphere-discovery-image.iso

    Replace <iso_datastore> with the name of the data store.

    Note

    All nodes in the cluster must boot from the discovery image.

  13. Boot three to five control plane nodes:

    $ govc vm.create -net.adapter <network_adapter_type> \
                     -disk.controller <disk_controller_type> \
                     -pool=<resource_pool> \
                     -c=16 \
                     -m=32768 \
                     -disk=120GB \
                     -disk-datastore=<datastore_file> \
                     -net.address="<nic_mac_address>" \
                     -iso-datastore=<iso_datastore> \
                     -iso="vsphere-discovery-image.iso" \
                     -folder="<inventory_folder>" \
                     <hostname>.<cluster_name>.example.com

    See vm.create for details.

    Note

    The foregoing example illustrates the minimum required resources for control plane nodes.

  14. Boot at least two worker nodes:

    $ govc vm.create -net.adapter <network_adapter_type> \
                     -disk.controller <disk_controller_type> \
                     -pool=<resource_pool> \
                     -c=4 \
                     -m=8192 \
                     -disk=120GB \
                     -disk-datastore=<datastore_file> \
                     -net.address="<nic_mac_address>" \
                     -iso-datastore=<iso_datastore> \
                     -iso="vsphere-discovery-image.iso" \
                     -folder="<inventory_folder>" \
                     <hostname>.<cluster_name>.example.com

    See vm.create for details.

    Note

    The foregoing example illustrates the minimum required resources for worker nodes.

  15. Ensure the VMs are running:

    $ govc ls /<datacenter>/vm/<folder_name>
    • Replace <datacenter> with the name of the data center.
    • Replace <folder_name> with the name of the VM inventory folder.
  16. After 2 minutes, shut down the VMs:

    $ for VM in $(govc ls /<datacenter>/vm/<folder_name>)
    do
         govc vm.power -s=true $VM
    done
    • Replace <datacenter> with the name of the data center.
    • Replace <folder_name> with the name of the VM inventory folder.
  17. Set the disk.EnableUUID setting to TRUE:

    $ for VM in $(govc ls /<datacenter>/vm/<folder_name>)
    do
         govc vm.change -vm $VM -e disk.EnableUUID=TRUE
    done
    • Replace <datacenter> with the name of the data center.
    • Replace <folder_name> with the name of the VM inventory folder.

      Note

      You must set disk.EnableUUID to TRUE on all of the nodes to enable autoscaling with vSphere.

  18. Restart the VMs:

    $ for VM in $(govc ls /<datacenter>/vm/<folder_name>)
    do
         govc vm.power -on=true $VM
    done
    • Replace <datacenter> with the name of the data center.
    • Replace <folder_name> with the name of the VM inventory folder.
  19. Return to the Assisted Installer user interface and wait until the Assisted Installer discovers the hosts and each of them have a Ready status.
  20. Select roles if needed.
  21. In Networking, clear the Allocate IPs via DHCP server checkbox.
  22. Set the API VIP address.
  23. Set the Ingress VIP address.
  24. Continue with the installation procedure.

After installing an OpenShift Container Platform cluster by using the Assisted Installer on vSphere with the platform integration feature enabled, you must update the following vSphere configuration settings manually:

  • vCenter username
  • vCenter password
  • vCenter address
  • vCenter cluster
  • Data center
  • Data store
  • Folder

Prerequisites

  • The Assisted Installer has finished installing the cluster successfully.
  • The cluster is connected to console.redhat.com.

Procedure

  1. Generate a base64-encoded username and password for vCenter:

    $ echo -n "<vcenter_username>" | base64 -w0

    Replace <vcenter_username> with your vCenter username.

    $ echo -n "<vcenter_password>" | base64 -w0

    Replace <vcenter_password> with your vCenter password.

  2. Backup the vSphere credentials:

    $ oc get secret vsphere-creds -o yaml -n kube-system > creds_backup.yaml
  3. Edit the vSphere credentials:

    $ cp creds_backup.yaml vsphere-creds.yaml
    $ vi vsphere-creds.yaml
    apiVersion: v1
    data:
      <vcenter_address>.username: <vcenter_username_encoded>
      <vcenter_address>.password: <vcenter_password_encoded>
    kind: Secret
    metadata:
      annotations:
        cloudcredential.openshift.io/mode: passthrough
      creationTimestamp: "2022-01-25T17:39:50Z"
      name: vsphere-creds
      namespace: kube-system
      resourceVersion: "2437"
      uid: 06971978-e3a5-4741-87f9-2ca3602f2658
    type: Opaque
    • Replace <vcenter_address> with the vCenter address.
    • Replace <vcenter_username_encoded> with the base64-encoded version of your vSphere username.
    • Replace <vcenter_password_encoded> with the base64-encoded version of your vSphere password.
  4. Replace the vSphere credentials:

    $ oc replace -f vsphere-creds.yaml
  5. Redeploy the kube-controller-manager pods:

    $ oc patch kubecontrollermanager cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' \
    --type=merge
  6. Backup the vSphere cloud provider configuration:

    $ oc get cm cloud-provider-config -o yaml -n openshift-config > \
    cloud-provider-config_backup.yaml
  7. Edit the cloud provider configuration:

    1. For OCP versions 4.17 and earlier, use the INI syntax for the data.config section of the .yaml file.

      $ vi cloud-provider-config.yaml
      apiVersion: v1
      data:
        config: |
          [Global]
          secret-name = "vsphere-creds"
          secret-namespace = "kube-system"
          insecure-flag = "1"
      
          [Workspace]
          server = "<vcenter_address>"
          datacenter = "<datacenter>"
          default-datastore = "<datastore>"
          folder = "/<datacenter>/vm/<folder>"
      
          [VirtualCenter "<vcenter_address>"]
          datacenters = "<datacenter>"
      kind: ConfigMap
      metadata:
        creationTimestamp: "2022-01-25T17:40:49Z"
        name: cloud-provider-config
        namespace: openshift-config
        resourceVersion: "2070"
        uid: 80bb8618-bf25-442b-b023-b31311918507
      • Replace <vcenter_address> with the vCenter address.
      • Replace <datacenter> with the name of the data center.
      • Replace <datastore> with the name of the data store.
      • Replace <folder> with the folder containing the cluster VMs.
    2. For OCP versions 4.18 and later, use YAML syntax for the data.config section of the .yaml file.

      $ cp cloud-provider-config_backup.yaml cloud-provider-config.yaml
      $ vi cloud-provider-config.yaml
      apiVersion: v1
      data:
        config: |
          global:
            user: ""
            password: ""
            server: ""
            port: 0
            insecureFlag: true
            datacenters: []
            soapRoundtripCount: 0
            caFile: ""
            thumbprint: ""
            secretName: vsphere-creds
            secretNamespace: kube-system
            secretsDirectory: ""
            apiDisable: false
            apiBinding: ""
            ipFamily: []
          vcenter:
            <vcenter_address>:
              user: ""
              password: ""
              tenantref: ""
              server: "<vcenter_address>"
              port: <vcenter_port>
              insecureFlag: true
              datacenters:
              - <datacenter>
              soapRoundtripCount: 0
              caFile: ""
              thumbprint: ""
              secretref: ""
              secretName: ""
              secretNamespace: ""
              ipFamily: []
          labels:
            zone: ""
            region: ""
      kind: ConfigMap
      • Replace <vcenter_address> with the vCenter address.
      • Replace <vcenter_port> with the vCenter port number. Port 443 is the default.
      • Replace <datacenter> with the name of the data center.
  8. Apply the cloud provider configuration:

    $ oc apply -f cloud-provider-config.yaml
  9. Taint the nodes with the uninitialized taint:

    Important

    Follow steps 9 through 12 if you are installing OpenShift Container Platform 4.13 or later.

    1. Identify the nodes to taint:

      $ oc get nodes
    2. Run the following command for each node:

      $ oc adm taint node <node_name> node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule

      Replace <node_name> with the name of the node.

    Example:

    $ oc get nodes
    NAME                STATUS   ROLES                  AGE   VERSION
    master-0   Ready    control-plane,master   45h   v1.26.3+379cd9f
    master-1   Ready    control-plane,master   45h   v1.26.3+379cd9f
    worker-0   Ready    worker                 45h   v1.26.3+379cd9f
    worker-1   Ready    worker                 45h   v1.26.3+379cd9f
    master-2   Ready    control-plane,master   45h   v1.26.3+379cd9f
    
    $ oc adm taint node master-0 node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule
    $ oc adm taint node master-1 node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule
    $ oc adm taint node master-2 node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule
    $ oc adm taint node worker-0 node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule
    $ oc adm taint node worker-1 node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule
  10. Back up the infrastructures configuration:

    $ oc get infrastructures.config.openshift.io -o yaml > infrastructures.config.openshift.io.yaml.backup
  11. Edit the infrastructures configuration:

    $ cp infrastructures.config.openshift.io.yaml.backup infrastructures.config.openshift.io.yaml
    $ vi infrastructures.config.openshift.io.yaml
    apiVersion: v1
    items:
    - apiVersion: config.openshift.io/v1
      kind: Infrastructure
      metadata:
        creationTimestamp: "2022-05-07T10:19:55Z"
        generation: 1
        name: cluster
        resourceVersion: "536"
        uid: e8a5742c-6d15-44e6-8a9e-064b26ab347d
      spec:
        cloudConfig:
          key: config
          name: cloud-provider-config
        platformSpec:
          type: VSphere
          vsphere:
            failureDomains:
            - name: assisted-generated-failure-domain
              region: assisted-generated-region
              server: <vcenter_address>
              topology:
                computeCluster: /<data_center>/host/<vcenter_cluster>
                datacenter: <data_center>
                datastore: /<data_center>/datastore/<datastore>
                folder: "/<data_center>/path/to/folder"
                networks:
                - "VM Network"
                resourcePool: /<data_center>/host/<vcenter_cluster>/Resources
              zone: assisted-generated-zone
            nodeNetworking:
              external: {}
              internal: {}
            vcenters:
            - datacenters:
              - <data_center>
              server: <vcenter_address>
    
    kind: List
    metadata:
      resourceVersion: ""
    • Replace <vcenter_address> with your vCenter address.
    • Replace <datacenter> with the name of your vCenter data center.
    • Replace <datastore> with the name of your vCenter data store.
    • Replace <folder> with the folder containing the cluster VMs.
    • Replace <vcenter_cluster> with the vSphere vCenter cluster where OpenShift Container Platform is installed.
  12. Apply the infrastructures configuration:

    $ oc apply -f infrastructures.config.openshift.io.yaml --overwrite=true

After installing an OpenShift Container Platform cluster by using the Assisted Installer on vSphere with the platform integration feature enabled, you must update the following vSphere configuration settings manually:

  • vCenter address
  • vCenter cluster
  • vCenter username
  • vCenter password
  • Data center
  • Default data store
  • Virtual machine folder

Prerequisites

  • The Assisted Installer has finished installing the cluster successfully.
  • The cluster is connected to console.redhat.com.

Procedure

  1. In the Administrator perspective, navigate to Home → Overview.
  2. Under Status, click vSphere connection to open the vSphere connection configuration wizard.
  3. In the vCenter field, enter the network address of the vSphere vCenter server. This can be either a domain name or an IP address. It appears in the vSphere web client URL; for example https://[your_vCenter_address]/ui.
  4. In the vCenter cluster field, enter the name of the vSphere vCenter cluster where OpenShift Container Platform is installed.

    Important

    This step is mandatory if you installed OpenShift Container Platform 4.13 or later.

  5. In the Username field, enter your vSphere vCenter username.
  6. In the Password field, enter your vSphere vCenter password.

    Warning

    The system stores the username and password in the vsphere-creds secret in the kube-system namespace of the cluster. An incorrect vCenter username or password makes the cluster nodes unschedulable.

  7. In the Datacenter field, enter the name of the vSphere data center that contains the virtual machines used to host the cluster; for example, SDDC-Datacenter.
  8. In the Default data store field, enter the vSphere data store that stores the persistent data volumes; for example, /SDDC-Datacenter/datastore/datastorename.

    Warning

    Updating the vSphere data center or default data store after the configuration has been saved detaches any active vSphere PersistentVolumes.

  9. In the Virtual Machine Folder field, enter the data center folder that contains the virtual machine of the cluster; for example, /SDDC-Datacenter/vm/ci-ln-hjg4vg2-c61657-t2gzr. For the OpenShift Container Platform installation to succeed, all virtual machines comprising the cluster must be located in a single data center folder.
  10. Click Save Configuration. This updates the cloud-provider-config file in the openshift-config namespace, and starts the configuration process.
  11. Reopen the vSphere connection configuration wizard and expand the Monitored operators panel. Check that the status of the operators is either Progressing or Healthy.

Verification

The connection configuration process updates operator statuses and control plane nodes. It takes approximately an hour to complete. During the configuration process, the nodes will reboot. Previously bound PersistentVolumeClaims objects might become disconnected.

Follow the steps below to monitor the configuration process.

  1. Check that the configuration process completed successfully:

    1. In the Administrator perspective, navigate to Home > Overview.
    2. Under Status click Operators. Wait for all operator statuses to change from Progressing to All succeeded. A Failed status indicates that the configuration failed.
    3. Under Status, click Control Plane. Wait for the response rate of all Control Pane components to return to 100%. A Failed control plane component indicates that the configuration failed.

    A failure indicates that at least one of the connection settings is incorrect. Change the settings in the vSphere connection configuration wizard and save the configuration again.

  2. Check that you are able to bind PersistentVolumeClaims objects by performing the following steps:

    1. Create a StorageClass object using the following YAML:

      kind: StorageClass
      apiVersion: storage.k8s.io/v1
      metadata:
       name: vsphere-sc
      provisioner: kubernetes.io/vsphere-volume
      parameters:
       datastore: YOURVCENTERDATASTORE
       diskformat: thin
      reclaimPolicy: Delete
      volumeBindingMode: Immediate
    2. Create a PersistentVolumeClaims object using the following YAML:

      kind: PersistentVolumeClaim
      apiVersion: v1
      metadata:
       name: test-pvc
       namespace: openshift-config
       annotations:
         volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/vsphere-volume
       finalizers:
         - kubernetes.io/pvc-protection
      spec:
       accessModes:
         - ReadWriteOnce
       resources:
         requests:
          storage: 10Gi
       storageClassName: vsphere-sc
       volumeMode: Filesystem

    For instructions, see Dynamic provisioning in the OpenShift Container Platform documentation. To troubleshoot a PersistentVolumeClaims object, navigate to StoragePersistentVolumeClaims in the Administrator perspective of the OpenShift Container Platform web console.

The Oracle® Cloud Infrastructure (OCI) platform provides services that can meet your needs for regulatory compliance, performance, and cost-effectiveness. You can use Oracle Cloud Infrastructure Resource Manager to provision and configure OCI resources.

The Assisted Installer supports cluster installation on Oracle Cloud Infrastructure for these OpenShift Container Platform versions:

  • From OpenShift Container Platform 4.14 and later, you can install a cluster on an Oracle Cloud Infrastructure virtual machine by using your own infrastructure.
  • From OpenShift Container Platform 4.16 and later, you can also install a cluster on an Oracle Cloud Infrastructure bare-metal machine.
Note

Bare-metal installations that use iSCSI boot drives require a secondary vNIC that is automatically created in the Terraform stack provided by Oracle.

The following sections show the OCI deployment options and their supported infrastructure in OpenShift Container Platform. Each infrastructure can have both VM and bare-metal servers.

14.1. Installing on Oracle Distributed Cloud

You can use the Assisted Installer to install an OpenShift Container Platform (RHOCP) cluster on the following Oracle® Distributed Cloud infrastructure:

Expand
InfrastructureDescriptionCertified fromSupport status

Commercial Public Cloud

Standard global OCI regions accessible to all customers.

RHOCP 4.14

General Availability

Dedicated Region

A complete OCI region hardware and software stack located in a customer data center.

RHOCP 4.14

General Availability

US Government Cloud

Physically isolated regions for US public sector, meeting FedRAMP High and DoD requirements.

RHOCP 4.20

Technology Preview

UK Government Cloud

Dedicated UK regions for public sector, ensuring data residency and UK-citizen operations.

RHOCP 4.20

Technology Preview

EU Sovereign Cloud

EU-based regions operated by EU residents to meet strict European data privacy standards.

RHOCP 4.20

Technology Preview

Isolated Region

Air-gapped regions disconnected from the internet for top-secret/national security workloads.

RHOCP 4.20

Technology Preview

Oracle Alloy

A partner-operated platform that allows third-party providers to offer their own branded OCI services.

RHOCP 4.20

Technology Preview

For installation instructions, see Installing a cluster on Oracle Distributed Cloud by using the Assisted Installer.

Important

The installation of OpenShift Container Platform on US Government Cloud, UK Government Cloud, EU Sovereign Cloud, Isolated Region, and Oracle Alloy infrastructures is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.

14.2. Installing on Oracle Edge Cloud

You can use the Assisted Installer to install an OpenShift Container Platform (RHOCP) cluster on the following Oracle® Edge Cloud infrastructure:

Expand
InfrastructureDescriptionCertified fromSupport status

Private Cloud Appliance

A customer-managed, rack-scale engineered system for on-premise private cloud modernization.

RHOCP 4.18

General Availability

Oracle Compute Cloud@Customer

An Oracle-managed compute rack in the customer’s data center that consumes OCI services via subscription.

RHOCP 4.18

General Availability

Roving Edge

Robust, portable tactical nodes for compute and storage in remote or mobile environments.

RHOCP 4.20

Technology Preview

For installation instructions, see Installing a cluster on Oracle Edge Cloud by using the Assisted Installer.

Important

The installation of OpenShift Container Platform on a Roving Edge device is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.

14.3. Installing on Oracle Database Appliance

Oracle Database Appliance (ODA) is an engineered system designed to simplify the deployment, management, and support of Oracle databases. It includes built-in KVM virtualization and can host application virtual machines, such as Red Hat OpenShift control plane and worker nodes, alongside database workloads.

You can install an OpenShift Container Platform cluster on Oracle Database Appliance by using the Assisted Installer. This deployment is available for OpenShift Container Platform 4.21 and later releases.

For installation instructions, see Installing on Oracle Database Appliance.

Chapter 15. Troubleshooting

There are cases where the Assisted Installer cannot begin the installation or the cluster fails to install properly. In these events, it is helpful to understand the likely failure modes and how to troubleshoot them.

15.1. Troubleshooting discovery ISO issues

The Assisted Installer uses an ISO image to run an agent that registers the host to the cluster and performs hardware and network validations before attempting to install OpenShift. You can follow these procedures to troubleshoot problems related to the host discovery.

Once you start the host with the discovery ISO image, the Assisted Installer discovers the host and presents it in the Assisted Service web console. For details, see "Configuring the discovery image".

15.1.1. Verifying the discovery agent is running

Check that the discovery agent is running on the host by verifying the host machine is powered on, networking is configured correctly, and the agent service is active.

Prerequisites

  • You have created an infrastructure environment by using the API or have created a cluster by using the web console.
  • You booted a host with the Infrastructure Environment discovery ISO and the host failed to register.
  • You have SSH access to the host.
  • You provided an SSH public key in the "Add hosts" dialog before generating the Discovery ISO so that you can SSH into your machine without a password.

Procedure

  1. Verify that your host machine is powered on.
  2. If you selected DHCP networking, check that the DHCP server is enabled.
  3. If you selected Static IP, bridges and bonds networking, check that your configurations are correct.
  4. Verify that you can access your host machine using SSH, a console such as the BMC, or a virtual machine console:

    $ ssh core@<host_ip_address>

    You can specify private key file by using the -i parameter if it is not stored in the default directory.

    $ ssh -i <ssh_private_key_file> core@<host_ip_address>

    If you fail to connect over SSH to the host, the host failed during boot or it failed to configure the network.

    Upon login, the following example shows the message you should see:

    screenshot of ISO login message

    If you are not seeing this message it means that the host did not boot with the Assisted Installer ISO image. Make sure you configured the boot order properly (The host should boot once from the live-ISO).

  5. Check the agent service logs:

    $ sudo journalctl -u agent.service

    The following is a screenshot of an agent service log. In the example, the errors indicate a network issue:

    screenshot of agent service log

    If there is an error pulling the agent image, check the proxy settings. Verify that the host is connected to the network. You can use nmcli to get additional information about your network configuration.

Check the agent logs to verify that the discovery agent can access the Assisted Service.

Prerequisites

  • You have created an Infrastructure Environment by using the API or have created a cluster by using the web console.
  • You booted a host with the Infrastructure Environment discovery ISO and the host failed to register.
  • You verified the discovery agent is running.

Procedure

  • Check the agent logs to verify the agent can access the Assisted Service:

    $ sudo journalctl TAG=agent

    The following is a screenshot of an agent log. In the example, the errors indicate that the agent failed to access the Assisted Service:

    screenshot of the agent log failing to access the Assisted Service

    Check the proxy settings you configured for the cluster. If configured, the proxy must allow access to the Assisted Service URL.

15.2. Troubleshooting minimal discovery ISO issues

Use the minimal ISO image when the virtual media connection has limited bandwidth. It includes only what the agent requires to boot a host with networking. The majority of the content is downloaded upon boot. The resulting ISO image is about 100MB in size compared to 1GB for the full ISO image.

If your environment requires static network configuration to access the Assisted Installer service, any issues with that configuration might prevent the minimal ISO from booting properly.

If the boot screen shows that the host has failed to download the root file system image, the network might not be configured correctly. You can interrupt the kernel boot early in the bootstrap process, before the root file system image is downloaded. This allows you to access the root console and review the network configurations.

The following example shows a rootfs download failure:

Failed root file system image download

Procedure

  1. Add the .spec.kernelArguments stanza to the infraEnv object of the cluster you are deploying:

    Note

    For details on modifying an infrastructure environment, see Modifying an infrastructure environment.

    # ...
    spec:
      clusterRef:
        name: sno1
        namespace: sno1
      cpuArchitecture: x86_64
      ipxeScriptType: DiscoveryImageAlways
      kernelArguments:
      - operation: append
        value: rd.break=initqueue
      nmStateConfigLabelSelector:
        matchLabels:
          nmstate-label: sno1
      pullSecretRef:
        name: assisted-deployment-pull-secret

    The value rd.break=initqueue interrupts the boot at the dracut main loop. See rd.break options for debugging kernel boot for details.

  2. Wait for the related nodes to reboot automatically and for the boot to stop at the iniqueue stage, before rootfs is downloaded. You will be redirected to the root console.
  3. Identify and change the incorrect network configurations. Here are some useful diagnostic commands:

    1. View system logs by using journalctl, for example:

      # journalctl -p err //Sorts logs by errors
      # journalctl -p crit //Sorts logs by critical errors
      # journalctl -p warning //Sorts logs by warnings
    2. View network connection information by using nmcli, as follows:

      # nmcli conn show
    3. Check the configuration files for incorrect network connections, for example:

      # cat /etc/assisted/network/host0/eno3.nmconnection
  4. Press control+d to resume the bootstrap process. The server downloads rootfs and completes the process.
  5. Reopen the infraEnv object and remove the .spec.kernelArguments stanza.

15.3. Correcting a host’s boot order

Once the installation that runs as part of the Discovery Image completes, the Assisted Installer reboots the host. The host must boot from its installation disk to continue forming the cluster. If you have not correctly configured the host’s boot order, it will boot from another disk instead, interrupting the installation.

If the host boots the discovery image again, the Assisted Installer will immediately detect this event and set the host’s status to Installing Pending User Action.  Alternatively, if the Assisted Installer does not detect that the host has booted the correct disk within the allotted time, it will also set this host status.

Procedure

  1. Reboot the host and set its boot order to boot from the installation disk. If you didn’t select an installation disk, the Assisted Installer selected one for you.
  2. To view the selected installation disk, click to expand the host’s information in the host inventory, and check which disk has the “Installation disk” role.

There are cases where the Assisted Installer declares an installation to be successful even though it encountered errors:

  • If you requested to install OLM operators and one or more failed to install, log in to the cluster’s console to remediate the failures.
  • If you requested to install more than two worker nodes and at least one failed to install, but at least two succeeded, add the failed workers to the installed cluster. For details, see "Replacing a control plane node in an unhealthy cluster".

When you add a node to an existing cluster as part of Day 2 operations, the node downloads the ignition configuration file from the Day 1 cluster. If the download fails and the node is unable to connect to the cluster, the status of the host in the Host discovery step changes to Insufficient. Clicking this status displays the following error message:

The host failed to download the ignition file from <URL>. You must ensure the host can reach the URL. Check your DNS and network configuration or update the IP address or domain used to reach the cluster.

error: ignition file download failed.... no route to host

There are several possible reasons for the connectivity failure. Here are some recommended actions.

Procedure

  1. Check the IP address and domain name of the cluster:

    1. Click the set the IP or domain used to reach the cluster hyperlink.
    2. In the Update cluster hostname window, enter the correct IP address or domain name for the cluster.
  2. Check your DNS settings to ensure that the DNS can resolve the domain that you provided.
  3. Ensure that port 22624 is open in all firewalls.
  4. Check the agent logs of the host to verify that the agent can access the Assisted Service via SSH:

    $ sudo journalctl TAG=agent
    Note

    For more details, see the "Verifying the agent can access the Assisted Service" troubleshooting section.

When using auto-assign for host roles, you might encounter validation errors related to the internal suggested_role field.

The Assisted Installer assigns two role types to each host:

  • role: This is the primary, visible role for the host. You can explicitly set the role as either control plane, arbiter, or worker. Alternatively, you can choose auto-assign, which allows the Assisted Installer to determine the appropriate role automatically.
  • suggested_role: When a host is set to auto-assign, the Assisted Installer assigns it a suggested_role value at the start of the installation process. This is an internal role type and is not visible to the user. The Assisted Installer determines the suggested_role based on its internal logic. For details, see Supported host roles.

The web console does not display the suggested_role field. Therefore, when using auto-assign, you might still encounter host pre-installation validation errors that suggest a host has a specific role. This happens because the Assisted Installer relies on suggested_role for certain validations.

Procedure

  • If you encounter an error, either manually assign a role to each affected node instead of using auto-assign, or follow the instructions provided in the error message, where applicable.

Legal Notice

Copyright © Red Hat

OpenShift documentation is licensed under the Apache License 2.0 (https://www.apache.org/licenses/LICENSE-2.0).

Modified versions must remove all Red Hat trademarks.

Portions adapted from https://github.com/kubernetes-incubator/service-catalog/ with modifications by Red Hat.

Red Hat, Red Hat Enterprise Linux, the Red Hat logo, the Shadowman logo, JBoss, OpenShift, Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries.

Linux® is the registered trademark of Linus Torvalds in the United States and other countries.

Java® is a registered trademark of Oracle and/or its affiliates.

XFS® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries.

MySQL® is a registered trademark of MySQL AB in the United States, the European Union and other countries.

Node.js® is an official trademark of the OpenJS Foundation.

The OpenStack® Word Mark and OpenStack logo are either registered trademarks/service marks or trademarks/service marks of the OpenStack Foundation, in the United States and other countries and are used with the OpenStack Foundation’s permission. We are not affiliated with, endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.

All other trademarks are the property of their respective owners.

Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2026 Red Hat
Back to top