Installing OpenShift Container Platform with the Assisted Installer
User Guide
Abstract
Preface Copy linkLink copied to clipboard!
Providing feedback on the Assisted Installer for OpenShift Container Platform documentation
You can give feedback or report an error in our documentation by creating a Jira issue. You must have a Red Hat Jira account.
- Log in to Jira.
- Click Create Issue to launch the form in a browser.
- Complete the Summary, Description, and Reporter fields.
Click Create to submit the form.
The form creates an issue in the Red Hat Hybrid Cloud Infrastructure (HCIDOCS) Jira project.
Chapter 1. About the Assisted Installer Copy linkLink copied to clipboard!
The Assisted Installer for Red Hat OpenShift Container Platform is a user-friendly installation solution offered on the Red Hat Hybrid Cloud Console. The Assisted Installer supports various deployment platforms with a focus on bare metal, Nutanix, vSphere, and Oracle Cloud Infrastructure. The Assisted Installer also supports various CPU architectures, including x86_64, s390x (IBM Z®), arm64, and ppc64le (IBM Power®).
You can install OpenShift Container Platform on premises in a connected environment, with an optional HTTP/S proxy, for the following platforms:
- Highly available OpenShift Container Platform or single-node OpenShift cluster
- OpenShift Container Platform on bare metal or vSphere with full platform integration, or other virtualization platforms without integration
- Optionally, OpenShift Virtualization and Red Hat OpenShift Data Foundation
1.1. Features Copy linkLink copied to clipboard!
The Assisted Installer provides installation functionality as a service. This software-as-a-service (SaaS) approach has the following features:
- Web interface
- You can install your cluster by using the Hybrid Cloud Console instead of creating installation configuration files manually.
- No bootstrap node
- You do not need a bootstrap node because the bootstrapping process runs on a node within the cluster.
- Streamlined installation workflow
- You do not need in-depth knowledge of OpenShift Container Platform to deploy a cluster. The Assisted Installer provides reasonable default configurations.
- You do not need to run the OpenShift Container Platform installer locally.
- You have access to the latest Assisted Installer for the latest tested z-stream releases.
- Advanced networking options
- The Assisted Installer supports IPv4 and dual stack networking with OVN only, NMState-based static IP addressing, and an HTTP/S proxy.
- OVN is the default Container Network Interface (CNI) for OpenShift Container Platform 4.12 and later.
- SDN is supported up to OpenShift Container Platform 4.14. SDN supports IPv4 only.
- Preinstallation validation
Before installing, the Assisted Installer checks the following configurations:
- Network connectivity
- Network bandwidth
- Connectivity to the registry
- Upstream DNS resolution of the domain name
- Time synchronization between cluster nodes
- Cluster node hardware
- Installation configuration parameters
- REST API
- You can automate the installation process by using the Assisted Installer REST API.
1.2. Host topology Copy linkLink copied to clipboard!
The OpenShift Container Platform architecture allows you to select a standard Kubernetes role for each of the discovered hosts. These roles define the function of the host within the cluster.
1.2.1. About assigning roles to hosts Copy linkLink copied to clipboard!
During the installation process, you can select a role for a host or configure the Assisted Installer to assign it for you.
The host must meet the minimum requirements for the role you selected. You can find the hardware requirements by referring to the Prerequisites section of this document or using the preflight requirement API.
If you do not select a role, the system selects one for you. You can change the role at any time before installation starts.
Each host can have any of the following roles:
- Control plane (master)
- The control plane nodes run the services that are required to control the cluster, including the API server. The control plane schedules workloads, maintains cluster state, and ensures stability.
- Compute (worker)
- The compute nodes are responsible for executing workloads for cluster users. Compute nodes advertise their capacity, so that the control plane scheduler can identify suitable compute nodes for running pods and containers.
- Arbiter
Arbiter nodes are a more cost-effective alternative to control plane nodes. They function similarly but run only the essential components required to maintain the etcd quorum and prevent a split-brain condition. Because they do not host the full control plane or any workloads, arbiter nodes can use less powerful hardware.
The Assisted Installer provides arbiter nodes for Two-Node OpenShift with Arbiter (TNA) clusters. Support for Two-Node OpenShift with Arbiter clusters begins with OpenShift Container Platform version 4.19 and later. For more details, see Two-Node OpenShift with Arbiter (TNA) resource requirements.
To install a Two-Node OpenShift with Arbiter cluster, assign the arbiter or auto-assign role to at least one of the nodes, and set the control plane node count for the cluster to 2.
Two-Node OpenShift with Arbiter (TNA) is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.
- Auto-assign
The Assisted Installer sets each host to an auto-assign role by default. Auto-assign allows the Assisted Installer to automatically determine whether the host should function as a control plane, arbiter, or compute (worker) role, based on detected hardware and network latency.
To determine the most suitable role, the Assisted Installer evaluates each host’s memory, CPU, disk space, and network performance. It assigns an internal
suggested_rolevalue to each host, which drives the auto-assignment process when auto-assigned is enabled. Pre-installation validations ensure the resulting role allocation is valid.The logic for auto-assigning roles is as follows:
- The Assisted Installer sorts the hosts by their hardware capabilities, from weakest to strongest. All hosts must meet the minimum requirements.
-
The Assisted Installer assigns the control plane role to the weakest hosts until it reaches the number of control plane nodes specified by the
control_plane_countfield. A host is designated as a control plane only if it passes the necessary control plane role validations. For details on specifying the control plane count, see "Additional resources". The Assisted Installer assigns the arbiter role to a host if the following conditions are met:
- The control plane count is 2.
The host meets the minimum hardware requirements for the cluster, and either of the following are true:
- The cluster already includes two control plane nodes, either manually assigned or through auto-assign; or
- The host does not meet the minimum hardware requirements for a control plane node.
- The Assisted Installer designates all remaining hosts as worker (compute) nodes. This approach ensures that the Assisted Installer prioritizes the most capable hosts for worker roles, while still maintaining the necessary number of valid control plane and arbiter nodes.
To assign a role to a host using the web console or the API, or to troubleshoot pre-installation validation errors for hosts with an auto-assign role, see "Additional resources".
1.2.2. About number of control plane nodes Copy linkLink copied to clipboard!
Using a higher number of control plane (master) nodes boosts fault tolerance and availability, minimizing downtime during failures. The number of control plane nodes that the Assisted Installer supports varies according to OpenShift Container Platform version:
- All versions of OpenShift Container Platform support one or three control plane nodes, where one control plane node is a Single-node OpenShift cluster.
- From OpenShift Container Platform version 4.18 and later, the Assisted Installer also supports four or five control plane nodes on a bare metal or user-managed networking platform with an x86_64 architecture. An implementation can support any number of compute nodes.
- From OpenShift Container Platform version 4.19 and later, the Assisted Installer also supports two control plane nodes for a Two-Node OpenShift with Arbiter (TNA) cluster topology.A cluster with only two control plane nodes must have at least one host with an arbiter role. For details, see About assigning roles to hosts.
Two-Node OpenShift with Arbiter (TNA) is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.
To specify the required number of control plane nodes for your cluster, see either of the following sections:
1.2.3. Workload scheduling on control plane nodes Copy linkLink copied to clipboard!
For smaller clusters, scheduling workloads to run on control plane nodes improves efficiency and maximizes resource utilization. You can enable this option during installation setup or as a postinstallation step.
Use the following guidelines to determine when to use this feature:
- Single-node OpenShift clusters, Two-Node OpenShift with Arbiter clusters, or clusters with up to one worker (compute) node: The system schedules workloads on control plane nodes by default. This setting cannot be changed.
- Clusters of between two to seven worker nodes: This configuration supports the manual scheduling of workloads on both control plane (master) and compute (worker) nodes.
- Clusters with more than seven worker nodes: Scheduling workloads on control plane nodes is not recommended.
Schedulable control plane nodes have the role Control plane, Worker.
For instructions on configuring control plane nodes as schedulable during the installation setup, see the following sections:
For instructions on configuring schedulable control plane nodes following an installation, see Configuring control plane nodes as schedulable in the OpenShift Container Platform documentation.
When you configure control plane nodes to be schedulable for workloads, an additional subscription is required for each control plane node that function as a compute (worker) node.
1.3. API support policy Copy linkLink copied to clipboard!
Assisted Installer APIs are supported for a minimum of three months from the announcement of deprecation.
1.3.1. API deprecation notice Copy linkLink copied to clipboard!
The following table presents the deprecated and modified APIs in the Assisted Installer.
assisted_service API
Expand Affected models Description of change -
cluster -
cluster-create-params
The
high_availability_modefield is deprecated starting from April 2025, and is planned to be removed in three months. Red Hat will provide bug fixes and support for this feature during the current release lifecycle, but this feature will no longer receive enhancements and will be removed.The alternative is to use
control_plane_countinstead. This change enables support for clusters with 4 or 5 control plane nodes, in addition to the previously supported configurations with 1 or 3 control plane nodes. The Assisted Installer supports 4 or 5 control plane nodes starting from OpenShift Container Platform version 4.18 and later.-
Chapter 2. Prerequisites Copy linkLink copied to clipboard!
The Assisted Installer validates the following prerequisites to ensure successful installation.
If you use a firewall, you must configure it so that Assisted Installer can access the resources it requires to function.
2.1. Supported CPU architectures Copy linkLink copied to clipboard!
The Assisted Installer is supported on the following CPU architectures:
- x86_64
- arm64
- ppc64le (IBM Power®)
- s390x (IBM Z®)
2.2. Supported drive types Copy linkLink copied to clipboard!
This section lists the installation drive types that you can and cannot use when installing Red Hat OpenShift Container Platform with the Assisted Installer.
Supported drive types
The table below shows the installation drive types supported for the different OpenShift Container Platform versions and CPU architectures.
| Drive types | RHOCP Version | Supported CPU Architectures | Comments |
|---|---|---|---|
| HDD | All | All | A hard disk drive. |
| SSD | All | All | An SSD or NVMe drive. |
| Multipath | All | All | A Linux multipath device that can aggregate paths for various protocols. Using multipath enhances availablity and performance. Currently, the Assisted Installer supports multipathing for Fibre Channel and iSCSI protocals. |
| FC (Fibre Channel) | All | s390x, x86_64 | Indicates a single path Fibre Channel (FC) drive. For a multipath Fibre Channel configuration, see 'Multipath' in this table. |
| iSCSI | 4.15 and later | x86_64 |
|
| RAID | 4.14 and later | All | A software RAID drive. The RAID should be configured via BIOS/UEFI. If this option is unavailable, you can configure OpenShift Container Platform to mirror the drives. For details, see Encrypting and mirroring disks during installation. |
| ECK | All | s390x | IBM drive. |
| ECKD (ESE) | All | s390x | IBM drive. |
| FBA | All | s390x | IBM drive. |
Unsupported drive types
The table below shows the installation drive types that are not supported.
| Drive types | Comments |
|---|---|
| Unknown | The system could not detect the drive type. |
| FDD | A floppy disk drive. |
| ODD | An optical disk drive (e.g., CD-ROM). |
| Virtual | A loopback device. |
| LVM | A Linux Logical Volume Management drive. |
2.3. Resource requirements Copy linkLink copied to clipboard!
This section describes the resource requirements for different clusters and installation options.
The multicluster engine for Kubernetes requires additional resources.
If you deploy the multicluster engine with storage, such as OpenShift Data Foundation or LVM Storage, you must also assign additional resources to each node.
2.3.1. Multi-node cluster resource requirements Copy linkLink copied to clipboard!
The resource requirements of a multi-node (high-availability) cluster depend on the installation options.
- Description
- A standard OpenShift Container Platform cluster configuration consists of three to five control plane nodes and two or more worker nodes. This configuration ensures full high availability for control plane services.
- Multi-node cluster basic installation
Control plane nodes:
- 4 CPU cores
- 16 GB RAM
100 GB storage
NoteThe disks must be reasonably fast, with an etcd
wal_fsync_duration_secondsp99 duration that is less than 10 ms. For more information, see the Red Hat Knowledgebase solution How to Use 'fio' to Check Etcd Disk Performance in OCP.
Compute nodes:
- 2 CPU cores
- 8 GB RAM
- 100 GB storage
- Multi-node cluster + multicluster engine
- Additional 4 CPU cores
Additional 16 GB RAM
NoteIf you deploy multicluster engine without OpenShift Data Foundation, no storage is configured. You configure the storage after the installation.
- Multi-node cluster + multicluster engine + OpenShift Data Foundation
- Additional 75 GB storage
2.3.2. Two-Node OpenShift with Arbiter (TNA) cluster resource requirements Copy linkLink copied to clipboard!
The resource requirements of a Two-Node OpenShift with Arbiter (TNA) cluster depend on the installation options.
- Description
A Two-Node OpenShift with Arbiter (TNA) cluster is a compact, cost-effective OpenShift Container Platform topology. It consists of two control plane nodes and a lightweight arbiter node. The arbiter node stores the full etcd data, maintaining an etcd quorum and preventing split brain. It does not run the additional control plane components
kube-apiserverandkube-controller-manager, nor does it run workloads. For details, see Overview of etcd.To install a Two-Node OpenShift with Arbiter cluster, assign an arbiter role to at least one of the nodes and set the control plane node count for the cluster to
2. Although OpenShift Container Platform does not currently impose a limit on the number of arbiter nodes, the typical deployment includes only one to minimize the use of hardware resources.Following installation, you can add additional arbiter nodes to a Two-Node OpenShift with Arbiter cluster but not to a standard multi-node cluster. It is also not possible to convert between a Two-Node OpenShift with Arbiter and standard topology.
Support for a Two-Node OpenShift with Arbiter cluster begins with OpenShift Container Platform version 4.19 and later. This configuration is available only for bare-metal installations.
Two-Node OpenShift with Arbiter (TNA) is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.
- Two-Node OpenShift with Arbiter basic installation
Control plane nodes:
- 4 CPU cores
- 16 GB RAM
100 GB storage
NoteThe disks must be reasonably fast, with an etcd
wal_fsync_duration_secondsp99 duration that is less than 10 ms. For more information, see the Red Hat Knowledgebase solution How to Use 'fio' to Check Etcd Disk Performance in OCP.
Arbiter node:
- 2 CPU cores
- 8 GB RAM
- 50 GB storage
- Two-Node OpenShift with Arbiter + multicluster engine
- Additional 4 CPU cores
Additional 16 GB RAM
NoteIf you deploy multicluster engine without OpenShift Data Foundation, no storage is configured. You configure the storage after the installation.
- Two-Node OpenShift with Arbiter + multicluster engine + OpenShift Data Foundation
- Additional 75 GB storage
2.3.3. Single-node OpenShift cluster resource requirements Copy linkLink copied to clipboard!
The resource requirements for single-node OpenShift depend on the installation options.
- Description
- A single-node OpenShift cluster is an OpenShift Container Platform deployment that runs entirely on a single node. Single-node OpenShift includes the control plane and worker functionality on one physical or virtual machine.
- Single-node OpenShift basic installation
- 8 CPU cores
- 16 GB RAM
- 100 GB storage
- Single-node OpenShift + multicluster engine
- Additional 8 CPU cores
Additional 32 GB RAM
NoteIf you deploy multicluster engine without OpenShift Data Foundation, LVM Storage is enabled.
- Single-node OpenShift + multicluster engine + OpenShift Data Foundation
- Additional 95 GB storage
2.4. Networking requirements Copy linkLink copied to clipboard!
For hosts of type VMware, set clusterSet disk.EnableUUID to TRUE, even when the platform is not vSphere.
2.4.1. General networking requirements Copy linkLink copied to clipboard!
The network must meet the following requirements:
- You have configured either dynamic (DHCP) or static IP addressing.
You have selected the correct route configuration for your IP addressing method:
- For dynamic IP addressing, ensure that you have configured your network routes dynamically via DHCP.
- For static IP addressing, ensure that you have configured the network routes manually via the static networking configurations.
ImportantYou cannot combine dynamic IP addresses with static route configurations. When the Assisted Installer receives a dynamic IP address (with a
/128prefix), it specifically looks for network routes that were also configured dynamically, such as those advertised via Router Advertisement (RA). If a network route is configured manually (with a/64prefix, for example), the Assisted Installer ignores it.-
You have opened port 6443 to allow the API URL access to the cluster using the
ocCLI tool when outside the firewall. - You have opened port 22624 in all firewalls. The Machine Config Operator (MCO) and new worker nodes use port 22624 to get the ignition data from the cluster API.
- You have opened port 443 to allow console access outside the firewall. Port 443 is also used for all ingress traffic.
- You have configured DNS to connect to the cluster API or ingress endpoints from outside the cluster.
- Optional: You have created a DNS Pointer record (PTR) for each node in the cluster if using static IP addressing.
You must create a DNS PTR record to boot with a preset hostname if the hostname will not come from another source (/etc/hosts or DHCP). Otherwise, the Assisted Installer’s automatic node renaming feature will rename the nodes to their network interface MAC address.
2.4.2. External DNS Copy linkLink copied to clipboard!
Installing multi-node cluster with user-managed networking requires external DNS. External DNS is not required to install multi-node clusters with cluster-managed networking or Single-node OpenShift with the Assisted Installer. Configure external DNS after installation to connect to the cluster from an external source.
External DNS requires the creation of the following record types:
- A/AAAA record for api.<cluster_name>.<base_domain>.
- A/AAAA record with a wildcard for *.apps.<cluster_name>.<base_domain>.
- A/AAAA record for each node in the cluster.
- Do not create a wildcard, such as *.<cluster_name>.<base_domain>, or the installation will not proceed.
- A/AAAA record settings at top-level domain registrars can take significant time to update. Ensure the newly created DNS Records are resolving before installation to prevent installation delays.
- For DNS record examples, see Example DNS configuration.
The OpenShift Container Platform cluster’s network must also meet the following requirements:
- Connectivity between all cluster nodes
- Connectivity for each node to the internet
- Access to an NTP server for time synchronization between the cluster nodes
2.4.2.1. Example DNS configuration Copy linkLink copied to clipboard!
The following DNS configuration provides A and PTR record configuration examples that meet the DNS requirements for deploying OpenShift Container Platform using the Assisted Installer. The examples are not meant to recommend one DNS solution over another.
In the examples, the cluster name is ocp4 and the base domain is example.com.
Example DNS A record configuration
The following example is a BIND zone file that shows sample A records for name resolution in a cluster installed using the Assisted Installer.
Example DNS zone database
- 1
- Provides name resolution for the Kubernetes API. The record refers to the IP address of the API load balancer.
- 2
- Provides name resolution for the Kubernetes API. The record refers to the IP address of the API load balancer and is used for internal cluster communications.
- 3
- Provides name resolution for the wildcard routes. The record refers to the IP address of the application ingress load balancer. The application ingress load balancer targets the machines that run the Ingress Controller pods. The Ingress Controller pods run on the worker machines by default.Note
In the example, the same load balancer is used for the Kubernetes API and application ingress traffic. In production scenarios, you can deploy the API and application ingress load balancers separately so that you can scale the load balancer infrastructure for each in isolation.
- 4
- Provides name resolution for the control plane machines.
- 5
- Provides name resolution for the worker machines.
Example DNS PTR record configuration
The following example is a BIND zone file that shows sample PTR records for reverse name resolution in a cluster installed using the Assisted Installer.
Example DNS zone database for reverse records
- 1
- Provides reverse DNS resolution for the Kubernetes API. The PTR record refers to the record name of the API load balancer.
- 2
- Provides reverse DNS resolution for the Kubernetes API. The PTR record refers to the record name of the API load balancer and is used for internal cluster communications.
- 3
- Provides reverse DNS resolution for the control plane machines.
- 4
- Provides reverse DNS resolution for the worker machines.
A PTR record is not required for the OpenShift Container Platform application wildcard.
2.4.3. Networking requirements for IBM Z Copy linkLink copied to clipboard!
In IBM Z® environments, advanced networking technologies like Open Systems Adapter (OSA), HiperSockets, and Remote Direct Memory Access (RDMA) over Converged Ethernet (RoCE) require specific configurations that deviate from the standard settings used in Assisted Installer deployments. These overrides are necessary to accommodate their unique requirements and ensure a successful and efficient deployment on IBM Z®.
The following table lists the network devices that are supported for the network configuration override functionality:
| Network device | z/VM | KVM | LPAR Classic | LPAR Dynamic Partition Manager (DPM) |
|---|---|---|---|---|
| OSA virtual switch | Supported | Not applicable | Not applicable | Not applicable |
| Direct attached OSA | Supported | Only through a Linux bridge | Supported | Supported |
| RDMA over Converged Ethernet (RoCE) | Supported | Only through a Linux bridge | Supported | Supported |
| HiperSockets | Supported | Only through a Linux bridge | Supported | Supported |
| Linux bridge | Not applicable | Supported | Not applicable | Not applicable |
2.4.3.1. Configuring network overrides in IBM Z Copy linkLink copied to clipboard!
You can specify a static IP address on IBM Z® machines that uses Logical Partition (LPAR) and z/VM. This is specially useful when the network devices do not have a static MAC address assigned to them.
If you have an existing .parm file, edit it to include the following entry:
ai.ip_cfg_override=1
ai.ip_cfg_override=1
This parameter allows the file to add the network settings to the CoreOS installer.
Example of the .parm file
- 1
- For the
coreos.live.rootfs_urlartifact, specify the matchingrootfsartifact for thekernelandinitramfsthat you are booting. Only HTTP and HTTPS protocols are supported. - 2
- For installations on direct access storage devices (DASD) type disks, use
rd.to specify the DASD where Red Hat Enterprise Linux (RHEL) is to be installed. For installations on Fibre Channel Protocol (FCP) disks, userd.zfcp=<adapter>,<wwpn>,<lun>to specify the FCP disk where RHEL is to be installed. - 3
- Specify values for
adapter,wwpn, andlunas in the following example:rd.zfcp=0.0.8002,0x500507630400d1e3,0x4000404600000000. - 4
- Specify this parameter when using an OSA network adapter or HiperSockets.
The override parameter overrides the host’s network configuration settings.
2.5. Preflight validations Copy linkLink copied to clipboard!
The Assisted Installer ensures the cluster meets the prerequisites before installation, because it eliminates complex postinstallation troubleshooting, thereby saving significant amounts of time and effort. Before installing software on the nodes, the Assisted Installer conducts the following validations:
- Ensures network connectivity
- Ensures sufficient network bandwidth
- Ensures connectivity to the registry
- Ensures that any upstream DNS can resolve the required domain name
- Ensures time synchronization between cluster nodes
- Verifies that the cluster nodes meet the minimum hardware requirements
- Validates the installation configuration parameters
If the Assisted Installer does not successfully validate the foregoing requirements, installation will not proceed.
Chapter 3. Customizing your environment using Operators and Operator Bundles Copy linkLink copied to clipboard!
You can customize the OpenShift Container Platform deployment by selecting one or more Operators or Operator bundles during the installation.
3.1. Customizing with Operators Copy linkLink copied to clipboard!
Operators are used to package, deploy, and manage services and applications. Before starting the installation, familiarize yourself with the Assisted Installer Operators, including their prerequisites and limitations. If you require advanced options, install the Operators after you have installed the cluster.
The additional requirements specified below apply to each Operator individually. If you select more than one Operator, or if the Assisted Installer automatically selects an Operator due to dependencies, the total required resources is the sum of the resource requirements for each Operator.
Additional resources
3.1.1. OpenShift Virtualization Operator Copy linkLink copied to clipboard!
You can deploy OpenShift Virtualization to perform the following tasks:
- Create and manage Linux and Windows virtual machines (VMs).
- Run pod and VM workloads alongside each other in a cluster.
- Connect to VMs through a variety of consoles and CLI tools.
- Import and clone existing VMs.
- Manage network interface controllers and storage drives attached to VMs.
- Live migrate VMs between nodes.
The OpenShift Virtualization Operator requires backend storage and might automatically activate a storage Operator in the background, according to the following criteria:
- None - If the CPU architecture is ARM64, no storage Operator is activated.
- LVM Storage - For single-node OpenShift clusters on any other CPU architecture deploying OpenShift Container Platform 4.12 or higher.
- Local Storage Operator (LSO) - For all other deployments.
Prerequisites
- Requires enabled CPU virtualization support in the firmware on all nodes.
- Requires an additional 360 MiB of memory and 2 CPU cores for each compute (worker) node.
- Requires an additional 150 MiB of memory and 4 CPU cores for each control plane node.
Requires Red Hat OpenShift Data Foundation (recommended for creating additional on-premise clusters), Logical Volume Manager Storage, or another persistent storage service.
ImportantDeploying OpenShift Virtualization without Red Hat OpenShift Data Foundation results in the following scenarios:
- Multi-node cluster: No storage is configured. You must configure storage after the OpenShift Data Foundation configuration.
- Single-node OpenShift: Logical Volume Manager Storage (LVM Storage) is installed.
You must review the prerequisites to ensure that your environment has sufficient additional resources for OpenShift Virtualization.
- OpenShift Virtualization is not supported on the following platforms: Nutanix, vSphere.
- OpenShift Virtualization is not compatible with the following CPU architectures: S390X, PPC64LE.
- OpenShift Virtualization is supported from OpenShift Container Platform 4:14 and later.
3.1.2. Migration Toolkit for Virtualization Operator Copy linkLink copied to clipboard!
The Migration Toolkit for Virtualization Operator allows you to migrate virtual machines at scale to a local or remote Red Hat OpenShift Virtualization cluster. You can perform the migration from any of the following source providers:
- VMware vSphere
- Red Hat Virtualization (RHV)
- Red Hat OpenShift Virtualization
- OpenStack
When you select the Migration Toolkit for Virtualization Operator, the Assisted Installer automatically activates the OpenShift Virtualization Operator. For a Single-node OpenShift installation, the Assisted Installer also activates the LVM Storage Operator.
You can install the Migration Toolkit for Virtualization Operator on OpenShift Container Platform using the Assisted Installer, either independently or as part of the OpenShift Virtualization Operator bundle.
Prerequisites
- Requires OpenShift Container Platform version 4.14 or later.
- Requires an x86_64 CPU architecture.
- Requires an additional 1024 MiB of memory and 1 CPU core for each control plane node and worker node.
- Requires the additional resources specified for the OpenShift Virtualization Operator, installed together with OpenShift Virtualization. For details, see the prerequisites in the OpenShift Virtualization Operator section.
Post-installation steps
After completing the installation, the Migration menu appears in the navigation pane of the Red Hat OpenShift web console.
The Migration menu provides access to the Migration Toolkit for Virtualization. Use the toolkit to create and execute a migration plan with the relevant source and destination providers.
For details, see either of the following chapters in the Migration Toolkit for Virtualization Guide:
Additional resources
3.1.3. Multicluster engine for Kubernetes Operator Copy linkLink copied to clipboard!
You can deploy the multicluster engine for Kubernetes to perform the following tasks in a large, multi-cluster environment:
- Provision and manage additional Kubernetes clusters from your initial cluster.
- Use hosted control planes to reduce management costs and optimize cluster deployment by decoupling the control and data planes.
- Use GitOps Zero Touch Provisioning to manage remote edge sites at scale.
You can deploy the multicluster engine with OpenShift Data Foundation on all OpenShift Container Platform clusters.
Prerequisites
- Requires an additional 16384 MiB of memory and 4 CPU cores for each compute (worker) node.
- Requires an additional 16384 MiB of memory and 4 CPU cores for each control plane node.
Requires OpenShift Data Foundation (recommended for creating additional on-premise clusters), LVM Storage, or another persistent storage service.
ImportantDeploying multicluster engine without OpenShift Data Foundation results in the following scenarios:
- Multi-node cluster: No storage is configured. You must configure storage after the installation process.
- Single-node OpenShift: LVM Storage is installed.
You must review the prerequisites to ensure that your environment has enough additional resources for the multicluster engine.
3.1.4. Logical Volume Manager Storage Operator Copy linkLink copied to clipboard!
You can use LVM Storage to dynamically provision block storage on a limited resources cluster.
Prerequisites
- Requires at least 1 non-boot drive per host.
- Requires 100 MiB of additional RAM.
- Requires 1 additional CPU core for each non-boot drive.
3.1.5. Red Hat OpenShift Data Foundation Operator Copy linkLink copied to clipboard!
You can use OpenShift Data Foundation for file, block, and object storage. This storage option is recommended for all OpenShift Container Platform clusters. OpenShift Data Foundation requires a separate subscription.
Prerequisites
- There are at least 3 compute (workers) nodes, each with 19 additional GiB of memory and 8 additional CPU cores.
- There are at least 2 drives per compute node. For each drive, there is an additional 5 GB of RAM.
- You comply to the additional requirements specified here: Planning your deployment.
You cannot install the OpenShift Data Foundation Operator on Oracle third-party platforms such as Oracle® Cloud Infrastructure or Oracle® Compute Cloud@Customer.
3.1.6. OpenShift Artificial Intelligence (AI) Operator Copy linkLink copied to clipboard!
Red Hat® OpenShift® Artificial Intelligence (AI) is a flexible, scalable artificial intelligence (AI) and machine learning (ML) platform that enables enterprises to create and deliver AI-enabled applications at scale across hybrid cloud environments. Red Hat® OpenShift® AI enables the following functionality:
- Data acquisition and preparation.
- Model training and fine-tuning.
- Model serving and model monitoring.
- Hardware acceleration.
The OpenShift AI Operator enables you to install Red Hat® OpenShift® AI on your OpenShift Container Platform cluster. From OpenShift Container Platform version 4.17 and later, you can use the Assisted Installer to deploy the OpenShift AI Operator to your cluster during the installation.
You can install the OpenShift Artificial Intelligence (AI) Operator either separately or as part of the OpenShift AI Operator bundle.
The integration of the OpenShift AI Operator into the Assisted Installer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.
Prerequisites
The prerequisites for installing the OpenShift AI Operator separately are as follows:
- You are installing OpenShift Container Platform version 4.17 or later.
You meet the following miminum requirements for the OpenShift AI Operator:
- Requires at least 2 compute (worker) nodes, each with 32 additional GiB of memory and 8 additional CPU cores.
- Requires at least 1 supported GPU. Both AMD and NVIDIA GPUs are supported.
- You meet the additional minimum requirements specified for the dependent Red Hat OpenShift Data Foundation Operator.
- You meet the additional requirements specified here: Requirements for OpenShift AI.
- See the additional prerequisites for the OpenShift AI Operator bundle, if you are installing the Operator as part of the bundle.
You cannot install the OpenShift AI Operator on Oracle third-party platforms such as Oracle® Cloud Infrastructure or Oracle® Compute Cloud@Customer.
Additional resources
3.1.7. OpenShift sandboxed containers Operator Copy linkLink copied to clipboard!
The OpenShift sandboxed containers Operator provides an additional virtual machine (VM) isolation layer for pods, which manages the installation, configuration, and updating of sandboxed containers runtime (Kata containers) on Red Hat OpenShift clusters. You can install the sandboxed containers runtime in an Red Hat OpenShift cluster by using the Assisted Installer.
The integration of the OpenShift sandboxed containers Operator into the Assisted Installer is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.
Prerequisites
The required functionality for the OpenShift Container Platform is supported by two main components:
- OpenShift Container Platform: Use OpenShift Container Platform version 4.17 or later to install OpenShift sandboxed containers on an Red Hat OpenShift cluster using the Assisted Installer. To learn more about the requirements for OpenShift sandboxed containers, see "Additional Resources".
Kata runtime: This includes Red Hat Enterprise Linux CoreOS (RHCOS) and updates with every OpenShift Container Platform release. The Operator depends on the features that come with the RHCOS host and the environment it runs in.
NoteYou must install Red Hat Enterprise Linux CoreOS (RHCOS) on the worker nodes. RHEL nodes are not supported.
Additional resources
3.1.8. Kubernetes NMState Operator Copy linkLink copied to clipboard!
NMState is a declarative NetworkManager API designed for configuring network settings using YAML or JSON-based instructions. The Kubernetes NMState Operator allows you to configure network interface types, DNS, and routing on the cluster nodes using NMState.
You can install the Kubernetes NMState Operator on OpenShift Container Platform using the Assisted Installer, either separately or as part of the OpenShift Virtualization Operator bundle. Installing the Kubernetes NMState Operator with the Assisted Installer automatically creates a kubernetes-nmstate instance, which deploys the NMState State Controller as a daemon set across all of the cluster nodes. The daemons on the cluster nodes periodically report on the state of each node’s network interfaces to the API server.
Prerequisites
- Supports OpenShift Container Platform 4.12 or later.
- Requires an x86_64 CPU architecture.
- Cannot be installed on the Nutanix and Oracle Cloud Infrastructure platforms.
3.1.9. Fence Agents Remediation Operator Copy linkLink copied to clipboard!
You can use Fence Agents Remediation Operator to automatically recover unhealthy nodes on environments with a traditional API end-point. When a node in the OpenShift Container Platform cluster becomes unhealthy or unresponsive, the Fence Agents Remediation Operator utilizes an external set of fencing agents to isolate it from the rest of the cluster. A fencing agent then resets the unhealthy node in an attempt to resolve transient hardware or software issues. Before or during the reboot process, the Fence Agents Remediation Operator safely moves workloads (pods) running on the unhealthy node to other healthy nodes in the cluster.
You can only install the Fence Agents Remediation Operator as part of the Virtualization Operator bundle.
Prerequisites
- See the prerequisites for the Virtualization Operator bundle.
Procedure
Post-installation steps
-
Create the
FenceAgentsRemediationTemplatecustom resource to define the required fencing agents and remediation parameters. For details, see Configuring the Fence Agents Remediation Operator. -
Configure the
NodeHealthCheckcustom resource by either replacing the defaultSelfNodeRemediationprovider withFenceAgentsRemediationor by addingFenceAgentsRemediationas an additional remediation provider.
Additional resources
3.1.10. Kube Descheduler Operator Copy linkLink copied to clipboard!
The Kube Descheduler Operator is a Kubernetes operator that automates the deployment, configuration, and management of the Kubernetes Descheduler within a cluster. You can use the Kube Descheduler Operator to evict pods (workloads) based on specific strategies, so that the pods can be rescheduled onto more appropriate nodes.
You can benefit from descheduling running pods in situations such as the following:
- Nodes are underutilized or overutilized.
- Pods and node affinity requirements, such as taints or labels, have changed and the original scheduling decisions are no longer appropriate for certain nodes.
- Node failure requires pods to be moved.
- New nodes are added to clusters.
- Pods have been restarted excessively.
You can only install the Kube Descheduler Operator as part of the Virtualization Operator bundle.
Prerequisites
- See the prerequisites for the Virtualization Operator bundle.
Procedure
Additional resources
3.1.11. Local Storage Operator Copy linkLink copied to clipboard!
The Local Storage Operator (LSO) enables the provisioning of persistent storage through local volumes. Local persistent volumes provide access to local storage devices, such as drives or partitions, by using the standard persistent volume claim interface.
You can perform the following actions using Local Storage Operator (LSO):
- Assign the storage devices to the storage classes without modifying the device configuration.
- Statically provision PVs and storage classes by configuring the LocalVolume custom resource (CR).
- Create workloads and PVCs while being aware of the underlying storage topology.
Selecting the OpenShift Virtualization Operator, either independently or as part of the Virtualization bundle, automatically activates Local Storage Operator (LSO) in the background.
Prerequisites
- See the prerequisites for the Virtualization Operator bundle.
Procedure
Additional resources
3.1.12. Node Health Check Operator Copy linkLink copied to clipboard!
The Node Health Check Operator monitors node conditions based on a defined set of criteria to assess their health status. When detecting an issue, the Operator delegates remediation tasks to the appropriate remediation provider to remediate the unhealthy nodes. The Assisted Installer supports the following remediation providers:
- Self Node Remediation Operator - An internal solution for rebooting unhealthy nodes.
- Fence Agents Remediation Operator - Leverages external management capabilities to forcefully isolate and reboot nodes.
You can only install the Node Health Check Operator as part the Virtualization Operator bundle.
Prerequisites
- See the prerequisites for the Virtualization Operator bundle.
Procedure
Additional resources
3.1.13. Node Maintenance Operator Copy linkLink copied to clipboard!
The Node Maintenance Operator facilitates planned maintenance by placing nodes into maintenance mode.
The Node Maintenance Operator watches for new or deleted NodeMaintenance custom resources (CRs). When it detects a new NodeMaintenance CR, it prevents new workloads from being scheduled on that node, and cordons off the node from the rest of the cluster. The Operator then evicts all pods that can be evicted from the node. When the administrator deletes the NodeMaintenance CR associated with the node, maintenance ends and the Operator makes the node available for new workloads.
You can only install the Node Maintenance Operator as part of the Virtualization Operator bundle.
Prerequisites
- See the prerequisites for the Virtualization Operator bundle.
Procedure
Additional resources
3.1.14. Self Node Remediation Operator Copy linkLink copied to clipboard!
The Self Node Remediation Operator automatically reboots unhealthy nodes. This remediation strategy minimizes downtime for stateful applications and ReadWriteOnce (RWO) volumes, and restores compute capacity if transient failures occur.
You can use the Self Node Remediation Operator as a remediation provider for the Node Health Check Operator. Currently, it is only possible to install the Self Node Remediation as a standalone Operator through the API.
The integration of the Self Node Remediation Operator into the Assisted Installer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.
Procedure
Additional resources
3.1.15. AMD GPU Operator Copy linkLink copied to clipboard!
The Advanced Micro Devices (AMD) Graphics Processing Unit (GPU) Operator simplifies the deployment and management of AMD Instinct™ GPUs within a Red Hat OpenShift Container Platform cluster. The hardware acceleration capabilities of the Operator automate several key tasks, making it easier to create artificial intelligence and machine learning (AI/ML) applications. Accelerating specific areas of GPU functions can minimize CPU processing and memory usage, improving overall application speed, memory consumption, and bandwidth restrictions.
You can install the AMD GPU Operator separately or as part of the OpenShift AI Operator bundle. Selecting the AMD GPU Operator automatically activates the Kernel Module Management Operator.
The integration of the AMD GPU Operator into the Assisted Installer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.
Prerequisites
- Requires at least 1 supported AMD GPU.
- See the additional prerequisites for the OpenShift AI Operator bundle if you are installing the Operator as part of the bundle.
Procedure
Additional resources
3.1.16. Authorino Operator Copy linkLink copied to clipboard!
The Authorino Operator provides an easy way to install Authorino, providing configurability options at the time of installation.
Authorino is a Kubernetes-native, external authorization service designed to secure APIs and applications. It intercepts requests to services and determines whether to allow or deny access based on configured authentication and authorization policies. Authorino provides a centralized and declarative way to manage access control for your Kubernetes-based applications without requiring code changes.
You can only install the Authorino Operator as part of the OpenShift AI Operator bundle.
The integration of the Authorino Operator into the Assisted Installer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.
Prerequisites
- See the prerequisites for the OpenShift AI Operator bundle.
Procedure
Additional resources
3.1.17. Kernel Module Management Operator Copy linkLink copied to clipboard!
The Kernel Module Management (KMM) Operator manages, builds, signs, and deploys out-of-tree kernel modules and device plugins on OpenShift Container Platform clusters.
KMM adds a new Module CRD which describes an out-of-tree kernel module and its associated device plugin. You can use Module resources to configure how to load the module, define ModuleLoader images for kernel versions, and include instructions for building and signing modules for specific kernel versions.
KMM is designed to accommodate multiple kernel versions at once for any kernel module, allowing for seamless node upgrades and reduced application downtime.
You can install the Kernel Module Management Operator either independently or as part of the OpenShift AI Operator bundle.
The integration of the Kernel Module Management Operator into the Assisted Installer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.
Prerequisites
- If you are installing the Operator as part of the OpenShift AI Operator bundle, see the bundle prerequisites.
- If you are installing the Operator separately, there are no additional prerequisites.
Procedure
Additional resources
3.1.18. Node Feature Discovery Operator Copy linkLink copied to clipboard!
The Node Feature Discovery (NFD) Operator automates the deployment and management of the Node Feature Discovery (NFD) add-on. The Node Feature Discovery add-on detects the configurations and hardware features of each node in an OpenShift Container Platform cluster. The add-on labels each node with hardware-specific information such as vendor, kernel configuration, or operating system version, making the cluster aware of the underlying hardware and software capabilities of the nodes.
With the Node Feature Discovery (NFD) Operator, administrators can easily gather information about the nodes to use for scheduling, resource management, and more by controlling the life-cycle of Node Feature Discovery (NFD).
You can install the Node Feature Discovery Operator separately or as part of the OpenShift AI Operator bundle.
The integration of the Node Feature Discovery Operator into the Assisted Installer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.
Prerequisites
- If you are installing the Operator as part of the OpenShift AI Operator bundle, see the bundle prerequisites.
- If you are installing the Operator separately, there are no additional prerequisites.
Procedure
3.1.19. NVIDIA GPU Operator Copy linkLink copied to clipboard!
The NVIDIA GPU Operator uses the operator framework within Kubernetes to automate the management of all NVIDIA software components needed to provision Graphical Processing Units (GPUs).
Some of these software components are as follows:
- NVIDIA drivers to enable Compute Unified Device Architecture (CUDA).
- The Kubernetes device plugin for GPUs.
- The NVIDIA Container Toolkit.
- Automatic node labelling using GPU Feature Discovery (GFD)
- GPU monitoring through the Data Center GPU Manager (DCGM).
In OpenShift Container Platform, the Operator provides a consistent, automated, and cloud-native way to leverage the power of NVIDIA GPUs for artificial intelligence, machine learning, high-performance computing, and other GPU-accelerated workloads.
You can install the NVIDIA GPU Operator either separately or as part of the OpenShift AI Operator bundle. Selecting the NVIDIA GPU Operator automatically activates the Node Feature Discovery Operator.
The integration of the NVIDIA GPU Operator into the Assisted Installer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.
Prerequisites
- Requires at least 1 supported NVIDIA GPU.
- See the additional prerequisites for the OpenShift AI Operator bundle if you are installing the Operator as part of the bundle.
Procedure
3.1.20. OpenShift Pipelines Operator Copy linkLink copied to clipboard!
Red Hat OpenShift Pipelines is a cloud-native, continuous integration and continuous delivery (CI/CD) solution based on Kubernetes resources. It uses Tekton building blocks to automate deployments across multiple platforms by abstracting away the underlying implementation details. Tekton introduces various standard custom resource definitions (CRDs) for defining CI/CD pipelines that are portable across Kubernetes distributions.
The Red Hat OpenShift Pipelines Operator handles the installation and management of OpenShift Pipelines. The Operator supports the following use cases:
- Continuous Integration (CI) - Automating code compilation, testing, and static analysis.
- Continuous Delivery/Deployment (CD) - Automating the deployment of applications to various environments (development, staging, production).
- Microservices Development - Supporting decentralized teams working on microservice-based architectures.
- Building Container Images - Efficiently building and pushing container images to registries.
- Orchestrating Complex Workflows - Defining multi-step processes for building, testing, and deploying applications across different platforms.
You can only install the OpenShift Pipelines Operator as part of the OpenShift AI Operator bundle.
The integration of the OpenShift Pipelines Operator into the Assisted Installer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.
Prerequisites
- See the prerequisites for the OpenShift AI Operator bundle.
Procedure
3.1.21. OpenShift Serverless Operator Copy linkLink copied to clipboard!
The Red Hat OpenShift Serverless Operator enables you to install and use the following components on your OpenShift Container Platform cluster:
- Knative Serving - Deploys and automatically scales stateless, containerized applications according to demand. It simplifies code deployment, and handles web requests and background processes.
- Knative Eventing - Provides the building blocks for an event-driven architecture on Kubernetes. It enables loose coupling between services by allowing them to communicate asynchronously through events, rather than through direct calls.
- Knative Broker for Apache Kafka - This is a specific implementation of a Knative Broker. It provides a robust, scalable, and high-performance mechanism for routing events within Knative Eventing, in environments where Apache Kafka is the preferred message broker.
The OpenShift Serverless Operator manages Knative custom resource definitions (CRDs) for your cluster and enables you to configure them without directly modifying individual config maps for each component.
You can only install the OpenShift Serverless Operator as part of the OpenShift AI Operator bundle.
The integration of the OpenShift Serverless Operator into the Assisted Installer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.
Prerequisites
- See the prerequisites for the OpenShift AI Operator bundle.
Procedure
Additional resources
3.1.22. OpenShift Service Mesh Operator Copy linkLink copied to clipboard!
Red Hat OpenShift Service Mesh addresses a variety of problems in a microservice architecture by creating a centralized point of control in an application. It adds a transparent layer on existing distributed applications without requiring any changes to the application code.
Microservice architectures split the work of enterprise applications into modular services, which can make scaling and maintenance easier. However, as an enterprise application built on a microservice architecture grows in size and complexity, it becomes difficult to understand and manage. Service Mesh can address those architecture problems by capturing or intercepting traffic between services and can modify, redirect, or create new requests to other services.
Service Mesh provides an easy way to create a network of deployed services that provides discovery, load balancing, service-to-service authentication, failure recovery, metrics, and monitoring. A service mesh also provides more complex operational functionality, including A/B testing, canary releases, access control, and end-to-end authentication
Red Hat OpenShift Service Mesh requires the use of the Red Hat OpenShift Service Mesh Operator which allows you to connect, secure, control, and observe the microservices that comprise your applications. You can also install other Operators to enhance your service mesh experience. Service mesh is based on the open source Istio project.
You can only install the OpenShift Service Mesh Operator as part of the OpenShift AI Operator bundle.
The integration of the OpenShift Service Mesh Operator into the Assisted Installer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.
Prerequisites
- See the prerequisites for the OpenShift AI Operator bundle.
- See Preparing to install service mesh in the OpenShift Container Platform documentation.
Procedure
Additional resources
3.1.23. Cluster Observability Operator Copy linkLink copied to clipboard!
The Cluster Observability Operator (COO) is an optional component of the OpenShift Container Platform designed for creating and managing highly customizable monitoring stacks. It enables cluster administrators to automate configuration and management of monitoring needs extensively, offering a more tailored and detailed view of each namespace compared to the default OpenShift Container Platform monitoring system.
The Cluster Observability Operator deploys the following monitoring components:
- Prometheus - A highly available Prometheus instance capable of sending metrics to an external endpoint by using remote write.
- Thanos Querier (optional) - Enables querying of Prometheus instances from a central location.
- Alertmanager (optional) - Provides alert configuration capabilities for different services.
- UI plugins (optional) - Enhances the observability capabilities with plugins for monitoring, logging, distributed tracing and troubleshooting.
- Korrel8r (optional) - Provides observability signal correlation, powered by the open source Korrel8r project.
You can install the Cluster Observability Operator Operator separately through the Assisted Installer API or as part of the Virtualization bundle in the Assisted Installer web console. For more information about the use of this Operator in OpenShift Container Platform, see "Additional resources".
Prerequisites
- See the prerequisites for the Virtualization Operator bundle.
3.1.24. MetalLB Operator Copy linkLink copied to clipboard!
You can install the MetalLB Operator to enable the use of LoadBalancer services in environments that do not have a built-in cloud load balancer, such as bare-metal clusters.
When you create a LoadBalancer service, MetalLB assigns an external IP address from a predefined pool. MetalLB advertises this IP address on the host network, making the service reachable from outside the cluster. When external traffic enters your OpenShift Container Platform cluster through the MetalLB LoadBalancer service, the return traffic to the client has the external IP address of the load balancer as the source IP.
You can install the MetalLB Operator separately through the Assisted Installer API or as part of the Virtualization bundle in the Assisted Installer web console. For more information about the use of this Operator in OpenShift Container Platform, see "Additional resources".
Prerequisites
- See the prerequisites for the Virtualization Operator bundle.
3.1.25. NUMA Resources Operator Copy linkLink copied to clipboard!
Non-Uniform Memory Access (NUMA) is a compute platform architecture that allows different CPUs to access different regions of memory at different speeds. NUMA resource topology refers to the locations of CPUs, memory, and PCI devices relative to each other in the compute node. Colocated resources are said to be in the same NUMA zone. For high-performance applications, the cluster needs to process pod workloads in a single NUMA zone.
The NUMA Resources Operator allows you to schedule high-performance workloads in the same NUMA zone. It deploys a node resources exporting agent that reports on available cluster node NUMA resources, and a secondary scheduler that manages the workloads.
You can install the NUMA Resources Operator separately through the Assisted Installer API or as part of the Virtualization bundle in the Assisted Installer web console. For more information about the use of this Operator in OpenShift Container Platform, see "Additional resources".
Prerequisites
- See the prerequisites for the Virtualization Operator bundle.
Procedure
Post-installation steps
Create the NUMAResourcesOperator custom resource and deploy the NUMA-aware secondary pod scheduler. For details, see Scheduling NUMA-aware workloads in "Additional resources".
3.1.26. OpenShift API for Data Protection (OADP) Operator Copy linkLink copied to clipboard!
The OpenShift API for Data Protection (OADP) product safeguards customer applications on OpenShift Container Platform. It offers comprehensive disaster recovery protection, covering OpenShift Container Platform applications, application-related cluster resources, persistent volumes, and internal images. OADP is also capable of backing up both containerized applications and virtual machines (VMs). OADP does not serve as a disaster recovery solution for etcd or OpenShift Operators.
You can install the OADP Operator separately through the Assisted Installer API or as part of the Virtualization bundle in the Assisted Installer web console. For more information about the use of this Operator in OpenShift Container Platform, see "Additional resources".
Prerequisites
- See the prerequisites for the Virtualization Operator bundle.
3.2. Customizing with Operator Bundles Copy linkLink copied to clipboard!
An Operator bundle is a recommended packaging format that combines related Operators to deliver a comprehensive set of capabilities. By selecting a bundle, administrators can extend functionality beyond that of a single Operator.
This approach makes the Assisted Installer more opinionated, offering an optimized platform for each selected bundle. It reduces the adoption barrier and minimizes the expertise required for customers to quickly access essential features. Additionally, it establishes a single, well-tested, and widely recognized path for platform deployments.
Meanwhile, individual Operators remain independent and free of unnecessary dependencies, ensuring a lightweight and flexible solution for small or specialized deployments, such as on single-node OpenShift.
When an administrator specifies an Operator bundle, the Assisted Installer automatically provisions the associated Operators included in the bundle. These Operators are predefined and cannot be deselected, ensuring consistency. Administrators can modify the selection after the installation has completed.
Additional resources
3.2.1. Virtualization Operator bundle Copy linkLink copied to clipboard!
Virtualization lets you create multiple simulated environments or resources from a single, physical hardware system. The Virtualization Operator bundle provides a recommended and proven path for virtualization platform deployments, minimizing obstacles. The solution supports the addition of nodes and Day-2 administrative operations.
The Virtualization Operator bundle prompts the Assisted Installer to install the following Operators together:
- Fence Agents Remediation Operator - Externally fences failed nodes using power controllers.
- Kube Descheduler Operator - Evicts pods to reschedule them on more suitable nodes.
- Local Storage Operator - Allows provisioning of persistent storage by using local volumes.
- Migration Toolkit for Virtualization Operator - Enables you to migrate virtual machines from VMware vSphere, Red Hat Virtualization, or OpenStack to OpenShift Virtualization running on Red Hat OpenShift Container Platform.
- Kubernetes NMState Operator - Enables you to configure various network interface types, DNS, and routing on cluster nodes.
- Node Health Check Operator - Identifies unhealthy nodes.
- Node Maintenance Operator - Places nodes into maintenance mode.
- OpenShift Virtualization Operator - Runs virtual machines alongside containers on one platform.
- Cluster Observability Operator - Provides observability and monitoring capabilities for your OpenShift cluster.
- MetalLB Operator - Provides load balancer services for bare metal OpenShift clusters.
- NUMA Resources Operator - Provides NUMA-aware scheduling to improve workload performance on NUMA systems.
- OpenShift API for Data Protection (OADP) Operator - Enables the backup and restoral of OpenShift Container Platform cluster resources and persistent volumes.
Prerequisites
- You are installing OpenShift Container Platform version 4.14 or later.
- There is enabled CPU virtualization support in BIOS (Intel-VT/AMD-V) on all nodes.
- Each control plane (master) node has an an additional 1024 MiB of memory and 3 CPU cores.
- Each compute (worker) node has an additional 1024 MiB of memory and 5 CPU cores.
- You have included the additional resources required to support the selected storage Operator.
- You are installing a cluster of three or more nodes. The Virtualization Operator bundle is not available on single-node OpenShift.
Procedure
Additional resources
3.2.2. OpenShift AI Operator bundle Copy linkLink copied to clipboard!
The OpenShift AI Operator bundle enables the training, serving, monitoring, and management of Artificial Intelligence (AI) and Machine Learning (ML) models and applications. It simplifies the deployment of AI and ML components on your OpenShift cluster.
The OpenShift AI Operator bundle bundle prompts the Assisted Installer to install the following Operators together:
- AMD GPU Operator - Automates the management of AMD software components needed to provision and monitor Graphics Processing Units (GPUs).
- Authorino Operator - Provides a lightweight external authorization service for tailor-made Zero Trust API security.
- Kernel Module Management Operator - Manages kernel modules and associated device plugins.
- Local Storage Operator - Allows provisioning of persistent storage by using local volumes.
- Node Feature Discovery Operator - Manages the detection of hardware features and configuration by labeling nodes with hardware-specific information.
- NVIDIA GPU Operator - Automates the management of NVIDIA software components needed to provision and monitor GPUs.
- OpenShift AI Operator - Trains, serves, monitors and manages AI/ML models and applications.
- Red Hat OpenShift Data Foundation Operator - Provides persistent software-defined storage for hybrid applications.
- OpenShift Pipelines Operator - Provides a cloud-native continuous integration and delivery (CI/CD) solution for building pipelines using Tekton.
- OpenShift Serverless Operator - Deploys workflow applications based on the CNCF (Cloud Native Computing Foundation) Serverless Workflow specification.
- OpenShift Service Mesh Operator - Provides behavioral insight and operational control over a service mesh.
The OpenShift AI Operator bundle is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.
Prerequisites
- The installation of the NVIDIA GPU, AMD GPU, and Kernel Module Management Operators depends on the Graphics Processing Unit (GPU) detected on your hosts following host discovery.
Procedure
Additional resources
Chapter 4. Installing with the Assisted Installer web console Copy linkLink copied to clipboard!
After you ensure the cluster nodes and network requirements are met, you can begin installing the cluster.
4.1. Preinstallation considerations Copy linkLink copied to clipboard!
Before installing OpenShift Container Platform with the Assisted Installer, you must consider the following configuration choices:
- Which base domain to use
- Which OpenShift Container Platform product version to install
- Whether to install a full cluster or single-node OpenShift
- Whether to use a DHCP server or a static network configuration
- Whether to use IPv4 or dual-stack networking
- Whether to install OpenShift Virtualization
- Whether to install Red Hat OpenShift Data Foundation
- Whether to install multicluster engine for Kubernetes
- Whether to integrate with the platform when installing on vSphere or Nutanix
- Whether to install a multi-architecture compute cluster
4.2. Setting the cluster details Copy linkLink copied to clipboard!
To create a cluster with the Assisted Installer web user interface, use the following procedure.
Procedure
- Log in to the Red Hat Hybrid Cloud Console.
- On the Red Hat OpenShift tile, click OpenShift.
- On the Red Hat OpenShift Container Platform tile, click Create cluster.
- Click the Datacenter tab.
- Under Assisted Installer, click Create cluster.
- Enter a name for the cluster in the Cluster name field.
Enter a base domain for the cluster in the Base domain field. All subdomains for the cluster will use this base domain.
NoteThe base domain must be a valid DNS name. You must not have a wildcard domain set up for the base domain.
From the OpenShift version dropdown list, select the version that you want to install and click Select. By default, the dropdown lists the latest OpenShift version. If you need an older version that is not displayed, click Show all available versions at the bottom of the list, and use the search box to find it.
Important-
For a multi-architecture compute cluster installation, select OpenShift Container Platform 4.12 or later, and use the
-multioption. For instructions on installing a multi-architecture compute cluster, see Installing multi-architecture compute clusters. - For IBM Power® and IBM Z® platforms, only OpenShift Container Platform 4.13 and later is supported.
- If you are booting from an iSCSI drive, select OpenShift Container Platform version 4.15 or later.
-
For a multi-architecture compute cluster installation, select OpenShift Container Platform 4.12 or later, and use the
Optional: Assisted Installer defaults to using x86_64 CPU architecture. If you are installing OpenShift Container Platform on a different architecture, select the architecture to use. Valid values are arm64, ppc64le, and s390x. Remember that some features are not available with arm64, ppc64le, and s390x CPU architectures.
ImportantFor a multi-architecture compute cluster installation, you can use
x86_64or 64-bit ARM CPU architecture for the control plane nodes. Automatic conversion fromx86_64to 64-bit ARM is only supported on Amazon Web Services (AWS). For instructions on installing a multi-architecture compute cluster, see Installing multi-architecture compute clusters.- Optional: The Assisted Installer already has the pull secret associated to your account. If you want to use a different pull secret, select Edit pull secret.
Optional: If you are installing OpenShift Container Platform on a third-party platform, select the platform from the Integrate with external partner platforms list. Valid values are
Nutanix,vSphereorOracle Cloud Infrastructure. Assisted Installer defaults to having no platform integration.Note- Assisted Installer supports Oracle Cloud Infrastructure (OCI) integration from OpenShift Container Platform 4.14 and later.
- For details on each of the external partner integrations, see Additional Resources.
From the Number of control plane nodes field, optionally change the default value of three control plane nodes for your installation. The possible options are
1(single-node OpenShift),2,3,4, or5.Important- Currently, single-node OpenShift is not supported on IBM Z® and IBM Power® platforms.
-
The Assisted Installer supports
4or5control plane nodes from OpenShift Container Platform 4.18 and later, on a bare metal or user-managed networking platform with an x86_64 CPU architecture. For details, see About specifying the number of control plane nodes. -
The Assisted Installer supports
2control plane nodes from OpenShift Container Platform 4.19 and later, for a Two-Node OpenShift with Arbiter cluster topology. If the number of control plane nodes for a cluster is2, then it must have at least one additional arbiter host. For details, see About specifying the number of control plane nodes.
Optional: Select Include custom manifests if you have at least one custom manifest to include in the installation. A custom manifest has additional configurations not currently supported in the Assisted Installer. Selecting the checkbox adds the Custom manifests step to the wizard, where you upload the manifests.
Important- If you are installing OpenShift Container Platform on the Oracle Cloud Infrastructure (OCI) third-party platform, it is mandatory to add the custom manifests provided by Oracle.
- If you have already added custom manifests, clearing the Include custom manifests checkbox automatically deletes them all. You must confirm the deletion.
Optional: The Assisted Installer defaults to DHCP networking. If you are using a static IP configuration, bridges or bonds for the cluster nodes instead of DHCP reservations, select Static IP, bridges, and bonds. Selecting the checkbox adds the Static network configurations step to the wizard. For details, see Configuring static networks.
ImportantA static IP configuration is not supported in the following scenarios:
- OpenShift Container Platform installations on Oracle Cloud Infrastructure.
- OpenShift Container Platform installations on iSCSI boot volumes.
Optional: If you want to enable encryption of the installation disks, under Enable encryption of installation disks you can select one of the following:
- For single-node OpenShift, select Control plane node, worker.
For multi-node clusters, select Control plane nodes to encrypt the control plane node installation disks. Select Workers to encrypt worker node installation disks. Select Arbiter to encrypt the arbiter node installation disks.
ImportantYou cannot change the base domain, the single-node OpenShift checkbox, the CPU architecture, the host’s network configuration, or the disk-encryption after installation begins.
4.3. Configuring static networks Copy linkLink copied to clipboard!
The Assisted Installer supports the following network configurations:
- IPv4 networking with SDN, supported up to OpenShift Container Platform 4.14.
- IPv4 and dual-stack networking with IPv4 as primary (OVN only), supported from OpenShift Container Platform 4.15 and later.
- Static network configuration using static interfaces with IP address and MAC address mapping.
- Host network interface configuration using the NMState library, a declarative network manager API for hosts. You can also use NMState to deploy hosts with bonds, VLANs, and other advanced networking features.
For installations on IBM Z® with z/VM, ensure that the z/VM nodes and vSwitches are properly configured for static networks and NMState. Also, the z/VM nodes must have a fixed MAC address assigned as the pool MAC addresses might cause issues with NMState. For more information about NMState, see NMState Declarative Network API.
4.3.1. Configuring static networks using form view Copy linkLink copied to clipboard!
You can configure networks by using the form view or the YAML view. Select Form view for basic configurations.
To add new hosts that will use the new or edited configurations, you’ll need to regenerate the Discovery ISO in the 'Host discovery' step and boot your new hosts from it.
Prerequisites
- You have selected the Static IP, bridges and bonds option under Hosts' network configuration on the Cluster details page. Selecting this option adds the Static network configurations step to the wizard.
Procedure
- Go to the Static network configurations page.
- From the Configure via options, select Form view.
Enter the network-wide configurations:
Select the Networking stack type. Valid options are IPv4 and Dual stack (with IPv4 as primary).
ImportantIPv6 is not currently supported in the following configurations:
- Single stack
- Primary within dual stack
- If the cluster hosts are on a shared VLAN, select the Use VLAN checkbox and enter the VLAN ID.
Enter the network-wide IP addresses. If you selected Dual stack networking, you must enter both IPv4 and IPv6 addresses.
- Enter the DNS server IP address.
- Enter the cluster subnet’s IP address range in CIDR notation.
- Enter the default gateway IP address.
Enter the host-specific configurations:
- If you are only setting a static IP address that uses a single network interface, use the form view to enter the IP address and the MAC address for each host.
Optional: You can use bonds to combine network interfaces for increased bandwidth and ensure redundancy. Creating a bond with a static IP address aggregates two network interfaces per host.
- Select the Use bond checkbox.
From the Bond type dropdown list, select the bond type. The default bond type is
Active-Backup (1).NoteFor a description of the bond types, see Bonding modes.
- Enter the MAC address for Port 1.
- Enter the MAC address for Port 2.
- Enter the IP address for the bond.
- Click Next.
4.3.2. Configuring static networks using YAML view Copy linkLink copied to clipboard!
If you use multiple interfaces or other advanced networking features, use the YAML view to enter the network state for each host that uses NMState syntax. For more information about NMState, see NMState Declarative Network API.
You can only create host-specific configurations using form view.
Prerequisites
- You have selected the Static IP, bridges and bonds option under Hosts' network configuration on the Cluster details page. Selecting this option adds the Static network configurations step to the wizard.
Procedure
- Go to the Static network configurations page.
- From the Configure via options, select YAML view.
- Upload, drag and drop, or copy and paste a YAML file containing NMState into the editor for network configurations.
- In the MAC to interface name mapping fields, enter the MAC address and interface name for each host interface used in your network configuration. Each host requires the MAC to interface name mapping to run the NMState YAML on the right machine. Assisted Installer uses the MAC to interface name mapping to replace any temporary interface names with the actual names.
- Select the Copy the YAML content checkbox to copy the YAML content between multiple hosts.
- Click Add another host configuration to configure additional hosts.
- Click Next.
4.4. Installing Operators and Operator bundles Copy linkLink copied to clipboard!
You can customize your deployment by selecting Operators and Operator bundles during the installation. If you require advanced options, add the Operators or bundles after you have installed the cluster.
When installing Operators and Operator bundles through the web console, the following conditions apply:
- Some Operators are only available as part of a bundle and cannot be selected individually.
- Some Operators can be selected either individually or as part of a bundle. If you select them as part of a bundle, you can only remove them by deselecting the bundle.
- Some Operators are only available as standalone installations.
This step is optional.
4.4.1. Installing standalone Operators Copy linkLink copied to clipboard!
You can select more than one standalone Operator and add Operator bundles as needed. Operators that appear greyed out are only available for installation as part of a bundle.
For instructions on installing Operator bundles, see Installing Operator bundles.
Prerequisites
- You have reviewed Customizing your installation using Operators for an overview of each Operator that you intend to install, together with its prerequisites and dependencies.
Procedure
- On the Operators page, expand the Single Operators arrow to display the full list of Operators.
Select one or more Operators from the following options:
ImportantThe integration of the AMD GPU, Kernel Module Management, Node Feature Discovery, NVIDIA GPU, and OpenShift AI Operators into the Assisted Installer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.
AMD GPU
NoteSelecting the AMD GPU Operator automatically activates the Kernel Module Management Operator.
- Kernel Module Management
- Logical Volume Manager Storage
Migration Toolkit for Virtualization
ImportantSelecting the Migration Toolkit for Virtualization Operator automatically activates the OpenShift Virtualization Operator. For a Single-node OpenShift installation, the Assisted Installer also activates the LVM Storage Operator.
Multicluster engine
ImportantYou can deploy the multicluster engine with OpenShift Data Foundation on all OpenShift Container Platform clusters. Deploying the multicluster engine without OpenShift Data Foundation results in the following storage configurations:
- Multi-node cluster: No storage is configured. You must configure storage after the installation.
- Single-node OpenShift: LVM Storage is installed.
NMState
NoteCurrently, you cannot install the Kubernetes NMState Operator on the Nutanix or Oracle Cloud Infrastructure (OCI) third-party platforms.
- Node Feature Discovery
NVIDIA GPU
NoteSelecting the NVIDIA GPU Operator automatically activates the Node Feature Discovery Operator.
- OpenShift AI
- OpenShift Data Foundation
OpenShift sandboxed containers
ImportantThe integration of the OpenShift sandboxed containers Operator into the Assisted Installer is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.
OpenShift Virtualization
ImportantThe OpenShift Virtualization Operator requires backend storage and might automatically activate a storage Operator in the background, according to the following criteria:
- None - If the CPU architecture is ARM64, no storage Operator is activated.
- LVM Storage - For single-node OpenShift clusters on any other CPU architecture deploying OpenShift Container Platform 4.12 or higher.
- Local Storage Operator (LSO) - For all other deployments.
- To install the Self Node Remediation Operator, see Installing Operators by using the API. It is not currently possible to install this Operator using the web console.
- Click Next.
4.4.2. Installing Operator bundles Copy linkLink copied to clipboard!
You can select more than one Operator bundle together with additional Operators as needed.
For instructions on installing individual Operators, see Installing Operators.
Prerequisites
- You have reviewed Customizing your installation using Operator bundles for an overview of each Operator bundle that you intend to install, together with its prerequisites and associated Operators.
Procedure
On the Operators page, select an Operator bundle:
Virtualization - Contains the following Operators:
- OpenShift Virtualization
- Kube Descheduler
- Node Maintenance
- Migration Toolkit for Virtualization
- Kubernetes NMState
- Fence Agents Remediation
- Node Health Check
- Local Storage Operator (LSO)
- Cluster Observability
- MetalLB
- NUMA Resources
- OADP
OpenShift AI - Contains the following Operators:
- Kubernetes Authorino
- OpenShift Data Foundation
- OpenShift AI
- AMD GPU
- Node Feature Discovery
- NVIDIA GPU
- OpenShift Pipelines
- OpenShift Service Mesh
- OpenShift Serverless
- Kernel Module Management
ImportantThe OpenShift AI Operator bundle is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.
- Click Next.
4.5. Adding hosts to the cluster Copy linkLink copied to clipboard!
You must add one or more hosts to the cluster. Adding a host to the cluster involves generating a discovery ISO. The discovery ISO runs Red Hat Enterprise Linux CoreOS (RHCOS) in-memory with an agent.
If you are installing the IBM Z® architecture, use the following table to identify the image file type:
Expand Architecture Boot method Image type Logical Partition-Classic
iPXE
Full image file: Download a self-contained ISO image
Logical Partition-Data Protection Manager
ISO or iPXE
Minimal image file: Download an ISO image that fetches content when booting up
- ISO images are not supported for installations on IBM Z (s390x) with z/VM or logical partitioning (LPAR) nodes; use the "Booting hosts with iPXE" procedure. ISO images and iPXE are supported for installations on RHEL KVM.
Perform the following procedure for each host on the cluster.
Procedure
Click the Add hosts button and select the provisioning type.
Select Minimal image file: Provision with virtual media to download a smaller image that will fetch the data needed to boot. The nodes must have virtual media capability. This is the recommended method for
x86_64andarm64architectures.ImportantThis option is mandatory in the following scenarios:
- If you are installing OpenShift Container Platform on Oracle Cloud Infrastructure.
- If you are installing OpenShift Container Platform on iSCSI boot volumes.
-
Select Full image file: Provision with physical media to download the larger full image. This is the recommended method for the
ppc64learchitecture and for thes390xarchitecture when installing with RHEL KVM. Select iPXE: Provision from your network server to boot the hosts using iPXE. This is the recommended method on IBM Z® with z/VM nodes and LPAR (both static and DPM). ISO boot is the recommended method on the RHEL KVM installation.
NoteIf you are installing OpenShift Container Platform on RHEL KVM, in some circumstances, the VMs on the KVM host are not rebooted on first boot and need to be restarted manually.
Optional: Activate the Run workloads on control plane nodes switch to schedule workloads to run on control plane nodes, in addition to the default worker nodes.
NoteThis option is available for clusters of five or more nodes. For clusters of under five nodes, the system runs workloads on the control plane nodes only, by default. For more details, see Configuring schedulable control plane nodes in Additional Resources.
Optional: If the cluster hosts require the use of a proxy, select Configure cluster-wide proxy settings. Enter the username, password, required domains or IP addresses, and port for the HTTP and HTTPS URLs of the proxy server. If the cluster hosts are behind a firewall, allow the nodes to access the required domains or IP addresses through the firewall. See Configuring your firewall for OpenShift Container Platform for more information.
NoteThe proxy username and password must be URL-encoded.
Optional: Add an SSH public key so that you can connect to the cluster nodes as the
coreuser. Having a login to the cluster nodes can provide you with debugging information during the installation.ImportantDo not skip this procedure in production environments, where disaster recovery and debugging is required.
- If you do not have an existing SSH key pair on your local machine, follow the steps in Generating a key pair for cluster node SSH access.
-
In the SSH public key field, click Browse to upload the
id_rsa.pubfile containing the SSH public key. Alternatively, drag and drop the file into the field from the file manager. To see the file in the file manager, select Show hidden files in the menu.
- Optional: If the cluster hosts are in a network with a re-encrypting man-in-the-middle (MITM) proxy, or if the cluster needs to trust certificates for other purposes such as container image registries, select Configure cluster-wide trusted certificates. Add additional certificates in X.509 format.
- Configure the discovery image if needed.
- Optional: If you are installing on a platform and want to integrate with the platform, select Integrate with your virtualization platform. You must boot all hosts and ensure they appear in the host inventory. All the hosts must be on the same platform.
- Click Generate Discovery ISO or Generate Script File.
- Download the discovery ISO or iPXE script.
- Boot the host(s) with the discovery image or iPXE script.
Additional resources
- Configuring the discovery image for additional details.
- Booting hosts with the discovery image for additional details.
- Red Hat Enterprise Linux 9 - Configuring and managing virtualization for additional details.
- How to configure a VIOS Media Repository/Virtual Media Library for additional details.
- Adding hosts on Nutanix with the web console
- Adding hosts on vSphere
- About scheduling workloads on control plane nodes
4.6. Configuring hosts Copy linkLink copied to clipboard!
After booting the hosts with the discovery ISO, the hosts will appear in a table on the Host Discovery page. You can optionally configure the hostname and role for each host. You can also delete a host if necessary.
Procedure
- Go to the Host Discovery tab.
In multi-host clusters, you can select a role for each host after host discovery:
- In the Role column, expand the Auto-assign arrow for the host.
Choose one of the following options:
-
Auto-assign - Automatically determines whether the host is a
control plane,workerorarbiternode. This is the default setting. - Control plane node - Assigns the control plane (master) role to the host, allowing the host to manage and coordinate the cluster.
- Worker - Assigns the compute (worker) role to the host, enabling the host to run application workloads.
- Arbiter - Assigns the arbiter role to a host, providing a cost-effective solution for components that require a quorum.
The minimum hardware requirements for control plane nodes exceed that of worker nodes. If you assign a role to a host, ensure that you assign the control plane role to hosts that meet the minimum hardware requirements.
For more details about the different host roles, see About assigning roles to hosts.
-
Auto-assign - Automatically determines whether the host is a
- Click the Status link to view hardware, network, and operator validations for the host.
- Optionally select or deselect all hosts by clicking the <number> selected checkbox or alternatively by selecting Select all or Select none from the dropdown list.
Optionally change one or more hostnames:
- To rename a single host, from the Options (⋮) menu for the host, select Change hostname. If necessary, enter a new name for the host and click Change. You must ensure that each host has a valid and unique hostname.
To rename multiple selected hosts, from the Actions list, select Change hostname. In the Change Hostname dialog, type the new name and include
{{n}}to make each hostname unique. Then click Change.NoteYou can see the new names appearing in the Preview pane as you type. The name will be identical for all selected hosts, with the exception of a single-digit increment per host.
Optionally remove one or more hosts:
- To remove a single host, from the Options (⋮) menu for the host, select Remove host. Click Remove host to confirm the deletion.
- To remove multiple selected hosts at the same time, from the Actions list, select Remove. Click Remove hosts to confirm the deletion.
NoteIn a regular deployment, a cluster can have three or more hosts, and at least three of these must be control plane nodes. If you delete a host that is also a control plane node, or if there are only two hosts, you will get a message saying that the system is not ready. To restore a host, you must reboot it from the discovery ISO.
- From the Options (⋮) menu for the host, optionally select View host events. The events in the list are presented chronologically.
- Click the arrow to the left of a host name to expand the host details.
Once all cluster hosts appear with a status of Ready, proceed to the next step.
4.7. Configuring storage disks Copy linkLink copied to clipboard!
Each of the hosts retrieved during host discovery can have multiple storage disks. The storage disks are listed for the host on the Storage page of the Assisted Installer wizard.
You can optionally modify the default configurations for each disk.
- Starting from OpenShift Container Platform 4.14, you can configure nodes with Intel® Virtual RAID on CPU (VROC) to manage NVMe RAIDs. For details, see Configuring an Intel® Virtual RAID on CPU (VROC) data volume.
- Starting from OpenShift Container Platform 4.15, you can install a cluster on a single or multipath iSCSI boot device using the Assisted Installer.
4.7.1. Changing the installation disk Copy linkLink copied to clipboard!
The Assisted Installer randomly assigns an installation disk by default. If there are multiple storage disks for a host, you can select a different disk to be the installation disk. This automatically unassigns the previous disk.
Red Hat Enterprise Linux CoreOS (RHCOS) supports multipathing over Fibre Channel on the installation disk, allowing stronger resilience to hardware failure to achieve higher host availability. Multipathing is enabled by default in the agent ISO image, with an /etc/multipath.conf configuration. For details, see Modifying the DM Multipath configuration file.
Procedure
- Navigate to the Storage page of the wizard.
- Expand a host to display the associated storage disks.
Select Installation disk from the Role list.
NoteMultipath devices are automatically discovered and listed in the host’s inventory. To assign a multipath Fibre Channel disk as the installation disk, choose a disk with Drive type set to
Multipath, rather than toFCwhich indicates a single path.- When all storage disks return to Ready status, proceed to the next step.
4.7.2. Disabling disk formatting Copy linkLink copied to clipboard!
The Assisted Installer marks all bootable disks for formatting during the installation process by default, regardless of whether or not they have been defined as the installation disk. Formatting causes data loss.
You can choose to disable the formatting of a specific disk. Perform this with caution, as bootable disks can interfere with the installation process, mainly in terms of boot order.
You cannot disable formatting for the installation disk.
Procedure
- Navigate to the Storage page of the wizard.
- Expand a host to display the associated storage disks.
- Clear Format for a disk.
- When all storage disks return to Ready status, proceed to the next step.
4.8. Configuring networking Copy linkLink copied to clipboard!
Before installing OpenShift Container Platform, you must configure the cluster network.
Procedure
In the Networking step, select one of the following network management types if it is not already selected for you:
Cluster-Managed Networking: Selecting cluster-managed networking means that the Assisted Installer will configure a standard network topology. This configuration includes an integrated load balancer and virtual routing for managing the API and Ingress VIP addresses. For details, see Network management types.
Note- Currently, cluster-managed networking is not supported on IBM Z® and IBM Power®.
- Cluster-managed networking is not supported on single-node OpenShift.
User-Managed Networking: Selecting user-managed networking deploys OpenShift Container Platform with a non-standard network topology. Select user-managed networking if you want to deploy with an external load balancer and DNS, or if you intend to deploy the cluster nodes across many distinct subnets. For details, see Network management types.
NoteOracle Cloud Infrastructure (OCI) is available for OpenShift Container Platform 4.14 with a user-managed networking configuration only.
ImportantThe Assisted Installer supports a third network management type called Cluster-Managed Networking with a User-Managed Load Balancer. This network management type provides automated cluster networking with an external load balancer. Currently you can configure this network management type through the API only. For details, see Installing cluster-managed networking with a user-managed load balancer.
For cluster-managed networking, configure the following settings:
Select your Networking stack type:
- IPv4: Select this type when your hosts are only using IPv4.
- Dual-stack: You can select dual-stack when your hosts are using IPv4 together with IPv6.
Define the Machine network. You can use the default network or select a subnet.
ImportantFor iSCSI boot volumes, the hosts connect over two machine networks: one designated for the OpenShift Container Platform installation and the other for iSCSI traffic. Ensure you select the OpenShift Container Platform network from the dropdown list. The iSCSI host IP address should not be on the machine network. Choosing the iSCSI network will result in an Insufficient status for the host in the Networking step.
- Define an API virtual IP. An API virtual IP provides an endpoint for all users to interact with, and configure the platform.
- Define an Ingress virtual IP. An Ingress virtual IP provides an endpoint for application traffic flowing from outside the cluster.
Optional: Select Use advanced networking to configure the following advanced networking properties:
- Cluster network CIDR: Define an IP address block to assign pod IP addresses.
- Cluster network host prefix: Define a subnet prefix length to assign to each node.
- Service network CIDR: Define an IP address block to assign service IP addresses.
- Optional: Select Host SSH Public Key for troubleshooting after installation to connect to hosts using a public SSH key for troubleshooting after installation.
Additional resources
4.9. Adding manifests and patches Copy linkLink copied to clipboard!
You can upload custom manifests and patches for system manifests in the Assisted Installer web console. You can also replace and remove these files.
For information about adding and modifying custom manifests by using the Assisted Installer API, see Adding custom manifests with the API.
4.9.1. Preparing custom manifests and manifest patches Copy linkLink copied to clipboard!
This section provides an overview of custom manifests and system manifest patches, including formatting considerations and the required naming conventions for uploading the files.
Follow these guidelines to ensure that the files you upload comply with the system requirements.
4.9.1.1. Custom manifests Copy linkLink copied to clipboard!
A custom manifest is a JSON or YAML file that contains advanced configurations not currently supported in the Assisted Installer user interface. You can create a custom manifest or use one provided by a third party.
You can upload a custom manifest from your file system to either the openshift folder or the manifests folder. The number of custom manifest files permitted is unlimited.
You can upload only one file at a time. However, each uploaded YAML file can contain multiple custom manifests. Uploading a multi-document YAML manifest is faster than adding the YAML files individually.
For a file containing a single custom manifest, accepted file extensions include .yaml, .yml, or .json. For a file containing multiple custom manifests, accepted file types include .yaml or .yml.
Single custom manifest example
Multiple custom manifest example
When you install OpenShift Container Platform on the Oracle Cloud Infrastructure (OCI) external platform, you must add the custom manifests provided by Oracle. For additional external partner integrations such as vSphere or Nutanix, this step is optional.
Additional resources
4.9.1.2. Patches for system manifests Copy linkLink copied to clipboard!
A manifest patch file conforms to the syntax of a YAML patch. Its purpose is to modify a system manifest that is automatically created by the Assisted Installer during installation preparation. Manifest patches are used to adjust onfigurations, manage updates, or apply changes in a structured and automated way. This approach ensures consistency and helps avoid errors when altering complex YAML documents.
4.9.1.2.1. General YAML syntax for system manifest patches Copy linkLink copied to clipboard!
The yaml-patch package is an implementation of JavaScript Object Notation (JSON) Patch, directly transposed to YAML. The general syntax of a system manifest YAML patch is the following:
- op: <add | remove | replace | move | copy | test> from: <source-path> path: <target-path> value: <any-yaml-structure>
- op: <add | remove | replace | move | copy | test>
from: <source-path>
path: <target-path>
value: <any-yaml-structure>
- 1
- See the JavaScript Object Notation (JSON) Patch for an explanation of each operation.
- 2
- Only valid for
moveandcopyoperations. - 3
- Always mandatory.
- 4
- Only valid for
add,replaceandtestoperations.
4.9.1.2.2. Naming conventions for system manifest patches Copy linkLink copied to clipboard!
When creating a new patch for a system manifest, use the following naming convention: <file to be patched>.patch_<suffix>. The name itself ensures that the correct manifest is overwritten, and the suffix allows for the application of many patches to the same manifest.
For example, if the original file has the name 50-masters-chrony-configuration.yaml, then the new patch should be called 50-masters-chrony-configuration.yaml.patch_1_apply-chrony-dhcp or similar.
The following example outlines the steps for patching a system manifest, and shows how the naming convention is applied:
The Assisted Installer automatically adds the following YAML file to the manifests of the cluster at the start of the installation.
Directory:
OpenShift
Filename:50-masters-chrony-configuration.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow To patch this YAML file with different content, you must generate a new
base64representation of the content and create a patch file:Generate
base64file content for/etc/chrony.conf:Copy to Clipboard Copied! Toggle word wrap Toggle overflow Create a patch file using this
base64string:Directory:
OpenShift
Filename:50-masters-chrony-configuration.yaml.patch_1_apply-chrony-dhcp--- - op: replace path: /spec/config/storage/files/0/contents value: data:text/plain;charset=utf-8;base64,ZHJpZnRmaWxlIC92YXIvbGliL2Nocm9ueS9kcmlmdAptYWtlc3RlcCAxLjAgMwpydGNzeW5jCmxvZ2RpciAvdmFyL2xvZy9jaHJvbnkKc291cmNlZGlyIC9ydW4vY2hyb255LWRoY3AK
--- - op: replace path: /spec/config/storage/files/0/contents value: data:text/plain;charset=utf-8;base64,ZHJpZnRmaWxlIC92YXIvbGliL2Nocm9ueS9kcmlmdAptYWtlc3RlcCAxLjAgMwpydGNzeW5jCmxvZ2RpciAvdmFyL2xvZy9jaHJvbnkKc291cmNlZGlyIC9ydW4vY2hyb255LWRoY3AKCopy to Clipboard Copied! Toggle word wrap Toggle overflow
- You then upload the patch file in the Assisted Installer web console. For details, see the following section.
4.9.2. Uploading custom manifests and manifest patches Copy linkLink copied to clipboard!
When uploading a custom manifest or patch, enter the filename and select a destination folder. The filename must be unique across both folders; you cannot use the same file name in both folders.
Prerequisites
- You have saved a custom manifest file to a local directory using an appropriate file name and extension.
Procedure
- On the Cluster details page of the wizard, select the Include custom manifests checkbox.
On the Custom manifest page, in the folder field, select the Assisted Installer folder where you want to save the manifest or patch.
NoteYou can upload a file to either the openShift or manifest folder. For a manifest patch, the system will look in both folders for the target file that it needs to patch.
In the Filename field, enter a name for the manifest file, including the extension:
-
For custom manifests, examples include
manifest1.jsonormultiple1.yaml. - For manifest patches, an example is 50-masters-chrony-configuration.yaml.patch_1_apply-chrony-dhcp.
-
For custom manifests, examples include
- Under Content, click the Upload icon or Browse button to upload a file. Alternatively, drag the file into the Content field from your file system.
- To upload another file, click Add another manifest and repeat the process. This saves any previously uploaded files.
- Click Next to save all files and proceed to the Review and create page. Custom manifests displays a list of the uploaded custom manifests and patches.
4.9.3. Modifying custom manifests and manifest patches Copy linkLink copied to clipboard!
You can rename uploaded custom manifest or patch files, and save custom manifest files to a different folder. Additionally, you can copy the contents of an existing file, or download it to the folder specified in your Chrome download settings.
It is not possible to edit the content of an uploaded manifest or patch file. Instead, you can overwrite the existing file.
Prerequisites
- You have uploaded at least one custom manifest or patch file.
Procedure
- To change the location of a custom manifest file, select a different folder from the Folder list.
- To change the file name, type the new name for the manifest or patch in the File name field. Patch files should respect the patch naming conventions discussed earlier in this section.
To overwrite a manifest or patch file, save a new file with the same file name in either the openshift or manifest folder.
NoteThe system will automatically detect and replace the original file, regardless of which folder it is in.
- To download a manifest or patch to your file system, click the Download icon.
- To copy a manifest or patch, click the Copy to clipboard icon.
- To apply the changes, click either Add another manifest or Next.
4.9.4. Removing custom manifests and manifest patches Copy linkLink copied to clipboard!
You can remove uploaded custom manifests or patches before installation in one of two ways:
- Removing a single custom manifest or patch.
- Removing all manifests and patches at the same time.
Once you have removed a manifest or patch file you cannot undo the action. The workaround is to upload the file again.
4.9.4.1. Removing all custom manifests and patches Copy linkLink copied to clipboard!
You can remove all custom manifests and patches at the same time. This also hides the Custom manifest page.
Prerequisites
- You have uploaded at least one custom manifest or patch file.
Procedure
- Browse to the Cluster details page of the wizard.
- Clear the Include custom manifests checkbox.
- In the Remove custom manifests dialog box, click Remove.
4.9.4.2. Removing a single custom manifest or patch Copy linkLink copied to clipboard!
You can delete one file at a time. This option does not allow deletion of the last remaining manifest or patch.
Prerequisites
- You have uploaded at least two custom manifest or patch files.
Procedure
- Browse to the Custom manifests page.
- Hover over the manifest name to display the Delete (minus) icon.
- Click the icon and then click Delete in the dialog box.
4.10. Preinstallation validations Copy linkLink copied to clipboard!
The Assisted Installer ensures the cluster meets the prerequisites before installation, because it eliminates complex postinstallation troubleshooting, thereby saving significant amounts of time and effort. Before installing the cluster, ensure the cluster and each host pass preinstallation validation.
4.11. Installing the cluster Copy linkLink copied to clipboard!
After you have completed the configuration and all the nodes are Ready, you can begin installation. The installation process takes a considerable amount of time, and you can monitor the installation from the Assisted Installer web console. Nodes will reboot during the installation, and they will initialize after installation.
Procedure
- Click Begin installation.
- Click the link in the Status column of the Host Inventory list to see the installation status of a particular host.
4.12. Completing the installation Copy linkLink copied to clipboard!
After the cluster is installed and initialized, the Assisted Installer indicates that the installation is finished. The Assisted Installer provides the console URL, the kubeadmin username and password, and the kubeconfig file. Additionally, the Assisted Installer provides cluster details including the OpenShift Container Platform version, base domain, CPU architecture, API and Ingress IP addresses, and the cluster and service network IP addresses.
Prerequisites
-
You have installed the
ocCLI tool.
Procedure
-
Make a copy of the
kubeadminusername and password. Download the
kubeconfigfile and copy it to theauthdirectory under your working directory:mkdir -p <working_directory>/auth
$ mkdir -p <working_directory>/authCopy to Clipboard Copied! Toggle word wrap Toggle overflow cp kubeconfig <working_directory>/auth
$ cp kubeconfig <working_directory>/authCopy to Clipboard Copied! Toggle word wrap Toggle overflow NoteThe
kubeconfigfile is available for download for 20 days after completing the installation.Add the
kubeconfigfile to your environment:export KUBECONFIG=<your working directory>/auth/kubeconfig
$ export KUBECONFIG=<your working directory>/auth/kubeconfigCopy to Clipboard Copied! Toggle word wrap Toggle overflow Login with the
ocCLI tool:oc login -u kubeadmin -p <password>
$ oc login -u kubeadmin -p <password>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
<password>with the password of thekubeadminuser.- Click the web console URL or click Launch OpenShift Console to open the console.
-
Enter the
kubeadminusername and password. Follow the instructions in the OpenShift Container Platform console to configure an identity provider and configure alert receivers. - Add a bookmark of the OpenShift Container Platform console.
- Complete any postinstallation platform integration steps.
Chapter 5. Installing with the Assisted Installer API Copy linkLink copied to clipboard!
After you ensure the cluster nodes and network requirements are met, you can begin installing the cluster by using the Assisted Installer API. To use the API, you must perform the following procedures:
- Set up the API authentication.
- Configure the pull secret.
- Register a new cluster definition.
- Create an infrastructure environment for the cluster.
Once you perform these steps, you can modify the cluster definition, create discovery ISOs, add hosts to the cluster, and install the cluster. This document does not cover every endpoint of the Assisted Installer API, but you can review all of the endpoints in the API viewer or the swagger.yaml file.
5.1. Generating the offline token Copy linkLink copied to clipboard!
Download the offline token from the Assisted Installer web console. You will use the offline token to set the API token.
Prerequisites
-
Install
jq. - Log in to the OpenShift Cluster Manager as a user with cluster creation privileges.
Procedure
- In the menu, click Downloads.
- In the Tokens section under OpenShift Cluster Manager API Token, click View API Token.
Click Load Token.
ImportantDisable pop-up blockers.
- In the Your API token section, copy the offline token.
In your terminal, set the offline token to the
OFFLINE_TOKENvariable:export OFFLINE_TOKEN=<copied_token>
$ export OFFLINE_TOKEN=<copied_token>Copy to Clipboard Copied! Toggle word wrap Toggle overflow TipTo make the offline token permanent, add it to your profile.
(Optional) Confirm the
OFFLINE_TOKENvariable definition.echo ${OFFLINE_TOKEN}$ echo ${OFFLINE_TOKEN}Copy to Clipboard Copied! Toggle word wrap Toggle overflow
5.2. Authenticating with the REST API Copy linkLink copied to clipboard!
API calls require authentication with the API token. Assuming you use API_TOKEN as a variable name, add -H "Authorization: Bearer ${API_TOKEN}" to API calls to authenticate with the REST API.
The API token expires after 15 minutes.
Prerequisites
-
You have generated the
OFFLINE_TOKENvariable.
Procedure
On the command line terminal, set the
API_TOKENvariable using theOFFLINE_TOKENto validate the user.Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Confirm the
API_TOKENvariable definition:echo ${API_TOKEN}$ echo ${API_TOKEN}Copy to Clipboard Copied! Toggle word wrap Toggle overflow Create a script in your path for one of the token generating methods. For example:
vim ~/.local/bin/refresh-token
$ vim ~/.local/bin/refresh-tokenCopy to Clipboard Copied! Toggle word wrap Toggle overflow Copy to Clipboard Copied! Toggle word wrap Toggle overflow Then, save the file.
Change the file mode to make it executable:
chmod +x ~/.local/bin/refresh-token
$ chmod +x ~/.local/bin/refresh-tokenCopy to Clipboard Copied! Toggle word wrap Toggle overflow Refresh the API token:
source refresh-token
$ source refresh-tokenCopy to Clipboard Copied! Toggle word wrap Toggle overflow Verify that you can access the API by running the following command:
curl -s https://api.openshift.com/api/assisted-install/v2/component-versions -H "Authorization: Bearer ${API_TOKEN}" | jq$ curl -s https://api.openshift.com/api/assisted-install/v2/component-versions -H "Authorization: Bearer ${API_TOKEN}" | jqCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
5.3. Configuring the pull secret Copy linkLink copied to clipboard!
Many of the Assisted Installer API calls require the pull secret. Download the pull secret to a file so that you can reference it in API calls. The pull secret is a JSON object that will be included as a value within the request’s JSON object. The pull secret JSON must be formatted to escape the quotes. For example:
Before
{"auths":{"cloud.openshift.com": ...
{"auths":{"cloud.openshift.com": ...
After
{\"auths\":{\"cloud.openshift.com\": ...
{\"auths\":{\"cloud.openshift.com\": ...
Procedure
- In the menu, click OpenShift.
- In the submenu, click Downloads.
- In the Tokens section under Pull secret, click Download.
To use the pull secret from a shell variable, execute the following command:
export PULL_SECRET=$(cat ~/Downloads/pull-secret.txt | jq -R .)
$ export PULL_SECRET=$(cat ~/Downloads/pull-secret.txt | jq -R .)Copy to Clipboard Copied! Toggle word wrap Toggle overflow To slurp the pull secret file using
jq, reference it in thepull_secretvariable, piping the value totojsonto ensure that it is properly formatted as escaped JSON. For example:Copy to Clipboard Copied! Toggle word wrap Toggle overflow Confirm the
PULL_SECRETvariable definition:echo ${PULL_SECRET}$ echo ${PULL_SECRET}Copy to Clipboard Copied! Toggle word wrap Toggle overflow
5.4. Generating the SSH public key Copy linkLink copied to clipboard!
During the installation of OpenShift Container Platform, you can optionally provide an SSH public key to the installation program. This is useful for initiating an SSH connection to a remote node when troubleshooting an installation error.
If you do not have an existing SSH key pair on your local machine to use for the authentication, create one now.
For more information, see Generating a key pair for cluster node SSH access.
Prerequisites
-
Generate the
OFFLINE_TOKENandAPI_TOKENvariables.
Procedure
From the root user in your terminal, get the SSH public key:
cat /root/.ssh/id_rsa.pub
$ cat /root/.ssh/id_rsa.pubCopy to Clipboard Copied! Toggle word wrap Toggle overflow Set the SSH public key to the
CLUSTER_SSHKEYvariable:CLUSTER_SSHKEY=<downloaded_ssh_key>
$ CLUSTER_SSHKEY=<downloaded_ssh_key>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Confirm the
CLUSTER_SSHKEYvariable definition:echo ${CLUSTER_SSHKEY}$ echo ${CLUSTER_SSHKEY}Copy to Clipboard Copied! Toggle word wrap Toggle overflow
5.5. Registering a new cluster Copy linkLink copied to clipboard!
To register a new cluster definition with the API, use the /v2/clusters endpoint.
The following parameters are mandatory:
-
name -
openshift-version -
pull_secret -
cpu_architecture
See the cluster-create-params model in the API viewer for details on the fields you can set when registering a new cluster. When setting the olm_operators field, see Additional Resources for details on installing Operators.
Prerequisites
-
You have generated a valid
API_TOKEN. Tokens expire every 15 minutes. - You have downloaded the pull secret.
-
Optional: You have assigned the pull secret to the
$PULL_SECRETvariable.
Procedure
Refresh the API token:
source refresh-token
$ source refresh-tokenCopy to Clipboard Copied! Toggle word wrap Toggle overflow Register a new cluster by using one of the following methods:
Reference the pull secret file in the request:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Write the configuration to a JSON file and then reference it in the request:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow curl -s -X POST "https://api.openshift.com/api/assisted-install/v2/clusters" \ -d @./cluster.json \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $API_TOKEN" \ | jq '.id'
$ curl -s -X POST "https://api.openshift.com/api/assisted-install/v2/clusters" \ -d @./cluster.json \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $API_TOKEN" \ | jq '.id'Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1 1
- Pay attention to the following:
-
To install the latest OpenShift version, use the
x.yformat, such as4.16for version 4.16.10. To install a specific OpenShift version, use thex.y.zformat, such as4.16.3for version 4.16.3. -
To install a multi-architecture compute cluster, add the
-multiextension, such as4.16-multifor the latest version or4.16.3-multifor a specific version. - If you are booting from an iSCSI drive, enter OpenShift Container Platform version 4.15 or later.
-
To install the latest OpenShift version, use the
- 2 2
- Optionally set the number of control plane nodes to
1for a single-node OpenShift cluster, to2or more for a Two-Node OpenShift with Arbiter cluster, or to3,4, or5for a multi-node OpenShift Container Platform cluster. If this setting is omitted, the Assisted Installer sets3as the default.Note-
The
control_plane_countfield replaces thehigh_availability_modefield, which is deprecated. For details, see API deprecation notice. - Currently, single-node OpenShift is not supported on IBM Z® and IBM Power® platforms.
-
The Assisted Installer supports
4or5control plane nodes from OpenShift Container Platform 4.18 and later, on a bare metal or user-managed networking platform with an x86_64 CPU architecture. For details, see About specifying the number of control plane nodes. -
The Assisted Installer supports
2control plane nodes from OpenShift Container Platform 4.19 and later, for a Two-Node OpenShift with Arbiter cluster topology. If the number of control plane nodes for a cluster is2, then it must have at least one additional arbiter host. For details, see Two-Node OpenShift with Arbiter resource requirements.
-
The
- 3
- Valid values are
x86_64,arm64,ppc64le,s390x, ormulti. Specifymultifor a multi-architecture compute cluster.
Assign the returned
cluster_idto theCLUSTER_IDvariable and export it:export CLUSTER_ID=<cluster_id>
$ export CLUSTER_ID=<cluster_id>Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteIf you close your terminal session, you need to export the
CLUSTER_IDvariable again in a new terminal session.Check the status of the new cluster:
curl -s -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $API_TOKEN" \ | jq
$ curl -s -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $API_TOKEN" \ | jqCopy to Clipboard Copied! Toggle word wrap Toggle overflow
Once you register a new cluster definition, create the infrastructure environment for the cluster.
You cannot see the cluster configuration settings in the Assisted Installer user interface until you create the infrastructure environment.
5.5.1. Installing Operators Copy linkLink copied to clipboard!
You can customize your deployment by adding Operators to the cluster during installation. You can install one or more Operators individually or add a group of Operators that form a bundle. If you require advanced options, add the Operators after you have installed the cluster.
This step is optional.
5.5.1.1. Installing standalone Operators Copy linkLink copied to clipboard!
Before selecting Operators for installation, you can verify which Operators are available in the Assisted Installer. You can also check whether an Operator is supported for a specific OCP version, CPU architecture, or platform.
You set the required Operator definitions by using the POST method for the assisted-service/v2/clusters/{cluster_id} endpoint and by setting the olm_operators parameter.
The Assisted Installer allows you to install the following standalone Operators. For additional Operators that you can select as part of a bundle, see Installing bundle Operators.
OpenShift Virtualization Operator (
cnv)Note- Currently, OpenShift Virtualization is not supported on IBM Z® and IBM Power®.
The OpenShift Virtualization Operator requires backend storage and might automatically activate a storage Operator in the background, according to the following criteria:
- None - If the CPU architecture is ARM64, no storage Operator is activated.
- LVM Storage - For single-node OpenShift clusters on any other CPU architecture deploying OpenShift Container Platform 4.12 or higher.
- Local Storage Operator (LSO) - For all other deployments.
Migration Toolkit for Virtualization Operator (
mtv)NoteSpecifying the Migration Toolkit for Virtualization (MTV) Operator automatically activates the OpenShift Virtualization Operator. For a Single-node OpenShift installation, the Assisted Installer also activates the LVM Storage Operator.
Multicluster engine Operator (
mce)NoteDeploying the multicluster engine without OpenShift Data Foundation results in the following storage configurations:
- Multi-node cluster: No storage is configured. You must configure storage after the installation.
- Single-node OpenShift: LVM Storage is installed.
-
OpenShift Data Foundation Operator (
odf) -
Logical Volume Manager Storage Operator (
lvm) -
OpenShift AI Operator (
openshift-ai) OpenShift sandboxed containers Operator (
osc)ImportantThe integration of the OpenShift sandboxed containers Operator into the Assisted Installer is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.
Kubernetes NMState Operator (
nmstate)NoteCurrently, you cannot install the Kubernetes NMState Operator on the Nutanix or Oracle Cloud Infrastructure (OCI) third-party platforms.
AMD GPU Operator (
amd-gpu)NoteInstalling the AMD GPU Operator automatically activates the Kernel Module Management Operator.
-
Kernel Module Management Operator (
kmm) -
Node Feature Discovery Operator (
node-feature-discovery) -
Self Node Remediation (
self-node-remediation) NVIDIA GPU Operator (
nvidia-gpu)NoteInstalling the NVIDIA GPU Operator automatically activates the Node Feature Discovery Operator.
The integration of the OpenShift AI, AMD GPU, Kernel Module Management, Node Feature Discovery, Self Node Remediation, and NVIDIA GPU Operators into the Assisted Installer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.
Prerequisites
- You have reviewed Customizing your installation using Operators for an overview of each Operator that you intend to install, together with its prerequisites and dependencies.
Procedure
Optional: Check which Operators are available in the Assisted Installer, by running the following command:
curl -s "https://api.openshift.com/api/assisted-install/v2/supported-operators" -H "Authorization: Bearer ${API_TOKEN}" | jq .$ curl -s "https://api.openshift.com/api/assisted-install/v2/supported-operators" -H "Authorization: Bearer ${API_TOKEN}" | jq .Copy to Clipboard Copied! Toggle word wrap Toggle overflow Check whether an Operator is supported for a specified OCP version, CPU architecture, or platform by running the following command:
curl -s "https://api.openshift.com/api/assisted-install/v2/support-levels/features?openshift_version=4.13&cpu_architecture=x86_64&platform_type=baremetal" -H "Authorization: Bearer ${API_TOKEN}" | jq .features.SNO$ curl -s "https://api.openshift.com/api/assisted-install/v2/support-levels/features?openshift_version=4.13&cpu_architecture=x86_64&platform_type=baremetal" -H "Authorization: Bearer ${API_TOKEN}" | jq .features.SNO1 2 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Replace the attributes as follows:
-
For
openshift_version, specify the OpenShift Container Platform version number. This attribute is mandatory. -
For
cpu_architecture, specifyx86_64,aarch64,arm64,ppc64le,s390x, ormulti. This attribute is optional. -
For
platform_type, specifybaremetal,none,nutanix,vsphere, orexternal. This attribute is optional.
-
For
- 2
- Specify the Operator in upper case, for example,
.NODE-FEATURE-DISCOVERYfor Node Feature Discovery,.OPENSHIFT-AIfor OpenShift AI,.OSCfor OpenShift sandboxed containers,.SELF-NODE-REMEDIATIONfor Self Node Remediation, or.MTVfor Migration Toolkit for Virtualization.Example output
"supported"
"supported"Copy to Clipboard Copied! Toggle word wrap Toggle overflow Where possible statuses are "supported", "dev-preview", "tech-preview", and "unavailable".
Get the full list of supported Operators and additional features for a specified OCP version, CPU architecture, or platform by running the following command:
curl -s "https://api.openshift.com/api/assisted-install/v2/support-levels/features?openshift_version=4.13&cpu_architecture=x86_64&platform_type=baremetal" -H "Authorization: Bearer ${API_TOKEN}" | jq$ curl -s "https://api.openshift.com/api/assisted-install/v2/support-levels/features?openshift_version=4.13&cpu_architecture=x86_64&platform_type=baremetal" -H "Authorization: Bearer ${API_TOKEN}" | jqCopy to Clipboard Copied! Toggle word wrap Toggle overflow Specify the Operators to install by running the following command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- List the Operators that you want to install. Specify
cnvfor OpenShift Virtualization,mtvfor Migration Toolkit for Virtualization,mcefor multicluster engine,odffor Red Hat OpenShift Data Foundation,lvmfor Logical Volume Manager Storage,openshift-aifor OpenShift AI,oscfor OpenShift sandboxed containers,nmstatefor Kubernetes NMState,amd-gpufor AMD GPU,kmmfor Kernel Module Management,node-feature-discoveryfor Node Feature Discovery,nvidia-gpufor NVIDIA GPU andself-node-remediationfor Self Node Remediation. Installing an Operator automatically activates any dependent Operators.
5.5.1.2. Installing bundle Operators Copy linkLink copied to clipboard!
Although you cannot install an Operator bundle directly through the API, you can verify which Operators are included in a bundle and specify each Operator individually.
The Assisted Installer currently supports the following Operator bundles:
Virtualization Operator bundle - Contains the following Operators:
-
Kube Descheduler Operator (
kube-descheduler) -
Node Maintenance Operator {
node-maintenance} -
Migration Toolkit for Virtualization Operator (
mtv) -
Kubernetes NMState Operator (
nmstate) -
Fence Agents Remediation Operator (
fence-agents-remediation) -
OpenShift Virtualization Operator (
cnv) -
Node Health Check Operator (
node-healthcheck) -
Local Storage Operator (LSO) Operator (
lso) -
Cluster Observability Operator (
cluster-observability) -
MetalLB Operator (
metallb) -
NUMA Resources Operator (
numaresources) -
OpenShift API for Data Protection Operator (
oadp)
-
Kube Descheduler Operator (
OpenShift AI Operator bundle - Contains the following Operators:
-
Kubernetes Authorino Operator (
authorino) -
OpenShift Data Foundation Operator (
odf) -
OpenShift AI Operator (
openshift-ai) -
AMD GPU Operator (
amd-gpu) -
Node Feature Discovery Operator (
node-feature-discovery) -
NVIDIA GPU Operator (
nvidia-gpu) -
OpenShift Pipelines Operator (
pipelines) -
OpenShift Service Mesh Operator (
servicemesh) -
OpenShift Serverless Operator (
serverless) -
Kernel Module Management Operator (
kmm)
-
Kubernetes Authorino Operator (
The OpenShift AI Operator bundle is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.
Prerequisites
- You have reviewed Customizing your installation using Operator bundles for an overview of the Operator bundles, together with their prerequisites and associated Operators.
Procedure
Optional: Check which Operator bundles are available in the Assisted Installer by running the following command:
curl -s "https://api.openshift.com/api/assisted-install/v2/operators/bundles" -H "Authorization: Bearer ${API_TOKEN}" | jq .$ curl -s "https://api.openshift.com/api/assisted-install/v2/operators/bundles" -H "Authorization: Bearer ${API_TOKEN}" | jq .Copy to Clipboard Copied! Toggle word wrap Toggle overflow Optional: Check which Operators are associated with a specific bundle by running the following command:
curl -s "https://api.openshift.com/api/assisted-install/v2/operators/bundles/virtualization" -H "Authorization: Bearer ${API_TOKEN}" | jq .$ curl -s "https://api.openshift.com/api/assisted-install/v2/operators/bundles/virtualization" -H "Authorization: Bearer ${API_TOKEN}" | jq .1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Specify
virtualizationfor the Virtualization Operator bundle oropenshift-aifor the OpenShift AI Operator bundle. The example specifies the Virtualization Operator bundle.
Example output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Install the Operators associated with the bundle by running the following command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Specify the Operators in the Operator bundle you are installing. The example lists the Operators for the Virtualization bundle.
- 2
- Note the following:
-
In the Virtualization bundle, specifying
cnvautomatically installslsoin the background. In the OpenShift AI Operator bundle:
-
Specifying
nvidia-gpuautomatically installsnode-feature-discovery. -
Specifying
amd-gpuautomatically installskmm.
-
Specifying
-
In the Virtualization bundle, specifying
5.5.2. Scheduling workloads to run on control plane nodes Copy linkLink copied to clipboard!
Use the schedulable_masters attribute to enable workloads to run on control plane nodes.
Prerequisites
-
You have generated a valid
API_TOKEN. Tokens expire every 15 minutes. -
You have created a
$PULL_SECRETvariable. - You are installing OpenShift Container Platform 4.14 or later.
Procedure
- Follow the instructions for installing Assisted Installer using the Assisted Installer API.
When you reach the step for registering a new cluster, set the
schedulable_mastersattribute as follows:Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Enables the scheduling of workloads on the control plane nodes.
5.5.3. Configuring the network management type Copy linkLink copied to clipboard!
The Assisted Installer lets you install the following network management types:
You define the network management type by adding the user_managed_networking and load_balancer attributes to the cluster definition, as in the example below:
Where:
-
user_managed_networkingis eithertrueorfalse. -
load_balancercan have the typeuser-managedorcluster-managed.
You can review the user_managed_networking and load_balancer valid values in the swagger.yaml file.
This step is optional. If you do not define a network management type, the Assisted Installer applies cluster-managed networking by default to all highly available clusters. For single-node OpenShift, the Assisted Installer applies user-managed networking by default.
5.5.3.1. Installing cluster-managed networking Copy linkLink copied to clipboard!
Selecting cluster-managed networking means that the Assisted Installer will configure a standard network topology. This configuration includes an integrated load balancer and virtual routing for managing the API and Ingress VIP addresses. For details, see Network management types.
Prerequisites
You are installing an OpenShift Container Platform cluster of three or more control plane nodes.
NoteCurrently, cluster-managed networking is not supported on IBM Z® and IBM Power®.
Procedure
To define cluster-managed networking, add the following attributes and values to your cluster definition:
"user_managed_networking": false, "load_balancer": { "type": "cluster-managed" }"user_managed_networking": false, "load_balancer": { "type": "cluster-managed" }Copy to Clipboard Copied! Toggle word wrap Toggle overflow Where the
load_balancerattribute is optional. If omitted for this configuration, thetypeis automatically set touser-managedfor single-node OpenShift or tocluster-managedfor all other implementations.
Additional resources
5.5.3.2. Installing user-managed networking Copy linkLink copied to clipboard!
Selecting user-managed networking deploys OpenShift Container Platform with a non-standard network topology. Select user-managed networking if you want to deploy a cluster with an external load balancer and DNS, or if you intend to deploy the cluster nodes across many distinct subnets.
For details, see Network management types.
The Assisted Installer lets you deploy more than one external load balancer for user-managed networking.
Oracle Cloud Infrastructure (OCI) is available for OpenShift Container Platform 4.14 with a user-managed networking configuration only.
Procedure
To define user-managed networking, add the following attributes to your cluster definition:
"user_managed_networking": true,
"user_managed_networking": true,Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteThe
load_balancerattribute is not required when user-managed networking is set totrue, because you will be provisioning your own load balancer.
Network Validations
When you enable user-managed networking, the following network validations change:
- The L3 connectivity check (ICMP) replaces the L2 check (ARP).
- The maximum transmission unit (MTU) validation verifies the MTU value for all interfaces and not only for the machine network.
Additional resources
5.5.3.3. Installing cluster-managed networking with a user-managed load balancer Copy linkLink copied to clipboard!
Cluster-managed networking with a user-managed load balancer is a hybrid network management type designed for scenarios that require automated cluster networking with external control over load balancing. This approach enables users to provide one or more external load balancers (for example, an API load balancer and an Ingress load balancer), while retaining the bare-metal features installed in cluster-managed networking.
For details, see Network management types.
Cluster-managed networking with a user-managed load balancer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.
Use the Assisted Installer API to deploy cluster-managed networking with a user-managed load balancer on a bare-metal or vSphere platform.
Prerequisites
- You are installing OpenShift Container Platform version 4.16 or higher.
- You are installing on a bare-metal or vSphere platform.
- You are using IPv4 single-stack networking.
- You are installing an OpenShift Container Platform cluster of three or more control plane nodes.
- For a vSphere platform installation, you meet the additional requirements specified in vSphere installation requirements.
Procedure
Configure the load balancer to be accessible from all hosts and have access to the following services:
- OpenShift Machine Config Operator (MCO) - On control plane nodes.
- OpenShift API - On control plane nodes.
- Ingress Controller - On compute (worker) nodes.
For details, see Configuring a user-managed load balancer (steps 1 and 2).
Configure the DNS records for your cluster to target the front-end IP addresses of the user-managed load balancer. You must update records to your DNS server for the cluster API and applications over the load balancer:
Configure the DNS record to make your primary API accessible:
<load_balancer_ip_address> <record_name> api.<cluster_name>.<base_domain>
<load_balancer_ip_address> <record_name> api.<cluster_name>.<base_domain>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Configure the DNS record to route external traffic to your applications via an ingress controller:
<load_balancer_ip_address> <record_name> apps.<cluster_name>.<base_domain>
<load_balancer_ip_address> <record_name> apps.<cluster_name>.<base_domain>Copy to Clipboard Copied! Toggle word wrap Toggle overflow For vSphere only, configure the DNS record to support internal API access within your network:
<load_balancer_ip_address> <record_name> api-int.<cluster_name>.<base_domain>
<load_balancer_ip_address> <record_name> api-int.<cluster_name>.<base_domain>Copy to Clipboard Copied! Toggle word wrap Toggle overflow
For details, see Configuring a user-managed load balancer (step 3).
Add the following configurations to the Assisted Installer API cluster definitions:
Set the
user_managed_networkingandload_balancerfields to the following values:"user_managed_networking": false, "load_balancer": { "type": "user-managed" }"user_managed_networking": false, "load_balancer": { "type": "user-managed" }Copy to Clipboard Copied! Toggle word wrap Toggle overflow For details, see Changing the network management type.
Specify the Ingress and API VIPs. These should correspond to the load balancer IP address:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Specify a list of machine networks to ensure the following:
- Each node has at least one network interface card (NIC) with an IP address in at least one machine network.
The load balancer IP, which is also the API VIP and Ingress VIP, is included in at least one of the machine networks.
Example
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
For more details, see Machine network.
Network Validations
When you enable this network management type, the following network validations change:
- The L3 connectivity check (ICMP) replaces the L2 check (ARP).
- The maximum transmission unit (MTU) validation verifies the MTU value for all interfaces and not only for the machine network.
5.6. Modifying a cluster Copy linkLink copied to clipboard!
To modify a cluster definition with the API, use the /v2/clusters/{cluster_id} endpoint. Modifying a cluster resource is a common operation for adding settings such as changing the network type or enabling user-managed networking. See the v2-cluster-update-params model in the API viewer for details on the fields you can set when modifying a cluster definition.
You can add or remove Operators from a cluster resource that has already been registered.
To create partitions on nodes, see Configuring storage on nodes in the OpenShift Container Platform documentation.
Prerequisites
- You have created a new cluster resource.
Procedure
Refresh the API token:
source refresh-token
$ source refresh-tokenCopy to Clipboard Copied! Toggle word wrap Toggle overflow Modify the cluster. For example, change the SSH key:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
5.6.1. Modifying Operators by using the API Copy linkLink copied to clipboard!
You can add or remove Operators from a cluster resource that has already been registered as part of a previous installation. This is only possible before you start the OpenShift Container Platform installation.
You modify the required Operator definition by using the PATCH method for the assisted-service/v2/clusters/{cluster_id} endpoint and by setting the olm_operators parameter.
Prerequisites
- You have refreshed the API token.
-
You have exported the
CLUSTER_IDas an environment variable.
Procedure
Run the following command to modify the Operators:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Specify
cnvfor OpenShift Virtualization,mtvfor Migration Toolkit for Virtualization,mcefor multicluster engine,odffor Red Hat OpenShift Data Foundation,lvmfor Logical Volume Manager Storage,openshift-aifor OpenShift AI,oscfor OpenShift sandboxed containers,nmstatefor Kubernetes NMState,amd-gpufor AMD GPU,kmmfor Kernel Module Management,node-feature-discoveryfor Node Feature Discovery,nvidia-gpufor NVIDIA GPU,self-node-remediationfor Self Node Remediation,pipelinesfor OpenShift Pipelines,servicemeshfor OpenShift Service Mesh,node-healthcheckfor Node Health Check,lsofor Local Storage Operator,fence-agents-remediationfor Fence Agents Remediation,kube-deschedulerfor Kube Descheduler,serverlessfor OpenShift Serverless,authorinofor Authorino,cluster-observabilityfor Cluster Observability Operator,metallbfor MetalLB,numaresourcesfor NUMA Resources, andoadpfor OpenShift API for Data Protection. - 2
- To modify the Operators, add a new complete list of Operators that you want to install, and not just the differences. To remove all Operators, specify an empty array:
"olm_operators": [].
Example output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteThe output is the description of the new cluster state. The
monitored_operatorsproperty in the output contains Operators of two types:-
"operator_type": "builtin": Operators of this type are an integral part of OpenShift Container Platform. -
"operator_type": "olm": Operators of this type are added manually by a user or automatically, as a dependency. In this example, the LVM Storage Operator is added automatically as a dependency of OpenShift Virtualization.
Additional resources
- See Customizing your installation using Operators and Operator Bundles for an overview of each Operator that you intend to install, together with its prerequisites and dependencies.
5.7. Registering a new infrastructure environment Copy linkLink copied to clipboard!
Once you register a new cluster definition with the Assisted Installer API, create an infrastructure environment using the v2/infra-envs endpoint. Registering a new infrastructure environment requires the following settings:
-
name -
pull_secret -
cpu_architecture
See the infra-env-create-params model in the API viewer for details on the fields you can set when registering a new infrastructure environment. You can modify an infrastructure environment after you create it. As a best practice, consider including the cluster_id when creating a new infrastructure environment. The cluster_id will associate the infrastructure environment with a cluster definition. When creating the new infrastructure environment, the Assisted Installer will also generate a discovery ISO.
Prerequisites
-
You have generated a valid
API_TOKEN. Tokens expire every 15 minutes. - You have downloaded the pull secret.
-
Optional: You have registered a new cluster definition and exported the
cluster_id.
Procedure
Refresh the API token:
source refresh-token
$ source refresh-tokenCopy to Clipboard Copied! Toggle word wrap Toggle overflow Register a new infrastructure environment. Provide a name, preferably something including the cluster name. This example provides the cluster ID to associate the infrastructure environment with the cluster resource. The following example specifies the
image_type. You can specify eitherfull-isoorminimal-iso. The default value isminimal-iso.Optional: You can register a new infrastructure environment by slurping the pull secret file in the request:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Note- 1
- Valid values are
x86_64,arm64,ppc64le,s390x, andmulti.
Optional: You can register a new infrastructure environment by writing the configuration to a JSON file and then referencing it in the request:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow curl -s -X POST "https://api.openshift.com/api/assisted-install/v2/infra-envs"
$ curl -s -X POST "https://api.openshift.com/api/assisted-install/v2/infra-envs" -d @./infra-envs.json -H "Content-Type: application/json" -H "Authorization: Bearer $API_TOKEN" | jq '.id'Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Assign the returned
idto theINFRA_ENV_IDvariable and export it:export INFRA_ENV_ID=<id>
$ export INFRA_ENV_ID=<id>Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Once you create an infrastructure environment and associate it to a cluster definition via the cluster_id, you can see the cluster settings in the Assisted Installer web user interface. If you close your terminal session, you need to re-export the id in a new terminal session.
5.8. Modifying an infrastructure environment Copy linkLink copied to clipboard!
You can modify an infrastructure environment using the /v2/infra-envs/{infra_env_id} endpoint. Modifying an infrastructure environment is a common operation for adding settings such as networking, SSH keys, or ignition configuration overrides.
See the infra-env-update-params model in the API viewer for details on the fields you can set when modifying an infrastructure environment. When modifying the new infrastructure environment, the Assisted Installer will also re-generate the discovery ISO.
Prerequisites
- You have created a new infrastructure environment.
Procedure
Refresh the API token:
source refresh-token
$ source refresh-tokenCopy to Clipboard Copied! Toggle word wrap Toggle overflow Modify the infrastructure environment:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
5.8.1. Adding kernel arguments Copy linkLink copied to clipboard!
Providing kernel arguments to the Red Hat Enterprise Linux CoreOS (RHCOS) kernel via the Assisted Installer means passing specific parameters or options to the kernel at boot time, particularly when you cannot customize the kernel parameters of the discovery ISO. Kernel parameters can control various aspects of the kernel’s behavior and the operating system’s configuration, affecting hardware interaction, system performance, and functionality. Kernel arguments are used to customize or inform the node’s RHCOS kernel about the hardware configuration, debugging preferences, system services, and other low-level settings.
The RHCOS installer kargs modify command supports the append, delete, and replace options.
You can modify an infrastructure environment using the /v2/infra-envs/{infra_env_id} endpoint. When modifying the new infrastructure environment, the Assisted Installer will also re-generate the discovery ISO.
Procedure
Refresh the API token:
source refresh-token
$ source refresh-tokenCopy to Clipboard Copied! Toggle word wrap Toggle overflow Modify the kernel arguments:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Replace
<karg>with the the kernel argument and<value>with the kernal argument value. For example:rd.net.timeout.carrier=60. You can specify multiple kernel arguments by adding a JSON object for each kernel argument.
5.9. Applying a static network configuration Copy linkLink copied to clipboard!
You can apply a static network configuration by using the Assisted Installer API. This step is optional.
A static IP configuration is not supported in the following scenarios:
- OpenShift Container Platform installations on Oracle Cloud Infrastructure.
- OpenShift Container Platform installations on iSCSI boot volumes.
Prerequisites
- You have created an infrastructure environment using the API or have created a cluster using the web console.
-
You have your infrastructure environment ID exported in your shell as
$INFRA_ENV_ID. -
You have credentials to use when accessing the API and have exported a token as
$API_TOKENin your shell. -
You have YAML files with a static network configuration available as
server-a.yamlandserver-b.yaml.
Procedure
Create a temporary file
/tmp/request-body.txtwith the API request:Copy to Clipboard Copied! Toggle word wrap Toggle overflow Refresh the API token:
source refresh-token
$ source refresh-tokenCopy to Clipboard Copied! Toggle word wrap Toggle overflow Send the request to the Assisted Service API endpoint:
curl -H "Content-Type: application/json" \ -X PATCH -d @/tmp/request-body.txt \ -H "Authorization: Bearer ${API_TOKEN}" \ https://api.openshift.com/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID$ curl -H "Content-Type: application/json" \ -X PATCH -d @/tmp/request-body.txt \ -H "Authorization: Bearer ${API_TOKEN}" \ https://api.openshift.com/api/assisted-install/v2/infra-envs/$INFRA_ENV_IDCopy to Clipboard Copied! Toggle word wrap Toggle overflow
5.10. Adding hosts Copy linkLink copied to clipboard!
After configuring the cluster resource and infrastructure environment, download the discovery ISO image. You can choose from two images:
-
Full ISO image: Use the full ISO image when booting must be self-contained. The image includes everything needed to boot and start the Assisted Installer agent. The ISO image is about 1GB in size. This is the recommended method for the
s390xarchitecture when installing with RHEL KVM. Minimal ISO image: Use the minimal ISO image when the virtual media connection has limited bandwidth. This is the default setting. The image includes only what the agent requires to boot a host with networking. The majority of the content is downloaded upon boot. The ISO image is about 100MB in size.
This option is mandatory in the following scenarios:
- If you are installing OpenShift Container Platform on Oracle Cloud Infrastructure.
- If you are installing OpenShift Container Platform on iSCSI boot volumes.
Currently, ISO images are supported on IBM Z® (s390x) with KVM, iPXE with z/VM, and LPAR (both static and DPM). For details, see Booting hosts using iPXE.
You can boot hosts with the discovery image using three methods. For details, see Booting hosts with the discovery image.
Prerequisites
- You have created a cluster.
- You have created an infrastructure environment.
- You have completed the configuration.
If the cluster hosts require the use of a proxy, select Configure cluster-wide proxy settings. Enter the username, password, required domains or IP addresses, and port for the HTTP and HTTPS URLs of the proxy server. If the cluster hosts are behind a firewall, allow the nodes to access the required domains or IP addresses through the firewall. See Configuring your firewall for OpenShift Container Platform for more information.
NoteThe proxy username and password must be URL-encoded.
-
You have selected an image type or will use the default
minimal-iso.
Procedure
- Configure the discovery image if needed. For details, see Configuring the discovery image.
Refresh the API token:
source refresh-token
$ source refresh-tokenCopy to Clipboard Copied! Toggle word wrap Toggle overflow Get the download URL:
curl -H "Authorization: Bearer ${API_TOKEN}" \ https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/downloads/image-url$ curl -H "Authorization: Bearer ${API_TOKEN}" \ https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/downloads/image-urlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
{ "expires_at": "2024-02-07T20:20:23.000Z", "url": "https://api.openshift.com/api/assisted-images/bytoken/<TOKEN>/<OCP_VERSION>/<CPU_ARCHITECTURE>/<FULL_OR_MINIMAL_IMAGE>.iso" }{ "expires_at": "2024-02-07T20:20:23.000Z", "url": "https://api.openshift.com/api/assisted-images/bytoken/<TOKEN>/<OCP_VERSION>/<CPU_ARCHITECTURE>/<FULL_OR_MINIMAL_IMAGE>.iso" }Copy to Clipboard Copied! Toggle word wrap Toggle overflow Download the discovery image:
wget -O discovery.iso <url>
$ wget -O discovery.iso <url>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
<url>with the download URL from the previous step.- Boot the host(s) with the discovery image.
- Assign a role to host(s).
5.10.1. Selecting a role Copy linkLink copied to clipboard!
You can select a role for the host by using the /v2/infra-envs/{infra_env_id}/hosts/{host_id} endpoint. A host can have one of the following roles:
-
master- Assigns the control plane role to the host, allowing the host to manage and coordinate the cluster. -
arbiter- Assigns the arbiter role to the host, providing a cost-effective solution for components that require a quorum. -
worker- Assigns the compute role to the host, enabling the host to run application workloads. -
auto-assign- Automatically determines whether the host is amaster,worker, orarbiter. This is the default setting.
Use this procedure to assign a role to the host. If the host_role setting is omitted, the host defaults to auto-assign.
Prerequisites
- You have added hosts to the cluster.
Procedure
Refresh the API token:
source refresh-token
$ source refresh-tokenCopy to Clipboard Copied! Toggle word wrap Toggle overflow Get the host IDs:
curl -s -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID" \ --header "Content-Type: application/json" \ -H "Authorization: Bearer $API_TOKEN" \ | jq '.host_networks[].host_ids'
$ curl -s -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID" \ --header "Content-Type: application/json" \ -H "Authorization: Bearer $API_TOKEN" \ | jq '.host_networks[].host_ids'Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
[ "1062663e-7989-8b2d-7fbb-e6f4d5bb28e5" ]
[ "1062663e-7989-8b2d-7fbb-e6f4d5bb28e5" ]Copy to Clipboard Copied! Toggle word wrap Toggle overflow Add the
host_rolesetting:Copy to Clipboard Copied! Toggle word wrap Toggle overflow Where:
-
<host_id>is the ID of the host. -
host_roleis"master", `"arbiter", or"worker". For details, see About assigning roles to hosts.
-
5.11. Modifying hosts Copy linkLink copied to clipboard!
After adding hosts, modify the hosts as needed. The most common modifications are to the host_name and the host_role parameters.
You can modify a host by using the /v2/infra-envs/{infra_env_id}/hosts/{host_id} endpoint. See the host-update-params model in the API viewer for details on the fields you can set when modifying a host.
A host might be one of the following roles:
-
master- Assigns the control plane role to the host, allowing the host to manage and coordinate the cluster. -
arbiter- Assigns the arbiter role to the host, providing a cost-effective solution for components that require a quorum. -
worker- Assigns the compute role to the host, enabling the host to run application workloads. -
auto-assign- Automatically determines whether the host is amaster,worker, or `arbiter' node.
Use the following procedure to set the host’s role. If the host_role setting is omitted, the host defaults to auto-assign.
Prerequisites
- You have added hosts to the cluster.
Procedure
Refresh the API token:
source refresh-token
$ source refresh-tokenCopy to Clipboard Copied! Toggle word wrap Toggle overflow Get the host IDs:
curl -s -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID" \ --header "Content-Type: application/json" \ -H "Authorization: Bearer $API_TOKEN" \ | jq '.host_networks[].host_ids'
$ curl -s -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID" \ --header "Content-Type: application/json" \ -H "Authorization: Bearer $API_TOKEN" \ | jq '.host_networks[].host_ids'Copy to Clipboard Copied! Toggle word wrap Toggle overflow Modify the host settings by using the example below:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Replace
<host_id>with the ID of the host.
5.11.1. Modifying storage disk configuration Copy linkLink copied to clipboard!
Each host retrieved during host discovery can have multiple storage disks. You can optionally change the default configurations for each disk.
- Starting from OpenShift Container Platform 4.14, you can configure nodes with Intel® Virtual RAID on CPU (VROC) to manage NVMe RAIDs. For details, see Configuring an Intel® Virtual RAID on CPU (VROC) data volume.
- Starting from OpenShift Container Platform 4.15, you can install a cluster on a single or multipath iSCSI boot device using the Assisted Installer.
Prerequisites
- Configure the cluster and discover the hosts. For details, see Additional resources.
5.11.1.1. Viewing the storage disks Copy linkLink copied to clipboard!
You can view the hosts in your cluster, and the disks on each host. You can then perform actions on a specific disk.
Procedure
Refresh the API token:
source refresh-token
$ source refresh-tokenCopy to Clipboard Copied! Toggle word wrap Toggle overflow Get the host IDs for the cluster:
curl -s "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID" \ -H "Authorization: Bearer $API_TOKEN" \ | jq '.host_networks[].host_ids'
$ curl -s "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID" \ -H "Authorization: Bearer $API_TOKEN" \ | jq '.host_networks[].host_ids'Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
"1022623e-7689-8b2d-7fbd-e6f4d5bb28e5"
"1022623e-7689-8b2d-7fbd-e6f4d5bb28e5"Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteThis is the ID of a single host. Multiple host IDs are separated by commas.
Get the disks for a specific host:
curl https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/hosts/<host_id> \ -H "Authorization: Bearer ${API_TOKEN}" \ | jq '.inventory | fromjson | .disks'$ curl https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/hosts/<host_id> \1 -H "Authorization: Bearer ${API_TOKEN}" \ | jq '.inventory | fromjson | .disks'Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Replace
<host_id>with the ID of the relevant host.
Example output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteThis is the output for one disk. It has the
disk_idandinstallation_eligibilityproperties for the disk.
5.11.1.2. Changing the installation disk Copy linkLink copied to clipboard!
The Assisted Installer randomly assigns an installation disk by default. If there are multiple storage disks for a host, you can select a different disk to be the installation disk. This automatically unassigns the previous disk.
You can select any disk whose installation_eligibility property is eligible: true to be the installation disk.
Red Hat Enterprise Linux CoreOS (RHCOS) supports multipathing over Fibre Channel on the installation disk, allowing stronger resilience to hardware failure to achieve higher host availability. Multipathing is enabled by default in the agent ISO image, with an /etc/multipath.conf configuration. For details, see Modifying the DM Multipath configuration file.
Procedure
- Get the host and storage disk IDs. For details, see Viewing the storage disks.
Optional: Identify the current installation disk:
curl https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/hosts/<host_id> \ -H "Authorization: Bearer ${API_TOKEN}" \ | jq '.installation_disk_id'$ curl https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/hosts/<host_id> \1 -H "Authorization: Bearer ${API_TOKEN}" \ | jq '.installation_disk_id'Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Replace
<host_id>with the ID of the relevant host.
Assign a new installation disk:
NoteMultipath devices are automatically discovered and listed in the host’s inventory. To assign a multipath Fibre Channel disk as the installation disk, choose a disk with
"drive_type"set to"Multipath", rather than to"FC"which indicates a single path.Copy to Clipboard Copied! Toggle word wrap Toggle overflow
5.11.1.3. Disabling disk formatting Copy linkLink copied to clipboard!
The Assisted Installer marks all bootable disks for formatting during the installation process by default, regardless of whether or not they have been defined as the installation disk. Formatting causes data loss.
You can choose to disable the formatting of a specific disk. Disable formatting with caution, as bootable disks can interfere with the installation process, specifically the boot order.
You cannot disable formatting for the installation disk.
Procedure
- Get the host and storage disk IDs. For details, see Viewing the storage disks.
Run the following command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Note
5.12. Adding custom manifests Copy linkLink copied to clipboard!
A custom manifest is a JSON or YAML file that contains advanced configurations not currently supported in the Assisted Installer user interface. You can create a custom manifest or use one provided by a third party. To create a custom manifest with the API, use the /v2/clusters/$CLUSTER_ID/manifests endpoint.
You can upload a base64-encoded custom manifest to either the openshift folder or the manifests folder with the Assisted Installer API. There is no limit to the number of custom manifests permitted.
You can only upload one base64-encoded JSON manifest at a time. However, each uploaded base64-encoded YAML file can contain multiple custom manifests. Uploading a multi-document YAML manifest is faster than adding the YAML files individually.
For a file containing a single custom manifest, accepted file extensions include .yaml, .yml, or .json.
Single custom manifest example
For a file containing multiple custom manifests, accepted file types include .yaml or .yml.
Multiple custom manifest example
- When you install OpenShift Container Platform on the Oracle Cloud Infrastructure (OCI) external platform, you must add the custom manifests provided by Oracle. For additional external partner integrations such as vSphere or Nutanix, this step is optional.
- For more information about custom manifests, see Additional Resources.
Prerequisites
-
You have generated a valid
API_TOKEN. Tokens expire every 15 minutes. -
You have registered a new cluster definition and exported the
cluster_idto the$CLUSTER_IDBASH variable.
Procedure
- Create a custom manifest file.
- Save the custom manifest file using the appropriate extension for the file format.
Refresh the API token:
source refresh-token
$ source refresh-tokenCopy to Clipboard Copied! Toggle word wrap Toggle overflow Add the custom manifest to the cluster by executing the following command:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
manifest.jsonwith the name of your manifest file. The second instance ofmanifest.jsonis the path to the file. Ensure the path is correct.Example output
{ "file_name": "manifest.json", "folder": "manifests" }{ "file_name": "manifest.json", "folder": "manifests" }Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteThe
base64 -w 0command base64-encodes the manifest as a string and omits carriage returns. Encoding with carriage returns will generate an exception.Verify that the Assisted Installer added the manifest:
curl -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID/manifests/files?folder=manifests&file_name=manifest.json" -H "Authorization: Bearer $API_TOKEN"
$ curl -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID/manifests/files?folder=manifests&file_name=manifest.json" -H "Authorization: Bearer $API_TOKEN"Copy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
manifest.jsonwith the name of your manifest file.
5.13. Preinstallation validations Copy linkLink copied to clipboard!
The Assisted Installer ensures the cluster meets the prerequisites before installation, because it eliminates complex postinstallation troubleshooting, thereby saving significant amounts of time and effort. Before installing the cluster, ensure the cluster and each host pass preinstallation validation.
5.14. Installing the cluster Copy linkLink copied to clipboard!
Once the cluster hosts past validation, you can install the cluster.
Prerequisites
- You have created a cluster and infrastructure environment.
- You have added hosts to the infrastructure environment.
- The hosts have passed validation.
Procedure
Refresh the API token:
source refresh-token
$ source refresh-tokenCopy to Clipboard Copied! Toggle word wrap Toggle overflow Install the cluster:
curl -H "Authorization: Bearer $API_TOKEN" \ -X POST \ https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID/actions/install | jq
$ curl -H "Authorization: Bearer $API_TOKEN" \ -X POST \ https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID/actions/install | jqCopy to Clipboard Copied! Toggle word wrap Toggle overflow - Complete any postinstallation platform integration steps.
Chapter 6. Enabling disk encryption Copy linkLink copied to clipboard!
You can enable encryption of installation disks using either the TPM v2 or Tang encryption modes.
In some situations, when you enable TPM disk encryption in the firmware for a bare-metal host and then boot it from an ISO that you generate with the Assisted Installer, the cluster deployment can get stuck. This can happen if there are left-over TPM encryption keys from a previous installation on the host. For more information, see BZ#2011634. If you experience this problem, contact Red Hat support.
6.1. Enabling TPM v2 encryption Copy linkLink copied to clipboard!
You can enable TPM v2 encryption from the Assisted Installer web console or API.
Prerequisites
-
Check to see if TPM v2 encryption is enabled in the BIOS on each host. Most Dell systems require this. Check the manual for your computer. The Assisted Installer will also validate that TPM is enabled in the firmware. See the
disk-encruptionmodel in the Assisted Installer API for additional details.
Verify that a TPM v2 encryption chip is installed on each node and enabled in the firmware.
Procedure
- Optional: Using the web console, in the Cluster details step, enable the encryption of installation disks for any of the following nodes: control plane nodes, workers, or arbiter.
Optional: Using the API, include the following settings in the "Modifying hosts" procedure to enable TPM v2 encryption:
Set the
disk_encryption.enable_onsetting to one of the following:-
"none" -
"all" -
"masters" -
"arbiters" -
"workers" -
"masters,arbiters" -
"masters,workers" -
"arbiters,workers" -
"masters,arbiters,workers"
-
Set the
disk_encryption.modesetting totpmv2.Example
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
6.2. Enabling Tang encryption Copy linkLink copied to clipboard!
You can enable Tang encryption from the Assisted Installer web console or API.
Prerequisites
- You have access to a Red Hat Enterprise Linux (RHEL) 8 machine that can be used to generate a thumbprint of the Tang exchange key.
Procedure
- Set up a Tang server or access an existing one. See Network-bound disk encryption for instructions. You can set multiple Tang servers, but the Assisted Installer must be able to connect to all of them during installation.
On the Tang server, retrieve the thumbprint for the Tang server using
tang-show-keys:tang-show-keys <port>
$ tang-show-keys <port>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Optional: Replace
<port>with the port number. The default port number is80.Example thumbprint
1gYTN_LpU9ZMB35yn5IbADY5OQ0
1gYTN_LpU9ZMB35yn5IbADY5OQ0Copy to Clipboard Copied! Toggle word wrap Toggle overflow Optional: Retrieve the thumbprint for the Tang server using
jose.Ensure
joseis installed on the Tang server:sudo dnf install jose
$ sudo dnf install joseCopy to Clipboard Copied! Toggle word wrap Toggle overflow On the Tang server, retrieve the thumbprint using
jose:sudo jose jwk thp -i /var/db/tang/<public_key>.jwk
$ sudo jose jwk thp -i /var/db/tang/<public_key>.jwkCopy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
<public_key>with the public exchange key for the Tang server.Example thumbprint
1gYTN_LpU9ZMB35yn5IbADY5OQ0
1gYTN_LpU9ZMB35yn5IbADY5OQ0Copy to Clipboard Copied! Toggle word wrap Toggle overflow
- Optional: Using the web console, in the Cluster details step, enable the encryption of installation disks for any of the following nodes: control plane nodes, workers, or arbiter. You will be required to enter URLs and thumbprints for the Tang servers.
Optional: Using the API, include the following settings in the "Modifying hosts" procedure to enable Tang encryption:
Set the
disk_encryption.enable_onsetting to one of the following:-
"none" -
"all" -
"masters" -
"arbiters" -
"workers" -
"masters,arbiters" -
"masters,workers" -
"arbiters,workers" -
"masters,arbiters,workers"
-
-
Set the
disk_encryption.modesetting totang. Set
disk_encyrption.tang_serversto provide the URL and thumbprint details about one or more Tang servers. Within thetang_serversvalue, comment out the quotes within the object(s).Example
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Chapter 7. Configuring the discovery image Copy linkLink copied to clipboard!
The Assisted Installer uses an initial image to run an agent that performs hardware and network validations before attempting to install OpenShift Container Platform. You can use Ignition to customize the discovery image.
Modifications to the discovery image will not persist in the system.
7.1. Creating an Ignition configuration file Copy linkLink copied to clipboard!
Ignition is a low-level system configuration utility, which is part of the temporary initial root filesystem, the initramfs. When Ignition runs on the first boot, it finds configuration data in the Ignition configuration file and applies it to the host before switch_root is called to pivot to the host’s root filesystem.
Ignition uses a JSON configuration specification file to represent the set of changes that occur on the first boot.
Ignition versions newer than 3.2 are not supported, and will raise an error.
Procedure
Create an Ignition file and specify the configuration specification version:
vim ~/ignition.conf
$ vim ~/ignition.confCopy to Clipboard Copied! Toggle word wrap Toggle overflow { "ignition": { "version": "3.1.0" } }{ "ignition": { "version": "3.1.0" } }Copy to Clipboard Copied! Toggle word wrap Toggle overflow Add configuration data to the Ignition file. For example, add a password to the
coreuser.Generate a password hash:
openssl passwd -6
$ openssl passwd -6Copy to Clipboard Copied! Toggle word wrap Toggle overflow Add the generated password hash to the
coreuser:Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Save the Ignition file and export it to the
IGNITION_FILEvariable:export IGNITION_FILE=~/ignition.conf
$ export IGNITION_FILE=~/ignition.confCopy to Clipboard Copied! Toggle word wrap Toggle overflow
7.2. Modifying the discovery image with Ignition Copy linkLink copied to clipboard!
Once you create an Ignition configuration file, you can modify the discovery image by patching the infrastructure environment using the Assisted Installer API.
Prerequisites
- If you used the web console to create the cluster, you have set up the API authentication.
-
You have an infrastructure environment and you have exported the infrastructure environment
idto theINFRA_ENV_IDvariable. -
You have a valid Ignition file and have exported the file name as
$IGNITION_FILE.
Procedure
Create an
ignition_config_overrideJSON object and redirect it to a file:jq -n \ --arg IGNITION "$(jq -c . $IGNITION_FILE)" \ '{ignition_config_override: $IGNITION}' \ > discovery_ignition.json$ jq -n \ --arg IGNITION "$(jq -c . $IGNITION_FILE)" \ '{ignition_config_override: $IGNITION}' \ > discovery_ignition.jsonCopy to Clipboard Copied! Toggle word wrap Toggle overflow Refresh the API token:
source refresh-token
$ source refresh-tokenCopy to Clipboard Copied! Toggle word wrap Toggle overflow Patch the infrastructure environment:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow The
ignition_config_overrideobject references the Ignition file.- Download the updated discovery image.
Chapter 8. Booting hosts with the discovery image Copy linkLink copied to clipboard!
The Assisted Installer uses an initial image to run an agent that performs hardware and network validations before attempting to install OpenShift Container Platform. You can boot hosts with the discovery image using three methods:
- USB drive
- Redfish virtual media
- iPXE
8.1. Creating an ISO image on a USB drive Copy linkLink copied to clipboard!
You can install the Assisted Installer agent using a USB drive that contains the discovery ISO image. Starting the host with the USB drive prepares the host for the software installation.
Procedure
- On the administration host, insert a USB drive into a USB port.
Copy the ISO image to the USB drive, for example:
dd if=<path_to_iso> of=<path_to_usb> status=progress
# dd if=<path_to_iso> of=<path_to_usb> status=progressCopy to Clipboard Copied! Toggle word wrap Toggle overflow where:
- <path_to_iso>
-
is the relative path to the downloaded discovery ISO file, for example,
discovery.iso. - <path_to_usb>
is the location of the connected USB drive, for example,
/dev/sdb.After the ISO is copied to the USB drive, you can use the USB drive to install the Assisted Installer agent on the cluster host.
8.2. Booting with a USB drive Copy linkLink copied to clipboard!
To register nodes with the Assisted Installer using a bootable USB drive, use the following procedure.
Procedure
- Insert the RHCOS discovery ISO USB drive into the target host.
- Configure the boot drive order in the server firmware settings to boot from the attached discovery ISO, and then reboot the server.
Wait for the host to boot up.
- For web console installations, on the administration host, return to the browser. Wait for the host to appear in the list of discovered hosts.
For API installations, refresh the token, check the enabled host count, and gather the host IDs:
source refresh-token
$ source refresh-tokenCopy to Clipboard Copied! Toggle word wrap Toggle overflow curl -s -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID" \ --header "Content-Type: application/json" \ -H "Authorization: Bearer $API_TOKEN" \ | jq '.enabled_host_count'
$ curl -s -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID" \ --header "Content-Type: application/json" \ -H "Authorization: Bearer $API_TOKEN" \ | jq '.enabled_host_count'Copy to Clipboard Copied! Toggle word wrap Toggle overflow curl -s -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID" \ --header "Content-Type: application/json" \ -H "Authorization: Bearer $API_TOKEN" \ | jq '.host_networks[].host_ids'
$ curl -s -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID" \ --header "Content-Type: application/json" \ -H "Authorization: Bearer $API_TOKEN" \ | jq '.host_networks[].host_ids'Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
[ "1062663e-7989-8b2d-7fbb-e6f4d5bb28e5" ]
[ "1062663e-7989-8b2d-7fbb-e6f4d5bb28e5" ]Copy to Clipboard Copied! Toggle word wrap Toggle overflow
8.3. Booting from an HTTP-hosted ISO image using the Redfish API Copy linkLink copied to clipboard!
You can provision hosts in your network using ISOs that you install using the Redfish Baseboard Management Controller (BMC) API.
Prerequisites
- Download the installation Red Hat Enterprise Linux CoreOS (RHCOS) ISO.
Procedure
- Copy the ISO file to an HTTP server accessible in your network.
Boot the host from the hosted ISO file, for example:
Call the redfish API to set the hosted ISO as the
VirtualMediaboot media by running the following command:curl -k -u <bmc_username>:<bmc_password> \ -d '{"Image":"<hosted_iso_file>", "Inserted": true}' \ -H "Content-Type: application/json" \ -X POST <host_bmc_address>/redfish/v1/Managers/iDRAC.Embedded.1/VirtualMedia/CD/Actions/VirtualMedia.InsertMedia$ curl -k -u <bmc_username>:<bmc_password> \ -d '{"Image":"<hosted_iso_file>", "Inserted": true}' \ -H "Content-Type: application/json" \ -X POST <host_bmc_address>/redfish/v1/Managers/iDRAC.Embedded.1/VirtualMedia/CD/Actions/VirtualMedia.InsertMediaCopy to Clipboard Copied! Toggle word wrap Toggle overflow Where:
- <bmc_username>:<bmc_password>
- Is the username and password for the target host BMC.
- <hosted_iso_file>
-
Is the URL for the hosted installation ISO, for example:
https://example.com/rhcos-live-minimal.iso. The ISO must be accessible from the target host machine. - <host_bmc_address>
- Is the BMC IP address of the target host machine.
Set the host to boot from the
VirtualMediadevice by running the following command:curl -k -u <bmc_username>:<bmc_password> \ -X PATCH -H 'Content-Type: application/json' \ -d '{"Boot": {"BootSourceOverrideTarget": "Cd", "BootSourceOverrideMode": "UEFI", "BootSourceOverrideEnabled": "Once"}}' \ <host_bmc_address>/redfish/v1/Systems/System.Embedded.1$ curl -k -u <bmc_username>:<bmc_password> \ -X PATCH -H 'Content-Type: application/json' \ -d '{"Boot": {"BootSourceOverrideTarget": "Cd", "BootSourceOverrideMode": "UEFI", "BootSourceOverrideEnabled": "Once"}}' \ <host_bmc_address>/redfish/v1/Systems/System.Embedded.1Copy to Clipboard Copied! Toggle word wrap Toggle overflow Reboot the host:
curl -k -u <bmc_username>:<bmc_password> \ -d '{"ResetType": "ForceRestart"}' \ -H 'Content-type: application/json' \ -X POST <host_bmc_address>/redfish/v1/Systems/System.Embedded.1/Actions/ComputerSystem.Reset$ curl -k -u <bmc_username>:<bmc_password> \ -d '{"ResetType": "ForceRestart"}' \ -H 'Content-type: application/json' \ -X POST <host_bmc_address>/redfish/v1/Systems/System.Embedded.1/Actions/ComputerSystem.ResetCopy to Clipboard Copied! Toggle word wrap Toggle overflow Optional: If the host is powered off, you can boot it using the
{"ResetType": "On"}switch. Run the following command:curl -k -u <bmc_username>:<bmc_password> \ -d '{"ResetType": "On"}' -H 'Content-type: application/json' \ -X POST <host_bmc_address>/redfish/v1/Systems/System.Embedded.1/Actions/ComputerSystem.Reset$ curl -k -u <bmc_username>:<bmc_password> \ -d '{"ResetType": "On"}' -H 'Content-type: application/json' \ -X POST <host_bmc_address>/redfish/v1/Systems/System.Embedded.1/Actions/ComputerSystem.ResetCopy to Clipboard Copied! Toggle word wrap Toggle overflow
8.4. Booting hosts using iPXE Copy linkLink copied to clipboard!
The Assisted Installer provides an iPXE script including all of the artifacts needed to boot the discovery image for an infrastructure environment. Due to the limitations of the current HTTPS implementation of iPXE, the recommendation is to download and expose the needed artifacts in an HTTP server. Currently, even if iPXE supports HTTPS protocol, the supported algorithms are old and not recommended.
The full list of supported ciphers is in https://ipxe.org/crypto.
Prerequisites
- You have created an infrastructure environment by using the API or you have created a cluster by using the web console.
-
You have your infrastructure environment ID exported in your shell as
$INFRA_ENV_ID. -
You have credentials to use when accessing the API and have exported a token as
$API_TOKENin your shell.
If you configure iPXE by using the web console, the $INFRA_ENV_ID and $API_TOKEN variables are preset.
- You have an HTTP server to host the images.
IBM Power® only supports PXE, which has the following requirements:
-
GRUB2 installed at
/var/lib/tftpboot - DHCP and TFTP for PXE
Procedure
Download the iPXE script directly from the web console, or get the iPXE script from the Assisted Installer by running the following command:
curl \ --silent \ --header "Authorization: Bearer $API_TOKEN" \ https://api.openshift.com/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID/downloads/files?file_name=ipxe-script > ipxe-script
$ curl \ --silent \ --header "Authorization: Bearer $API_TOKEN" \ https://api.openshift.com/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID/downloads/files?file_name=ipxe-script > ipxe-scriptCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example
#!ipxe initrd --name initrd http://api.openshift.com/api/assisted-images/images/<infra_env_id>/pxe-initrd?arch=x86_64&image_token=<token_string>&version=4.10 kernel http://api.openshift.com/api/assisted-images/boot-artifacts/kernel?arch=x86_64&version=4.10 initrd=initrd coreos.live.rootfs_url=http://api.openshift.com/api/assisted-images/boot-artifacts/rootfs?arch=x86_64&version=4.10 random.trust_cpu=on rd.luks.options=discard ignition.firstboot ignition.platform.id=metal console=tty1 console=ttyS1,115200n8 coreos.inst.persistent-kargs="console=tty1 console=ttyS1,115200n8" boot
#!ipxe initrd --name initrd http://api.openshift.com/api/assisted-images/images/<infra_env_id>/pxe-initrd?arch=x86_64&image_token=<token_string>&version=4.10 kernel http://api.openshift.com/api/assisted-images/boot-artifacts/kernel?arch=x86_64&version=4.10 initrd=initrd coreos.live.rootfs_url=http://api.openshift.com/api/assisted-images/boot-artifacts/rootfs?arch=x86_64&version=4.10 random.trust_cpu=on rd.luks.options=discard ignition.firstboot ignition.platform.id=metal console=tty1 console=ttyS1,115200n8 coreos.inst.persistent-kargs="console=tty1 console=ttyS1,115200n8" bootCopy to Clipboard Copied! Toggle word wrap Toggle overflow Download the required artifacts by extracting URLs from the
ipxe-script:Download the initial RAM disk by running the following command:
awk '/^initrd /{print $NF}' ipxe-script \ | xargs curl -o initrd.img -L$ awk '/^initrd /{print $NF}' ipxe-script \ | xargs curl -o initrd.img -LCopy to Clipboard Copied! Toggle word wrap Toggle overflow Download the Linux kernel by running the following command:
awk '/^kernel /{print $2}' ipxe-script | xargs curl -o kernel -L$ awk '/^kernel /{print $2}' ipxe-script | xargs curl -o kernel -LCopy to Clipboard Copied! Toggle word wrap Toggle overflow Download the root filesystem by running the following command:
grep ^kernel ipxe_script | xargs -n1 | grep ^coreos.live.rootfs_url | cut -d = -f 2,3,4 | xargs curl -o rootfs.img -L
$ grep ^kernel ipxe_script | xargs -n1 | grep ^coreos.live.rootfs_url | cut -d = -f 2,3,4 | xargs curl -o rootfs.img -LCopy to Clipboard Copied! Toggle word wrap Toggle overflow
Change the URLs to the different artifacts in the
ipxe-scriptto match your local HTTP server. For example:#!ipxe set webserver http://192.168.0.1 initrd --name initrd $webserver/initrd.img kernel $webserver/kernel initrd=initrd coreos.live.rootfs_url=$webserver/rootfs.img random.trust_cpu=on rd.luks.options=discard ignition.firstboot ignition.platform.id=metal console=tty1 console=ttyS1,115200n8 coreos.inst.persistent-kargs="console=tty1 console=ttyS1,115200n8" boot
#!ipxe set webserver http://192.168.0.1 initrd --name initrd $webserver/initrd.img kernel $webserver/kernel initrd=initrd coreos.live.rootfs_url=$webserver/rootfs.img random.trust_cpu=on rd.luks.options=discard ignition.firstboot ignition.platform.id=metal console=tty1 console=ttyS1,115200n8 coreos.inst.persistent-kargs="console=tty1 console=ttyS1,115200n8" bootCopy to Clipboard Copied! Toggle word wrap Toggle overflow Optional: When installing with RHEL KVM on IBM Z® you must boot the host by specifying additional kernel arguments:
random.trust_cpu=on rd.luks.options=discard ignition.firstboot ignition.platform.id=metal console=tty1 console=ttyS1,115200n8 coreos.inst.persistent-kargs="console=tty1 console=ttyS1,115200n8
random.trust_cpu=on rd.luks.options=discard ignition.firstboot ignition.platform.id=metal console=tty1 console=ttyS1,115200n8 coreos.inst.persistent-kargs="console=tty1 console=ttyS1,115200n8Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteWhen you install with iPXE on RHEL KVM, the VMs on the VM host might not start on the first boot. You must start them manually.
Optional: When installing on IBM Power® you must download the
initramfs,kernel, androotimages as follows:-
Copy the
initrd.imgandkernel.imgimages to the/var/lib/tftpboot/rhcosPXE directory. -
Copy the
rootfs.imgto the/var/www/html/installHTTPD directory. Add the following entry to the
/var/lib/tftpboot/boot/grub2/grub.cfgdirectory:Copy to Clipboard Copied! Toggle word wrap Toggle overflow
-
Copy the
Chapter 9. Preinstallation validations Copy linkLink copied to clipboard!
9.1. Definition of preinstallation validations Copy linkLink copied to clipboard!
The Assisted Installer aims to make cluster installation as simple, efficient, and error-free as possible. The Assisted Installer performs validation checks on the configuration and the gathered telemetry before starting an installation.
The Assisted Installer uses the information provided before installation, such as control plane topology, network configuration and hostnames. It will also use real time telemetry from the hosts you are attempting to install.
When a host boots the discovery ISO, an agent will start on the host. The agent will send information about the state of the host to the Assisted Installer.
The Assisted Installer uses all of this information to compute real time preinstallation validations. All validations are either blocking or non-blocking to the installation.
9.2. Blocking and non-blocking validations Copy linkLink copied to clipboard!
A blocking validation will prevent progress of the installation, meaning that you will need to resolve the issue and pass the blocking validation before you can proceed.
A non-blocking validation is a warning and will tell you of things that might cause you a problem.
9.3. Validation types Copy linkLink copied to clipboard!
The Assisted Installer performs two types of validation:
Host
Host validations ensure that the configuration of a given host is valid for installation.
Cluster
Cluster validations ensure that the configuration of the whole cluster is valid for installation.
9.4. Host validations Copy linkLink copied to clipboard!
9.4.1. Getting host validations by using the REST API Copy linkLink copied to clipboard!
If you use the web console, many of these validations will not show up by name. To get a list of validations consistent with the labels, use the following procedure.
Prerequisites
-
You have installed the
jqutility. - You have created an Infrastructure Environment by using the API or have created a cluster by using the web console.
- You have hosts booted with the discovery ISO
-
You have your Cluster ID exported in your shell as
CLUSTER_ID. -
You have credentials to use when accessing the API and have exported a token as
API_TOKENin your shell.
Procedures
Refresh the API token:
source refresh-token
$ source refresh-tokenCopy to Clipboard Copied! Toggle word wrap Toggle overflow Get all validations for all hosts:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Get non-passing validations for all hosts:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
9.4.2. Host validations in detail Copy linkLink copied to clipboard!
| Parameter | Validation type | Description |
|---|---|---|
|
| non-blocking | Checks that the host has recently communicated with the Assisted Installer. |
|
| non-blocking | Checks that the Assisted Installer received the inventory from the host. |
|
| non-blocking | Checks that the number of CPU cores meets the minimum requirements. |
|
| non-blocking | Checks that the amount of memory meets the minimum requirements. |
|
| non-blocking | Checks that at least one available disk meets the eligibility criteria. |
|
| blocking | Checks that the number of cores meets the minimum requirements for the host role. |
|
| blocking | Checks that the amount of memory meets the minimum requirements for the host role. |
|
| blocking | For Day 2 hosts, checks that the host can download ignition configuration from the Day 1 cluster. |
|
| blocking | The majority group is the largest full-mesh connectivity group on the cluster, where all members can communicate with all other members. This validation checks that hosts in a multi-node, Day 1 cluster are in the majority group. |
|
| blocking | Checks that the platform is valid for the network settings. |
|
| non-blocking | Checks if an NTP server has been successfully used to synchronize time on the host. |
|
| non-blocking | Checks if container images have been successfully pulled from the image registry. |
|
| blocking | Checks that disk speed metrics from an earlier installation meet requirements, if they exist. |
|
| blocking | Checks that the average network latency between hosts in the cluster meets the requirements. |
|
| blocking | Checks that the network packet loss between hosts in the cluster meets the requirements. |
|
| blocking | Checks that the host has a default route configured. |
|
| blocking | For a multi node cluster with user managed networking. Checks that the host is able to resolve the API domain name for the cluster. |
|
| blocking | For a multi node cluster with user managed networking. Checks that the host is able to resolve the internal API domain name for the cluster. |
|
| blocking | For a multi node cluster with user managed networking. Checks that the host is able to resolve the internal apps domain name for the cluster. |
|
| non-blocking | Checks that the host is compatible with the cluster platform |
|
| blocking | Checks that the wildcard DNS *.<cluster_name>.<base_domain> is not configured, because this causes known problems for OpenShift |
|
| non-blocking | Checks that the type of host and disk encryption configured meet the requirements. |
|
| blocking | Checks that this host does not have any overlapping subnets. |
|
| blocking | Checks that the hostname is unique in the cluster. |
|
| blocking | Checks the validity of the hostname, meaning that it matches the general form of hostnames and is not forbidden.
|
|
| blocking | Checks that the host IP is in the address range of the machine CIDR. |
|
| blocking | Validates that the host meets the requirements of the Local Storage Operator. |
|
| blocking | Validates that the host meets the requirements of the OpenShift Data Foundation Operator.
|
|
| blocking | Validates that the host meets the requirements of Container Native Virtualization.
|
|
| blocking | Validates that the host meets the requirements of the Logical Volume Manager Storage Operator.
|
|
| non-blocking |
Verifies that each valid disk sets |
|
| blocking | Checks that the discovery agent version is compatible with the agent docker image version. |
|
| blocking | Checks that installation disk is not skipping disk formatting. |
|
| blocking | Checks that all disks marked to skip formatting are in the inventory. A disk ID can change on reboot, and this validation prevents issues caused by that. |
|
| blocking | Checks the connection of the installation media to the host. |
|
| non-blocking | Checks that the machine network definition exists for the cluster. |
|
| blocking | Checks that the platform is compatible with the network settings. Some platforms are only permitted when installing Single Node Openshift or when using User Managed Networking. |
|
| non-blocking | Checks the maximum transmission unit (MTU) of hosts and networking devices in the cluster environment to identify compatibility issues. For more information, see Additional resources. |
Additional resources
9.5. Cluster validations Copy linkLink copied to clipboard!
9.5.1. Getting cluster validations by using the REST API Copy linkLink copied to clipboard!
If you use the web console, many of these validations will not show up by name. To obtain a list of validations consistent with the labels, use the following procedure.
Prerequisites
-
You have installed the
jqutility. - You have created an Infrastructure Environment by using the API or have created a cluster by using the web console.
-
You have your Cluster ID exported in your shell as
CLUSTER_ID. -
You have credentials to use when accessing the API and have exported a token as
API_TOKENin your shell.
Procedures
Refresh the API token:
source refresh-token
$ source refresh-tokenCopy to Clipboard Copied! Toggle word wrap Toggle overflow Get all cluster validations:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Get non-passing cluster validations:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
9.5.2. Cluster validations in detail Copy linkLink copied to clipboard!
| Parameter | Validation type | Description |
|---|---|---|
|
| non-blocking | Checks that the machine network definition exists for the cluster. |
|
| non-blocking | Checks that the cluster network definition exists for the cluster. |
|
| non-blocking | Checks that the service network definition exists for the cluster. |
|
| blocking | Checks that the defined networks do not overlap. |
|
| blocking | Checks that the defined networks share the same address families (valid address families are IPv4, IPv6) |
|
| blocking | Checks the cluster network prefix to ensure that it is valid and allows enough address space for all hosts. |
|
| blocking |
For a non user managed networking cluster. Checks that |
|
| non-blocking |
For a non user managed networking cluster. Checks that |
|
| blocking |
For a non user managed networking cluster. Checks if the |
|
| blocking |
For a non user managed networking cluster. Checks that |
|
| non-blocking |
For a non user managed networking cluster. Checks if the |
|
| blocking | Checks that all hosts in the cluster are in the "ready to install" status. |
|
| blocking |
|
|
| non-blocking | Checks that the base DNS domain exists for the cluster. |
|
| non-blocking | Checks that the pull secret exists. Does not check that the pull secret is valid or authorized. |
|
| blocking | Checks that each of the host clocks are no more than 4 minutes out of sync with each other. |
|
| blocking | Validates that the cluster meets the requirements of the Local Storage Operator. |
|
| blocking | Validates that the cluster meets the requirements of the OpenShift Data Foundation Operator.
|
|
| blocking | Validates that the cluster meets the requirements of Container Native Virtualization.
|
|
| blocking | Validates that the cluster meets the requirements of the Logical Volume Manager Storage Operator.
|
|
| blocking | Checks the validity of the network type if it exists.
|
Chapter 10. Network configuration Copy linkLink copied to clipboard!
The following sections describe the basics of network configuration with the Assisted Installer.
10.1. Cluster networking Copy linkLink copied to clipboard!
There are various network types and addresses used by OpenShift and listed in the following table.
IPv6 is not currently supported in the following configurations:
- Single stack
- Primary within dual stack
| Type | DNS | Description |
|---|---|---|
|
| The IP address pools from which pod IP addresses are allocated. | |
|
| The IP address pool for services. | |
|
| The IP address blocks for machines forming the cluster. | |
|
|
| The VIP to use for API communication. You must provide this setting or preconfigure the address in the DNS so that the default name resolves correctly. If you are deploying with dual-stack networking, this must be the IPv4 address. |
|
|
|
The VIPs to use for API communication. You must provide this setting or preconfigure the address in the DNS so that the default name resolves correctly. If using dual stack networking, the first address must be the IPv4 address and the second address must be the IPv6 address. You must also set the |
|
|
| The VIP to use for ingress traffic. If you are deploying with dual-stack networking, this must be the IPv4 address. |
|
|
|
The VIPs to use for ingress traffic. If you are deploying with dual-stack networking, the first address must be the IPv4 address and the second address must be the IPv6 address. You must also set the |
OpenShift Container Platform 4.12 introduces the new apiVIPs and ingressVIPs settings to accept many IP addresses for dual-stack networking. When using dual-stack networking, the first IP address must be the IPv4 address and the second IP address must be the IPv6 address. The new settings will replace apiVIP and IngressVIP, but you must set both the new and old settings when modifying the configuration by using the API.
Currently, the Assisted Service can deploy OpenShift Container Platform clusters by using one of the following configurations:
- IPv4
- Dual-stack (IPv4 + IPv6 with IPv4 as primary)
OVN is the default Container Network Interface (CNI) in OpenShift Container Platform 4.12 and later releases. SDN is supported up to OpenShift Container Platform 4.14, but not for OpenShift Container Platform 4.15 and later releases.
10.1.1. Limitations Copy linkLink copied to clipboard!
Cluster networking has the following limitations.
- SDN
- The SDN controller is not supported with single-node OpenShift.
- The SDN controller does not support dual-stack networking.
- The SDN controller is not supported for OpenShift Container Platform 4.15 and later releases. For more information, see Deprecation of the OpenShift SDN network plugin in the OpenShift Container Platform release notes.
- OVN-Kubernetes
- For more information, see About the OVN-Kubernetes network plugin.
10.1.2. Cluster network Copy linkLink copied to clipboard!
The cluster network is a network from which every pod deployed in the cluster gets its IP address. Given that the workload might live across many nodes forming the cluster, it is important for the network provider to be able to easily find an individual node based on the pod’s IP address. To do this, clusterNetwork.cidr is further split into subnets of the size defined in clusterNetwork.hostPrefix.
The host prefix specifies a length of the subnet assigned to each individual node in the cluster. An example of how a cluster might assign addresses for the multi-node cluster:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
Creating a 3-node cluster by using this snippet might create the following network topology:
-
Pods scheduled in node #1 get IPs from
10.128.0.0/23 -
Pods scheduled in node #2 get IPs from
10.128.2.0/23 -
Pods scheduled in node #3 get IPs from
10.128.4.0/23
Explaining OVN-Kubernetes internals is out of scope for this document, but the pattern previously described provides a way to route Pod-to-Pod traffic between different nodes without keeping a big list of mapping between Pods and their corresponding nodes.
10.1.3. Machine network Copy linkLink copied to clipboard!
Machine networks are IP networks that connect all the cluster nodes within OpenShift Container Platform.
The Assisted Installer supports a single machine network for most cluster installations. In such cases, the Assisted Installer automatically determines the appropriate machine network based on the API and Ingress virtual IPs (VIPs) that you specify.
The Assisted Installer supports two machine networks in the following scenarios:
- For dual stack configurations, the Assisted Installer automatically allocates two machine networks, based on the IPv4 and IPv6 subnets and the API and Ingress VIPs that you specify.
- For iSCSI boot volumes, the hosts are automatically connected over two machine networks: one designated for the OpenShift Container Platform installation and the other for iSCSI traffic. During the installation process, ensure that you select the OpenShift Container Platform network. Using the iSCSI network will result in an error for the host.
The Assisted Installer supports multiple machine networks for the "cluster-managed networking with a user-managed load balancer" network management type. When installing this network management type, you must manually define the machine networks in the API cluster definitions, with the following conditions:
- Each node must have at least one network interface in at least one machine network.
- The load balancer IPs (VIPs) should be included in at least one of the machine networks.
Currently, you can install cluster-managed networking with a user-managed load balancer using the Assisted Installer API only.
10.1.4. Single-node OpenShift compared to multi-node cluster Copy linkLink copied to clipboard!
Depending on whether you are deploying single-node OpenShift or a multi-node cluster, different values are mandatory. The following table explains this in more detail.
| Parameter | Single-node OpenShift | Multi-node cluster with DHCP mode | Multi-node cluster without DHCP mode |
|---|---|---|---|
|
| Required | Required | Required |
|
| Required | Required | Required |
|
| Auto-assign possible (*) | Auto-assign possible (*) | Auto-assign possible (*) |
|
| Forbidden | Forbidden | Required |
|
| Forbidden | Forbidden | Required in 4.12 and later releases |
|
| Forbidden | Forbidden | Required |
|
| Forbidden | Forbidden | Required in 4.12 and later releases |
(*) Auto assignment of the machine network CIDR happens if there is only a single host network. Otherwise you need to specify it explicitly.
10.1.5. Air-gapped environments Copy linkLink copied to clipboard!
The workflow for deploying a cluster without Internet access has some prerequisites, which are out of scope of this document. You can consult the Zero Touch Provisioning the hard way Git repository for some insights.
10.2. VIP DHCP allocation Copy linkLink copied to clipboard!
The VIP DHCP allocation is a feature allowing users to skip the requirement of manually providing virtual IPs for API and Ingress by leveraging the ability of a service to automatically assign those IP addresses from the DHCP server.
If you enable the VIP DHCP allocation feature, the service will not use the api_vips and ingress_vips defined in the cluster configuration. Instead, it will request IP addresses from the DHCP server on the machine network and use the assigned VIPs accordingly.
Please note this is not an OpenShift Container Platform feature and it has been implemented in the Assisted Service to make the configuration easier. For a more detailed explanation of the syntax for the VIP addresses, see "Additional resources".
VIP DHCP allocation is currently limited to the OpenShift Container Platform SDN network type. SDN is not supported from OpenShift Container Platform version 4.15 and later. Therefore, support for VIP DHCP allocation is also ending from OpenShift Container Platform 4.15 and later.
10.2.1. Enabling VIP DHCP allocation Copy linkLink copied to clipboard!
You can enable automatic VIP allocation through DHCP.
Procedure
- Follow the instructions for registering a new cluster by using the API. For details, see Registering a new cluster.
Add the following payload settings to the cluster configuration:
-
Set
vip_dhcp_allocationtotrue. -
Set
network_typetoOpenShiftSDN. -
Include your network configurations for
cluster_networks,service_networks, andmachine_networks.
Example payload to enable autoallocation
Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
Set
Submit the payload to the Assisted Service API to apply the configuration, by running the following command:
curl -s -X POST "https://api.openshift.com/api/assisted-install/v2/clusters/<cluster-id>" \ -d @./payload.json \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $API_TOKEN" \ | jq '.id'
$ curl -s -X POST "https://api.openshift.com/api/assisted-install/v2/clusters/<cluster-id>" \ -d @./payload.json \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $API_TOKEN" \ | jq '.id'Copy to Clipboard Copied! Toggle word wrap Toggle overflow
10.2.2. Disabling VIP DHCP allocation Copy linkLink copied to clipboard!
If you want to manually control your VIP assignments, you can disable VIP DHCP allocation.
Procedure
- Follow the instructions for registering a new cluster by using the API. For details, see Registering a new cluster.
Add the following payload settings to the cluster configuration:
-
Set
vip_dhcp_allocationtofalse. -
Specify the IP addresses for
api_vipsandingress_vips. You can take these IPs from yourmachine_networksconfiguration. -
Set
network_typetoOVNKubernetes,OpenShiftSDN, or another supported SDN type, if applicable. -
Include your network configurations for
cluster_networksandservice_networks.
Example payload to disable autoallocation
Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
Set
Submit the payload to the Assisted Service API to apply the configuration, by running the following command:
curl -s -X POST "https://api.openshift.com/api/assisted-install/v2/clusters/<cluster-id>" \ -d @./payload.json \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $API_TOKEN" \ | jq '.id'
$ curl -s -X POST "https://api.openshift.com/api/assisted-install/v2/clusters/<cluster-id>" \ -d @./payload.json \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $API_TOKEN" \ | jq '.id'Copy to Clipboard Copied! Toggle word wrap Toggle overflow
10.3. Network management types Copy linkLink copied to clipboard!
The Assisted Installer supports the following network management types:
- Cluster-managed networking
Cluster-managed networking is the default option for deploying OpenShift Container Platform clusters. It minimizes user intervention by automatically provisioning and managing key network components.
The main characteristics of cluster-managed networking are the following:
- Integrates automated load balancing and virtual routing for managing the Virtual IP (VIP) addresses to ensure redundancy.
- Automatically supports an extensive internal DNS (CoreDNS) for service discovery.
- Hosts all control plane nodes within a single, contiguous subnet, simplifying routing and connectivity within the cluster.
- Supports the installation of platform-specific features such as the Bare Metal Operator for bare metal.
- Available for clusters with three or more control plane nodes; not available for single-node OpenShift.
You can configure cluster-managed networking both the web console or API. If you do not define a network management type, the Assisted Installer applies cluster-managed networking automatically for highly available clusters.
- User-managed networking
User-managed networking allows customers with custom or non-standard network topologies to deploy OpenShift Container Platform clusters. It provides control and flexibility, allowing you to integrate OpenShift Container Platform with existing and complex network infrastructures.
The main characteristics of user-managed networking are the following:
- Allows users to configure one or more external load balancers for handling API and Ingress IP addresses.
- Enables control plane nodes to span multiple subnets.
- Can be deployed on both single-node OpenShift and high-availability clusters.
You can configure user-managed networking in both the Assisted Installer web console or API.
- Cluster managed networking with a user-managed load balancer
Cluster-managed networking with a user-managed load balancer is a hybrid network management type designed for scenarios that require automated cluster networking with external control over load balancing.
This approach combines elements from both cluster-managed and user-managed networking. The main characteristics of this network management type are as follows:
- Allows users to configure one or more external load balancers for handling API and Ingress IP addresses.
- Automatically supports an extensive internal DNS (CoreDNS) for service discovery.
- Enables control plane nodes to span multiple subnets.
- Supports the installation of platform specific features such as the Bare Metal Operator for bare metal.
- Provides high fault tolerance and disaster recovery for the control plane nodes.
The Assisted Installer supports cluster-managed networking with a user-managed load balancer on a bare-metal or vSphere platform. Currently you can configure this network management type through the API only.
Cluster-managed networking with a user-managed load balancer is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.
10.4. Static network configuration Copy linkLink copied to clipboard!
You may use static network configurations when generating or updating the discovery ISO.
Prerequisites
- You are familiar with NMState.
10.4.1. NMState configuration Copy linkLink copied to clipboard!
The NMState file in YAML format specifies the desired network configuration for the host. It has the logical names of the interfaces that will be replaced with the actual name of the interface at discovery time.
Example of NMState configuration
10.4.2. MAC interface mapping Copy linkLink copied to clipboard!
MAC interface map is an attribute that maps logical interfaces defined in the NMState configuration with the actual interfaces present on the host.
The mapping should always use physical interfaces present on the host. For example, when the NMState configuration defines a bond or VLAN, the mapping should only contain an entry for parent interfaces.
Example of MAC interface mapping
10.4.3. Additional NMState configuration examples Copy linkLink copied to clipboard!
The following examples are only meant to show a partial configuration. They are not meant for use as-is, and you should always adjust to the environment where they will be used. If used incorrectly, they can leave your machines with no network connectivity.
Tagged VLAN example
Network bond example
10.5. Converting to dual-stack networking Copy linkLink copied to clipboard!
Dual-stack IPv4/IPv6 configuration allows deployment of a cluster with pods residing in both IPv4 and IPv6 subnets.
10.5.1. Prerequisites Copy linkLink copied to clipboard!
- You are familiar with OVN-K8s documentation
10.5.2. Example payload for single-node OpenShift Copy linkLink copied to clipboard!
10.5.3. Example payload for an OpenShift Container Platform cluster consisting of many nodes Copy linkLink copied to clipboard!
10.5.4. Limitations Copy linkLink copied to clipboard!
The api_vips IP address and ingress_vips IP address settings must be of the primary IP address family when using dual-stack networking, which must be IPv4 addresses. Currently, Red Hat does not support dual-stack VIPs or dual-stack networking with IPv6 as the primary IP address family. Red Hat supports dual-stack networking with IPv4 as the primary IP address family and IPv6 as the secondary IP address family. Therefore, you must place the IPv4 entries before the IPv6 entries when entering the IP address values.
Chapter 11. Expanding the cluster Copy linkLink copied to clipboard!
You can expand a cluster installed with the Assisted Installer by adding hosts using the user interface or the API.
11.1. Checking for multi-architecture support Copy linkLink copied to clipboard!
You must check that your cluster can support multiple architectures before you add a node with a different architecture.
Procedure
- Log in to the cluster using the CLI.
Check that your cluster uses the architecture payload by running the following command:
oc adm release info -o json | jq .metadata.metadata
$ oc adm release info -o json | jq .metadata.metadataCopy to Clipboard Copied! Toggle word wrap Toggle overflow
Verification
If you see the following output, your cluster supports multiple architectures:
{ "release.openshift.io/architecture": "multi" }{ "release.openshift.io/architecture": "multi" }Copy to Clipboard Copied! Toggle word wrap Toggle overflow
11.2. Installing multi-architecture compute clusters Copy linkLink copied to clipboard!
A cluster with an x86_64 or arm64 control plane can support worker nodes that have two different CPU architectures. Multi-architecture clusters combine the strengths of each architecture and support a variety of workloads.
For example, you can add arm64, IBM Power® (ppc64le), or IBM Z® (s390x) worker nodes to an existing OpenShift Container Platform cluster with an x86_64.
The main steps of the installation are as follows:
- Create and register a multi-architecture compute cluster.
-
Create an
x86_64orarm64infrastructure environment, download the ISO discovery image for the environment, and add the control plane. Anarm64infrastructure environment is available for Amazon Web Services (AWS) and Google Cloud (GC) only. -
Create an
arm64,ppc64le, ors390xinfrastructure environment, download the ISO discovery images forarm64,ppc64le, ors390x, and add the worker nodes.
Supported platforms
For the supported platforms for each OpenShift Container Platform version, see About clusters with multi-architecture compute machines. Use the appropriate platforms for the version you are installing.
Main steps
- Start the procedure for installing OpenShift Container Platform using the API. For details, see Installing with the Assisted Installer API in the Additional Resources section.
When you reach the "Registering a new cluster" step of the installation, register the cluster as a multi-architecture compute cluster:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Note- 1
- Use the
multi-option for the OpenShift Container Platform version number; for example,"4.19-multi". - 2
- Set the CPU architecture to
"multi". - 3
- Set the number of control plane nodes to "3", "4", or "5". The option of 4 or 5 control plane nodes is available from OpenShift Container Platform 4.18 and later. Single-node OpenShift is not supported for a multi-architecture compute cluster. The
control_plane_countfield replaceshigh_availability_mode, which is deprecated.
When you reach the "Registering a new infrastructure environment" step of the installation, set
cpu_architecturetox86_64:Copy to Clipboard Copied! Toggle word wrap Toggle overflow When you reach the "Adding hosts" step of the installation, set
host_roletomaster:NoteFor more information, see Assigning Roles to Hosts in Additional Resources.
Copy to Clipboard Copied! Toggle word wrap Toggle overflow -
Download the discovery image for the
x86_64architecture. -
Boot the
x86_64architecture hosts using the generated discovery image. - Start the installation and wait for the cluster to be fully installed.
Repeat the "Registering a new infrastructure environment" step of the installation. This time, set
cpu_architectureto one of the following:ppc64le(for IBM Power®),s390x(for IBM Z®), orarm64. For example:Copy to Clipboard Copied! Toggle word wrap Toggle overflow Repeat the "Adding hosts" step of the installation. This time, set
host_roletoworker:NoteFor more details, see Assigning Roles to Hosts in Additional Resources.
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - Download the discovery image for the arm64, ppc64le or s390x architecture.
- Boot the architecture hosts using the generated discovery image.
- Start the installation and wait for the cluster to be fully installed.
Verification
View the arm64, ppc64le, or s390x worker nodes in the cluster by running the following command:
oc get nodes -o wide
$ oc get nodes -o wideCopy to Clipboard Copied! Toggle word wrap Toggle overflow
11.3. Adding hosts with the web console Copy linkLink copied to clipboard!
You can add hosts to clusters that were created using the Assisted Installer.
- Adding hosts to Assisted Installer clusters is only supported for clusters running OpenShift Container Platform version 4.11 and later.
-
When adding a control plane node during Day 2 operations, ensure that the new node shares the same subnet as the Day 1 network. The subnet is specified in the
machineNetworkfield of theinstall-config.yamlfile. This requirement applies to cluster-managed networks such as bare metal or vSphere, and not to user-managed networks.
Procedure
- Log in to OpenShift Cluster Manager and click the cluster that you want to expand.
- Click Add hosts and download the discovery ISO for the new host, adding an SSH public key and configuring cluster-wide proxy settings as needed.
- Optional: Modify ignition files as needed.
- Boot the target host using the discovery ISO, and wait for the host to be discovered in the console.
-
Select the host role. It can be either a
workeror acontrol planehost. - Start the installation.
As the installation proceeds, the installation generates pending certificate signing requests (CSRs) for the host. When prompted, approve the pending CSRs to complete the installation.
When the host is successfully installed, it is listed as a host in the cluster web console.
New hosts will be encrypted using the same method as the original cluster.
11.4. Adding hosts with the API Copy linkLink copied to clipboard!
You can add hosts to clusters using the Assisted Installer REST API.
Prerequisites
-
Install the Red Hat OpenShift Cluster Manager CLI (
ocm). - Log in to Red Hat OpenShift Cluster Manager as a user with cluster creation privileges.
-
Install
jq. - Ensure that all the required DNS records exist for the cluster that you want to expand.
When adding a control plane node during Day 2 operations, ensure that the new node shares the same subnet as the Day 1 network. The subnet is specified in the machineNetwork field of the install-config.yaml file. This requirement applies to cluster-managed networks such as bare metal or vSphere, and not to user-managed networks.
Procedure
- Authenticate against the Assisted Installer REST API and generate an API token for your session. The generated token is valid for 15 minutes only.
Set the
$API_URLvariable by running the following command:export API_URL=<api_url>
$ export API_URL=<api_url>1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Replace
<api_url>with the Assisted Installer API URL, for example,https://api.openshift.com
Import the cluster by running the following commands:
Set the
$CLUSTER_IDvariable:Log in to the cluster and run the following command:
export CLUSTER_ID=$(oc get clusterversion -o jsonpath='{.items[].spec.clusterID}')$ export CLUSTER_ID=$(oc get clusterversion -o jsonpath='{.items[].spec.clusterID}')Copy to Clipboard Copied! Toggle word wrap Toggle overflow Display the
$CLUSTER_IDvariable output:echo ${CLUSTER_ID}$ echo ${CLUSTER_ID}Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Set the
$CLUSTER_REQUESTvariable that is used to import the cluster:Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Replace
<api_vip>with the hostname for the cluster’s API server. This can be the DNS domain for the API server or the IP address of the single node which the host can reach. For example,api.compute-1.example.com. - 2
- Replace
<cluster_id>with the$CLUSTER_IDoutput from the previous substep. - 3
- Replace
<openshift_cluster_name>with the plain text name for the cluster. The cluster name should match the cluster name that was set during the Day 1 cluster installation.
Import the cluster and set the
$CLUSTER_IDvariable. Run the following command:CLUSTER_ID=$(curl "$API_URL/api/assisted-install/v2/clusters/import" -H "Authorization: Bearer ${API_TOKEN}" -H 'accept: application/json' -H 'Content-Type: application/json' \ -d "$CLUSTER_REQUEST" | tee /dev/stderr | jq -r '.id')$ CLUSTER_ID=$(curl "$API_URL/api/assisted-install/v2/clusters/import" -H "Authorization: Bearer ${API_TOKEN}" -H 'accept: application/json' -H 'Content-Type: application/json' \ -d "$CLUSTER_REQUEST" | tee /dev/stderr | jq -r '.id')Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Generate the
InfraEnvresource for the cluster and set the$INFRA_ENV_IDvariable by running the following commands:- Download the pull secret file from Red Hat OpenShift Cluster Manager at console.redhat.com.
Set the
$INFRA_ENV_REQUESTvariable:Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Replace
<path_to_pull_secret_file>with the path to the local file containing the downloaded pull secret from Red Hat OpenShift Cluster Manager at console.redhat.com. - 2
- Replace
<path_to_ssh_pub_key>with the path to the public SSH key required to access the host. If you do not set this value, you cannot access the host while in discovery mode. - 3
- Replace
<infraenv_name>with the plain text name for theInfraEnvresource. - 4
- Replace
<iso_image_type>with the ISO image type, eitherfull-isoorminimal-iso.
Post the
$INFRA_ENV_REQUESTto the /v2/infra-envs API and set the$INFRA_ENV_IDvariable:INFRA_ENV_ID=$(curl "$API_URL/api/assisted-install/v2/infra-envs" -H "Authorization: Bearer ${API_TOKEN}" -H 'accept: application/json' -H 'Content-Type: application/json' -d "$INFRA_ENV_REQUEST" | tee /dev/stderr | jq -r '.id')$ INFRA_ENV_ID=$(curl "$API_URL/api/assisted-install/v2/infra-envs" -H "Authorization: Bearer ${API_TOKEN}" -H 'accept: application/json' -H 'Content-Type: application/json' -d "$INFRA_ENV_REQUEST" | tee /dev/stderr | jq -r '.id')Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Get the URL of the discovery ISO for the cluster host by running the following command:
curl -s "$API_URL/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID" -H "Authorization: Bearer ${API_TOKEN}" | jq -r '.download_url'$ curl -s "$API_URL/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID" -H "Authorization: Bearer ${API_TOKEN}" | jq -r '.download_url'Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
https://api.openshift.com/api/assisted-images/images/41b91e72-c33e-42ee-b80f-b5c5bbf6431a?arch=x86_64&image_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NTYwMjYzNzEsInN1YiI6IjQxYjkxZTcyLWMzM2UtNDJlZS1iODBmLWI1YzViYmY2NDMxYSJ9.1EX_VGaMNejMhrAvVRBS7PDPIQtbOOc8LtG8OukE1a4&type=minimal-iso&version=4.12
https://api.openshift.com/api/assisted-images/images/41b91e72-c33e-42ee-b80f-b5c5bbf6431a?arch=x86_64&image_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NTYwMjYzNzEsInN1YiI6IjQxYjkxZTcyLWMzM2UtNDJlZS1iODBmLWI1YzViYmY2NDMxYSJ9.1EX_VGaMNejMhrAvVRBS7PDPIQtbOOc8LtG8OukE1a4&type=minimal-iso&version=4.12Copy to Clipboard Copied! Toggle word wrap Toggle overflow Download the ISO:
curl -L -s '<iso_url>' --output rhcos-live-minimal.iso
$ curl -L -s '<iso_url>' --output rhcos-live-minimal.iso1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Replace
<iso_url>with the URL for the ISO from the previous step.
-
Boot the new worker host from the downloaded
rhcos-live-minimal.iso. Get the list of hosts in the cluster that are not installed. Keep running the following command until the new host shows up:
curl -s "$API_URL/api/assisted-install/v2/clusters/$CLUSTER_ID" -H "Authorization: Bearer ${API_TOKEN}" | jq -r '.hosts[] | select(.status != "installed").id'$ curl -s "$API_URL/api/assisted-install/v2/clusters/$CLUSTER_ID" -H "Authorization: Bearer ${API_TOKEN}" | jq -r '.hosts[] | select(.status != "installed").id'Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
2294ba03-c264-4f11-ac08-2f1bb2f8c296
2294ba03-c264-4f11-ac08-2f1bb2f8c296Copy to Clipboard Copied! Toggle word wrap Toggle overflow Set the
$HOST_IDvariable for the new host, for example:HOST_ID=<host_id>
$ HOST_ID=<host_id>1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Replace
<host_id>with the host ID from the previous step.
Check that the host is ready to install by running the following command:
NoteEnsure that you copy the entire command including the complete
jqexpression.Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow When the previous command shows that the host is ready, start the installation using the /v2/infra-envs/{infra_env_id}/hosts/{host_id}/actions/install API by running the following command:
curl -X POST -s "$API_URL/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID/hosts/$HOST_ID/actions/install" -H "Authorization: Bearer ${API_TOKEN}"$ curl -X POST -s "$API_URL/api/assisted-install/v2/infra-envs/$INFRA_ENV_ID/hosts/$HOST_ID/actions/install" -H "Authorization: Bearer ${API_TOKEN}"Copy to Clipboard Copied! Toggle word wrap Toggle overflow As the installation proceeds, the installation generates pending certificate signing requests (CSRs) for the host.
ImportantYou must approve the CSRs to complete the installation.
Keep running the following API call to monitor the cluster installation:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Optional: Run the following command to see all the events for the cluster:
curl -s "$API_URL/api/assisted-install/v2/events?cluster_id=$CLUSTER_ID" -H "Authorization: Bearer ${API_TOKEN}" | jq -c '.[] | {severity, message, event_time, host_id}'$ curl -s "$API_URL/api/assisted-install/v2/events?cluster_id=$CLUSTER_ID" -H "Authorization: Bearer ${API_TOKEN}" | jq -c '.[] | {severity, message, event_time, host_id}'Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - Log in to the cluster and approve the pending CSRs to complete the installation.
Verification
Check that the new host was successfully added to the cluster with a status of
Ready:oc get nodes
$ oc get nodesCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
NAME STATUS ROLES AGE VERSION control-plane-1.example.com Ready master,worker 56m v1.25.0 compute-1.example.com Ready worker 11m v1.25.0
NAME STATUS ROLES AGE VERSION control-plane-1.example.com Ready master,worker 56m v1.25.0 compute-1.example.com Ready worker 11m v1.25.0Copy to Clipboard Copied! Toggle word wrap Toggle overflow
11.5. Replacing a control plane node in a healthy cluster Copy linkLink copied to clipboard!
You can replace a control plane (master) node in a healthy OpenShift Container Platform cluster that has three to five control plane nodes, by adding a new control plane node and removing an existing control plane node.
If the cluster is unhealthy, you must peform additional operations before you can manage the control plane nodes. See Replacing a control plane node in an unhealthy cluster for more information.
11.5.1. Adding a new control plane node Copy linkLink copied to clipboard!
Add the new control plane node, and verify that it is healthy. In the example below, the new node is node-5.
Prerequisites
- You are using OpenShift Container Platform 4.11 or later.
- You have installed a healthy cluster with at least three control plane nodes.
- You have created a single control plane node to be added to the cluster for Day 2.
Procedure
Retrieve pending Certificate Signing Requests (CSRs) for the new Day 2 control plane node:
oc get csr | grep Pending
$ oc get csr | grep PendingCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
csr-5sd59 8m19s kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper <none> Pending csr-xzqts 10s kubernetes.io/kubelet-serving system:node:node-5 <none> Pending
csr-5sd59 8m19s kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper <none> Pending csr-xzqts 10s kubernetes.io/kubelet-serving system:node:node-5 <none> PendingCopy to Clipboard Copied! Toggle word wrap Toggle overflow Approve all pending CSRs for the new node (
node-5in this example):oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve$ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approveCopy to Clipboard Copied! Toggle word wrap Toggle overflow ImportantYou must approve the CSRs to complete the installation.
Confirm that the new control plane node is in
Readystatus:oc get nodes
$ oc get nodesCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteThe
etcdoperator requires aMachinecustom resource (CR) that references the new node when the cluster runs with a Machine API. The machine API is automatically activated when the cluster has three or more control plane nodes.Create the
BareMetalHostandMachineCRs and link them to the new control plane’sNodeCR.Create the
BareMetalHostCR with a unique.metadata.namevalue:Copy to Clipboard Copied! Toggle word wrap Toggle overflow Apply the
BareMetalHostCR:oc apply -f <filename>
$ oc apply -f <filename>1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Replace <filename> with the name of the
BareMetalHostCR.
Create the
MachineCR using the unique.metadata.namevalue:Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Replace
<cluster_name>with the name of the specific cluster, for example,test-day2-1-6qv96.
To get the cluster name, run the following command:
oc get infrastructure cluster -o=jsonpath='{.status.infrastructureName}{"\n"}'$ oc get infrastructure cluster -o=jsonpath='{.status.infrastructureName}{"\n"}'Copy to Clipboard Copied! Toggle word wrap Toggle overflow Apply the
MachineCR:oc apply -f <filename>
$ oc apply -f <filename>1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Replace
<filename>with the name of theMachineCR.
Link
BareMetalHost,Machine, andNodeby running thelink-machine-and-node.shscript:Copy the
link-machine-and-node.shscript below to a local machine:Copy to Clipboard Copied! Toggle word wrap Toggle overflow Make the script executable:
chmod +x link-machine-and-node.sh
$ chmod +x link-machine-and-node.shCopy to Clipboard Copied! Toggle word wrap Toggle overflow Run the script:
bash link-machine-and-node.sh node-5 node-5
$ bash link-machine-and-node.sh node-5 node-5Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteThe first
node-5instance represents the machine, and the second represents the node.
Confirm members of
etcdby executing into one of the pre-existing control plane nodes:Open a remote shell session to the control plane node:
oc rsh -n openshift-etcd etcd-node-0
$ oc rsh -n openshift-etcd etcd-node-0Copy to Clipboard Copied! Toggle word wrap Toggle overflow List
etcdmembers:etcdctl member list -w table
# etcdctl member list -w tableCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Monitor the
etcdoperator configuration process until completion:oc get clusteroperator etcd
$ oc get clusteroperator etcdCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output (upon completion)
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE etcd 4.11.5 True False False 5h54m
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE etcd 4.11.5 True False False 5h54mCopy to Clipboard Copied! Toggle word wrap Toggle overflow Confirm
etcdhealth by running the following commands:Open a remote shell session to the control plane node:
oc rsh -n openshift-etcd etcd-node-0
$ oc rsh -n openshift-etcd etcd-node-0Copy to Clipboard Copied! Toggle word wrap Toggle overflow Check endpoint health:
etcdctl endpoint health
# etcdctl endpoint healthCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
192.168.111.24 is healthy: committed proposal: took = 10.383651ms 192.168.111.26 is healthy: committed proposal: took = 11.297561ms 192.168.111.25 is healthy: committed proposal: took = 13.892416ms 192.168.111.28 is healthy: committed proposal: took = 11.870755ms
192.168.111.24 is healthy: committed proposal: took = 10.383651ms 192.168.111.26 is healthy: committed proposal: took = 11.297561ms 192.168.111.25 is healthy: committed proposal: took = 13.892416ms 192.168.111.28 is healthy: committed proposal: took = 11.870755msCopy to Clipboard Copied! Toggle word wrap Toggle overflow
Verify that all nodes are ready:
oc get nodes
$ oc get nodesCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Verify that the cluster Operators are all available:
oc get ClusterOperators
$ oc get ClusterOperatorsCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Verify that the cluster version is correct:
oc get ClusterVersion
$ oc get ClusterVersionCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.11.5 True False 5h57m Cluster version is 4.11.5
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.11.5 True False 5h57m Cluster version is 4.11.5Copy to Clipboard Copied! Toggle word wrap Toggle overflow
11.5.2. Removing the existing control plane node Copy linkLink copied to clipboard!
Remove the control plane node that you are replacing. This is node-0 in the example below.
Prerequisites
- You have added a new healthy control plane node.
Procedure
Delete the
BareMetalHostCR of the pre-existing control plane node:oc delete bmh -n openshift-machine-api node-0
$ oc delete bmh -n openshift-machine-api node-0Copy to Clipboard Copied! Toggle word wrap Toggle overflow Confirm that the machine is unhealthy:
oc get machine -A
$ oc get machine -ACopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Delete the
MachineCR:oc delete machine -n openshift-machine-api node-0
$ oc delete machine -n openshift-machine-api node-0 machine.machine.openshift.io "node-0" deletedCopy to Clipboard Copied! Toggle word wrap Toggle overflow Confirm removal of the
NodeCR:oc get nodes
$ oc get nodesCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Check
etcd-operatorlogs to confirm status of theetcdcluster:oc logs -n openshift-etcd-operator etcd-operator-8668df65d-lvpjf
$ oc logs -n openshift-etcd-operator etcd-operator-8668df65d-lvpjfCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
E0927 07:53:10.597523 1 base_controller.go:272] ClusterMemberRemovalController reconciliation failed: cannot remove member: 192.168.111.23 because it is reported as healthy but it doesn't have a machine nor a node resource
E0927 07:53:10.597523 1 base_controller.go:272] ClusterMemberRemovalController reconciliation failed: cannot remove member: 192.168.111.23 because it is reported as healthy but it doesn't have a machine nor a node resourceCopy to Clipboard Copied! Toggle word wrap Toggle overflow Remove the physical machine to allow the
etcdoperator to reconcile the cluster members:Open a remote shell session to the control plane node:
oc rsh -n openshift-etcd etcd-node-1
$ oc rsh -n openshift-etcd etcd-node-1Copy to Clipboard Copied! Toggle word wrap Toggle overflow Monitor the progress of
etcdoperator reconciliation by checking members and endpoint health:etcdctl member list -w table; etcdctl endpoint health
# etcdctl member list -w table; etcdctl endpoint healthCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
11.6. Replacing a control plane node in an unhealthy cluster Copy linkLink copied to clipboard!
You can replace an unhealthy control plane (master) node in an OpenShift Container Platform cluster that has three to five control plane nodes, by removing the unhealthy control plane node and adding a new one.
For details on replacing a control plane node in a healthy cluster, see Replacing a control plane node in a healthy cluster.
11.6.1. Removing an unhealthy control plane node Copy linkLink copied to clipboard!
Remove the unhealthy control plane node from the cluster. This is node-0 in the example below.
Prerequisites
- You have installed a cluster with at least three control plane nodes.
- At least one of the control plane nodes is not ready.
Procedure
Check the node status to confirm that a control plane node is not ready:
oc get nodes
$ oc get nodesCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Confirm in the
etcd-operatorlogs that the cluster is unhealthy:oc logs -n openshift-etcd-operator etcd-operator deployment/etcd-operator
$ oc logs -n openshift-etcd-operator etcd-operator deployment/etcd-operatorCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
E0927 08:24:23.983733 1 base_controller.go:272] DefragController reconciliation failed: cluster is unhealthy: 2 of 3 members are available, node-0 is unhealthy
E0927 08:24:23.983733 1 base_controller.go:272] DefragController reconciliation failed: cluster is unhealthy: 2 of 3 members are available, node-0 is unhealthyCopy to Clipboard Copied! Toggle word wrap Toggle overflow Confirm the
etcdmembers by running the following commands:Open a remote shell session to the control plane node:
oc rsh -n openshift-etcd node-1
$ oc rsh -n openshift-etcd node-1Copy to Clipboard Copied! Toggle word wrap Toggle overflow List the
etcdctlmembers:etcdctl member list -w table
# etcdctl member list -w tableCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Confirm that
etcdctlendpoint health reports an unhealthy member of the cluster:etcdctl endpoint health
# etcdctl endpoint healthCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
{"level":"warn","ts":"2022-09-27T08:25:35.953Z","logger":"client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000680380/192.168.111.25","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing dial tcp 192.168.111.25: connect: no route to host\""} 192.168.111.28 is healthy: committed proposal: took = 12.465641ms 192.168.111.26 is healthy: committed proposal: took = 12.297059ms 192.168.111.25 is unhealthy: failed to commit proposal: context deadline exceeded Error: unhealthy cluster{"level":"warn","ts":"2022-09-27T08:25:35.953Z","logger":"client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000680380/192.168.111.25","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing dial tcp 192.168.111.25: connect: no route to host\""} 192.168.111.28 is healthy: committed proposal: took = 12.465641ms 192.168.111.26 is healthy: committed proposal: took = 12.297059ms 192.168.111.25 is unhealthy: failed to commit proposal: context deadline exceeded Error: unhealthy clusterCopy to Clipboard Copied! Toggle word wrap Toggle overflow Remove the unhealthy control plane by deleting the
Machinecustom resource (CR):oc delete machine -n openshift-machine-api node-0
$ oc delete machine -n openshift-machine-api node-0Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteThe
MachineandNodeCRs might not be deleted because they are protected by finalizers. If this occurs, you must delete theMachineCR manually by removing all finalizers.Verify in the
etcd-operatorlogs whether the unhealthy machine has been removed:oc logs -n openshift-etcd-operator etcd-operator deployment/ettcd-operator
$ oc logs -n openshift-etcd-operator etcd-operator deployment/ettcd-operatorCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
I0927 08:58:41.249222 1 machinedeletionhooks.go:135] skip removing the deletion hook from machine node-0 since its member is still present with any of: [{InternalIP } {InternalIP 192.168.111.25}]I0927 08:58:41.249222 1 machinedeletionhooks.go:135] skip removing the deletion hook from machine node-0 since its member is still present with any of: [{InternalIP } {InternalIP 192.168.111.25}]Copy to Clipboard Copied! Toggle word wrap Toggle overflow If you see that removal has been skipped, as in the above log example, manually remove the unhealthy
etcdctlmember:Open a remote shell session to the control plane node:
oc rsh -n openshift-etcd node-1
$ oc rsh -n openshift-etcd node-1Copy to Clipboard Copied! Toggle word wrap Toggle overflow List the
etcdctlmembers:etcdctl member list -w table
# etcdctl member list -w tableCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Confirm that
etcdctlendpoint health reports an unhealthy member of the cluster:etcdctl endpoint health
# etcdctl endpoint healthCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
{"level":"warn","ts":"2022-09-27T10:31:07.227Z","logger":"client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0000d6e00/192.168.111.25","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing dial tcp 192.168.111.25: connect: no route to host\""} 192.168.111.28 is healthy: committed proposal: took = 13.038278ms 192.168.111.26 is healthy: committed proposal: took = 12.950355ms 192.168.111.25 is unhealthy: failed to commit proposal: context deadline exceeded Error: unhealthy cluster{"level":"warn","ts":"2022-09-27T10:31:07.227Z","logger":"client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0000d6e00/192.168.111.25","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing dial tcp 192.168.111.25: connect: no route to host\""} 192.168.111.28 is healthy: committed proposal: took = 13.038278ms 192.168.111.26 is healthy: committed proposal: took = 12.950355ms 192.168.111.25 is unhealthy: failed to commit proposal: context deadline exceeded Error: unhealthy clusterCopy to Clipboard Copied! Toggle word wrap Toggle overflow Remove the unhealthy
etcdctlmember from the cluster:etcdctl member remove 61e2a86084aafa62
# etcdctl member remove 61e2a86084aafa62Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
Member 61e2a86084aafa62 removed from cluster 6881c977b97990d7
Member 61e2a86084aafa62 removed from cluster 6881c977b97990d7Copy to Clipboard Copied! Toggle word wrap Toggle overflow Verify that the unhealthy
etcdctlmember was removed by running the following command:etcdctl member list -w table
# etcdctl member list -w tableCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
11.6.2. Adding a new control plane node Copy linkLink copied to clipboard!
Add a new control plane node to replace the unhealthy node that you removed. In the example below, the new node is node-5.
Prerequisites
- You have installed a control plane node for Day 2. For more information, see Adding hosts with the web console or Adding hosts with the API.
Procedure
Retrieve pending Certificate Signing Requests (CSRs) for the new Day 2 control plane node:
oc get csr | grep Pending
$ oc get csr | grep PendingCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
csr-5sd59 8m19s kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper <none> Pending csr-xzqts 10s kubernetes.io/kubelet-serving system:node:node-5 <none> Pending
csr-5sd59 8m19s kubernetes.io/kube-apiserver-client-kubelet system:serviceaccount:openshift-machine-config-operator:node-bootstrapper <none> Pending csr-xzqts 10s kubernetes.io/kubelet-serving system:node:node-5 <none> PendingCopy to Clipboard Copied! Toggle word wrap Toggle overflow Approve all pending CSRs for the new node (
node-5in this example):oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approve$ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | xargs --no-run-if-empty oc adm certificate approveCopy to Clipboard Copied! Toggle word wrap Toggle overflow NoteYou must approve the CSRs to complete the installation.
Confirm that the control plane node is in
Readystatus:oc get nodes
$ oc get nodesCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow The
etcdoperator requires aMachineCR referencing the new node when the cluster runs with a Machine API. The machine API is automatically activated when the cluster has three control plane nodes.Create the
BareMetalHostandMachineCRs and link them to the new control plane’sNodeCR.ImportantBoot-it-yourself will not create
BareMetalHostandMachineCRs, so you must create them. Failure to create theBareMetalHostandMachineCRs will generate errors in theetcdoperator.Create the
BareMetalHostCR with a unique.metadata.namevalue:Copy to Clipboard Copied! Toggle word wrap Toggle overflow Apply the
BareMetalHostCR:oc apply -f <filename>
$ oc apply -f <filename>1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Replace <filename> with the name of the
BareMetalHostCR.
Create the
MachineCR using the unique.metadata.namevalue:Copy to Clipboard Copied! Toggle word wrap Toggle overflow Apply the
MachineCR:oc apply -f <filename>
$ oc apply -f <filename>1 Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Replace <filename> with the name of the
MachineCR.
Link
BareMetalHost,Machine, andNodeby running thelink-machine-and-node.shscript:Copy the
link-machine-and-node.shscript below to a local machine:Copy to Clipboard Copied! Toggle word wrap Toggle overflow Make the script executable:
chmod +x link-machine-and-node.sh
$ chmod +x link-machine-and-node.shCopy to Clipboard Copied! Toggle word wrap Toggle overflow Run the script:
bash link-machine-and-node.sh node-5 node-5
$ bash link-machine-and-node.sh node-5 node-5Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteThe first
node-5instance represents the machine, and the second represents the node.
Confirm members of
etcdby running the following commands:Open a remote shell session to the control plane node:
oc rsh -n openshift-etcd node-1
$ oc rsh -n openshift-etcd node-1Copy to Clipboard Copied! Toggle word wrap Toggle overflow List the
etcdctlmembers:etcdctl member list -w table
# etcdctl member list -w tableCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Monitor the
etcdoperator configuration process until completion:oc get clusteroperator etcd
$ oc get clusteroperator etcdCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output (upon completion)
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE etcd 4.11.5 True False False 22h
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE etcd 4.11.5 True False False 22hCopy to Clipboard Copied! Toggle word wrap Toggle overflow Confirm
etcdctlhealth by running the following commands:Open a remote shell session to the control plane node:
oc rsh -n openshift-etcd node-1
$ oc rsh -n openshift-etcd node-1Copy to Clipboard Copied! Toggle word wrap Toggle overflow Check endpoint health:
etcdctl endpoint health
# etcdctl endpoint healthCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
192.168.111.26 is healthy: committed proposal: took = 9.105375ms 192.168.111.28 is healthy: committed proposal: took = 9.15205ms 192.168.111.29 is healthy: committed proposal: took = 10.277577ms
192.168.111.26 is healthy: committed proposal: took = 9.105375ms 192.168.111.28 is healthy: committed proposal: took = 9.15205ms 192.168.111.29 is healthy: committed proposal: took = 10.277577msCopy to Clipboard Copied! Toggle word wrap Toggle overflow
Confirm the health of the nodes:
oc get Nodes
$ oc get NodesCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Verify that the cluster Operators are all available:
oc get ClusterOperators
$ oc get ClusterOperatorsCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Verify that the cluster version is correct:
oc get ClusterVersion
$ oc get ClusterVersionCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.11.5 True False 22h Cluster version is 4.11.5
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.11.5 True False 22h Cluster version is 4.11.5Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Chapter 12. Installing on Nutanix Copy linkLink copied to clipboard!
If you install OpenShift Container Platform on Nutanix, the Assisted Installer can integrate the OpenShift Container Platform cluster with the Nutanix platform, which exposes the Machine API to Nutanix and enables autoscaling and the dynamic provisioning of storage containers with the Nutanix Container Storage Interface (CSI).
To deploy an OpenShift Container Platform cluster and maintain its daily operation, you need access to a Nutanix account with the necessary environment requirements. For details, see Environment requirements.
12.1. Adding hosts on Nutanix with the UI Copy linkLink copied to clipboard!
To add hosts on Nutanix with the user interface (UI), generate the minimal discovery image ISO from the Assisted Installer. This downloads a smaller image that will fetch the data needed to boot a host with networking and is the default setting. The majority of the content downloads upon boot. The ISO image is about 100MB in size.
After this is complete, you must create an image for the Nutanix platform and create the Nutanix virtual machines.
Prerequisites
- You have created a cluster profile in the Assisted Installer UI.
- You have a Nutanix cluster environment set up, and made a note of the cluster name and subnet name.
Procedure
- In the Cluster details page, select Nutanix from the Integrate with external partner platforms dropdown list. The Include custom manifest checkbox is optional.
- In the Host discovery page, click the Add hosts button.
Optional: Add an SSH public key so that you can connect to the Nutanix VMs as the
coreuser. Having a login to the cluster hosts can provide you with debugging information during the installation.- If you do not have an existing SSH key pair on your local machine, follow the steps in Generating a key pair for cluster node SSH access.
-
In the SSH public key field, click Browse to upload the
id_rsa.pubfile containing the SSH public key or drag and drop the file into the field from the file manager. To see the file in the file manager, select Show hidden files in the menu.
Select the required provisioning type.
NoteMinimal image file: Provision with virtual media downloads a smaller image that will fetch the data needed to boot.
In Networking, select Cluster-managed networking. Nutanix does not support User-managed networking.
Optional: If the cluster hosts require the use of a proxy, select Configure cluster-wide proxy settings. Enter the username, password, required domains or IP addresses, and port for the HTTP and HTTPS URLs of the proxy server. If the cluster hosts are behind a firewall, allow the nodes to access the required domains or IP addresses through the firewall. See Configuring your firewall for OpenShift Container Platform for more information.
NoteThe proxy username and password must be URL-encoded.
- Optional: Configure the discovery image if you want to boot it with an ignition file. See Configuring the discovery image for additional details.
- Click Generate Discovery ISO.
- Copy the Discovery ISO URL.
- In the Nutanix Prism UI, follow the directions to upload the discovery image from the Assisted Installer.
In the Nutanix Prism UI, create the control plane (master) VMs through Prism Central.
-
Enter the Name. For example,
control-planeormaster. - Enter the Number of VMs. This should be 3, 4, or 5 for the control plane.
- Ensure the remaining settings meet the minimum requirements for control plane hosts.
-
Enter the Name. For example,
In the Nutanix Prism UI, create the worker VMs through Prism Central.
-
Enter the Name. For example,
worker. - Enter the Number of VMs. You should create at least 2 worker nodes.
- Ensure the remaining settings meet the minimum requirements for worker hosts.
-
Enter the Name. For example,
-
Return to the Assisted Installer user interface and wait until the Assisted Installer discovers the hosts and each of them have a
Readystatus. - Continue with the installation procedure.
12.2. Adding hosts on Nutanix with the API Copy linkLink copied to clipboard!
To add hosts on Nutanix with the API, generate the discovery image ISO from the Assisted Installer. Use the minimal discovery image ISO. This is the default setting. The image includes only what is required to boot a host with networking. The majority of the content is downloaded upon boot. The ISO image is about 100MB in size.
Once this is complete, you must create an image for the Nutanix platform and create the Nutanix virtual machines.
Prerequisites
- You have set up the Assisted Installer API authentication.
- You have created an Assisted Installer cluster profile.
- You have created an Assisted Installer infrastructure environment.
-
You have your infrastructure environment ID exported in your shell as
$INFRA_ENV_ID. - You have completed the Assisted Installer cluster configuration.
- You have a Nutanix cluster environment set up, and made a note of the cluster name and subnet name.
Procedure
- Configure the discovery image if you want it to boot with an ignition file.
Create a Nutanix cluster configuration file to hold the environment variables:
touch ~/nutanix-cluster-env.sh
$ touch ~/nutanix-cluster-env.shCopy to Clipboard Copied! Toggle word wrap Toggle overflow chmod +x ~/nutanix-cluster-env.sh
$ chmod +x ~/nutanix-cluster-env.shCopy to Clipboard Copied! Toggle word wrap Toggle overflow If you have to start a new terminal session, you can reload the environment variables easily. For example:
source ~/nutanix-cluster-env.sh
$ source ~/nutanix-cluster-env.shCopy to Clipboard Copied! Toggle word wrap Toggle overflow Assign the Nutanix cluster’s name to the
NTX_CLUSTER_NAMEenvironment variable in the configuration file:cat << EOF >> ~/nutanix-cluster-env.sh export NTX_CLUSTER_NAME=<cluster_name> EOF
$ cat << EOF >> ~/nutanix-cluster-env.sh export NTX_CLUSTER_NAME=<cluster_name> EOFCopy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
<cluster_name>with the name of the Nutanix cluster.Assign the Nutanix cluster’s subnet name to the
NTX_SUBNET_NAMEenvironment variable in the configuration file:cat << EOF >> ~/nutanix-cluster-env.sh export NTX_SUBNET_NAME=<subnet_name> EOF
$ cat << EOF >> ~/nutanix-cluster-env.sh export NTX_SUBNET_NAME=<subnet_name> EOFCopy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
<subnet_name>with the name of the Nutanix cluster’s subnet.Refresh the API token:
source refresh-token
$ source refresh-tokenCopy to Clipboard Copied! Toggle word wrap Toggle overflow Get the download URL:
curl -H "Authorization: Bearer ${API_TOKEN}" \ https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/downloads/image-url$ curl -H "Authorization: Bearer ${API_TOKEN}" \ https://api.openshift.com/api/assisted-install/v2/infra-envs/${INFRA_ENV_ID}/downloads/image-urlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Create the Nutanix image configuration file:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
<image_url>with the image URL downloaded from the previous step.Create the Nutanix image:
curl -k -u <user>:'<password>' -X 'POST' \ 'https://<domain-or-ip>:<port>/api/nutanix/v3/images \ -H 'accept: application/json' \ -H 'Content-Type: application/json' \ -d @./create-image.json | jq '.metadata.uuid'
$ curl -k -u <user>:'<password>' -X 'POST' \ 'https://<domain-or-ip>:<port>/api/nutanix/v3/images \ -H 'accept: application/json' \ -H 'Content-Type: application/json' \ -d @./create-image.json | jq '.metadata.uuid'Copy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
<user>with the Nutanix user name. Replace'<password>'with the Nutanix password. Replace<domain-or-ip>with the domain name or IP address of the Nutanix plaform. Replace<port>with the port for the Nutanix server. The port defaults to9440.Assign the returned UUID to the
NTX_IMAGE_UUIDenvironment variable in the configuration file:cat << EOF >> ~/nutanix-cluster-env.sh export NTX_IMAGE_UUID=<uuid> EOF
$ cat << EOF >> ~/nutanix-cluster-env.sh export NTX_IMAGE_UUID=<uuid> EOFCopy to Clipboard Copied! Toggle word wrap Toggle overflow Get the Nutanix cluster UUID:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
<user>with the Nutanix user name. Replace'<password>'with the Nutanix password. Replace<domain-or-ip>with the domain name or IP address of the Nutanix plaform. Replace<port>with the port for the Nutanix server. The port defaults to9440. Replace<nutanix_cluster_name>with the name of the Nutanix cluster.Assign the returned Nutanix cluster UUID to the
NTX_CLUSTER_UUIDenvironment variable in the configuration file:cat << EOF >> ~/nutanix-cluster-env.sh export NTX_CLUSTER_UUID=<uuid> EOF
$ cat << EOF >> ~/nutanix-cluster-env.sh export NTX_CLUSTER_UUID=<uuid> EOFCopy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
<uuid>with the returned UUID of the Nutanix cluster.Get the Nutanix cluster’s subnet UUID:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
<user>with the Nutanix user name. Replace'<password>'with the Nutanix password. Replace<domain-or-ip>with the domain name or IP address of the Nutanix plaform. Replace<port>with the port for the Nutanix server. The port defaults to9440. Replace<subnet_name>with the name of the cluster’s subnet.Assign the returned Nutanix subnet UUID to the
NTX_CLUSTER_UUIDenvironment variable in the configuration file:cat << EOF >> ~/nutanix-cluster-env.sh export NTX_SUBNET_UUID=<uuid> EOF
$ cat << EOF >> ~/nutanix-cluster-env.sh export NTX_SUBNET_UUID=<uuid> EOFCopy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
<uuid>with the returned UUID of the cluster subnet.Ensure the Nutanix environment variables are set:
source ~/nutanix-cluster-env.sh
$ source ~/nutanix-cluster-env.shCopy to Clipboard Copied! Toggle word wrap Toggle overflow Create a VM configuration file for each Nutanix host. Create three to five control plane (master) VMs and at least two worker VMs. For example:
touch create-master-0.json
$ touch create-master-0.jsonCopy to Clipboard Copied! Toggle word wrap Toggle overflow Copy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
<host_name>with the name of the host.Boot each Nutanix virtual machine:
curl -k -u <user>:'<password>' -X 'POST' \ 'https://<domain-or-ip>:<port>/api/nutanix/v3/vms' \ -H 'accept: application/json' \ -H 'Content-Type: application/json' \ -d @./<vm_config_file_name> | jq '.metadata.uuid'
$ curl -k -u <user>:'<password>' -X 'POST' \ 'https://<domain-or-ip>:<port>/api/nutanix/v3/vms' \ -H 'accept: application/json' \ -H 'Content-Type: application/json' \ -d @./<vm_config_file_name> | jq '.metadata.uuid'Copy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
<user>with the Nutanix user name. Replace'<password>'with the Nutanix password. Replace<domain-or-ip>with the domain name or IP address of the Nutanix plaform. Replace<port>with the port for the Nutanix server. The port defaults to9440. Replace<vm_config_file_name>with the name of the VM configuration file.Assign the returned VM UUID to a unique environment variable in the configuration file:
cat << EOF >> ~/nutanix-cluster-env.sh export NTX_MASTER_0_UUID=<uuid> EOF
$ cat << EOF >> ~/nutanix-cluster-env.sh export NTX_MASTER_0_UUID=<uuid> EOFCopy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
<uuid>with the returned UUID of the VM.NoteThe environment variable must have a unique name for each VM.
Wait until the Assisted Installer has discovered each VM and they have passed validation.
curl -s -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID"
$ curl -s -X GET "https://api.openshift.com/api/assisted-install/v2/clusters/$CLUSTER_ID" --header "Content-Type: application/json" -H "Authorization: Bearer $API_TOKEN" | jq '.enabled_host_count'Copy to Clipboard Copied! Toggle word wrap Toggle overflow Modify the cluster definition to enable integration with Nutanix:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - Continue with the installation procedure.
12.3. Nutanix postinstallation configuration Copy linkLink copied to clipboard!
Complete and validate the OpenShift Container Platform integration with the Nutanix cloud provider.
By default, the installation program downloads and installs the Red Hat Enterprise Linux CoreOS (RHCOS) image. If Prism Central does not have internet access, you can host the RHCOS image on any HTTP server and point the installation program to the image or you can use Prism Central to upload the image manually.
12.3.1. Updating the Nutanix configuration settings Copy linkLink copied to clipboard!
After installing OpenShift Container Platform on the Nutanix platform by using the Assisted Installer, update the following Nutanix configuration settings manually.
Prerequisites
- You have your Nutanix Prism Element username.
- You have your Nutanix Prism Element password.
- You have your Nutanix Prism storage container.
- The Assisted Installer has finished installing the cluster successfully.
- You have connected the cluster to console.redhat.com.
- You have access to the Red Hat OpenShift Container Platform command line interface.
Procedure
In the OpenShift Container Platform command line interface, update the Nutanix cluster configuration settings:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Replace
<prismcentral_address>with the Nutanix Prism Central address. - 2
- Replace
<prismcentral_port>with the Nutanix Prism Central port. - 3
- Replace
<prismelement_address>with the Nutanix Prism Element address. - 4
- Replace
<prismelement_port>with the Nutanix Prism Element port. - 5
- Replace
<prismelement_clustername>with the Nutanix Prism Element cluster name.
Example output
infrastructure.config.openshift.io/cluster patched
infrastructure.config.openshift.io/cluster patchedCopy to Clipboard Copied! Toggle word wrap Toggle overflow For additional details, see Creating a compute machine set on Nutanix.
NoteOptional: You can define prism category key and value pairs. These category key-value pairs must exist in Prism Central. You can define the key-value pairs in separate categories for compute nodes, control plane nodes, or all nodes.
Create the Nutanix secret:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
secret/nutanix-credentials created
secret/nutanix-credentials createdCopy to Clipboard Copied! Toggle word wrap Toggle overflow When installing OpenShift Container Platform version 4.13 or later, update the Nutanix cloud provider configuration:
Get the Nutanix cloud provider configuration YAML file:
oc get cm cloud-provider-config -o yaml -n openshift-config > cloud-provider-config-backup.yaml
$ oc get cm cloud-provider-config -o yaml -n openshift-config > cloud-provider-config-backup.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Create a backup of the configuration file:
cp cloud-provider-config_backup.yaml cloud-provider-config.yaml
$ cp cloud-provider-config_backup.yaml cloud-provider-config.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Open the configuration YAML file:
$ vi cloud-provider-config.yaml
$ vi cloud-provider-config.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Edit the configuration YAML file as follows:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Apply the configuration updates:
oc apply -f cloud-provider-config.yaml
$ oc apply -f cloud-provider-config.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
Warning: resource configmaps/cloud-provider-config is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. configmap/cloud-provider-config configured
Warning: resource configmaps/cloud-provider-config is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically. configmap/cloud-provider-config configuredCopy to Clipboard Copied! Toggle word wrap Toggle overflow
12.3.2. Creating the Nutanix CSI Operator group Copy linkLink copied to clipboard!
Create an Operator group for the Nutanix CSI Operator.
For a description of operator groups and related concepts, see Common Operator Framework terms.
Prerequisites
- You have updated the Nutanix configuration settings.
Procedure
Open the Nutanix CSI Operator Group YAML file:
$ vi openshift-cluster-csi-drivers-operator-group.yaml
$ vi openshift-cluster-csi-drivers-operator-group.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Edit the YAML file as follows:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Create the Operator Group:
oc create -f openshift-cluster-csi-drivers-operator-group.yaml
$ oc create -f openshift-cluster-csi-drivers-operator-group.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
operatorgroup.operators.coreos.com/openshift-cluster-csi-driversjw9cd created
operatorgroup.operators.coreos.com/openshift-cluster-csi-driversjw9cd createdCopy to Clipboard Copied! Toggle word wrap Toggle overflow
12.3.3. Installing the Nutanix CSI Operator Copy linkLink copied to clipboard!
The Nutanix Container Storage Interface (CSI) Operator for Kubernetes deploys and manages the Nutanix CSI Driver.
For instructions on performing this step through the OpenShift Container Platform web console, see the Installing the Operator section of the Nutanix CSI Operator documentation.
Prerequisites
- You have created the Nutanix CSI Operator group.
Procedure
Get the parameter values for the Nutanix CSI Operator YAML file:
Check that the Nutanix CSI Operator exists:
oc get packagemanifests | grep nutanix
$ oc get packagemanifests | grep nutanixCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
nutanixcsioperator Certified Operators 129m
nutanixcsioperator Certified Operators 129mCopy to Clipboard Copied! Toggle word wrap Toggle overflow Assign the default channel for the Operator to a BASH variable:
DEFAULT_CHANNEL=$(oc get packagemanifests nutanixcsioperator -o jsonpath={.status.defaultChannel})$ DEFAULT_CHANNEL=$(oc get packagemanifests nutanixcsioperator -o jsonpath={.status.defaultChannel})Copy to Clipboard Copied! Toggle word wrap Toggle overflow Assign the starting cluster service version (CSV) for the Operator to a BASH variable:
STARTING_CSV=$(oc get packagemanifests nutanixcsioperator -o jsonpath=\{.status.channels[*].currentCSV\})$ STARTING_CSV=$(oc get packagemanifests nutanixcsioperator -o jsonpath=\{.status.channels[*].currentCSV\})Copy to Clipboard Copied! Toggle word wrap Toggle overflow Assign the catalog source for the subscription to a BASH variable:
CATALOG_SOURCE=$(oc get packagemanifests nutanixcsioperator -o jsonpath=\{.status.catalogSource\})$ CATALOG_SOURCE=$(oc get packagemanifests nutanixcsioperator -o jsonpath=\{.status.catalogSource\})Copy to Clipboard Copied! Toggle word wrap Toggle overflow Assign the Nutanix CSI Operator source namespace to a BASH variable:
SOURCE_NAMESPACE=$(oc get packagemanifests nutanixcsioperator -o jsonpath=\{.status.catalogSourceNamespace\})$ SOURCE_NAMESPACE=$(oc get packagemanifests nutanixcsioperator -o jsonpath=\{.status.catalogSourceNamespace\})Copy to Clipboard Copied! Toggle word wrap Toggle overflow
Create the Nutanix CSI Operator YAML file using the BASH variables:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Create the CSI Nutanix Operator:
oc apply -f nutanixcsioperator.yaml
$ oc apply -f nutanixcsioperator.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
subscription.operators.coreos.com/nutanixcsioperator created
subscription.operators.coreos.com/nutanixcsioperator createdCopy to Clipboard Copied! Toggle word wrap Toggle overflow Run the following command until the Operator subscription state changes to
AtLatestKnown. This indicates that the Operator subscription has been created, and might take some time.oc get subscription nutanixcsioperator -n openshift-cluster-csi-drivers -o 'jsonpath={..status.state}'$ oc get subscription nutanixcsioperator -n openshift-cluster-csi-drivers -o 'jsonpath={..status.state}'Copy to Clipboard Copied! Toggle word wrap Toggle overflow
12.3.4. Deploying the Nutanix CSI storage driver Copy linkLink copied to clipboard!
The Nutanix Container Storage Interface (CSI) Driver for Kubernetes provides scalable and persistent storage for stateful applications.
For instructions on performing this step through the OpenShift Container Platform web console, see the Installing the CSI Driver using the Operator section of the Nutanix CSI Operator documentation.
Prerequisites
- You have installed the Nutanix CSI Operator.
Procedure
Create a
NutanixCsiStorageresource to deploy the driver:Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
snutanixcsistorage.crd.nutanix.com/nutanixcsistorage created
snutanixcsistorage.crd.nutanix.com/nutanixcsistorage createdCopy to Clipboard Copied! Toggle word wrap Toggle overflow Create a Nutanix secret YAML file for the CSI storage driver:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
- Replace these parameters with actual values while keeping the same format.
Example output
secret/nutanix-secret created
secret/nutanix-secret createdCopy to Clipboard Copied! Toggle word wrap Toggle overflow
12.3.5. Validating the postinstallation configurations Copy linkLink copied to clipboard!
Verify that you can create a storage class and a bound persistent volume claim.
Prerequisites
- You have deployed the Nutanix CSI storage driver.
Procedure
Verify that you can create a storage class:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Note- 1
- Take <nutanix_storage_container> from the Nutanix configuration; for example, SelfServiceContainer.
Example output
storageclass.storage.k8s.io/nutanix-volume created
storageclass.storage.k8s.io/nutanix-volume createdCopy to Clipboard Copied! Toggle word wrap Toggle overflow Verify that you can create the Nutanix persistent volume claim (PVC):
Create the persistent volume claim (PVC):
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
persistentvolumeclaim/nutanix-volume-pvc created
persistentvolumeclaim/nutanix-volume-pvc createdCopy to Clipboard Copied! Toggle word wrap Toggle overflow Validate that the persistent volume claim (PVC) status is Bound:
oc get pvc -n openshift-cluster-csi-drivers
$ oc get pvc -n openshift-cluster-csi-driversCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE nutanix-volume-pvc Bound nutanix-volume 52s
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE nutanix-volume-pvc Bound nutanix-volume 52sCopy to Clipboard Copied! Toggle word wrap Toggle overflow
Chapter 13. Installing on vSphere Copy linkLink copied to clipboard!
The Assisted Installer integrates the OpenShift Container Platform cluster with the vSphere platform, which exposes the Machine API to vSphere and enables autoscaling.
General support for the following Broadcom products ends on October 2nd, 2025:
- VMware vSphere 7.x
- VMware vCenter 7.x
- VMware Cloud Foundation (VCF) 4.x
This change follows Broadcom’s earlier EoGS announcement, which extended support for these products up to this date.
For details, see the following resources:
13.1. Adding hosts on vSphere Copy linkLink copied to clipboard!
You can add hosts to the Assisted Installer cluster using the online vSphere client or the govc vSphere CLI tool. The following procedure demonstrates adding hosts with the govc CLI tool. To use the online vSphere Client, refer to the documentation for vSphere.
To add hosts on vSphere with the vSphere govc CLI, generate the discovery image ISO from the Assisted Installer. The minimal discovery image ISO is the default setting. This image includes only what is required to boot a host with networking. The majority of the content is downloaded upon boot. The ISO image is about 100MB in size.
After this is complete, you must create an image for the vSphere platform and create the vSphere virtual machines.
Prerequisites
- You are using vSphere 8.0 or higher.
-
You have the vSphere
govcCLI tool installed and configured. -
You have set
clusterSet disk.EnableUUIDto TRUE in vSphere. - You have created a cluster in the Assisted Installer web console, or
- You have created an Assisted Installer cluster profile and infrastructure environment with the API.
-
You have exported your infrastructure environment ID in your shell as
$INFRA_ENV_ID.
Procedure
- Configure the discovery image if you want it to boot with an ignition file.
- In Cluster details, select vSphere from the Integrate with external partner platforms dropdown list. The Include custom manifest checkbox is optional.
- In Host discovery, click the Add hosts button and select the provisioning type.
Add an SSH public key so that you can connect to the vSphere VMs as the
coreuser. Having a login to the cluster hosts can provide you with debugging information during the installation.- If you do not have an existing SSH key pair on your local machine, follow the steps in Generating a key pair for cluster node SSH access.
-
In the SSH public key field, click Browse to upload the
id_rsa.pubfile containing the SSH public key. Alternatively, drag and drop the file into the field from the file manager. To see the file in the file manager, select Show hidden files in the menu.
Select the required discovery image ISO.
NoteMinimal image file: Provision with virtual media downloads a smaller image that will fetch the data needed to boot.
In Networking, select Cluster-managed networking or User-managed networking:
Optional: If the cluster hosts require the use of a proxy, select Configure cluster-wide proxy settings. Enter the username, password, required domains or IP addresses, and port for the HTTP and HTTPS URLs of the proxy server. If the cluster hosts are behind a firewall, allow the nodes to access the required domains or IP addresses through the firewall. See Configuring your firewall for OpenShift Container Platform for more information.
NoteThe proxy username and password must be URL-encoded.
- Optional: If the cluster hosts are in a network with a re-encrypting man-in-the-middle (MITM) proxy or the cluster needs to trust certificates for other purposes such as container image registries, select Configure cluster-wide trusted certificates and add the additional certificates.
- Optional: Configure the discovery image if you want to boot it with an ignition file. For more information, see Additional Resources.
- Click Generate Discovery ISO.
- Copy the Discovery ISO URL.
Download the discovery ISO:
wget - O vsphere-discovery-image.iso <discovery_url>
$ wget - O vsphere-discovery-image.iso <discovery_url>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
<discovery_url>with the Discovery ISO URL from the preceding step.On the command line, power off and delete any preexisting virtual machines:
for VM in $(/usr/local/bin/govc ls /<datacenter>/vm/<folder_name>) do /usr/local/bin/govc vm.power -off $VM /usr/local/bin/govc vm.destroy $VM done
$ for VM in $(/usr/local/bin/govc ls /<datacenter>/vm/<folder_name>) do /usr/local/bin/govc vm.power -off $VM /usr/local/bin/govc vm.destroy $VM doneCopy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
<datacenter>with the name of the data center. Replace<folder_name>with the name of the VM inventory folder.Remove preexisting ISO images from the data store, if there are any:
govc datastore.rm -ds <iso_datastore> <image>
$ govc datastore.rm -ds <iso_datastore> <image>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
<iso_datastore>with the name of the data store. Replaceimagewith the name of the ISO image.Upload the Assisted Installer discovery ISO:
govc datastore.upload -ds <iso_datastore> vsphere-discovery-image.iso
$ govc datastore.upload -ds <iso_datastore> vsphere-discovery-image.isoCopy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
<iso_datastore>with the name of the data store.NoteAll nodes in the cluster must boot from the discovery image.
Boot three to five control plane nodes:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow See vm.create for details.
NoteThe foregoing example illustrates the minimum required resources for control plane nodes.
Boot at least two worker nodes:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow See vm.create for details.
NoteThe foregoing example illustrates the minimum required resources for worker nodes.
Ensure the VMs are running:
govc ls /<datacenter>/vm/<folder_name>
$ govc ls /<datacenter>/vm/<folder_name>Copy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
<datacenter>with the name of the data center. Replace<folder_name>with the name of the VM inventory folder.After 2 minutes, shut down the VMs:
for VM in $(govc ls /<datacenter>/vm/<folder_name>) do govc vm.power -s=true $VM done$ for VM in $(govc ls /<datacenter>/vm/<folder_name>) do govc vm.power -s=true $VM doneCopy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
<datacenter>with the name of the data center. Replace<folder_name>with the name of the VM inventory folder.Set the
disk.EnableUUIDsetting toTRUE:for VM in $(govc ls /<datacenter>/vm/<folder_name>) do govc vm.change -vm $VM -e disk.EnableUUID=TRUE done$ for VM in $(govc ls /<datacenter>/vm/<folder_name>) do govc vm.change -vm $VM -e disk.EnableUUID=TRUE doneCopy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
<datacenter>with the name of the data center. Replace<folder_name>with the name of the VM inventory folder.NoteYou must set
disk.EnableUUIDtoTRUEon all of the nodes to enable autoscaling with vSphere.Restart the VMs:
for VM in $(govc ls /<datacenter>/vm/<folder_name>) do govc vm.power -on=true $VM done$ for VM in $(govc ls /<datacenter>/vm/<folder_name>) do govc vm.power -on=true $VM doneCopy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
<datacenter>with the name of the data center. Replace<folder_name>with the name of the VM inventory folder.-
Return to the Assisted Installer user interface and wait until the Assisted Installer discovers the hosts and each of them have a
Readystatus. - Select roles if needed.
- In Networking, clear the Allocate IPs via DHCP server checkbox.
- Set the API VIP address.
- Set the Ingress VIP address.
- Continue with the installation procedure.
13.2. vSphere postinstallation configuration using the CLI Copy linkLink copied to clipboard!
After installing an OpenShift Container Platform cluster by using the Assisted Installer on vSphere with the platform integration feature enabled, you must update the following vSphere configuration settings manually:
- vCenter username
- vCenter password
- vCenter address
- vCenter cluster
- Data center
- Data store
- Folder
Prerequisites
- The Assisted Installer has finished installing the cluster successfully.
- The cluster is connected to console.redhat.com.
Procedure
Generate a base64-encoded username and password for vCenter:
echo -n "<vcenter_username>" | base64 -w0
$ echo -n "<vcenter_username>" | base64 -w0Copy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
<vcenter_username>with your vCenter username.echo -n "<vcenter_password>" | base64 -w0
$ echo -n "<vcenter_password>" | base64 -w0Copy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
<vcenter_password>with your vCenter password.Backup the vSphere credentials:
oc get secret vsphere-creds -o yaml -n kube-system > creds_backup.yaml
$ oc get secret vsphere-creds -o yaml -n kube-system > creds_backup.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Edit the vSphere credentials:
cp creds_backup.yaml vsphere-creds.yaml
$ cp creds_backup.yaml vsphere-creds.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow vi vsphere-creds.yaml
$ vi vsphere-creds.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Copy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
<vcenter_address>with the vCenter address. Replace<vcenter_username_encoded>with the base64-encoded version of your vSphere username. Replace<vcenter_password_encoded>with the base64-encoded version of your vSphere password.Replace the vSphere credentials:
oc replace -f vsphere-creds.yaml
$ oc replace -f vsphere-creds.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Redeploy the kube-controller-manager pods:
oc patch kubecontrollermanager cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' \ --type=merge$ oc patch kubecontrollermanager cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' \ --type=mergeCopy to Clipboard Copied! Toggle word wrap Toggle overflow Backup the vSphere cloud provider configuration:
oc get cm cloud-provider-config -o yaml -n openshift-config > \ cloud-provider-config_backup.yaml
$ oc get cm cloud-provider-config -o yaml -n openshift-config > \ cloud-provider-config_backup.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Edit the cloud provider configuration:
For OCP versions 4.17 and earlier, use the INI syntax for the
data.configsection of the.yamlfile.vi cloud-provider-config.yaml
$ vi cloud-provider-config.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Copy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
<vcenter_address>with the vCenter address. Replace<datacenter>with the name of the data center. Replace<datastore>with the name of the data store. Replace<folder>with the folder containing the cluster VMs.For OCP versions 4.18 and later, use YAML syntax for the
data.configsection of the.yamlfile.cp cloud-provider-config_backup.yaml cloud-provider-config.yaml
$ cp cloud-provider-config_backup.yaml cloud-provider-config.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow vi cloud-provider-config.yaml
$ vi cloud-provider-config.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Copy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
<vcenter_address>with the vCenter address. Replace<vcenter_port>with the vCenter port number. Port 443 is the default. Replace<datacenter>with the name of the data center.
Apply the cloud provider configuration:
oc apply -f cloud-provider-config.yaml
$ oc apply -f cloud-provider-config.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Taint the nodes with the
uninitializedtaint:ImportantFollow steps 9 through 12 if you are installing OpenShift Container Platform 4.13 or later.
Identify the nodes to taint:
oc get nodes
$ oc get nodesCopy to Clipboard Copied! Toggle word wrap Toggle overflow Run the following command for each node:
oc adm taint node <node_name> node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule
$ oc adm taint node <node_name> node.cloudprovider.kubernetes.io/uninitialized=true:NoScheduleCopy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
<node_name>with the name of the node.
Example
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Back up the infrastructures configuration:
oc get infrastructures.config.openshift.io -o yaml > infrastructures.config.openshift.io.yaml.backup
$ oc get infrastructures.config.openshift.io -o yaml > infrastructures.config.openshift.io.yaml.backupCopy to Clipboard Copied! Toggle word wrap Toggle overflow Edit the infrastructures configuration:
cp infrastructures.config.openshift.io.yaml.backup infrastructures.config.openshift.io.yaml
$ cp infrastructures.config.openshift.io.yaml.backup infrastructures.config.openshift.io.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow vi infrastructures.config.openshift.io.yaml
$ vi infrastructures.config.openshift.io.yamlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Copy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
<vcenter_address>with your vCenter address. Replace<datacenter>with the name of your vCenter data center. Replace<datastore>with the name of your vCenter data store. Replace<folder>with the folder containing the cluster VMs. Replace<vcenter_cluster>with the vSphere vCenter cluster where OpenShift Container Platform is installed.Apply the infrastructures configuration:
oc apply -f infrastructures.config.openshift.io.yaml --overwrite=true
$ oc apply -f infrastructures.config.openshift.io.yaml --overwrite=trueCopy to Clipboard Copied! Toggle word wrap Toggle overflow
13.3. vSphere postinstallation configuration using the web console Copy linkLink copied to clipboard!
After installing an OpenShift Container Platform cluster by using the Assisted Installer on vSphere with the platform integration feature enabled, you must update the following vSphere configuration settings manually:
- vCenter address
- vCenter cluster
- vCenter username
- vCenter password
- Data center
- Default data store
- Virtual machine folder
Prerequisites
- The Assisted Installer has finished installing the cluster successfully.
- The cluster is connected to console.redhat.com.
Procedure
- In the Administrator perspective, navigate to Home → Overview.
- Under Status, click vSphere connection to open the vSphere connection configuration wizard.
-
In the vCenter field, enter the network address of the vSphere vCenter server. This can be either a domain name or an IP address. It appears in the vSphere web client URL; for example
https://[your_vCenter_address]/ui. In the vCenter cluster field, enter the name of the vSphere vCenter cluster where OpenShift Container Platform is installed.
ImportantThis step is mandatory if you installed OpenShift Container Platform 4.13 or later.
- In the Username field, enter your vSphere vCenter username.
In the Password field, enter your vSphere vCenter password.
WarningThe system stores the username and password in the
vsphere-credssecret in thekube-systemnamespace of the cluster. An incorrect vCenter username or password makes the cluster nodes unschedulable.-
In the Datacenter field, enter the name of the vSphere data center that contains the virtual machines used to host the cluster; for example,
SDDC-Datacenter. In the Default data store field, enter the vSphere data store that stores the persistent data volumes; for example,
/SDDC-Datacenter/datastore/datastorename.WarningUpdating the vSphere data center or default data store after the configuration has been saved detaches any active vSphere
PersistentVolumes.-
In the Virtual Machine Folder field, enter the data center folder that contains the virtual machine of the cluster; for example,
/SDDC-Datacenter/vm/ci-ln-hjg4vg2-c61657-t2gzr. For the OpenShift Container Platform installation to succeed, all virtual machines comprising the cluster must be located in a single data center folder. -
Click Save Configuration. This updates the
cloud-provider-configfile in theopenshift-confignamespace, and starts the configuration process. - Reopen the vSphere connection configuration wizard and expand the Monitored operators panel. Check that the status of the operators is either Progressing or Healthy.
Verification
The connection configuration process updates operator statuses and control plane nodes. It takes approximately an hour to complete. During the configuration process, the nodes will reboot. Previously bound PersistentVolumeClaims objects might become disconnected.
Follow the steps below to monitor the configuration process.
Check that the configuration process completed successfully:
- In the Administrator perspective, navigate to Home > Overview.
- Under Status click Operators. Wait for all operator statuses to change from Progressing to All succeeded. A Failed status indicates that the configuration failed.
- Under Status, click Control Plane. Wait for the response rate of all Control Pane components to return to 100%. A Failed control plane component indicates that the configuration failed.
A failure indicates that at least one of the connection settings is incorrect. Change the settings in the vSphere connection configuration wizard and save the configuration again.
Check that you are able to bind
PersistentVolumeClaimsobjects by performing the following steps:Create a
StorageClassobject using the following YAML:Copy to Clipboard Copied! Toggle word wrap Toggle overflow Create a
PersistentVolumeClaimsobject using the following YAML:Copy to Clipboard Copied! Toggle word wrap Toggle overflow
For instructions, see Dynamic provisioning in the OpenShift Container Platform documentation. To troubleshoot a
PersistentVolumeClaimsobject, navigate to Storage → PersistentVolumeClaims in the Administrator perspective of the OpenShift Container Platform web console.
Chapter 14. Installing on Oracle Cloud Infrastructure (OCI) Copy linkLink copied to clipboard!
From OpenShift Container Platform 4.14 and later versions, you can use the Assisted Installer to install a cluster on an Oracle Cloud Infrastructure virtual machine (VM) by using infrastructure that you provide. Oracle Cloud Infrastructure provides services that can meet your needs for regulatory compliance, performance, and cost-effectiveness. You can access OCI Resource Manager configurations to provision and configure OCI resources.
From OpenShift Container Platform 4.16 and later versions, the integration between OpenShift Container Platform and Oracle Cloud Infrastructure is certified on both virtual machines (VMs) and bare metal (BM) machines. Bare-metal installations using iSCSI boot drives require a secondary vNIC that is automatically created in the Terraform stack provided by Oracle.
For details and installation instructions, see Installing a cluster on Oracle Cloud Infrastructure by using the Assisted Installer in the OpenShift Container Platform documentation.
Chapter 15. Troubleshooting Copy linkLink copied to clipboard!
There are cases where the Assisted Installer cannot begin the installation or the cluster fails to install properly. In these events, it is helpful to understand the likely failure modes as well as how to troubleshoot the failure.
15.1. Troubleshooting discovery ISO issues Copy linkLink copied to clipboard!
The Assisted Installer uses an ISO image to run an agent that registers the host to the cluster and performs hardware and network validations before attempting to install OpenShift. You can follow these procedures to troubleshoot problems related to the host discovery.
Once you start the host with the discovery ISO image, the Assisted Installer discovers the host and presents it in the Assisted Service web console. See Configuring the discovery image for additional details.
15.1.1. Verify the discovery agent is running Copy linkLink copied to clipboard!
Prerequisites
- You have created an infrastructure environment by using the API or have created a cluster by using the web console.
- You booted a host with the Infrastructure Environment discovery ISO and the host failed to register.
- You have SSH access to the host.
- You provided an SSH public key in the "Add hosts" dialog before generating the Discovery ISO so that you can SSH into your machine without a password.
Procedure
- Verify that your host machine is powered on.
- If you selected DHCP networking, check that the DHCP server is enabled.
- If you selected Static IP, bridges and bonds networking, check that your configurations are correct.
Verify that you can access your host machine using SSH, a console such as the BMC, or a virtual machine console:
ssh core@<host_ip_address>
$ ssh core@<host_ip_address>Copy to Clipboard Copied! Toggle word wrap Toggle overflow You can specify private key file by using the
-iparameter if it is not stored in the default directory.ssh -i <ssh_private_key_file> core@<host_ip_address>
$ ssh -i <ssh_private_key_file> core@<host_ip_address>Copy to Clipboard Copied! Toggle word wrap Toggle overflow If you fail to connect over SSH to the host, the host failed during boot or it failed to configure the network.
Upon login you should see this message:
Example login
If you are not seeing this message it means that the host did not boot with the Assisted Installer ISO image. Make sure you configured the boot order properly (The host should boot once from the live-ISO).
Check the agent service logs:
sudo journalctl -u agent.service
$ sudo journalctl -u agent.serviceCopy to Clipboard Copied! Toggle word wrap Toggle overflow In the following example, the errors indicate there is a network issue:
Example agent service log screenshot of agent service log
If there is an error pulling the agent image, check the proxy settings. Verify that the host is connected to the network. You can use
nmclito get additional information about your network configuration.
15.1.2. Verify the agent can access the assisted-service Copy linkLink copied to clipboard!
Prerequisites
- You have created an Infrastructure Environment by using the API or have created a cluster by using the web console.
- You booted a host with the Infrastructure Environment discovery ISO and the host failed to register.
- You verified the discovery agent is running.
Procedure
Check the agent logs to verify the agent can access the Assisted Service:
sudo journalctl TAG=agent
$ sudo journalctl TAG=agentCopy to Clipboard Copied! Toggle word wrap Toggle overflow The errors in the following example indicate that the agent failed to access the Assisted Service.
Example agent log
Check the proxy settings you configured for the cluster. If configured, the proxy must allow access to the Assisted Service URL.
15.2. Troubleshooting minimal discovery ISO issues Copy linkLink copied to clipboard!
Use the minimal ISO image when the virtual media connection has limited bandwidth. It includes only what the agent requires to boot a host with networking. The majority of the content is downloaded upon boot. The resulting ISO image is about 100MB in size compared to 1GB for the full ISO image.
15.2.1. Troubleshooting minimal ISO boot failure by interrupting the boot process Copy linkLink copied to clipboard!
If your environment requires static network configuration to access the Assisted Installer service, any issues with that configuration might prevent the minimal ISO from booting properly. If the boot screen shows that the host has failed to download the root file system image, the network might not be configured correctly.
You can interrupt the kernel boot early in the bootstrap process, before the root file system image is downloaded. This allows you to access the root console and review the network configurations.
Example rootfs download failure
Procedure
Add the
.spec.kernelArgumentsstanza to theinfraEnvobject of the cluster you are deploying:NoteFor details on modifying an infrastructure environment, see Additional Resources.
Copy to Clipboard Copied! Toggle word wrap Toggle overflow - 1
rd.break=initqueueinterrupts the boot at thedracutmain loop. See rd.break options for debugging kernel boot for details.
-
Wait for the related nodes to reboot automatically and for the boot to stop at the
iniqueuestage, beforerootfsis downloaded. You will be redirected to the root console. Identify and change the incorrect network configurations. Here are some useful diagnostic commands:
View system logs by using
journalctl, for example:journalctl -p err //Sorts logs by errors journalctl -p crit //Sorts logs by critical errors journalctl -p warning //Sorts logs by warnings
# journalctl -p err //Sorts logs by errors # journalctl -p crit //Sorts logs by critical errors # journalctl -p warning //Sorts logs by warningsCopy to Clipboard Copied! Toggle word wrap Toggle overflow View network connection information by using
nmcli, as follows:nmcli conn show
# nmcli conn showCopy to Clipboard Copied! Toggle word wrap Toggle overflow Check the configuration files for incorrect network connections, for example:
cat /etc/assisted/network/host0/eno3.nmconnection
# cat /etc/assisted/network/host0/eno3.nmconnectionCopy to Clipboard Copied! Toggle word wrap Toggle overflow
-
Press
control+dto resume the bootstrap process. The server downloadsrootfsand completes the process. -
Reopen the
infraEnvobject and remove the.spec.kernelArgumentsstanza.
15.3. Correcting a host’s boot order Copy linkLink copied to clipboard!
Once the installation that runs as part of the Discovery Image completes, the Assisted Installer reboots the host. The host must boot from its installation disk to continue forming the cluster. If you have not correctly configured the host’s boot order, it will boot from another disk instead, interrupting the installation.
If the host boots the discovery image again, the Assisted Installer will immediately detect this event and set the host’s status to Installing Pending User Action. Alternatively, if the Assisted Installer does not detect that the host has booted the correct disk within the allotted time, it will also set this host status.
Procedure
- Reboot the host and set its boot order to boot from the installation disk. If you didn’t select an installation disk, the Assisted Installer selected one for you. To view the selected installation disk, click to expand the host’s information in the host inventory, and check which disk has the “Installation disk” role.
15.4. Rectifying partially-successful installations Copy linkLink copied to clipboard!
There are cases where the Assisted Installer declares an installation to be successful even though it encountered errors:
- If you requested to install OLM operators and one or more failed to install, log in to the cluster’s console to remediate the failures.
- If you requested to install more than two worker nodes and at least one failed to install, but at least two succeeded, add the failed workers to the installed cluster.
15.5. API connectivity failure when adding nodes to a cluster Copy linkLink copied to clipboard!
When you add a node to an existing cluster as part of Day 2 operations, the node downloads the ignition configuration file from the Day 1 cluster. If the download fails and the node is unable to connect to the cluster, the status of the host in the Host discovery step changes to Insufficient. Clicking this status displays the following error message:
The host failed to download the ignition file from <URL>. You must ensure the host can reach the URL. Check your DNS and network configuration or update the IP address or domain used to reach the cluster. error: ignition file download failed.... no route to host
The host failed to download the ignition file from <URL>. You must ensure the host can reach the URL. Check your DNS and network configuration or update the IP address or domain used to reach the cluster.
error: ignition file download failed.... no route to host
There are several possible reasons for the connectivity failure. Here are some recommended actions.
Procedure
Check the IP address and domain name of the cluster:
- Click the set the IP or domain used to reach the cluster hyperlink.
- In the Update cluster hostname window, enter the correct IP address or domain name for the cluster.
- Check your DNS settings to ensure that the DNS can resolve the domain that you provided.
-
Ensure that port
22624is open in all firewalls. Check the agent logs of the host to verify that the agent can access the Assisted Service via SSH:
sudo journalctl TAG=agent
$ sudo journalctl TAG=agentCopy to Clipboard Copied! Toggle word wrap Toggle overflow NoteFor more details, see Verify the agent can access the Assisted Service.
15.6. Troubleshooting auto-assign validation errors Copy linkLink copied to clipboard!
The Assisted Installer assigns two role types to each host:
-
role: This is the primary, visible role for the host. You can explicitly set the role as either control plane, arbiter, or worker. Alternatively, you can choose auto-assign, which allows the Assisted Installer to determine the appropriate role automatically. -
suggested_role: When a host is set to auto-assign, the Assisted Installer assigns it asuggested_rolevalue at the start of the installation process. This is an internal role type and is not visible to the user. The Assisted Installer determines thesuggested_rolebased on its internal logic. For details, see the About assigning roles to hosts section in "Additional resources".
The web console does not display the suggested_role field. Therefore, when using auto-assign, you might still encounter host pre-installation validation errors that suggest a host has a specific role. This happens because the Assisted Installer relies on suggested_role for certain validations.
If you encounter an error, either manually assign a role to each affected node instead of using auto-assign, or follow the instructions provided in the error message, where applicable.
Legal Notice
Copy linkLink copied to clipboard!
Copyright © 2025 Red Hat
OpenShift documentation is licensed under the Apache License 2.0 (https://www.apache.org/licenses/LICENSE-2.0).
Modified versions must remove all Red Hat trademarks.
Portions adapted from https://github.com/kubernetes-incubator/service-catalog/ with modifications by Red Hat.
Red Hat, Red Hat Enterprise Linux, the Red Hat logo, the Shadowman logo, JBoss, OpenShift, Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries.
Linux® is the registered trademark of Linus Torvalds in the United States and other countries.
Java® is a registered trademark of Oracle and/or its affiliates.
XFS® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries.
MySQL® is a registered trademark of MySQL AB in the United States, the European Union and other countries.
Node.js® is an official trademark of Joyent. Red Hat Software Collections is not formally related to or endorsed by the official Joyent Node.js open source or commercial project.
The OpenStack® Word Mark and OpenStack logo are either registered trademarks/service marks or trademarks/service marks of the OpenStack Foundation, in the United States and other countries and are used with the OpenStack Foundation’s permission. We are not affiliated with, endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.
All other trademarks are the property of their respective owners.