Installing on OCI

Procedure

1. Log in to your Oracle Cloud Infrastructure (OCI) account with administrator privileges.

2. Configure the account by defining the Cloud Accounts and Resources (Oracle documentation). Ensure that you create the following resources:

   1. Create a child compartment for organizing, restricting access, and setting usage limits to OCI resources. For the full procedure, see Creating a Compartment (Oracle documentation).
   2. Create a new object storage bucket into which you will upload the discovery ISO image. For the full procedure, see Creating an Object Storage Bucket (Oracle documentation).

3. Download the latest version of the create-cluster-vX.X.X.zip configuration file from the oracle-quickstart/oci-openshift repository. This file provides the infrastructure for the cluster and contains configurations for the following:

   • Terraform Stacks: The Terraform stack code for provisioning OCI resources to create and manage OpenShift Container Platform clusters on OCI.
   • Custom Manifests: The manifest files needed for the installation of OpenShift Container Platform clusters on OCI.
   Note

   To make any changes to the manifests, you can clone the entire Oracle GitHub repository and access the custom_manifests and terraform-stacks directories directly.

   For details, see Configuration Files (Oracle documentation).

Procedure

1. Log in to your Oracle Cloud Infrastructure (OCI) account.
2. Upload the discovery ISO image from your local drive to the new object storage bucket you created. For the full procedure, see Uploading an Object Storage Object to a Bucket (Oracle documentation).

3. Locate the uploaded discovery ISO, and complete the following steps:

   1. Create a Pre-Authenticated Request (PAR) for the ISO from the adjacent options menu.
   2. Copy the generated URL to use as the OpenShift Image Source URI in the next step.

   For the full procedure, see Creating a Pre-Authenticated Requests in Object Storage (Oracle documentation).

4. Create and apply the Terraform stack:

   Important

   The Terraform stack includes files for creating cluster resources and custom manifests. The stack also includes a script, and when you apply the stack, the script creates OCI resources, such as DNS records, an instance, and so on. For a list of the resources, see Terraform Defined Resources for OpenShift on OCI README file.

   1. Upload the Terraform stack template create-cluster-vX.X.X.zip to the new object storage bucket.

   2. Complete the stack information and click Next.

      Important
      • Make sure that Cluster Name matches Cluster Name in Assisted Installer, and Zone DNS matches Base Domain in Assisted Installer.
      • In the OpenShift Image Source URI field, paste the Pre-Authenticated Request URL link that you generated in the previous step.
      • Ensure that the correct Compute Shape field value is defined, depending on whether you are installing on bare metal or a virtual machine. If not, select a different shape from the list. For details, see Compute Shapes (Oracle documentation).
   3. Click Apply to apply the stack.

   For the full procedure, see Creating OpenShift Container Platform Infrastructure Using Resource Manager (Oracle documentation).

5. Copy the dynamic_custom_manifest.yml file from the Outputs page of the Terraform stack.

   Note

   The YAML file contains all the required manifests, concatenated and preformatted with the configuration values. For details, see the Custom Manifests README file.

   For the full procedure, see Getting the OpenShift Container Platform Custom Manifests for Installation (Oracle documentation).

Procedure

1. From the Red Hat Hybrid Cloud Console, go to Clusters > Assisted Clusters and select your cluster’s name.
2. On the Installation Progress page, check that the Installation progress bar is at 100% and a message displays indicating Installation completed successfully.

3. Under Host inventory, confirm that the status of all control plane and compute nodes is Installed.

   Note

   OpenShift Container Platform designates one of the control plane nodes as the bootstrap virtual machine, eliminating the need for a separate bootstrap machine.

4. Click the Web Console URL, to access the OpenShift Container Platform web console.
5. From the menu, select Compute > Nodes.
6. Locate your node from the Nodes table.
7. From the Terminal tab, verify that iSCSI appears next to the serial number.
8. From the Overview tab, check that your node has a Ready status.
9. Select the YAML tab.
10. Check the labels parameter, and verify that the listed labels apply to your configuration. For example, the topology.kubernetes.io/region=us-sanjose-1 label indicates in what OCI region the node was deployed.

1. Create OCI resources and services (Oracle).
2. Disconnected environments: Prepare a web server that is accessible by OCI instances (Red Hat).
3. Prepare configuration files for the Agent-based Installer (Red Hat).
4. Generate the agent ISO image (Red Hat).
5. Disconnected environments: Upload the rootfs image to the web server (Red Hat).
6. Configure your firewall for OpenShift Container Platform (Red Hat).
7. Upload the agent ISO image to a storage bucket (Oracle).
8. Create a custom image from the uploaded agent ISO image (Oracle).
9. Create compute instances on OCI (Oracle).
10. Verify that your cluster runs on OCI (Oracle).

Procedure

1. Create an installation directory to store configuration files in by running the following command:

   $ mkdir ~/<directory_name>

2. Configure the install-config.yaml configuration file to meet the needs of your organization and save the file in the directory you created.

   install-config.yaml file that sets an external platform

   # install-config.yaml
   apiVersion: v1
   baseDomain: <base_domain> 1
   networking:
     clusterNetwork:
     - cidr: 10.128.0.0/14
       hostPrefix: 23
     network type: OVNKubernetes
     machineNetwork:
     - cidr: <ip_address_from_cidr> 2
     serviceNetwork:
     - 172.30.0.0/16
   compute:
     - architecture: amd64 3
     hyperthreading: Enabled
     name: worker
     replicas: 0
   controlPlane:
     architecture: amd64 4
     hyperthreading: Enabled
     name: master
     replicas: 3
   platform:
      external:
       platformName: oci 5
       cloudControllerManager: External
   sshKey: <public_ssh_key> 6
   pullSecret: '<pull_secret>' 7
   # ...

   1

   The base domain of your cloud provider.

   2

   The IP address from the virtual cloud network (VCN) that the CIDR allocates to resources and components that operate on your network.

   3 4

   Depending on your infrastructure, you can select either arm64 or amd64.

   5

   Set OCI as the external platform, so that OpenShift Container Platform can integrate with OCI.

   6

   Specify your SSH public key.

   7

   The pull secret that you need for authenticate purposes when downloading container images for OpenShift Container Platform components and services, such as Quay.io. See Install OpenShift Container Platform 4 from the Red Hat Hybrid Cloud Console.

3. Create a directory on your local system named openshift. This must be a subdirectory of the installation directory.

   Important

   Do not move the install-config.yaml or agent-config.yaml configuration files to the openshift directory.

4. If you used a stack to provision OCI infrastructure resources: Copy and paste the dynamic_custom_manifest output of the OCI stack into a file titled manifest.yaml and save the file in the openshift directory.

5. If you did not use a stack to provision OCI infrastructure resources: Download and prepare custom manifests to create an Agent ISO image:

   1. Go to Configuration Files (Oracle documentation) and follow the link to the custom manifests directory on GitHub.
   2. Copy the contents of the condensed-manifest.yml file and save it locally to a file in the openshift directory.
   3. In the condensed-manifest.yml file, update the sections marked with TODO to specify the compartment Oracle® Cloud Identifier (OCID), VCN OCID, subnet OCID from the load balancer, and the security lists OCID.

6. Configure the agent-config.yaml configuration file to meet your organization’s requirements.

   Sample agent-config.yaml file for an IPv4 network.

   apiVersion: v1beta1
   metadata:
     name: <cluster_name> 1
     namespace: <cluster_namespace> 2
   rendezvousIP: <ip_address_from_CIDR> 3
   bootArtifactsBaseURL: <server_URL> 4
   # ...

   1

   The cluster name that you specified in your DNS record.

   2

   The namespace of your cluster on OpenShift Container Platform.

   3

   If you use IPv4 as the network IP address format, ensure that you set the rendezvousIP parameter to an IPv4 address that the VCN’s Classless Inter-Domain Routing (CIDR) method allocates on your network. Also ensure that at least one instance from the pool of instances that you booted with the ISO matches the IP address value you set for the rendezvousIP parameter.

   4

   The URL of the server where you want to upload the rootfs image. This parameter is required only for disconnected environments.

7. Generate a minimal ISO image, which excludes the rootfs image, by entering the following command in your installation directory:

   $ ./openshift-install agent create image --log-level debug

   The command also completes the following actions:

   • Creates a subdirectory, ./<installation_directory>/auth directory:, and places kubeadmin-password and kubeconfig files in the subdirectory.
   • Creates a rendezvousIP file based on the IP address that you specified in the agent-config.yaml configuration file.

   • Optional: Any modifications you made to agent-config.yaml and install-config.yaml configuration files get imported to the Zero Touch Provisioning (ZTP) custom resources.

     Important

     The Agent-based Installer uses Red Hat Enterprise Linux CoreOS (RHCOS). The rootfs image, which is mentioned in a later step, is required for booting, recovering, and repairing your operating system.

8. Disconnected environments only: Upload the rootfs image to a web server.

   1. Go to the ./<installation_directory>/boot-artifacts directory that was generated when you created the minimal ISO image.

   2. Use your preferred web server, such as any Hypertext Transfer Protocol daemon (httpd), to upload the rootfs image to the location specified in the bootArtifactsBaseURL parameter of the agent-config.yaml file.

      For example, if the bootArtifactsBaseURL parameter states http://192.168.122.20, you would upload the generated rootfs image to this location so that the Agent-based installer can access the image from http://192.168.122.20/agent.x86_64-rootfs.img. After the Agent-based installer boots the minimal ISO for the external platform, the Agent-based Installer downloads the rootfs image from the http://192.168.122.20/agent.x86_64-rootfs.img location into the system memory.

      Note

      The Agent-based Installer also adds the value of the bootArtifactsBaseURL to the minimal ISO Image’s configuration, so that when the Operator boots a cluster’s node, the Agent-based Installer downloads the rootfs image into system memory.

      Important

      Consider that the full ISO image, which is in excess of 1 GB, includes the rootfs image. The image is larger than the minimal ISO Image, which is typically less than 150 MB.

Procedure

1. Set the following registry URLs for your firewall’s allowlist:

   URL	Port	Function
   registry.redhat.io	443	Provides core container images
   access.redhat.com	443	Hosts a signature store that a container client requires for verifying images pulled from registry.access.redhat.com. In a firewall environment, ensure that this resource is on the allowlist.
   registry.access.redhat.com	443	Hosts all the container images that are stored on the Red Hat Ecosystem Catalog, including core container images.
   quay.io	443	Provides core container images
   cdn.quay.io	443	Provides core container images
   cdn01.quay.io	443	Provides core container images
   cdn02.quay.io	443	Provides core container images
   cdn03.quay.io	443	Provides core container images
   cdn04.quay.io	443	Provides core container images
   cdn05.quay.io	443	Provides core container images
   cdn06.quay.io	443	Provides core container images
   sso.redhat.com	443	The https://console.redhat.com site uses authentication from sso.redhat.com

   • You can use the wildcards *.quay.io and *.openshiftapps.com instead of cdn.quay.io and cdn0[1-6].quay.io in your allowlist.
   • You can use the wildcard *.access.redhat.com to simplify the configuration and ensure that all subdomains, including registry.access.redhat.com, are allowed.
   • When you add a site, such as quay.io, to your allowlist, do not add a wildcard entry, such as *.quay.io, to your denylist. In most cases, image registries use a content delivery network (CDN) to serve images. If a firewall blocks access, image downloads are denied when the initial download request redirects to a hostname such as cdn01.quay.io.
2. Set your firewall’s allowlist to include any site that provides resources for a language or framework that your builds require.

3. If you do not disable Telemetry, you must grant access to the following URLs to access Red Hat Insights:

   URL	Port	Function
   cert-api.access.redhat.com	443	Required for Telemetry
   api.access.redhat.com	443	Required for Telemetry
   infogw.api.openshift.com	443	Required for Telemetry
   console.redhat.com	443	Required for Telemetry and for insights-operator

4. Set your firewall’s allowlist to include the following registry URLs:

   URL	Port	Function
   api.openshift.com	443	Required both for your cluster token and to check if updates are available for the cluster.
   rhcos.mirror.openshift.com	443	Required to download Red Hat Enterprise Linux CoreOS (RHCOS) images.

5. Set your firewall’s allowlist to include the following external URLs. Each repository URL hosts OCI containers. Consider mirroring images to as few repositories as possible to reduce any performance issues.

   URL	Port	Function
   k8s.gcr.io	port	A Kubernetes registry that hosts container images for a community-based image registry. This image registry is hosted on a custom Google Container Registry (GCR) domain.
   ghcr.io	port	A GitHub image registry where you can store and manage Open Container Initiative images. Requires an access token to publish, install, and delete private, internal, and public packages.
   storage.googleapis.com	443	A source of release image signatures, although the Cluster Version Operator needs only a single functioning source.
   registry.k8s.io	port	Replaces the k8s.gcr.io image registry because the k8s.gcr.io image registry does not support other platforms and vendors.

Procedure

1. Upload the agent ISO image to Oracle’s default Object Storage bucket and import the agent ISO image as a custom image to this bucket. Ensure you that you configure the custom image to boot in Unified Extensible Firmware Interface (UEFI) mode. For more information, see Creating the OpenShift Container Platform ISO Image (Oracle documentation).

2. Create a compute instance from the supplied base image for your cluster topology. See Creating the OpenShift Container Platform cluster on OCI (Oracle documentation).

   Important

   Before you create the compute instance, check that you have enough memory and disk resources for your cluster. Additionally, ensure that at least one compute instance has the same IP address as the address stated under rendezvousIP in the agent-config.yaml file.

1. Create Compute Cloud@Customer resources and services (Oracle).
2. Prepare configuration files for the Agent-based Installer (Red Hat).
3. Generate the agent ISO image (Red Hat).
4. Convert the ISO image to an Oracle Cloud Infrastructure (OCI) image, upload it to an OCI Home Region Bucket, and then import the uploaded image to the Compute Cloud@Customer system (Oracle).
5. Disconnected environments: Prepare a web server that is accessible by OCI instances (Red Hat).
6. Disconnected environments: Upload the rootfs image to the web server (Red Hat).
7. Configure your firewall for OpenShift Container Platform (Red Hat).
8. Create control plane nodes and configure load balancers (Oracle).
9. Create compute nodes and configure load balancers (Oracle).
10. Verify that your cluster runs on OCI (Oracle).

Procedure

1. Create an installation directory to store configuration files in by running the following command:

   $ mkdir ~/<directory_name>

2. Configure the install-config.yaml configuration file to meet the needs of your organization and save the file in the directory you created.

   install-config.yaml file that sets an external platform

   # install-config.yaml
   apiVersion: v1
   baseDomain: <base_domain> 1
   networking:
     clusterNetwork:
     - cidr: 10.128.0.0/14
       hostPrefix: 23
     network type: OVNKubernetes
     machineNetwork:
     - cidr: <ip_address_from_cidr> 2
     serviceNetwork:
     - 172.30.0.0/16
   compute:
     - architecture: amd64 3
     hyperthreading: Enabled
     name: worker
     replicas: 0
   controlPlane:
     architecture: amd64 4
     hyperthreading: Enabled
     name: master
     replicas: 3
   platform:
      external:
       platformName: oci 5
       cloudControllerManager: External
   sshKey: <public_ssh_key> 6
   pullSecret: '<pull_secret>' 7
   # ...

   1

   The base domain of your cloud provider.

   2

   The IP address from the virtual cloud network (VCN) that the CIDR allocates to resources and components that operate on your network.

   3 4

   Depending on your infrastructure, you can select either arm64 or amd64.

   5

   Set OCI as the external platform, so that OpenShift Container Platform can integrate with OCI.

   6

   Specify your SSH public key.

   7

   The pull secret that you need for authenticate purposes when downloading container images for OpenShift Container Platform components and services, such as Quay.io. See Install OpenShift Container Platform 4 from the Red Hat Hybrid Cloud Console.

3. Create a directory on your local system named openshift. This must be a subdirectory of the installation directory.

   Important

   Do not move the install-config.yaml or agent-config.yaml configuration files to the openshift directory.

4. Configure the Oracle custom manifest files.

   1. Go to "Prepare the OpenShift Master Images" in OpenShift Cluster Setup with Agent Based Installer on Compute Cloud@Customer (Oracle documentation).
   2. Copy and paste the oci-ccm.yml, oci-csi.yml, and machineconfig-ccm.yml files into your openshift directory.
   3. Edit the oci-ccm.yml and oci-csi.yml files to specify the compartment Oracle® Cloud Identifier (OCID), VCN OCID, subnet OCID from the load balancer, the security lists OCID, and the c3-cert.pem section.

5. Configure the agent-config.yaml configuration file to meet your organization’s requirements.

   Sample agent-config.yaml file for an IPv4 network.

   apiVersion: v1beta1
   metadata:
     name: <cluster_name> 1
     namespace: <cluster_namespace> 2
   rendezvousIP: <ip_address_from_CIDR> 3
   bootArtifactsBaseURL: <server_URL> 4
   # ...

   1

   The cluster name that you specified in your DNS record.

   2

   The namespace of your cluster on OpenShift Container Platform.

   3

   If you use IPv4 as the network IP address format, ensure that you set the rendezvousIP parameter to an IPv4 address that the VCN’s Classless Inter-Domain Routing (CIDR) method allocates on your network. Also ensure that at least one instance from the pool of instances that you booted with the ISO matches the IP address value you set for the rendezvousIP parameter.

   4

   The URL of the server where you want to upload the rootfs image. This parameter is required only for disconnected environments.

6. Generate a minimal ISO image, which excludes the rootfs image, by entering the following command in your installation directory:

   $ ./openshift-install agent create image --log-level debug

   The command also completes the following actions:

   • Creates a subdirectory, ./<installation_directory>/auth directory:, and places kubeadmin-password and kubeconfig files in the subdirectory.
   • Creates a rendezvousIP file based on the IP address that you specified in the agent-config.yaml configuration file.

   • Optional: Any modifications you made to agent-config.yaml and install-config.yaml configuration files get imported to the Zero Touch Provisioning (ZTP) custom resources.

     Important

     The Agent-based Installer uses Red Hat Enterprise Linux CoreOS (RHCOS). The rootfs image, which is mentioned in a later step, is required for booting, recovering, and repairing your operating system.

7. Disconnected environments only: Upload the rootfs image to a web server.

   1. Go to the ./<installation_directory>/boot-artifacts directory that was generated when you created the minimal ISO image.

   2. Use your preferred web server, such as any Hypertext Transfer Protocol daemon (httpd), to upload the rootfs image to the location specified in the bootArtifactsBaseURL parameter of the agent-config.yaml file.

      For example, if the bootArtifactsBaseURL parameter states http://192.168.122.20, you would upload the generated rootfs image to this location so that the Agent-based installer can access the image from http://192.168.122.20/agent.x86_64-rootfs.img. After the Agent-based installer boots the minimal ISO for the external platform, the Agent-based Installer downloads the rootfs image from the http://192.168.122.20/agent.x86_64-rootfs.img location into the system memory.

      Note

      The Agent-based Installer also adds the value of the bootArtifactsBaseURL to the minimal ISO Image’s configuration, so that when the Operator boots a cluster’s node, the Agent-based Installer downloads the rootfs image into system memory.

      Important

      Consider that the full ISO image, which is in excess of 1 GB, includes the rootfs image. The image is larger than the minimal ISO Image, which is typically less than 150 MB.

Procedure

1. Set the following registry URLs for your firewall’s allowlist:

   URL	Port	Function
   registry.redhat.io	443	Provides core container images
   access.redhat.com	443	Hosts a signature store that a container client requires for verifying images pulled from registry.access.redhat.com. In a firewall environment, ensure that this resource is on the allowlist.
   registry.access.redhat.com	443	Hosts all the container images that are stored on the Red Hat Ecosystem Catalog, including core container images.
   quay.io	443	Provides core container images
   cdn.quay.io	443	Provides core container images
   cdn01.quay.io	443	Provides core container images
   cdn02.quay.io	443	Provides core container images
   cdn03.quay.io	443	Provides core container images
   cdn04.quay.io	443	Provides core container images
   cdn05.quay.io	443	Provides core container images
   cdn06.quay.io	443	Provides core container images
   sso.redhat.com	443	The https://console.redhat.com site uses authentication from sso.redhat.com

   • You can use the wildcards *.quay.io and *.openshiftapps.com instead of cdn.quay.io and cdn0[1-6].quay.io in your allowlist.
   • You can use the wildcard *.access.redhat.com to simplify the configuration and ensure that all subdomains, including registry.access.redhat.com, are allowed.
   • When you add a site, such as quay.io, to your allowlist, do not add a wildcard entry, such as *.quay.io, to your denylist. In most cases, image registries use a content delivery network (CDN) to serve images. If a firewall blocks access, image downloads are denied when the initial download request redirects to a hostname such as cdn01.quay.io.
2. Set your firewall’s allowlist to include any site that provides resources for a language or framework that your builds require.

3. If you do not disable Telemetry, you must grant access to the following URLs to access Red Hat Insights:

   URL	Port	Function
   cert-api.access.redhat.com	443	Required for Telemetry
   api.access.redhat.com	443	Required for Telemetry
   infogw.api.openshift.com	443	Required for Telemetry
   console.redhat.com	443	Required for Telemetry and for insights-operator

4. If you use Alibaba Cloud, Amazon Web Services (AWS), Microsoft Azure, or Google Cloud Platform (GCP) to host your cluster, you must grant access to the URLs that offer the cloud provider API and DNS for that cloud:

   Cloud	URL	Port	Function
   Alibaba	*.aliyuncs.com	443	Required to access Alibaba Cloud services and resources. Review the Alibaba endpoints_config.go file to find the exact endpoints to allow for the regions that you use.
   AWS	aws.amazon.com	443	Used to install and manage clusters in an AWS environment.
   	*.amazonaws.com Alternatively, if you choose to not use a wildcard for AWS APIs, you must include the following URLs in your allowlist:	443	Required to access AWS services and resources. Review the AWS Service Endpoints in the AWS documentation to find the exact endpoints to allow for the regions that you use.
   	ec2.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	events.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	iam.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	route53.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	*.s3.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	*.s3.<aws_region>.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	*.s3.dualstack.<aws_region>.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	sts.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	sts.<aws_region>.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	tagging.us-east-1.amazonaws.com	443	Used to install and manage clusters in an AWS environment. This endpoint is always us-east-1, regardless of the region the cluster is deployed in.
   	ec2.<aws_region>.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	elasticloadbalancing.<aws_region>.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	servicequotas.<aws_region>.amazonaws.com	443	Required. Used to confirm quotas for deploying the service.
   	tagging.<aws_region>.amazonaws.com	443	Allows the assignment of metadata about AWS resources in the form of tags.
   	*.cloudfront.net	443	Used to provide access to CloudFront. If you use the AWS Security Token Service (STS) and the private S3 bucket, you must provide access to CloudFront.
   GCP	*.googleapis.com	443	Required to access GCP services and resources. Review Cloud Endpoints in the GCP documentation to find the endpoints to allow for your APIs.
   	accounts.google.com	443	Required to access your GCP account.
   Microsoft Azure	management.azure.com	443	Required to access Microsoft Azure services and resources. Review the Microsoft Azure REST API reference in the Microsoft Azure documentation to find the endpoints to allow for your APIs.
   	*.blob.core.windows.net	443	Required to download Ignition files.
   	login.microsoftonline.com	443	Required to access Microsoft Azure services and resources. Review the Azure REST API reference in the Microsoft Azure documentation to find the endpoints to allow for your APIs.

5. Allowlist the following URLs:

   URL	Port	Function
   *.apps.<cluster_name>.<base_domain>	443	Required to access the default cluster routes unless you set an ingress wildcard during installation.
   api.openshift.com	443	Required both for your cluster token and to check if updates are available for the cluster.
   console.redhat.com	443	Required for your cluster token.
   mirror.openshift.com	443	Required to access mirrored installation content and images. This site is also a source of release image signatures, although the Cluster Version Operator needs only a single functioning source.
   quayio-production-s3.s3.amazonaws.com	443	Required to access Quay image content in AWS.
   rhcos.mirror.openshift.com	443	Required to download Red Hat Enterprise Linux CoreOS (RHCOS) images.
   sso.redhat.com	443	The https://console.redhat.com site uses authentication from sso.redhat.com
   storage.googleapis.com/openshift-release	443	A source of release image signatures, although the Cluster Version Operator needs only a single functioning source.

   Operators require route access to perform health checks. Specifically, the authentication and web console Operators connect to two routes to verify that the routes work. If you are the cluster administrator and do not want to allow *.apps.<cluster_name>.<base_domain>, then allow these routes:

   • oauth-openshift.apps.<cluster_name>.<base_domain>
   • canary-openshift-ingress-canary.apps.<cluster_name>.<base_domain>
   • console-openshift-console.apps.<cluster_name>.<base_domain>, or the hostname that is specified in the spec.route.hostname field of the consoles.operator/cluster object if the field is not empty.

6. Allowlist the following URLs for optional third-party content:

   URL	Port	Function
   registry.connect.redhat.com	443	Required for all third-party images and certified operators.
   rhc4tp-prod-z8cxf-image-registry-us-east-1-evenkyleffocxqvofrk.s3.dualstack.us-east-1.amazonaws.com	443	Provides access to container images hosted on registry.connect.redhat.com
   oso-rhc4tp-docker-registry.s3-us-west-2.amazonaws.com	443	Required for Sonatype Nexus, F5 Big IP operators.

7. If you use a default Red Hat Network Time Protocol (NTP) server allow the following URLs:

   • 1.rhel.pool.ntp.org
   • 2.rhel.pool.ntp.org
   • 3.rhel.pool.ntp.org

Procedure

1. Convert the agent ISO image to an OCI image, upload it to an OCI Home Region Bucket, and then import the uploaded image to the Compute Cloud@Customer system. See "Prepare the OpenShift Master Images" in OpenShift Cluster Setup with Agent Based Installer on Compute Cloud@Customer (Oracle documentation) for instructions.
2. Create control plane instances on Compute Cloud@Customer. See "Create control plane instances on C3 and Master Node LB Backend Sets" in OpenShift Cluster Setup with Agent Based Installer on Compute Cloud@Customer (Oracle documentation) for instructions.

3. Create a compute instance from the supplied base image for your cluster topology. See "Add worker nodes" in OpenShift Cluster Setup with Agent Based Installer on Compute Cloud@Customer (Oracle documentation) for instructions.

   Important

   Before you create the compute instance, check that you have enough memory and disk resources for your cluster. Additionally, ensure that at least one compute instance has the same IP address as the address stated under rendezvousIP in the agent-config.yaml file.

1. Create Private Cloud Appliance resources and services (Oracle).
2. Prepare configuration files for the Agent-based Installer (Red Hat).
3. Generate the agent ISO image (Red Hat).
4. Convert the ISO image to an Oracle Cloud Infrastructure (OCI) image, upload it to an OCI Home Region Bucket, and then import the uploaded image to the Private Cloud Appliance system (Oracle).
5. Disconnected environments: Prepare a web server that is accessible by OCI instances (Red Hat).
6. Disconnected environments: Upload the rootfs image to the web server (Red Hat).
7. Configure your firewall for OpenShift Container Platform (Red Hat).
8. Create control plane nodes and configure load balancers (Oracle).
9. Create compute nodes and configure load balancers (Oracle).
10. Verify that your cluster runs on OCI (Oracle).

Procedure

1. Create an installation directory to store configuration files in by running the following command:

   $ mkdir ~/<directory_name>

2. Configure the install-config.yaml configuration file to meet the needs of your organization and save the file in the directory you created.

   install-config.yaml file that sets an external platform

   # install-config.yaml
   apiVersion: v1
   baseDomain: <base_domain> 1
   networking:
     clusterNetwork:
     - cidr: 10.128.0.0/14
       hostPrefix: 23
     network type: OVNKubernetes
     machineNetwork:
     - cidr: <ip_address_from_cidr> 2
     serviceNetwork:
     - 172.30.0.0/16
   compute:
     - architecture: amd64 3
     hyperthreading: Enabled
     name: worker
     replicas: 0
   controlPlane:
     architecture: amd64 4
     hyperthreading: Enabled
     name: master
     replicas: 3
   platform:
      external:
       platformName: oci 5
       cloudControllerManager: External
   sshKey: <public_ssh_key> 6
   pullSecret: '<pull_secret>' 7
   # ...

   1

   The base domain of your cloud provider.

   2

   The IP address from the virtual cloud network (VCN) that the CIDR allocates to resources and components that operate on your network.

   3 4

   Depending on your infrastructure, you can select either arm64 or amd64.

   5

   Set OCI as the external platform, so that OpenShift Container Platform can integrate with OCI.

   6

   Specify your SSH public key.

   7

   The pull secret that you need for authenticate purposes when downloading container images for OpenShift Container Platform components and services, such as Quay.io. See Install OpenShift Container Platform 4 from the Red Hat Hybrid Cloud Console.

3. Create a directory on your local system named openshift. This must be a subdirectory of the installation directory.

   Important

   Do not move the install-config.yaml or agent-config.yaml configuration files to the openshift directory.

4. Configure the Oracle custom manifest files.

   1. Go to "Prepare the OpenShift Master Images" in OpenShift Cluster Setup with Agent Based Installer on Private Cloud Appliance (Oracle documentation).
   2. Copy and paste the oci-ccm.yml, oci-csi.yml, and machineconfig-ccm.yml files into your openshift directory.
   3. Edit the oci-ccm.yml and oci-csi.yml files to specify the compartment Oracle® Cloud Identifier (OCID), VCN OCID, subnet OCID from the load balancer, the security lists OCID, and the c3-cert.pem section.

5. Configure the agent-config.yaml configuration file to meet your organization’s requirements.

   Sample agent-config.yaml file for an IPv4 network.

   apiVersion: v1beta1
   metadata:
     name: <cluster_name> 1
     namespace: <cluster_namespace> 2
   rendezvousIP: <ip_address_from_CIDR> 3
   bootArtifactsBaseURL: <server_URL> 4
   # ...

   1

   The cluster name that you specified in your DNS record.

   2

   The namespace of your cluster on OpenShift Container Platform.

   3

   If you use IPv4 as the network IP address format, ensure that you set the rendezvousIP parameter to an IPv4 address that the VCN’s Classless Inter-Domain Routing (CIDR) method allocates on your network. Also ensure that at least one instance from the pool of instances that you booted with the ISO matches the IP address value you set for the rendezvousIP parameter.

   4

   The URL of the server where you want to upload the rootfs image. This parameter is required only for disconnected environments.

6. Generate a minimal ISO image, which excludes the rootfs image, by entering the following command in your installation directory:

   $ ./openshift-install agent create image --log-level debug

   The command also completes the following actions:

   • Creates a subdirectory, ./<installation_directory>/auth directory:, and places kubeadmin-password and kubeconfig files in the subdirectory.
   • Creates a rendezvousIP file based on the IP address that you specified in the agent-config.yaml configuration file.

   • Optional: Any modifications you made to agent-config.yaml and install-config.yaml configuration files get imported to the Zero Touch Provisioning (ZTP) custom resources.

     Important

     The Agent-based Installer uses Red Hat Enterprise Linux CoreOS (RHCOS). The rootfs image, which is mentioned in a later step, is required for booting, recovering, and repairing your operating system.

7. Disconnected environments only: Upload the rootfs image to a web server.

   1. Go to the ./<installation_directory>/boot-artifacts directory that was generated when you created the minimal ISO image.

   2. Use your preferred web server, such as any Hypertext Transfer Protocol daemon (httpd), to upload the rootfs image to the location specified in the bootArtifactsBaseURL parameter of the agent-config.yaml file.

      For example, if the bootArtifactsBaseURL parameter states http://192.168.122.20, you would upload the generated rootfs image to this location so that the Agent-based installer can access the image from http://192.168.122.20/agent.x86_64-rootfs.img. After the Agent-based installer boots the minimal ISO for the external platform, the Agent-based Installer downloads the rootfs image from the http://192.168.122.20/agent.x86_64-rootfs.img location into the system memory.

      Note

      The Agent-based Installer also adds the value of the bootArtifactsBaseURL to the minimal ISO Image’s configuration, so that when the Operator boots a cluster’s node, the Agent-based Installer downloads the rootfs image into system memory.

      Important

      Consider that the full ISO image, which is in excess of 1 GB, includes the rootfs image. The image is larger than the minimal ISO Image, which is typically less than 150 MB.

Procedure

1. Set the following registry URLs for your firewall’s allowlist:

   URL	Port	Function
   registry.redhat.io	443	Provides core container images
   access.redhat.com	443	Hosts a signature store that a container client requires for verifying images pulled from registry.access.redhat.com. In a firewall environment, ensure that this resource is on the allowlist.
   registry.access.redhat.com	443	Hosts all the container images that are stored on the Red Hat Ecosystem Catalog, including core container images.
   quay.io	443	Provides core container images
   cdn.quay.io	443	Provides core container images
   cdn01.quay.io	443	Provides core container images
   cdn02.quay.io	443	Provides core container images
   cdn03.quay.io	443	Provides core container images
   cdn04.quay.io	443	Provides core container images
   cdn05.quay.io	443	Provides core container images
   cdn06.quay.io	443	Provides core container images
   sso.redhat.com	443	The https://console.redhat.com site uses authentication from sso.redhat.com

   • You can use the wildcards *.quay.io and *.openshiftapps.com instead of cdn.quay.io and cdn0[1-6].quay.io in your allowlist.
   • You can use the wildcard *.access.redhat.com to simplify the configuration and ensure that all subdomains, including registry.access.redhat.com, are allowed.
   • When you add a site, such as quay.io, to your allowlist, do not add a wildcard entry, such as *.quay.io, to your denylist. In most cases, image registries use a content delivery network (CDN) to serve images. If a firewall blocks access, image downloads are denied when the initial download request redirects to a hostname such as cdn01.quay.io.
2. Set your firewall’s allowlist to include any site that provides resources for a language or framework that your builds require.

3. If you do not disable Telemetry, you must grant access to the following URLs to access Red Hat Insights:

   URL	Port	Function
   cert-api.access.redhat.com	443	Required for Telemetry
   api.access.redhat.com	443	Required for Telemetry
   infogw.api.openshift.com	443	Required for Telemetry
   console.redhat.com	443	Required for Telemetry and for insights-operator

4. If you use Alibaba Cloud, Amazon Web Services (AWS), Microsoft Azure, or Google Cloud Platform (GCP) to host your cluster, you must grant access to the URLs that offer the cloud provider API and DNS for that cloud:

   Cloud	URL	Port	Function
   Alibaba	*.aliyuncs.com	443	Required to access Alibaba Cloud services and resources. Review the Alibaba endpoints_config.go file to find the exact endpoints to allow for the regions that you use.
   AWS	aws.amazon.com	443	Used to install and manage clusters in an AWS environment.
   	*.amazonaws.com Alternatively, if you choose to not use a wildcard for AWS APIs, you must include the following URLs in your allowlist:	443	Required to access AWS services and resources. Review the AWS Service Endpoints in the AWS documentation to find the exact endpoints to allow for the regions that you use.
   	ec2.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	events.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	iam.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	route53.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	*.s3.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	*.s3.<aws_region>.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	*.s3.dualstack.<aws_region>.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	sts.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	sts.<aws_region>.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	tagging.us-east-1.amazonaws.com	443	Used to install and manage clusters in an AWS environment. This endpoint is always us-east-1, regardless of the region the cluster is deployed in.
   	ec2.<aws_region>.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	elasticloadbalancing.<aws_region>.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	servicequotas.<aws_region>.amazonaws.com	443	Required. Used to confirm quotas for deploying the service.
   	tagging.<aws_region>.amazonaws.com	443	Allows the assignment of metadata about AWS resources in the form of tags.
   	*.cloudfront.net	443	Used to provide access to CloudFront. If you use the AWS Security Token Service (STS) and the private S3 bucket, you must provide access to CloudFront.
   GCP	*.googleapis.com	443	Required to access GCP services and resources. Review Cloud Endpoints in the GCP documentation to find the endpoints to allow for your APIs.
   	accounts.google.com	443	Required to access your GCP account.
   Microsoft Azure	management.azure.com	443	Required to access Microsoft Azure services and resources. Review the Microsoft Azure REST API reference in the Microsoft Azure documentation to find the endpoints to allow for your APIs.
   	*.blob.core.windows.net	443	Required to download Ignition files.
   	login.microsoftonline.com	443	Required to access Microsoft Azure services and resources. Review the Azure REST API reference in the Microsoft Azure documentation to find the endpoints to allow for your APIs.

5. Allowlist the following URLs:

   URL	Port	Function
   *.apps.<cluster_name>.<base_domain>	443	Required to access the default cluster routes unless you set an ingress wildcard during installation.
   api.openshift.com	443	Required both for your cluster token and to check if updates are available for the cluster.
   console.redhat.com	443	Required for your cluster token.
   mirror.openshift.com	443	Required to access mirrored installation content and images. This site is also a source of release image signatures, although the Cluster Version Operator needs only a single functioning source.
   quayio-production-s3.s3.amazonaws.com	443	Required to access Quay image content in AWS.
   rhcos.mirror.openshift.com	443	Required to download Red Hat Enterprise Linux CoreOS (RHCOS) images.
   sso.redhat.com	443	The https://console.redhat.com site uses authentication from sso.redhat.com
   storage.googleapis.com/openshift-release	443	A source of release image signatures, although the Cluster Version Operator needs only a single functioning source.

   Operators require route access to perform health checks. Specifically, the authentication and web console Operators connect to two routes to verify that the routes work. If you are the cluster administrator and do not want to allow *.apps.<cluster_name>.<base_domain>, then allow these routes:

   • oauth-openshift.apps.<cluster_name>.<base_domain>
   • canary-openshift-ingress-canary.apps.<cluster_name>.<base_domain>
   • console-openshift-console.apps.<cluster_name>.<base_domain>, or the hostname that is specified in the spec.route.hostname field of the consoles.operator/cluster object if the field is not empty.

6. Allowlist the following URLs for optional third-party content:

   URL	Port	Function
   registry.connect.redhat.com	443	Required for all third-party images and certified operators.
   rhc4tp-prod-z8cxf-image-registry-us-east-1-evenkyleffocxqvofrk.s3.dualstack.us-east-1.amazonaws.com	443	Provides access to container images hosted on registry.connect.redhat.com
   oso-rhc4tp-docker-registry.s3-us-west-2.amazonaws.com	443	Required for Sonatype Nexus, F5 Big IP operators.

7. If you use a default Red Hat Network Time Protocol (NTP) server allow the following URLs:

   • 1.rhel.pool.ntp.org
   • 2.rhel.pool.ntp.org
   • 3.rhel.pool.ntp.org

Procedure

1. Convert the agent ISO image to an OCI image, upload it to an OCI Home Region Bucket, and then import the uploaded image to the Private Cloud Appliance system. See "Prepare the OpenShift Master Images" in OpenShift Cluster Setup with Agent Based Installer on Private Cloud Appliance (Oracle documentation) for instructions.
2. Create control plane instances on Private Cloud Appliance. See "Create control plane instances on PCA and Master Node LB Backend Sets" in OpenShift Cluster Setup with Agent Based Installer on Private Cloud Appliance (Oracle documentation) for instructions.

3. Create a compute instance from the supplied base image for your cluster topology. See "Add worker nodes" in OpenShift Cluster Setup with Agent Based Installer on Private Cloud Appliance (Oracle documentation) for instructions.

   Important

   Before you create the compute instance, check that you have enough memory and disk resources for your cluster. Additionally, ensure that at least one compute instance has the same IP address as the address stated under rendezvousIP in the agent-config.yaml file.

1. Preparing the Compute Cloud@Customer bastion server.
2. Running the Terraform script via the Home region.
3. Preparing the OpenShift Container Platform image for Oracle Cloud Infrastructure (OCI).
4. Running the Terraform script via the Compute Cloud@Customer region.
5. Installing the cluster by using the Assisted Installer web console.

Procedure

1. Install the bastion server. For details, see the "Bastion Installation" section in the Oracle documentation.
2. Install the Terraform application which is used to run the Terraform script. For details, see the "Terraform Installation" section in the Oracle documentation.
3. Install and configure the OCI command line interface (CLI). For details, see the "Installing and Configuring the OCI CLI" section in the Oracle documentation.

Procedure

1. Connect to the bastion server via SSH.
2. Create OpenShift\createResourceOnHomeRegion folders.
3. Copy the createInfraResources.tf and terraform.tfvars files from the C3_PCA GitHub repository into the createResourceOnHomeRegion folder.
4. Ensure that you have access to the source environment, and that your C3 certificate has been exported.
5. Run the createInfraResources.tf Terraform script.

Procedure

1. Connect to the bastion server via SSH.
2. Set the C3 Certificate location and export the certificate.
3. Run the terraform.tfvars script to create three control plane nodes and three compute nodes.
4. Update the labels for the control plane and compute nodes.
5. Stop and restart the instances one by one on the Compute Cloud@Customer portal.