Chapter 3. Installing a cluster on Oracle Compute Cloud@Customer by using the Agent-based Installer

1. Create Compute Cloud@Customer resources and services (Oracle).
2. Prepare configuration files for the Agent-based Installer (Red Hat).
3. Generate the agent ISO image (Red Hat).
4. Convert the ISO image to an Oracle Cloud Infrastructure (OCI) image, upload it to an OCI Home Region Bucket, and then import the uploaded image to the Compute Cloud@Customer system (Oracle).
5. Disconnected environments: Prepare a web server that is accessible by OCI instances (Red Hat).
6. Disconnected environments: Upload the rootfs image to the web server (Red Hat).
7. Configure your firewall for OpenShift Container Platform (Red Hat).
8. Create control plane nodes and configure load balancers (Oracle).
9. Create compute nodes and configure load balancers (Oracle).
10. Verify that your cluster runs on OCI (Oracle).

Procedure

1. Create an installation directory to store configuration files in by running the following command:

   $ mkdir ~/<directory_name>

2. Configure the install-config.yaml configuration file to meet the needs of your organization and save the file in the directory you created.

   install-config.yaml file that sets an external platform

   # install-config.yaml
   apiVersion: v1
   baseDomain: <base_domain> 1
   networking:
     clusterNetwork:
     - cidr: 10.128.0.0/14
       hostPrefix: 23
     network type: OVNKubernetes
     machineNetwork:
     - cidr: <ip_address_from_cidr> 2
     serviceNetwork:
     - 172.30.0.0/16
   compute:
     - architecture: amd64 3
     hyperthreading: Enabled
     name: worker
     replicas: 0
   controlPlane:
     architecture: amd64 4
     hyperthreading: Enabled
     name: master
     replicas: 3
   platform:
      external:
       platformName: oci 5
       cloudControllerManager: External
   sshKey: <public_ssh_key> 6
   pullSecret: '<pull_secret>' 7
   # ...

   1

   The base domain of your cloud provider.

   2

   The IP address from the virtual cloud network (VCN) that the CIDR allocates to resources and components that operate on your network.

   3 4

   Depending on your infrastructure, you can select either arm64 or amd64.

   5

   Set OCI as the external platform, so that OpenShift Container Platform can integrate with OCI.

   6

   Specify your SSH public key.

   7

   The pull secret that you need for authenticate purposes when downloading container images for OpenShift Container Platform components and services, such as Quay.io. See Install OpenShift Container Platform 4 from the Red Hat Hybrid Cloud Console.

3. Create a directory on your local system named openshift. This must be a subdirectory of the installation directory.

   Important

   Do not move the install-config.yaml or agent-config.yaml configuration files to the openshift directory.

4. Configure the Oracle custom manifest files.

   1. Go to "Prepare the OpenShift Master Images" in OpenShift Cluster Setup with Agent Based Installer on Compute Cloud@Customer (Oracle documentation).
   2. Copy and paste the oci-ccm.yml, oci-csi.yml, and machineconfig-ccm.yml files into your openshift directory.
   3. Edit the oci-ccm.yml and oci-csi.yml files to specify the compartment Oracle® Cloud Identifier (OCID), VCN OCID, subnet OCID from the load balancer, the security lists OCID, and the c3-cert.pem section.

5. Configure the agent-config.yaml configuration file to meet your organization’s requirements.

   Sample agent-config.yaml file for an IPv4 network.

   apiVersion: v1beta1
   metadata:
     name: <cluster_name> 1
     namespace: <cluster_namespace> 2
   rendezvousIP: <ip_address_from_CIDR> 3
   bootArtifactsBaseURL: <server_URL> 4
   # ...

   1

   The cluster name that you specified in your DNS record.

   2

   The namespace of your cluster on OpenShift Container Platform.

   3

   If you use IPv4 as the network IP address format, ensure that you set the rendezvousIP parameter to an IPv4 address that the VCN’s Classless Inter-Domain Routing (CIDR) method allocates on your network. Also ensure that at least one instance from the pool of instances that you booted with the ISO matches the IP address value you set for the rendezvousIP parameter.

   4

   The URL of the server where you want to upload the rootfs image. This parameter is required only for disconnected environments.

6. Generate a minimal ISO image, which excludes the rootfs image, by entering the following command in your installation directory:

   $ ./openshift-install agent create image --log-level debug

   The command also completes the following actions:

   • Creates a subdirectory, ./<installation_directory>/auth directory:, and places kubeadmin-password and kubeconfig files in the subdirectory.
   • Creates a rendezvousIP file based on the IP address that you specified in the agent-config.yaml configuration file.

   • Optional: Any modifications you made to agent-config.yaml and install-config.yaml configuration files get imported to the Zero Touch Provisioning (ZTP) custom resources.

     Important

     The Agent-based Installer uses Red Hat Enterprise Linux CoreOS (RHCOS). The rootfs image, which is mentioned in a later step, is required for booting, recovering, and repairing your operating system.

7. Disconnected environments only: Upload the rootfs image to a web server.

   1. Go to the ./<installation_directory>/boot-artifacts directory that was generated when you created the minimal ISO image.

   2. Use your preferred web server, such as any Hypertext Transfer Protocol daemon (httpd), to upload the rootfs image to the location specified in the bootArtifactsBaseURL parameter of the agent-config.yaml file.

      For example, if the bootArtifactsBaseURL parameter states http://192.168.122.20, you would upload the generated rootfs image to this location so that the Agent-based installer can access the image from http://192.168.122.20/agent.x86_64-rootfs.img. After the Agent-based installer boots the minimal ISO for the external platform, the Agent-based Installer downloads the rootfs image from the http://192.168.122.20/agent.x86_64-rootfs.img location into the system memory.

      Note

      The Agent-based Installer also adds the value of the bootArtifactsBaseURL to the minimal ISO Image’s configuration, so that when the Operator boots a cluster’s node, the Agent-based Installer downloads the rootfs image into system memory.

      Important

      Consider that the full ISO image, which is in excess of 1 GB, includes the rootfs image. The image is larger than the minimal ISO Image, which is typically less than 150 MB.

Procedure

1. Set the following registry URLs for your firewall’s allowlist:

   URL	Port	Function
   registry.redhat.io	443	Provides core container images
   access.redhat.com	443	Hosts a signature store that a container client requires for verifying images pulled from registry.access.redhat.com. In a firewall environment, ensure that this resource is on the allowlist.
   registry.access.redhat.com	443	Hosts all the container images that are stored on the Red Hat Ecosystem Catalog, including core container images.
   quay.io	443	Provides core container images
   cdn.quay.io	443	Provides core container images
   cdn01.quay.io	443	Provides core container images
   cdn02.quay.io	443	Provides core container images
   cdn03.quay.io	443	Provides core container images
   cdn04.quay.io	443	Provides core container images
   cdn05.quay.io	443	Provides core container images
   cdn06.quay.io	443	Provides core container images
   sso.redhat.com	443	The https://console.redhat.com site uses authentication from sso.redhat.com

   • You can use the wildcards *.quay.io and *.openshiftapps.com instead of cdn.quay.io and cdn0[1-6].quay.io in your allowlist.
   • You can use the wildcard *.access.redhat.com to simplify the configuration and ensure that all subdomains, including registry.access.redhat.com, are allowed.
   • When you add a site, such as quay.io, to your allowlist, do not add a wildcard entry, such as *.quay.io, to your denylist. In most cases, image registries use a content delivery network (CDN) to serve images. If a firewall blocks access, image downloads are denied when the initial download request redirects to a hostname such as cdn01.quay.io.
2. Set your firewall’s allowlist to include any site that provides resources for a language or framework that your builds require.

3. If you do not disable Telemetry, you must grant access to the following URLs to access Red Hat Insights:

   URL	Port	Function
   cert-api.access.redhat.com	443	Required for Telemetry
   api.access.redhat.com	443	Required for Telemetry
   infogw.api.openshift.com	443	Required for Telemetry
   console.redhat.com	443	Required for Telemetry and for insights-operator

4. If you use Alibaba Cloud, Amazon Web Services (AWS), Microsoft Azure, or Google Cloud Platform (GCP) to host your cluster, you must grant access to the URLs that offer the cloud provider API and DNS for that cloud:

   Cloud	URL	Port	Function
   Alibaba	*.aliyuncs.com	443	Required to access Alibaba Cloud services and resources. Review the Alibaba endpoints_config.go file to find the exact endpoints to allow for the regions that you use.
   AWS	aws.amazon.com	443	Used to install and manage clusters in an AWS environment.
   	*.amazonaws.com Alternatively, if you choose to not use a wildcard for AWS APIs, you must include the following URLs in your allowlist:	443	Required to access AWS services and resources. Review the AWS Service Endpoints in the AWS documentation to find the exact endpoints to allow for the regions that you use.
   	ec2.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	events.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	iam.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	route53.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	*.s3.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	*.s3.<aws_region>.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	*.s3.dualstack.<aws_region>.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	sts.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	sts.<aws_region>.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	tagging.us-east-1.amazonaws.com	443	Used to install and manage clusters in an AWS environment. This endpoint is always us-east-1, regardless of the region the cluster is deployed in.
   	ec2.<aws_region>.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	elasticloadbalancing.<aws_region>.amazonaws.com	443	Used to install and manage clusters in an AWS environment.
   	servicequotas.<aws_region>.amazonaws.com	443	Required. Used to confirm quotas for deploying the service.
   	tagging.<aws_region>.amazonaws.com	443	Allows the assignment of metadata about AWS resources in the form of tags.
   	*.cloudfront.net	443	Used to provide access to CloudFront. If you use the AWS Security Token Service (STS) and the private S3 bucket, you must provide access to CloudFront.
   GCP	*.googleapis.com	443	Required to access GCP services and resources. Review Cloud Endpoints in the GCP documentation to find the endpoints to allow for your APIs.
   	accounts.google.com	443	Required to access your GCP account.
   Microsoft Azure	management.azure.com	443	Required to access Microsoft Azure services and resources. Review the Microsoft Azure REST API reference in the Microsoft Azure documentation to find the endpoints to allow for your APIs.
   	*.blob.core.windows.net	443	Required to download Ignition files.
   	login.microsoftonline.com	443	Required to access Microsoft Azure services and resources. Review the Azure REST API reference in the Microsoft Azure documentation to find the endpoints to allow for your APIs.

5. Allowlist the following URLs:

   URL	Port	Function
   *.apps.<cluster_name>.<base_domain>	443	Required to access the default cluster routes unless you set an ingress wildcard during installation.
   api.openshift.com	443	Required both for your cluster token and to check if updates are available for the cluster.
   console.redhat.com	443	Required for your cluster token.
   mirror.openshift.com	443	Required to access mirrored installation content and images. This site is also a source of release image signatures, although the Cluster Version Operator needs only a single functioning source.
   quayio-production-s3.s3.amazonaws.com	443	Required to access Quay image content in AWS.
   rhcos.mirror.openshift.com	443	Required to download Red Hat Enterprise Linux CoreOS (RHCOS) images.
   sso.redhat.com	443	The https://console.redhat.com site uses authentication from sso.redhat.com
   storage.googleapis.com/openshift-release	443	A source of release image signatures, although the Cluster Version Operator needs only a single functioning source.

   Operators require route access to perform health checks. Specifically, the authentication and web console Operators connect to two routes to verify that the routes work. If you are the cluster administrator and do not want to allow *.apps.<cluster_name>.<base_domain>, then allow these routes:

   • oauth-openshift.apps.<cluster_name>.<base_domain>
   • canary-openshift-ingress-canary.apps.<cluster_name>.<base_domain>
   • console-openshift-console.apps.<cluster_name>.<base_domain>, or the hostname that is specified in the spec.route.hostname field of the consoles.operator/cluster object if the field is not empty.

6. Allowlist the following URLs for optional third-party content:

   URL	Port	Function
   registry.connect.redhat.com	443	Required for all third-party images and certified operators.
   rhc4tp-prod-z8cxf-image-registry-us-east-1-evenkyleffocxqvofrk.s3.dualstack.us-east-1.amazonaws.com	443	Provides access to container images hosted on registry.connect.redhat.com
   oso-rhc4tp-docker-registry.s3-us-west-2.amazonaws.com	443	Required for Sonatype Nexus, F5 Big IP operators.

7. If you use a default Red Hat Network Time Protocol (NTP) server allow the following URLs:

   • 1.rhel.pool.ntp.org
   • 2.rhel.pool.ntp.org
   • 3.rhel.pool.ntp.org

Procedure

1. Convert the agent ISO image to an OCI image, upload it to an OCI Home Region Bucket, and then import the uploaded image to the Compute Cloud@Customer system. See "Prepare the OpenShift Master Images" in OpenShift Cluster Setup with Agent Based Installer on Compute Cloud@Customer (Oracle documentation) for instructions.
2. Create control plane instances on Compute Cloud@Customer. See "Create control plane instances on C3 and Master Node LB Backend Sets" in OpenShift Cluster Setup with Agent Based Installer on Compute Cloud@Customer (Oracle documentation) for instructions.

3. Create a compute instance from the supplied base image for your cluster topology. See "Add worker nodes" in OpenShift Cluster Setup with Agent Based Installer on Compute Cloud@Customer (Oracle documentation) for instructions.

   Important

   Before you create the compute instance, check that you have enough memory and disk resources for your cluster. Additionally, ensure that at least one compute instance has the same IP address as the address stated under rendezvousIP in the agent-config.yaml file.