Chapter 3. Installation configuration

Procedure

1. Generate the Kubernetes manifests for the cluster:

   $ ./openshift-install create manifests --dir=<installation_directory>

2. Decide if you want to add kernel arguments to worker or master nodes.

3. In the openshift directory, create a file (for example, 99-openshift-machineconfig-master-kargs.yaml) to define a MachineConfig object to add the kernel settings. This example adds a loglevel=7 kernel argument to master nodes:

   $ cat << EOF > 99-openshift-machineconfig-master-kargs.yaml
   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: master
     name: 99-openshift-machineconfig-master-kargs
   spec:
     kernelArguments:
       - 'loglevel=7'
   EOF

   You can change master to worker to add kernel arguments to worker nodes instead. Create a separate YAML file to add to both master and worker nodes.

Procedure

1. Create the contents of the chrony.conf file and encode it as base64. For example:

   $ cat << EOF | base64
       server clock.redhat.com iburst
       driftfile /var/lib/chrony/drift
       makestep 1.0 3
       rtcsync
       logdir /var/log/chrony
       EOF
   
   ICAgIHNlcnZlciBjbG9jay5yZWRoYXQuY29tIGlidXJzdAogICAgZHJpZnRmaWxlIC92YXIvbGli
   L2Nocm9ueS9kcmlmdAogICAgbWFrZXN0ZXAgMS4wIDMKICAgIHJ0Y3N5bmMKICAgIGxvZ2RpciAv
   dmFyL2xvZy9jaHJvbnkK

2. Create the MachineConfig object file, replacing the base64 string with the one you just created yourself. This example adds the file to master nodes. You can change it to worker or make an additional MachineConfig object for the worker role:

   $ cat << EOF > ./masters-chrony-configuration.yaml
   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     labels:
       machineconfiguration.openshift.io/role: master
     name: masters-chrony-configuration
   spec:
     config:
       ignition:
         config: {}
         security:
           tls: {}
         timeouts: {}
         version: 2.2.0
       networkd: {}
       passwd: {}
       storage:
         files:
         - contents:
             source: data:text/plain;charset=utf-8;base64,ICAgIHNlcnZlciBjbG9jay5yZWRoYXQuY29tIGlidXJzdAogICAgZHJpZnRmaWxlIC92YXIvbGliL2Nocm9ueS9kcmlmdAogICAgbWFrZXN0ZXAgMS4wIDMKICAgIHJ0Y3N5bmMKICAgIGxvZ2RpciAvdmFyL2xvZy9jaHJvbnkK
             verification: {}
           filesystem: root
           mode: 420
           path: /etc/chrony.conf
     osImageURL: ""
   EOF

3. Make a backup copy of the configuration file.
4. If the cluster is not up yet, generate manifest files, add this file to the openshift directory, then continue to create the cluster.

5. If the cluster is already running, apply the file as follows:

    $ oc apply -f ./masters-chrony-configuration.yaml

1. Download your registry.redhat.io pull secret from the Pull Secret page on the Red Hat OpenShift Cluster Manager site and save it to a .json file.

2. Generate the base64-encoded user name and password or token for your mirror registry:

   $ echo -n '<user_name>:<password>' | base64 -w0 1
   BGVtbYk3ZHAtqXs=

   1

   For <user_name> and <password>, specify the user name and password that you configured for your registry.

3. Make a copy of your pull secret in JSON format:

   $ cat ./pull-secret.text | jq .  > <path>/<pull-secret-file>1

   1

   Specify the path to the folder to store the pull secret in and a name for the JSON file that you create.

   The contents of the file resemble the following example:

   {
     "auths": {
       "cloud.openshift.com": {
         "auth": "b3BlbnNo...",
         "email": "you@example.com"
       },
       "quay.io": {
         "auth": "b3BlbnNo...",
         "email": "you@example.com"
       },
       "registry.connect.redhat.com": {
         "auth": "NTE3Njg5Nj...",
         "email": "you@example.com"
       },
       "registry.redhat.io": {
         "auth": "NTE3Njg5Nj...",
         "email": "you@example.com"
       }
     }
   }

4. Edit the new file and add a section that describes your registry to it:

     "auths": {
       "<mirror_registry>": { 1
         "auth": "<credentials>", 2
         "email": "you@example.com"
     },

   1

   For <mirror_registry>, specify the registry domain name, and optionally the port, that your mirror registry uses to serve content. For example, registry.example.com or registry.example.com:5000

   2

   For <credentials>, specify the base64-encoded user name and password for the mirror registry.

   The file resembles the following example:

   {
     "auths": {
       "<mirror_registry>": {
         "auth": "<credentials>",
         "email": "you@example.com"
       },
       "cloud.openshift.com": {
         "auth": "b3BlbnNo...",
         "email": "you@example.com"
       },
       "quay.io": {
         "auth": "b3BlbnNo...",
         "email": "you@example.com"
       },
       "registry.connect.redhat.com": {
         "auth": "NTE3Njg5Nj...",
         "email": "you@example.com"
       },
       "registry.redhat.io": {
         "auth": "NTE3Njg5Nj...",
         "email": "you@example.com"
       }
     }
   }

1. Review the OpenShift Container Platform downloads page to determine the version of OpenShift Container Platform that you want to install and determine the corresponding tag on the Repository Tags page.

2. Set the required environment variables:

   $ export OCP_RELEASE=<release_version> 1
   $ export LOCAL_REGISTRY='<local_registry_host_name>:<local_registry_host_port>' 2
   $ export LOCAL_REPOSITORY='<local_repository_name>' 3
   $ export PRODUCT_REPO='openshift-release-dev' 4
   $ export LOCAL_SECRET_JSON='<path_to_pull_secret>' 5
   $ export RELEASE_NAME="ocp-release" 6
   $ export ARCHITECTURE=<server_architecture> 7
   $ REMOVABLE_MEDIA_PATH=<path> 8

   1

   For <release_version>, specify the tag that corresponds to the version of OpenShift Container Platform to install, such as 4.5.0.

   2

   For <local_registry_host_name>, specify the registry domain name for your mirror repository, and for <local_registry_host_port>, specify the port that it serves content on.

   3

   For <local_repository_name>, specify the name of the repository to create in your registry, such as ocp4/openshift4.

   4

   The repository to mirror. For a production release, you must specify openshift-release-dev.

   5

   For <path_to_pull_secret>, specify the absolute path to and file name of the pull secret for your mirror registry that you created.

   6

   The release mirror. For a production release, you must specify ocp-release.

   7

   For server_architecture, specify the architecture of the server, such as x86_64.

   8

   For <path>, specify the full path, including the initial forward slash (/) character.

3. Mirror the version images to the internal container registry:

   • If your mirror host does not have Internet access, take the following actions:

     1. Connect the removable media to a system that is connected to the Internet.

     2. Review the images and configuration manifests to mirror:

        $ oc adm release mirror -a ${LOCAL_SECRET_JSON}  \
             --from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE} \
             --to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} \
             --to-release-image=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE} --dry-run

     3. Record the entire imageContentSources section from the output of the previous command. The information about your mirrors is unique to your mirrored repository, and you must add the imageContentSources section to the install-config.yaml file during installation.

     4. Mirror the images to a directory on the removable media:

        $ oc adm release mirror -a ${LOCAL_SECRET_JSON} --to-dir=${REMOVABLE_MEDIA_PATH}/mirror quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE}

     5. Take the media to the restricted network environment and upload the images to the local container registry.

        $ oc image mirror -a ${LOCAL_SECRET_JSON} --from-dir=${REMOVABLE_MEDIA_PATH}/mirror "file://openshift/release:${OCP_RELEASE}*" ${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} 1

        1

        For REMOVABLE_MEDIA_PATH, you must use the same path that you specified when you mirrored the images.

   • If the local container registry is connected to the mirror host, take the following actions:

     1. Directly push the release images to the local registry by using following command:

        $ oc adm release mirror -a ${LOCAL_SECRET_JSON}  \
             --from=quay.io/${PRODUCT_REPO}/${RELEASE_NAME}:${OCP_RELEASE}-${ARCHITECTURE} \
             --to=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY} \
             --to-release-image=${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE}

        This command pulls the release information as a digest, and its output includes the imageContentSources data that you require when you install your cluster.

     2. Record the entire imageContentSources section from the output of the previous command. The information about your mirrors is unique to your mirrored repository, and you must add the imageContentSources section to the install-config.yaml file during installation.

        Note

        The image name gets patched to Quay.io during the mirroring process, and the podman images will show Quay.io in the registry on the bootstrap virtual machine.

4. To create the installation program that is based on the content that you mirrored, extract it and pin it to the release:

   • If your mirror host does not have Internet access, run the following command:

     $ oc adm release extract -a ${LOCAL_SECRET_JSON} --command=openshift-install "${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}"

   • If the local container registry is connected to the mirror host, run the following command:

     $ oc adm release extract -a ${LOCAL_SECRET_JSON} --command=openshift-install "${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE}"

Procedure

1. Import the default must-gather image from your installation payload:

   $ oc import-image is/must-gather -n openshift

Procedure

1. Access the images of a specific image stream to mirror, for example:

   $ oc get is <imagestream> -n openshift -o json | jq .spec.tags[].from.name | grep registry.redhat.io

2. Mirror images from registry.redhat.io associated with any image streams you need in the restricted network environment into one of the defined mirrors, for example:

   $ oc image mirror registry.redhat.io/rhscl/ruby-25-rhel7:latest ${MIRROR_ADDR}/rhscl/ruby-25-rhel7:latest

3. Add the required trusted CAs for the mirror in the cluster’s image configuration object:

   $ oc create configmap registry-config --from-file=${MIRROR_ADDR_HOSTNAME}..5000=$path/ca.crt -n openshift-config
   $ oc patch image.config.openshift.io/cluster --patch '{"spec":{"additionalTrustedCA":{"name":"registry-config"}}}' --type=merge

4. Update the samplesRegistry field in the Samples Operator configuration object to contain the hostname portion of the mirror location defined in the mirror configuration:

   $ oc edit configs.samples.operator.openshift.io -n openshift-cluster-samples-operator

   Note

   This is required because the image stream import process does not use the mirror or search mechanism at this time.

5. Add any image streams that are not mirrored into the skippedImagestreams field of the Samples Operator configuration object. Or if you do not want to support any of the sample image streams, set the Samples Operator to Removed in the Samples Operator configuration object.

   Note

   Any unmirrored image streams that are not skipped, or if the Samples Operator is not changed to Removed, will result in the Samples Operator reporting a Degraded status two hours after the image stream imports start failing.

   Many of the templates in the openshift namespace reference the image streams. So using Removed to purge both the image streams and templates will eliminate the possibility of attempts to use them if they are not functional because of any missing image streams.

Procedure

1. Allowlist the following registry URLs:

   URL	Function
   registry.redhat.io	Provides core container images
   quay.io	Provides core container images
   sso.redhat.com	The https://cloud.redhat.com/openshift site uses authentication from sso.redhat.com
   openshift.org	Provides Red Hat Enterprise Linux CoreOS (RHCOS) images

   When you add a site such as quay.io to your allowlist, do not add a wildcard entry such as *.quay.io to your denylist. In most cases, image registries use a content delivery network (CDN) to serve images. If a firewall blocks access, then image downloads are denied when the initial download request is redirected to a host name such as cdn01.quay.io.

2. Allowlist any site that provides resources for a language or framework that your builds require.

3. If you do not disable Telemetry, you must grant access to the following URLs to access Red Hat Insights:

   URL	Function
   cert-api.access.redhat.com	Required for Telemetry
   api.access.redhat.com	Required for Telemetry
   infogw.api.openshift.com	Required for Telemetry
   https://cloud.redhat.com/api/ingress	Required for Telemetry and for insights-operator

4. If you use Amazon Web Services (AWS), Microsoft Azure, or Google Cloud Platform (GCP) to host your cluster, you must grant access to the URLs that provide the cloud provider API and DNS for that cloud:

   Cloud	URL	Function
   AWS	*.amazonaws.com	Required to access AWS services and resources. Review the AWS Service Endpoints in the AWS documentation to determine the exact endpoints to allow for the regions that you use.
   	oso-rhc4tp-docker-registry.s3-us-west-2.amazonaws.com	Required to access AWS services and resources when using strict security requirements. Review the AWS Service Endpoints in the AWS documentation to determine the exact endpoints to allow for the regions that you use.
   GCP	*.googleapis.com	Required to access GCP services and resources. Review Cloud Endpoints in the GCP documentation to determine the endpoints to allow for your APIs.
   	accounts.google.com	Required to access your GCP account.
   Azure	management.azure.com	Required to access Azure services and resources. Review the Azure REST API Reference in the Azure documentation to determine the endpoints to allow for your APIs.

5. Allowlist the following URLs:

   URL	Function
   mirror.openshift.com	Required to access mirrored installation content and images. This site is also a source of release image signatures, although the Cluster Version Operator needs only a single functioning source.
   storage.googleapis.com/openshift-release	A source of release image signatures, although the Cluster Version Operator needs only a single functioning source.
   *.apps.<cluster_name>.<base_domain>	Required to access the default cluster routes unless you set an ingress wildcard during installation.
   quay-registry.s3.amazonaws.com	Required to access Quay image content in AWS.
   api.openshift.com	Required to check if updates are available for the cluster.
   art-rhcos-ci.s3.amazonaws.com	Required to download Red Hat Enterprise Linux CoreOS (RHCOS) images.
   api.openshift.com	Required for your cluster token.
   cloud.redhat.com/openshift	Required for your cluster token.

   Operators require route access to perform health checks. Specifically, the authentication and web console Operators connect to two routes to verify that the routes work. If you are the cluster administrator and do not want to allow *.apps.<cluster_name>.<base_domain>, then allow these routes:

   • oauth-openshift.apps.<cluster_name>.<base_domain>
   • console-openshift-console.apps.<cluster_name>.<base_domain>, or the host name that is specified in the spec.route.hostname field of the consoles.operator/cluster object if the field is not empty.

Procedure

1. Review the DNS custom resource for your cluster:

   $ oc get dnses.config.openshift.io/cluster -o yaml
   apiVersion: config.openshift.io/v1
   kind: DNS
   metadata:
     creationTimestamp: "2019-10-25T18:27:09Z"
     generation: 2
     name: cluster
     resourceVersion: "37966"
     selfLink: /apis/config.openshift.io/v1/dnses/cluster
     uid: 0e714746-f755-11f9-9cb1-02ff55d8f976
   spec:
     baseDomain: <base_domain>
     privateZone:
       tags:
         Name: <infrastructureID>-int
         kubernetes.io/cluster/<infrastructureID>: owned
     publicZone:
       id: Z2XXXXXXXXXXA4
   status: {}

   Note that the spec section contains both a private and a public zone.

2. Patch the DNS custom resource to remove the public zone:

   $ oc patch dnses.config.openshift.io/cluster --type=merge --patch='{"spec": {"publicZone": null}}'
   dns.config.openshift.io/cluster patched

   Because the Ingress Controller consults the DNS definition when it creates Ingress objects, when you create or modify Ingress objects, only private records are created.

   Important

   DNS records for the existing Ingress objects are not modified when you remove the public zone.

3. Optional: Review the DNS custom resource for your cluster and confirm that the public zone was removed:

   $ oc get dnses.config.openshift.io/cluster -o yaml
   apiVersion: config.openshift.io/v1
   kind: DNS
   metadata:
     creationTimestamp: "2019-10-25T18:27:09Z"
     generation: 2
     name: cluster
     resourceVersion: "37966"
     selfLink: /apis/config.openshift.io/v1/dnses/cluster
     uid: 0e714746-f755-11f9-9cb1-02ff55d8f976
   spec:
     baseDomain: <base_domain>
     privateZone:
       tags:
         Name: <infrastructureID>-int
         kubernetes.io/cluster/<infrastructureID>-wfpg4: owned
   status: {}

Procedure

1. Modify the default Ingress Controller to use only an internal endpoint:

   $ oc replace --force --wait --filename - <<EOF
   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     namespace: openshift-ingress-operator
     name: default
   spec:
     endpointPublishingStrategy:
       type: LoadBalancerService
       loadBalancer:
         scope: Internal
   EOF
   ingresscontroller.operator.openshift.io "default" deleted
   ingresscontroller.operator.openshift.io/default replaced

   The public DNS entry is removed, and the private zone entry is updated.

Procedure

1. In the web portal or console for AWS or Azure, take the following actions:

   1. Locate and delete appropriate load balancer component.

      • For AWS, delete the external load balancer. The API DNS entry in the private zone already points to the internal load balancer, which uses an identical configuration, so you do not need to modify the internal load balancer.
      • For Azure, delete the api-internal rule for the load balancer.
   2. Delete the api.$clustername.$yourdomain DNS entry in the public zone.

2. From your terminal, list the cluster machines:

   $ oc get machine -n openshift-machine-api
   NAME                            STATE     TYPE        REGION      ZONE         AGE
   lk4pj-master-0                  running   m4.xlarge   us-east-1   us-east-1a   17m
   lk4pj-master-1                  running   m4.xlarge   us-east-1   us-east-1b   17m
   lk4pj-master-2                  running   m4.xlarge   us-east-1   us-east-1a   17m
   lk4pj-worker-us-east-1a-5fzfj   running   m4.xlarge   us-east-1   us-east-1a   15m
   lk4pj-worker-us-east-1a-vbghs   running   m4.xlarge   us-east-1   us-east-1a   15m
   lk4pj-worker-us-east-1b-zgpzg   running   m4.xlarge   us-east-1   us-east-1b   15m

   You modify the control plane machines, which contain master in the name, in the following step.

3. Remove the external load balancer from each control plane machine.

   1. Edit a control plane Machine object to remove the reference to the external load balancer.

      $ oc edit machines -n openshift-machine-api <master_name> 1

      1

      Specify the name of the control plane, or master, Machine object to modify.

   2. Remove the lines that describe the external load balancer, which are marked in the following example, and save and exit the object specification:

      ...
      spec:
        providerSpec:
          value:
          ...
            loadBalancers:
            - name: lk4pj-ext 1
              type: network 2
            - name: lk4pj-int
              type: network

      1 2

      Delete this line.

   3. Repeat this process for each of the machines that contains master in the name.