Configuring network settings

Procedure

1. Create a network attachment definition, such as tuning-example.yaml, with the following content:

   apiVersion: "k8s.cni.cncf.io/v1"
   kind: NetworkAttachmentDefinition
   metadata:
     name: <name> 

   1

   
     namespace: default 

   2

   
   spec:
     config: '{
       "cniVersion": "0.4.0", 

   3

   
       "name": "<name>", 

   4

   
       "plugins": [{
          "type": "<main_CNI_plugin>" 

   5

   
         },
         {
          "type": "tuning", 

   6

   
          "sysctl": {
               "net.ipv4.conf.IFNAME.accept_redirects": "1" 

   7

   
           }
         }
        ]
   }

   Copy to Clipboard Toggle word wrap

   1

   Specifies the name for the additional network attachment to create. The name must be unique within the specified namespace.

   2

   Specifies the namespace that the object is associated with.

   3

   Specifies the CNI specification version.

   4

   Specifies the name for the configuration. It is recommended to match the configuration name to the name value of the network attachment definition.

   5

   Specifies the name of the main CNI plugin to configure.

   6

   Specifies the name of the CNI meta plugin.

   7

   Specifies the sysctl to set. The interface name is represented by the IFNAME token and is replaced with the actual name of the interface at runtime.

   An example YAML file is shown here:

   apiVersion: "k8s.cni.cncf.io/v1"
   kind: NetworkAttachmentDefinition
   metadata:
     name: tuningnad
     namespace: default
   spec:
     config: '{
       "cniVersion": "0.4.0",
       "name": "tuningnad",
       "plugins": [{
         "type": "bridge"
         },
         {
         "type": "tuning",
         "sysctl": {
            "net.ipv4.conf.IFNAME.accept_redirects": "1"
           }
       }
     ]
   }'

   Copy to Clipboard Toggle word wrap

2. Apply the YAML by running the following command:

   $ oc apply -f tuning-example.yaml

   Copy to Clipboard Toggle word wrap

   Example output

   networkattachmentdefinition.k8.cni.cncf.io/tuningnad created

   Copy to Clipboard Toggle word wrap

3. Create a pod such as examplepod.yaml with the network attachment definition similar to the following:

   apiVersion: v1
   kind: Pod
   metadata:
     name: tunepod
     namespace: default
     annotations:
       k8s.v1.cni.cncf.io/networks: tuningnad 

   1

   
   spec:
     containers:
     - name: podexample
       image: centos
       command: ["/bin/bash", "-c", "sleep INF"]
       securityContext:
         runAsUser: 2000 

   2

   
         runAsGroup: 3000 

   3

   
         allowPrivilegeEscalation: false 

   4

   
         capabilities: 

   5

   
           drop: ["ALL"]
     securityContext:
       runAsNonRoot: true 

   6

   
       seccompProfile: 

   7

   
         type: RuntimeDefault

   Copy to Clipboard Toggle word wrap

   1

   Specify the name of the configured NetworkAttachmentDefinition.

   2

   runAsUser controls which user ID the container is run with.

   3

   runAsGroup controls which primary group ID the containers is run with.

   4

   allowPrivilegeEscalation determines if a pod can request to allow privilege escalation. If unspecified, it defaults to true. This boolean directly controls whether the no_new_privs flag gets set on the container process.

   5

   capabilities permit privileged actions without giving full root access. This policy ensures all capabilities are dropped from the pod.

   6

   runAsNonRoot: true requires that the container will run with a user with any UID other than 0.

   7

   RuntimeDefault enables the default seccomp profile for a pod or container workload.

4. Apply the yaml by running the following command:

   $ oc apply -f examplepod.yaml

   Copy to Clipboard Toggle word wrap

5. Verify that the pod is created by running the following command:

   $ oc get pod

   Copy to Clipboard Toggle word wrap

   Example output

   NAME      READY   STATUS    RESTARTS   AGE
   tunepod   1/1     Running   0          47s

   Copy to Clipboard Toggle word wrap

6. Log in to the pod by running the following command:

   $ oc rsh tunepod

   Copy to Clipboard Toggle word wrap

7. Verify the values of the configured sysctl flags. For example, find the value net.ipv4.conf.net1.accept_redirects by running the following command:

   sh-4.4# sysctl net.ipv4.conf.net1.accept_redirects

   Copy to Clipboard Toggle word wrap

   Expected output

   net.ipv4.conf.net1.accept_redirects = 1

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a network attachment definition, such as tuning-example.yaml, with the following content:

   apiVersion: "k8s.cni.cncf.io/v1"
   kind: NetworkAttachmentDefinition
   metadata:
     name: <name> 

   1

   
     namespace: default 

   2

   
   spec:
     config: '{
       "cniVersion": "0.4.0", 

   3

   
       "name": "<name>", 

   4

   
       "plugins": [{
          "type": "<main_CNI_plugin>" 

   5

   
         },
         {
          "type": "tuning", 

   6

   
          "allmulti": true 

   7

   
           }
         }
        ]
   }

   Copy to Clipboard Toggle word wrap

   1

   Specifies the name for the additional network attachment to create. The name must be unique within the specified namespace.

   2

   Specifies the namespace that the object is associated with.

   3

   Specifies the CNI specification version.

   4

   Specifies the name for the configuration. Match the configuration name to the name value of the network attachment definition.

   5

   Specifies the name of the main CNI plugin to configure.

   6

   Specifies the name of the CNI meta plugin.

   7

   Changes the all-multicast mode of interface. If enabled, all multicast packets on the network will be received by the interface.

   An example YAML file is shown here:

   apiVersion: "k8s.cni.cncf.io/v1"
   kind: NetworkAttachmentDefinition
   metadata:
     name: setallmulti
     namespace: default
   spec:
     config: '{
       "cniVersion": "0.4.0",
       "name": "setallmulti",
       "plugins": [
         {
           "type": "bridge"
         },
         {
           "type": "tuning",
           "allmulti": true
         }
       ]
     }'

   Copy to Clipboard Toggle word wrap

2. Apply the settings specified in the YAML file by running the following command:

   $ oc apply -f tuning-allmulti.yaml

   Copy to Clipboard Toggle word wrap

   Example output

   networkattachmentdefinition.k8s.cni.cncf.io/setallmulti created

   Copy to Clipboard Toggle word wrap

3. Create a pod with a network attachment definition similar to that specified in the following examplepod.yaml sample file:

   apiVersion: v1
   kind: Pod
   metadata:
     name: allmultipod
     namespace: default
     annotations:
       k8s.v1.cni.cncf.io/networks: setallmulti 

   1

   
   spec:
     containers:
     - name: podexample
       image: centos
       command: ["/bin/bash", "-c", "sleep INF"]
       securityContext:
         runAsUser: 2000 

   2

   
         runAsGroup: 3000 

   3

   
         allowPrivilegeEscalation: false 

   4

   
         capabilities: 

   5

   
           drop: ["ALL"]
     securityContext:
       runAsNonRoot: true 

   6

   
       seccompProfile: 

   7

   
         type: RuntimeDefault

   Copy to Clipboard Toggle word wrap

   1

   Specifies the name of the configured NetworkAttachmentDefinition.

   2

   Specifies the user ID the container is run with.

   3

   Specifies which primary group ID the containers is run with.

   4

   Specifies if a pod can request privilege escalation. If unspecified, it defaults to true. This boolean directly controls whether the no_new_privs flag gets set on the container process.

   5

   Specifies the container capabilities. The drop: ["ALL"] statement indicates that all Linux capabilities are dropped from the pod, providing a more restrictive security profile.

   6

   Specifies that the container will run with a user with any UID other than 0.

   7

   Specifies the container’s seccomp profile. In this case, the type is set to RuntimeDefault. Seccomp is a Linux kernel feature that restricts the system calls available to a process, enhancing security by minimizing the attack surface.

4. Apply the settings specified in the YAML file by running the following command:

   $ oc apply -f examplepod.yaml

   Copy to Clipboard Toggle word wrap

5. Verify that the pod is created by running the following command:

   $ oc get pod

   Copy to Clipboard Toggle word wrap

   Example output

   NAME          READY   STATUS    RESTARTS   AGE
   allmultipod   1/1     Running   0          23s

   Copy to Clipboard Toggle word wrap

6. Log in to the pod by running the following command:

   $ oc rsh allmultipod

   Copy to Clipboard Toggle word wrap

7. List all the interfaces associated with the pod by running the following command:

   sh-4.4# ip link

   Copy to Clipboard Toggle word wrap

   Example output

   1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
       link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
   2: eth0@if22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8901 qdisc noqueue state UP mode DEFAULT group default
       link/ether 0a:58:0a:83:00:10 brd ff:ff:ff:ff:ff:ff link-netnsid 0 

   1

   
   3: net1@if24: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
       link/ether ee:9b:66:a4:ec:1d brd ff:ff:ff:ff:ff:ff link-netnsid 0 

   2

   Copy to Clipboard Toggle word wrap

   1

   eth0@if22 is the primary interface

   2

   net1@if24 is the secondary interface configured with the network-attachment-definition that supports the all-multicast mode (ALLMULTI flag)

Procedure

1. To obtain the cluster network range and host prefix for your cluster, enter the following command:

   $ oc get network.operator.openshift.io \
     -o jsonpath="{.items[0].spec.clusterNetwork}"

   Copy to Clipboard Toggle word wrap

   Example output

   [{"cidr":"10.217.0.0/22","hostPrefix":23}]

   Copy to Clipboard Toggle word wrap

2. To expand the cluster network IP address range, enter the following command. Use the CIDR IP address range and host prefix returned from the output of the previous command.

   $ oc patch Network.config.openshift.io cluster --type='merge' --patch \
     '{
       "spec":{
         "clusterNetwork": [ {"cidr":"<network>/<cidr>","hostPrefix":<prefix>} ],
         "networkType": "OVNKubernetes"
       }
     }'

   Copy to Clipboard Toggle word wrap

   where:

   <network>

   Specifies the network part of the cidr field that you obtained from the previous step. You cannot change this value.

   <cidr>

   Specifies the network prefix length. For example, 14. Change this value to a smaller number than the value from the output in the previous step to expand the cluster network range.

   <prefix>

   Specifies the current host prefix for your cluster. This value must be the same value for the hostPrefix field that you obtained from the previous step.

   Example command

   $ oc patch Network.config.openshift.io cluster --type='merge' --patch \
     '{
       "spec":{
         "clusterNetwork": [ {"cidr":"10.217.0.0/14","hostPrefix": 23} ],
         "networkType": "OVNKubernetes"
       }
     }'

   Copy to Clipboard Toggle word wrap

   Example output

   network.config.openshift.io/cluster patched

   Copy to Clipboard Toggle word wrap

3. To confirm that the configuration is active, enter the following command. It can take up to 30 minutes for this change to take effect.

   $ oc get network.operator.openshift.io \
     -o jsonpath="{.items[0].spec.clusterNetwork}"

   Copy to Clipboard Toggle word wrap

   Example output

   [{"cidr":"10.217.0.0/14","hostPrefix":23}]

   Copy to Clipboard Toggle word wrap

Procedure

1. Create an IP failover service account:

   $ oc create sa ipfailover

   Copy to Clipboard Toggle word wrap

2. Update security context constraints (SCC) for hostNetwork:

   $ oc adm policy add-scc-to-user privileged -z ipfailover

   Copy to Clipboard Toggle word wrap

   $ oc adm policy add-scc-to-user hostnetwork -z ipfailover

   Copy to Clipboard Toggle word wrap

3. Red Hat OpenStack Platform (RHOSP) only: Complete the following steps to make a failover VIP address reachable on RHOSP ports.

   1. Use the RHOSP CLI to show the default RHOSP API and VIP addresses in the allowed_address_pairs parameter of your RHOSP cluster:

      $ openstack port show <cluster_name> -c allowed_address_pairs

      Copy to Clipboard Toggle word wrap

      Output example

      *Field*                  *Value*
      allowed_address_pairs    ip_address='192.168.0.5', mac_address='fa:16:3e:31:f9:cb'
                               ip_address='192.168.0.7', mac_address='fa:16:3e:31:f9:cb'

      Copy to Clipboard Toggle word wrap

   2. Set a different VIP address for the IP failover deployment and make the address reachable on RHOSP ports by entering the following command in the RHOSP CLI. Do not set any default RHOSP API and VIP addresses as the failover VIP address for the IP failover deployment.

      Example of adding the 1.1.1.1 failover IP address as an allowed address on RHOSP ports.

      $ openstack port set <cluster_name> --allowed-address ip-address=1.1.1.1,mac-address=fa:fa:16:3e:31:f9:cb

      Copy to Clipboard Toggle word wrap

   3. Create a deployment YAML file to configure IP failover for your deployment. See "Example deployment YAML for IP failover configuration" in a later step.

   4. Specify the following specification in the IP failover deployment so that you pass the failover VIP address to the OPENSHIFT_HA_VIRTUAL_IPS environment variable:

      Example of adding the 1.1.1.1 VIP address to OPENSHIFT_HA_VIRTUAL_IPS

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: ipfailover-keepalived
      # ...
            spec:
                env:
                - name: OPENSHIFT_HA_VIRTUAL_IPS
                value: "1.1.1.1"
      # ...

      Copy to Clipboard Toggle word wrap

4. Create a deployment YAML file to configure IP failover.

   Note

   For Red Hat OpenStack Platform (RHOSP), you do not need to re-create the deployment YAML file. You already created this file as part of the earlier instructions.

   Example deployment YAML for IP failover configuration

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: ipfailover-keepalived 

   1

   
     labels:
       ipfailover: hello-openshift
   spec:
     strategy:
       type: Recreate
     replicas: 2
     selector:
       matchLabels:
         ipfailover: hello-openshift
     template:
       metadata:
         labels:
           ipfailover: hello-openshift
       spec:
         serviceAccountName: ipfailover
         privileged: true
         hostNetwork: true
         nodeSelector:
           node-role.kubernetes.io/worker: ""
         containers:
         - name: openshift-ipfailover
           image: registry.redhat.io/openshift4/ose-keepalived-ipfailover-rhel9:v4.19
           ports:
           - containerPort: 63000
             hostPort: 63000
           imagePullPolicy: IfNotPresent
           securityContext:
             privileged: true
           volumeMounts:
           - name: lib-modules
             mountPath: /lib/modules
             readOnly: true
           - name: host-slash
             mountPath: /host
             readOnly: true
             mountPropagation: HostToContainer
           - name: etc-sysconfig
             mountPath: /etc/sysconfig
             readOnly: true
           - name: config-volume
             mountPath: /etc/keepalive
           env:
           - name: OPENSHIFT_HA_CONFIG_NAME
             value: "ipfailover"
           - name: OPENSHIFT_HA_VIRTUAL_IPS 

   2

   
             value: "1.1.1.1-2"
           - name: OPENSHIFT_HA_VIP_GROUPS 

   3

   
             value: "10"
           - name: OPENSHIFT_HA_NETWORK_INTERFACE 

   4

   
             value: "ens3" #The host interface to assign the VIPs
           - name: OPENSHIFT_HA_MONITOR_PORT 

   5

   
             value: "30060"
           - name: OPENSHIFT_HA_VRRP_ID_OFFSET 

   6

   
             value: "10"
           - name: OPENSHIFT_HA_REPLICA_COUNT 

   7

   
             value: "2" #Must match the number of replicas in the deployment
           - name: OPENSHIFT_HA_USE_UNICAST
             value: "false"
           #- name: OPENSHIFT_HA_UNICAST_PEERS
             #value: "10.0.148.40,10.0.160.234,10.0.199.110"
           - name: OPENSHIFT_HA_IPTABLES_CHAIN 

   8

   
             value: "INPUT"
           #- name: OPENSHIFT_HA_NOTIFY_SCRIPT 

   9

   
           #  value: /etc/keepalive/mynotifyscript.sh
           - name: OPENSHIFT_HA_CHECK_SCRIPT 

   10

   
             value: "/etc/keepalive/mycheckscript.sh"
           - name: OPENSHIFT_HA_PREEMPTION 

   11

   
             value: "preempt_delay 300"
           - name: OPENSHIFT_HA_CHECK_INTERVAL 

   12

   
             value: "2"
           livenessProbe:
             initialDelaySeconds: 10
             exec:
               command:
               - pgrep
               - keepalived
         volumes:
         - name: lib-modules
           hostPath:
             path: /lib/modules
         - name: host-slash
           hostPath:
             path: /
         - name: etc-sysconfig
           hostPath:
             path: /etc/sysconfig
         # config-volume contains the check script
         # created with `oc create configmap keepalived-checkscript --from-file=mycheckscript.sh`
         - configMap:
             defaultMode: 0755
             name: keepalived-checkscript
           name: config-volume
         imagePullSecrets:
           - name: openshift-pull-secret 

   13

   Copy to Clipboard Toggle word wrap

   1

   The name of the IP failover deployment.

   2

   The list of IP address ranges to replicate. This must be provided. For example, 1.2.3.4-6,1.2.3.9.

   3

   The number of groups to create for VRRP. If not set, a group is created for each virtual IP range specified with the OPENSHIFT_HA_VIP_GROUPS variable.

   4

   The interface name that IP failover uses to send VRRP traffic. By default, eth0 is used.

   5

   The IP failover pod tries to open a TCP connection to this port on each VIP. If connection is established, the service is considered to be running. If this port is set to 0, the test always passes. The default value is 80.

   6

   The offset value used to set the virtual router IDs. Using different offset values allows multiple IP failover configurations to exist within the same cluster. The default offset is 10, and the allowed range is 0 through 255.

   7

   The number of replicas to create. This must match spec.replicas value in IP failover deployment configuration. The default value is 2.

   8

   The name of the iptables chain to automatically add an iptables rule to allow the VRRP traffic on. If the value is not set, an iptables rule is not added. If the chain does not exist, it is not created, and Keepalived operates in unicast mode. The default is INPUT.

   9

   The full path name in the pod file system of a script that is run whenever the state changes.

   10

   The full path name in the pod file system of a script that is periodically run to verify the application is operating.

   11

   The strategy for handling a new higher priority host. The default value is preempt_delay 300, which causes a Keepalived instance to take over a VIP after 5 minutes if a lower-priority master is holding the VIP.

   12

   The period, in seconds, that the check script is run. The default value is 2.

   13

   Create the pull secret before creating the deployment, otherwise you will get an error when creating the deployment.

Procedure

1. Create the desired script and create a ConfigMap object to hold it. The script has no input arguments and must return 0 for OK and 1 for fail.

   The check script, mycheckscript.sh:

   #!/bin/bash
       # Whatever tests are needed
       # E.g., send request and verify response
   exit 0

   Copy to Clipboard Toggle word wrap

2. Create the ConfigMap object :

   $ oc create configmap mycustomcheck --from-file=mycheckscript.sh

   Copy to Clipboard Toggle word wrap

3. Add the script to the pod. The defaultMode for the mounted ConfigMap object files must able to run by using oc commands or by editing the deployment configuration. A value of 0755, 493 decimal, is typical:

   $ oc set env deploy/ipfailover-keepalived \
       OPENSHIFT_HA_CHECK_SCRIPT=/etc/keepalive/mycheckscript.sh

   Copy to Clipboard Toggle word wrap

   $ oc set volume deploy/ipfailover-keepalived --add --overwrite \
       --name=config-volume \
       --mount-path=/etc/keepalive \
       --source='{"configMap": { "name": "mycustomcheck", "defaultMode": 493}}'

   Copy to Clipboard Toggle word wrap
   Note

   The oc set env command is whitespace sensitive. There must be no whitespace on either side of the = sign.

   Tip

   You can alternatively edit the ipfailover-keepalived deployment configuration:

   $ oc edit deploy ipfailover-keepalived

   Copy to Clipboard Toggle word wrap

       spec:
         containers:
         - env:
           - name: OPENSHIFT_HA_CHECK_SCRIPT  

   1

   
             value: /etc/keepalive/mycheckscript.sh
   ...
           volumeMounts: 

   2

   
           - mountPath: /etc/keepalive
             name: config-volume
         dnsPolicy: ClusterFirst
   ...
         volumes: 

   3

   
         - configMap:
             defaultMode: 0755 

   4

   
             name: customrouter
           name: config-volume
   ...

   Copy to Clipboard Toggle word wrap

   1

   In the spec.container.env field, add the OPENSHIFT_HA_CHECK_SCRIPT environment variable to point to the mounted script file.

   2

   Add the spec.container.volumeMounts field to create the mount point.

   3

   Add a new spec.volumes field to mention the config map.

   4

   This sets run permission on the files. When read back, it is displayed in decimal, 493.

   Save the changes and exit the editor. This restarts ipfailover-keepalived.

Procedure

1. Optional: Identify and delete any check and notify scripts that are stored as config maps:

   1. Identify whether any pods for IP failover use a config map as a volume:

      $ oc get pod -l ipfailover \
        -o jsonpath="\
      {range .items[?(@.spec.volumes[*].configMap)]}
      {'Namespace: '}{.metadata.namespace}
      {'Pod:       '}{.metadata.name}
      {'Volumes that use config maps:'}
      {range .spec.volumes[?(@.configMap)]}  {'volume:    '}{.name}
        {'configMap: '}{.configMap.name}{'\n'}{end}
      {end}"

      Copy to Clipboard Toggle word wrap

      Example output

      Namespace: default
      Pod:       keepalived-worker-59df45db9c-2x9mn
      Volumes that use config maps:
        volume:    config-volume
        configMap: mycustomcheck

      Copy to Clipboard Toggle word wrap

   2. If the preceding step provided the names of config maps that are used as volumes, delete the config maps:

      $ oc delete configmap <configmap_name>

      Copy to Clipboard Toggle word wrap

2. Identify an existing deployment for IP failover:

   $ oc get deployment -l ipfailover

   Copy to Clipboard Toggle word wrap

   Example output

   NAMESPACE   NAME         READY   UP-TO-DATE   AVAILABLE   AGE
   default     ipfailover   2/2     2            2           105d

   Copy to Clipboard Toggle word wrap

3. Delete the deployment:

   $ oc delete deployment <ipfailover_deployment_name>

   Copy to Clipboard Toggle word wrap

4. Remove the ipfailover service account:

   $ oc delete sa ipfailover

   Copy to Clipboard Toggle word wrap

5. Run a job that removes the IP tables rule that was added when IP failover was initially configured:

   1. Create a file such as remove-ipfailover-job.yaml with contents that are similar to the following example:

      apiVersion: batch/v1
      kind: Job
      metadata:
        generateName: remove-ipfailover-
        labels:
          app: remove-ipfailover
      spec:
        template:
          metadata:
            name: remove-ipfailover
          spec:
            containers:
            - name: remove-ipfailover
              image: registry.redhat.io/openshift4/ose-keepalived-ipfailover-rhel9:v4.19
              command: ["/var/lib/ipfailover/keepalived/remove-failover.sh"]
            nodeSelector: 

      1

      
              kubernetes.io/hostname: <host_name>  

      2

      
            restartPolicy: Never

      Copy to Clipboard Toggle word wrap

      1

      The nodeSelector is likely the same as the selector used in the old IP failover deployment.

      2

      Run the job for each node in your cluster that was configured for IP failover and replace the hostname each time.

   2. Run the job:

      $ oc create -f remove-ipfailover-job.yaml

      Copy to Clipboard Toggle word wrap

      Example output

      job.batch/remove-ipfailover-2h8dm created

      Copy to Clipboard Toggle word wrap

Procedure

1. Create a config map that contains any additional CA certificates required for proxying HTTPS connections.

   Note

   You can skip this step if the identity certificate of the proxy is signed by an authority from the Red Hat Enterprise Linux CoreOS (RHCOS) trust bundle.

   1. Create a file called user-ca-bundle.yaml, and provide the values of your PEM-encoded certificates:

      apiVersion: v1
      data:
        ca-bundle.crt: | 

      1

      
          <MY_PEM_ENCODED_CERTS> 

      2

      
      kind: ConfigMap
      metadata:
        name: user-ca-bundle 

      3

      
        namespace: openshift-config 

      4

      Copy to Clipboard Toggle word wrap

      1

      This data key must be named ca-bundle.crt.

      2

      One or more PEM-encoded X.509 certificates used to sign the proxy’s identity certificate.

      3

      The config map name that is referenced from the Proxy object.

      4

      The config map must exist in the openshift-config namespace.

   2. Create the config map from the user-ca-bundle.yaml file by entering the following command:

      $ oc create -f user-ca-bundle.yaml

      Copy to Clipboard Toggle word wrap

2. Use the oc edit command to modify the Proxy object:

   $ oc edit proxy/cluster

   Copy to Clipboard Toggle word wrap

3. Configure the necessary fields for the proxy:

   apiVersion: config.openshift.io/v1
   kind: Proxy
   metadata:
     name: cluster
   spec:
     httpProxy: http://<username>:<pswd>@<ip>:<port> 

   1

   
     httpsProxy: https://<username>:<pswd>@<ip>:<port> 

   2

   
     noProxy: example.com 

   3

   
     readinessEndpoints:
     - http://www.google.com 

   4

   
     - https://www.google.com
     trustedCA:
       name: user-ca-bundle 

   5

   Copy to Clipboard Toggle word wrap

   1

   A proxy URL to use for creating HTTP connections outside the cluster. The URL scheme must be http.

   2

   A proxy URL to use for creating HTTPS connections outside the cluster. The URL scheme must be either http or https. Specify a URL for the proxy that supports the URL scheme. For example, most proxies report an error if they are configured to use https but they only support http. This failure message may not propagate to the logs and can appear to be a network connection failure instead. If using a proxy that listens for https connections from the cluster, you might need to configure the cluster to accept the CAs and certificates that the proxy uses.

   3

   A comma-separated list of destination domain names, domains, IP addresses (or other network CIDRs), and port numbers to exclude proxying.

   Note

   Port numbers are only supported when configuring IPv6 addresses. Port numbers are not supported when configuring IPv4 addresses.

   Preface a domain with . to match subdomains only. For example, .y.com matches x.y.com, but not y.com. Use * to bypass proxy for all destinations.

   If your noproxy field needs to include a domain address, you must explicitly specify that FQDN, or prefix-matched subdomain, in the noproxy field. You cannot use the IP address or CIDR range that encapsulates the domain. This is because the cluster does not wait for DNS to return the IP address before assigning the route connection, and checks explicitly against the request being made. For example, if you have a CIDR block value, such as 10.0.0.0/24, for the noproxy field and the field attempts to access https://10.0.0.11, the addresses successfully match. However, attempting to access https://exampleserver.externaldomain.com, whose A record entry is 10.0.0.11, fails. An additional value of .externaldomain.com for your noproxy field is necessary.

   If you scale up compute nodes that are not included in the network defined by the networking.machineNetwork[].cidr field from the installation configuration, you must add them to this list to prevent connection issues.

   This field is ignored if neither the httpProxy or httpsProxy fields are set.

   4

   One or more URLs external to the cluster to use to perform a readiness check before writing the httpProxy and httpsProxy values to status.

   5

   A reference to the config map in the openshift-config namespace that contains additional CA certificates required for proxying HTTPS connections. Note that the config map must already exist before referencing it here. This field is required unless the proxy’s identity certificate is signed by an authority from the RHCOS trust bundle.

4. Save the file to apply the changes.

Procedure

1. Use the oc edit command to modify the proxy:

   $ oc edit proxy/cluster

   Copy to Clipboard Toggle word wrap

2. Remove all spec fields from the Proxy object. For example:

   apiVersion: config.openshift.io/v1
   kind: Proxy
   metadata:
     name: cluster
   spec: {}

   Copy to Clipboard Toggle word wrap
3. Save the file to apply the changes.

Procedure

1. Check the proxy configuration status using the oc command:

   $ oc get proxy/cluster -o yaml

   Copy to Clipboard Toggle word wrap
2. Verify the proxy fields in the output to ensure they match your configuration. Specifically, check the spec.httpProxy, spec.httpsProxy, spec.noProxy, and spec.trustedCA fields.

3. Inspect the status of the Proxy object:

   $ oc get proxy/cluster -o jsonpath='{.status}'

   Copy to Clipboard Toggle word wrap

   Example output

   {
   status:
       httpProxy: http://user:xxx@xxxx:3128
       httpsProxy: http://user:xxx@xxxx:3128
       noProxy: .cluster.local,.svc,10.0.0.0/16,10.128.0.0/14,127.0.0.1,169.254.169.254,172.30.0.0/16,localhost,test.no-proxy.com
   }

   Copy to Clipboard Toggle word wrap

4. Check the logs of the Machine Config Operator (MCO) to ensure that the configuration changes were applied successfully:

   $ oc logs -n openshift-machine-config-operator $(oc get pods -n openshift-machine-config-operator -l k8s-app=machine-config-operator -o name)

   Copy to Clipboard Toggle word wrap
5. Look for messages that indicate the proxy settings were applied and the nodes were rebooted if necessary.

6. Verify that system components are using the proxy by checking the logs of a component that makes external requests, such as the Cluster Version Operator (CVO):

   $ oc logs -n openshift-cluster-version $(oc get pods -n openshift-cluster-version -l k8s-app=machine-config-operator -o name)

   Copy to Clipboard Toggle word wrap
7. Look for log entries that show that external requests have been routed through the proxy.

Procedure

1. Edit your install-config.yaml file and add the proxy settings. For example:

   apiVersion: v1
   baseDomain: my.domain.com
   proxy:
     httpProxy: http://<username>:<pswd>@<ip>:<port> 

   1

   
     httpsProxy: https://<username>:<pswd>@<ip>:<port> 

   2

   
     noProxy: example.com 

   3

   
   additionalTrustBundle: | 

   4

   
       -----BEGIN CERTIFICATE-----
       <MY_TRUSTED_CA_CERT>
       -----END CERTIFICATE-----
   additionalTrustBundlePolicy: <policy_to_add_additionalTrustBundle> 

   5

   Copy to Clipboard Toggle word wrap

   1

   A proxy URL to use for creating HTTP connections outside the cluster. The URL scheme must be http.

   2

   A proxy URL to use for creating HTTPS connections outside the cluster.

   3

   A comma-separated list of destination domain names, IP addresses, or other network CIDRs to exclude from proxying. Preface a domain with . to match subdomains only. For example, .y.com matches x.y.com, but not y.com. Use * to bypass the proxy for all destinations.

   4

   If provided, the installation program generates a config map that is named user-ca-bundle in the openshift-config namespace that contains one or more additional CA certificates that are required for proxying HTTPS connections. The Cluster Network Operator then creates a trusted-ca-bundle config map that merges these contents with the Red Hat Enterprise Linux CoreOS (RHCOS) trust bundle, and this config map is referenced in the trustedCA field of the Proxy object. The additionalTrustBundle field is required unless the proxy’s identity certificate is signed by an authority from the RHCOS trust bundle.

   5

   Optional: The policy to determine the configuration of the Proxy object to reference the user-ca-bundle config map in the trustedCA field. The allowed values are Proxyonly and Always. Use Proxyonly to reference the user-ca-bundle config map only when http/https proxy is configured. Use Always to always reference the user-ca-bundle config map. The default value is Proxyonly.

   Note

   The installation program does not support the proxy readinessEndpoints field.

   Note

   If the installer times out, restart and then complete the deployment by using the wait-for command of the installer. For example:

   $ ./openshift-install wait-for install-complete --log-level debug

   Copy to Clipboard Toggle word wrap
2. Save the file and reference it when installing OpenShift Container Platform.

Procedure

1. Create a config map that contains any additional CA certificates required for proxying HTTPS connections.

   Note

   You can skip this step if the identity certificate of the proxy is signed by an authority from the Red Hat Enterprise Linux CoreOS (RHCOS) trust bundle.

   1. Create a file called user-ca-bundle.yaml, and provide the values of your PEM-encoded certificates:

      apiVersion: v1
      data:
        ca-bundle.crt: | 

      1

      
          <MY_PEM_ENCODED_CERTS> 

      2

      
      kind: ConfigMap
      metadata:
        name: user-ca-bundle 

      3

      
        namespace: openshift-config 

      4

      Copy to Clipboard Toggle word wrap

      1

      This data key must be named ca-bundle.crt.

      2

      One or more PEM-encoded X.509 certificates used to sign the proxy’s identity certificate.

      3

      The config map name that is referenced from the Proxy object.

      4

      The config map must exist in the openshift-config namespace.

   2. Create the config map from the user-ca-bundle.yaml file by entering the following command:

      $ oc create -f user-ca-bundle.yaml

      Copy to Clipboard Toggle word wrap

2. Use the oc edit command to modify the Proxy object:

   $ oc edit proxy/cluster

   Copy to Clipboard Toggle word wrap

3. Configure the necessary fields for the proxy:

   apiVersion: config.openshift.io/v1
   kind: Proxy
   metadata:
     name: cluster
   spec:
     httpProxy: http://<username>:<pswd>@<ip>:<port> 

   1

   
     httpsProxy: https://<username>:<pswd>@<ip>:<port> 

   2

   
     noProxy: example.com 

   3

   
     readinessEndpoints:
     - http://www.google.com 

   4

   
     - https://www.google.com
     trustedCA:
       name: user-ca-bundle 

   5

   Copy to Clipboard Toggle word wrap

   1

   A proxy URL to use for creating HTTP connections outside the cluster. The URL scheme must be http.

   2

   A proxy URL to use for creating HTTPS connections outside the cluster. The URL scheme must be either http or https. Specify a URL for the proxy that supports the URL scheme. For example, most proxies report an error if they are configured to use https but they only support http. This failure message may not propagate to the logs and can appear to be a network connection failure instead. If using a proxy that listens for https connections from the cluster, you might need to configure the cluster to accept the CAs and certificates that the proxy uses.

   3

   A comma-separated list of destination domain names, domains, IP addresses (or other network CIDRs), and port numbers to exclude proxying.

   Note

   Port numbers are only supported when configuring IPv6 addresses. Port numbers are not supported when configuring IPv4 addresses.

   Preface a domain with . to match subdomains only. For example, .y.com matches x.y.com, but not y.com. Use * to bypass proxy for all destinations.

   If your noproxy field needs to include a domain address, you must explicitly specify that FQDN, or prefix-matched subdomain, in the noproxy field. You cannot use the IP address or CIDR range that encapsulates the domain. This is because the cluster does not wait for DNS to return the IP address before assigning the route connection, and checks explicitly against the request being made. For example, if you have a CIDR block value, such as 10.0.0.0/24, for the noproxy field and the field attempts to access https://10.0.0.11, the addresses successfully match. However, attempting to access https://exampleserver.externaldomain.com, whose A record entry is 10.0.0.11, fails. An additional value of .externaldomain.com for your noproxy field is necessary.

   If you scale up compute nodes that are not included in the network defined by the networking.machineNetwork[].cidr field from the installation configuration, you must add them to this list to prevent connection issues.

   This field is ignored if neither the httpProxy or httpsProxy fields are set.

   4

   One or more URLs external to the cluster to use to perform a readiness check before writing the httpProxy and httpsProxy values to status.

   5

   A reference to the config map in the openshift-config namespace that contains additional CA certificates required for proxying HTTPS connections. Note that the config map must already exist before referencing it here. This field is required unless the proxy’s identity certificate is signed by an authority from the RHCOS trust bundle.

4. Save the file to apply the changes.