Chapter 26. OpenShift SDN network plugin

1. Egress firewall is also known as egress network policy in OpenShift SDN. This is not the same as network policy egress.
2. Egress router for OVN-Kubernetes supports only redirect mode.
3. IPv6 single-stack networking on a bare-metal platform.
4. IPv4/IPv6 dual-stack networking on bare-metal, VMware vSphere (installer-provisioned infrastructure installations only), IBM Power®, and IBM Z® platforms. On VMware vSphere, dual-stack networking limitations exist.
5. IPv6/IPv4 dual-stack networking on bare-metal and IBM Power® platforms.

Procedure

1. Stop all of the machine configuration pools managed by the Machine Config Operator (MCO):

   • Stop the master configuration pool by entering the following command in your CLI:

     $ oc patch MachineConfigPool master --type='merge' --patch \
       '{ "spec": { "paused": true } }'

   • Stop the worker machine configuration pool by entering the following command in your CLI:

     $ oc patch MachineConfigPool worker --type='merge' --patch \
       '{ "spec":{ "paused": true } }'

2. To prepare for the migration, set the migration field to null by entering the following command in your CLI:

   $ oc patch Network.operator.openshift.io cluster --type='merge' \
     --patch '{ "spec": { "migration": null } }'

3. Check that the migration status is empty for the Network.config.openshift.io object by entering the following command in your CLI. Empty command output indicates that the object is not in a migration operation.

   $ oc get Network.config cluster -o jsonpath='{.status.migration}'

4. Apply the patch to the Network.operator.openshift.io object to set the network plugin back to OpenShift SDN by entering the following command in your CLI:

   $ oc patch Network.operator.openshift.io cluster --type='merge' \
     --patch '{ "spec": { "migration": { "networkType": "OpenShiftSDN" } } }'

   Important

   If you applied the patch to the Network.config.openshift.io object before the patch operation finalizes on the Network.operator.openshift.io object, the Cluster Network Operator (CNO) enters into a degradation state and this causes a slight delay until the CNO recovers from the degraded state.

5. Confirm that the migration status of the network plugin for the Network.config.openshift.io cluster object is OpenShiftSDN by entering the following command in your CLI:

   $ oc get Network.config cluster -o jsonpath='{.status.migration.networkType}'

6. Apply the patch to the Network.config.openshift.io object to set the network plugin back to OpenShift SDN by entering the following command in your CLI:

   $ oc patch Network.config.openshift.io cluster --type='merge' \
     --patch '{ "spec": { "networkType": "OpenShiftSDN" } }'

7. Optional: Disable automatic migration of several OVN-Kubernetes capabilities to the OpenShift SDN equivalents:

   • Egress IPs
   • Egress firewall
   • Multicast

   To disable automatic migration of the configuration for any of the previously noted OpenShift SDN features, specify the following keys:

   $ oc patch Network.operator.openshift.io cluster --type='merge' \
     --patch '{
       "spec": {
         "migration": {
           "networkType": "OpenShiftSDN",
           "features": {
             "egressIP": <bool>,
             "egressFirewall": <bool>,
             "multicast": <bool>
           }
         }
       }
     }'

   where:

   bool: Specifies whether to enable migration of the feature. The default is true.

8. Optional: You can customize the following settings for OpenShift SDN to meet your network infrastructure requirements:

   • Maximum transmission unit (MTU)
   • VXLAN port

   To customize either or both of the previously noted settings, customize and enter the following command in your CLI. If you do not need to change the default value, omit the key from the patch.

   $ oc patch Network.operator.openshift.io cluster --type=merge \
     --patch '{
       "spec":{
         "defaultNetwork":{
           "openshiftSDNConfig":{
             "mtu":<mtu>,
             "vxlanPort":<port>
       }}}}'

   mtu

   The MTU for the VXLAN overlay network. This value is normally configured automatically, but if the nodes in your cluster do not all use the same MTU, then you must set this explicitly to 50 less than the smallest node MTU value.

   port

   The UDP port for the VXLAN overlay network. If a value is not specified, the default is 4789. The port cannot be the same as the Geneve port that is used by OVN-Kubernetes. The default value for the Geneve port is 6081.

   Example patch command

   $ oc patch Network.operator.openshift.io cluster --type=merge \
     --patch '{
       "spec":{
         "defaultNetwork":{
           "openshiftSDNConfig":{
             "mtu":1200
       }}}}'

9. Reboot each node in your cluster. You can reboot the nodes in your cluster with either of the following approaches:

   • With the oc rsh command, you can use a bash script similar to the following:

     #!/bin/bash
     readarray -t POD_NODES <<< "$(oc get pod -n openshift-machine-config-operator -o wide| grep daemon|awk '{print $1" "$7}')"
     
     for i in "${POD_NODES[@]}"
     do
       read -r POD NODE <<< "$i"
       until oc rsh -n openshift-machine-config-operator "$POD" chroot /rootfs shutdown -r +1
         do
           echo "cannot reboot node $NODE, retry" && sleep 3
         done
     done

   • With the ssh command, you can use a bash script similar to the following. The script assumes that you have configured sudo to not prompt for a password.

     #!/bin/bash
     
     for ip in $(oc get nodes  -o jsonpath='{.items[*].status.addresses[?(@.type=="InternalIP")].address}')
     do
        echo "reboot node $ip"
        ssh -o StrictHostKeyChecking=no core@$ip sudo shutdown -r -t 3
     done

10. Wait until the Multus daemon set rollout completes. Run the following command to see your rollout status:

    $ oc -n openshift-multus rollout status daemonset/multus

    The name of the Multus pods is in the form of multus-<xxxxx> where <xxxxx> is a random sequence of letters. It might take several moments for the pods to restart.

    Example output

    Waiting for daemon set "multus" rollout to finish: 1 out of 6 new pods have been updated...
    ...
    Waiting for daemon set "multus" rollout to finish: 5 of 6 updated pods are available...
    daemon set "multus" successfully rolled out

11. After the nodes in your cluster have rebooted and the multus pods are rolled out, start all of the machine configuration pools by running the following commands::

    • Start the master configuration pool:

      $ oc patch MachineConfigPool master --type='merge' --patch \
        '{ "spec": { "paused": false } }'

    • Start the worker configuration pool:

      $ oc patch MachineConfigPool worker --type='merge' --patch \
        '{ "spec": { "paused": false } }'

    As the MCO updates machines in each config pool, it reboots each node.

    By default the MCO updates a single machine per pool at a time, so the time that the migration requires to complete grows with the size of the cluster.

12. Confirm the status of the new machine configuration on the hosts:

    1. To list the machine configuration state and the name of the applied machine configuration, enter the following command in your CLI:

       $ oc describe node | egrep "hostname|machineconfig"

       Example output

       kubernetes.io/hostname=master-0
       machineconfiguration.openshift.io/currentConfig: rendered-master-c53e221d9d24e1c8bb6ee89dd3d8ad7b
       machineconfiguration.openshift.io/desiredConfig: rendered-master-c53e221d9d24e1c8bb6ee89dd3d8ad7b
       machineconfiguration.openshift.io/reason:
       machineconfiguration.openshift.io/state: Done

       Verify that the following statements are true:

       • The value of machineconfiguration.openshift.io/state field is Done.
       • The value of the machineconfiguration.openshift.io/currentConfig field is equal to the value of the machineconfiguration.openshift.io/desiredConfig field.

    2. To confirm that the machine config is correct, enter the following command in your CLI:

       $ oc get machineconfig <config_name> -o yaml

       where <config_name> is the name of the machine config from the machineconfiguration.openshift.io/currentConfig field.

13. Confirm that the migration succeeded:

    1. To confirm that the network plugin is OpenShift SDN, enter the following command in your CLI. The value of status.networkType must be OpenShiftSDN.

       $ oc get Network.config/cluster -o jsonpath='{.status.networkType}{"\n"}'

    2. To confirm that the cluster nodes are in the Ready state, enter the following command in your CLI:

       $ oc get nodes

    3. If a node is stuck in the NotReady state, investigate the machine config daemon pod logs and resolve any errors.

       1. To list the pods, enter the following command in your CLI:

          $ oc get pod -n openshift-machine-config-operator

          Example output

          NAME                                         READY   STATUS    RESTARTS   AGE
          machine-config-controller-75f756f89d-sjp8b   1/1     Running   0          37m
          machine-config-daemon-5cf4b                  2/2     Running   0          43h
          machine-config-daemon-7wzcd                  2/2     Running   0          43h
          machine-config-daemon-fc946                  2/2     Running   0          43h
          machine-config-daemon-g2v28                  2/2     Running   0          43h
          machine-config-daemon-gcl4f                  2/2     Running   0          43h
          machine-config-daemon-l5tnv                  2/2     Running   0          43h
          machine-config-operator-79d9c55d5-hth92      1/1     Running   0          37m
          machine-config-server-bsc8h                  1/1     Running   0          43h
          machine-config-server-hklrm                  1/1     Running   0          43h
          machine-config-server-k9rtx                  1/1     Running   0          43h

          The names for the config daemon pods are in the following format: machine-config-daemon-<seq>. The <seq> value is a random five character alphanumeric sequence.

       2. To display the pod log for each machine config daemon pod shown in the previous output, enter the following command in your CLI:

          $ oc logs <pod> -n openshift-machine-config-operator

          where pod is the name of a machine config daemon pod.

       3. Resolve any errors in the logs shown by the output from the previous command.

    4. To confirm that your pods are not in an error state, enter the following command in your CLI:

       $ oc get pods --all-namespaces -o wide --sort-by='{.spec.nodeName}'

       If pods on a node are in an error state, reboot that node.

14. Complete the following steps only if the migration succeeds and your cluster is in a good state:

    1. To remove the migration configuration from the Cluster Network Operator configuration object, enter the following command in your CLI:

       $ oc patch Network.operator.openshift.io cluster --type='merge' \
         --patch '{ "spec": { "migration": null } }'

    2. To remove the OVN-Kubernetes configuration, enter the following command in your CLI:

       $ oc patch Network.operator.openshift.io cluster --type='merge' \
         --patch '{ "spec": { "defaultNetwork": { "ovnKubernetesConfig":null } } }'

    3. To remove the OVN-Kubernetes network provider namespace, enter the following command in your CLI:

       $ oc delete namespace openshift-ovn-kubernetes

Procedure

1. To backup the configuration for the cluster network, enter the following command:

   $ oc get Network.config.openshift.io cluster -o yaml > cluster-openshift-sdn.yaml

2. Verify that the OVN_SDN_MIGRATION_TIMEOUT environment variable is set and is equal to 0s by running the following command:

   #!/bin/bash
   
   if [ -n "$OVN_SDN_MIGRATION_TIMEOUT" ] && [ "$OVN_SDN_MIGRATION_TIMEOUT" = "0s" ]; then
       unset OVN_SDN_MIGRATION_TIMEOUT
   fi
   
   #loops the timeout command of the script to repeatedly check the cluster Operators until all are available.
   
   co_timeout=${OVN_SDN_MIGRATION_TIMEOUT:-1200s}
   timeout "$co_timeout" bash <<EOT
   until
     oc wait co --all --for='condition=AVAILABLE=True' --timeout=10s && \
     oc wait co --all --for='condition=PROGRESSING=False' --timeout=10s && \
     oc wait co --all --for='condition=DEGRADED=False' --timeout=10s;
   do
     sleep 10
     echo "Some ClusterOperators Degraded=False,Progressing=True,or Available=False";
   done
   EOT

3. Remove the configuration from the Cluster Network Operator (CNO) configuration object by running the following command:

   $ oc patch Network.operator.openshift.io cluster --type='merge' \
   --patch '{"spec":{"migration":null}}'

4. Delete the NodeNetworkConfigurationPolicy (NNCP) custom resource (CR) that defines the primary network interface for the OpenShift SDN network plugin by completing the following steps:

   1. Check that the existing NNCP CR bonded the primary interface to your cluster by entering the following command:

      $ oc get nncp

      Example output

      NAME          STATUS      REASON
      bondmaster0   Available   SuccessfullyConfigured

      Network Manager stores the connection profile for the bonded primary interface in the /etc/NetworkManager/system-connections system path.

   2. Remove the NNCP from your cluster:

      $ oc delete nncp <nncp_manifest_filename>

5. To prepare all the nodes for the migration, set the migration field on the CNO configuration object by running the following command:

   $ oc patch Network.operator.openshift.io cluster --type='merge' \
     --patch '{ "spec": { "migration": { "networkType": "OVNKubernetes" } } }'

   Note

   This step does not deploy OVN-Kubernetes immediately. Instead, specifying the migration field triggers the Machine Config Operator (MCO) to apply new machine configs to all the nodes in the cluster in preparation for the OVN-Kubernetes deployment.

   1. Check that the reboot is finished by running the following command:

      $ oc get mcp

   2. Check that all cluster Operators are available by running the following command:

      $ oc get co

   3. Alternatively: You can disable automatic migration of several OpenShift SDN capabilities to the OVN-Kubernetes equivalents:

      • Egress IPs
      • Egress firewall
      • Multicast

      To disable automatic migration of the configuration for any of the previously noted OpenShift SDN features, specify the following keys:

      $ oc patch Network.operator.openshift.io cluster --type='merge' \
        --patch '{
          "spec": {
            "migration": {
              "networkType": "OVNKubernetes",
              "features": {
                "egressIP": <bool>,
                "egressFirewall": <bool>,
                "multicast": <bool>
              }
            }
          }
        }'

      where:

      bool: Specifies whether to enable migration of the feature. The default is true.

6. Optional: You can customize the following settings for OVN-Kubernetes to meet your network infrastructure requirements:

   • Maximum transmission unit (MTU). Consider the following before customizing the MTU for this optional step:

     • If you use the default MTU, and you want to keep the default MTU during migration, this step can be ignored.
     • If you used a custom MTU, and you want to keep the custom MTU during migration, you must declare the custom MTU value in this step.

     • This step does not work if you want to change the MTU value during migration. Instead, you must first follow the instructions for "Changing the cluster MTU". You can then keep the custom MTU value by performing this procedure and declaring the custom MTU value in this step.

       Note

       OpenShift-SDN and OVN-Kubernetes have different overlay overhead. MTU values should be selected by following the guidelines found on the "MTU value selection" page.

   • Geneve (Generic Network Virtualization Encapsulation) overlay network port
   • OVN-Kubernetes IPv4 internal subnet
   • OVN-Kubernetes IPv6 internal subnet

   To customize either of the previously noted settings, enter and customize the following command. If you do not need to change the default value, omit the key from the patch.

   $ oc patch Network.operator.openshift.io cluster --type=merge \
     --patch '{
       "spec":{
         "defaultNetwork":{
           "ovnKubernetesConfig":{
             "mtu":<mtu>,
             "genevePort":<port>,
             "v4InternalSubnet":"<ipv4_subnet>",
             "v6InternalSubnet":"<ipv6_subnet>"
       }}}}'

   where:

   mtu

   The MTU for the Geneve overlay network. This value is normally configured automatically, but if the nodes in your cluster do not all use the same MTU, then you must set this explicitly to 100 less than the smallest node MTU value.

   port

   The UDP port for the Geneve overlay network. If a value is not specified, the default is 6081. The port cannot be the same as the VXLAN port that is used by OpenShift SDN. The default value for the VXLAN port is 4789.

   ipv4_subnet

   An IPv4 address range for internal use by OVN-Kubernetes. You must ensure that the IP address range does not overlap with any other subnet used by your OpenShift Container Platform installation. The IP address range must be larger than the maximum number of nodes that can be added to the cluster. The default value is 100.64.0.0/16.

   ipv6_subnet

   An IPv6 address range for internal use by OVN-Kubernetes. You must ensure that the IP address range does not overlap with any other subnet used by your OpenShift Container Platform installation. The IP address range must be larger than the maximum number of nodes that can be added to the cluster. The default value is fd98::/48.

   Example patch command to update mtu field

   $ oc patch Network.operator.openshift.io cluster --type=merge \
     --patch '{
       "spec":{
         "defaultNetwork":{
           "ovnKubernetesConfig":{
             "mtu":1200
       }}}}'

7. As the MCO updates machines in each machine config pool, it reboots each node one by one. You must wait until all the nodes are updated. Check the machine config pool status by entering the following command:

   $ oc get mcp

   A successfully updated node has the following status: UPDATED=true, UPDATING=false, DEGRADED=false.

   Note

   By default, the MCO updates one machine per pool at a time, causing the total time the migration takes to increase with the size of the cluster.

8. Confirm the status of the new machine configuration on the hosts:

   1. To list the machine configuration state and the name of the applied machine configuration, enter the following command:

      $ oc describe node | egrep "hostname|machineconfig"

      Example output

      kubernetes.io/hostname=master-0
      machineconfiguration.openshift.io/currentConfig: rendered-master-c53e221d9d24e1c8bb6ee89dd3d8ad7b
      machineconfiguration.openshift.io/desiredConfig: rendered-master-c53e221d9d24e1c8bb6ee89dd3d8ad7b
      machineconfiguration.openshift.io/reason:
      machineconfiguration.openshift.io/state: Done

      Verify that the following statements are true:

      • The value of machineconfiguration.openshift.io/state field is Done.
      • The value of the machineconfiguration.openshift.io/currentConfig field is equal to the value of the machineconfiguration.openshift.io/desiredConfig field.

   2. To confirm that the machine config is correct, enter the following command:

      $ oc get machineconfig <config_name> -o yaml | grep ExecStart

      where <config_name> is the name of the machine config from the machineconfiguration.openshift.io/currentConfig field.

      The machine config must include the following update to the systemd configuration:

      ExecStart=/usr/local/bin/configure-ovs.sh OVNKubernetes

   3. If a node is stuck in the NotReady state, investigate the machine config daemon pod logs and resolve any errors.

      1. To list the pods, enter the following command:

         $ oc get pod -n openshift-machine-config-operator

         Example output

         NAME                                         READY   STATUS    RESTARTS   AGE
         machine-config-controller-75f756f89d-sjp8b   1/1     Running   0          37m
         machine-config-daemon-5cf4b                  2/2     Running   0          43h
         machine-config-daemon-7wzcd                  2/2     Running   0          43h
         machine-config-daemon-fc946                  2/2     Running   0          43h
         machine-config-daemon-g2v28                  2/2     Running   0          43h
         machine-config-daemon-gcl4f                  2/2     Running   0          43h
         machine-config-daemon-l5tnv                  2/2     Running   0          43h
         machine-config-operator-79d9c55d5-hth92      1/1     Running   0          37m
         machine-config-server-bsc8h                  1/1     Running   0          43h
         machine-config-server-hklrm                  1/1     Running   0          43h
         machine-config-server-k9rtx                  1/1     Running   0          43h

         The names for the config daemon pods are in the following format: machine-config-daemon-<seq>. The <seq> value is a random five character alphanumeric sequence.

      2. Display the pod log for the first machine config daemon pod shown in the previous output by enter the following command:

         $ oc logs <pod> -n openshift-machine-config-operator

         where pod is the name of a machine config daemon pod.

      3. Resolve any errors in the logs shown by the output from the previous command.

9. To start the migration, configure the OVN-Kubernetes network plugin by using one of the following commands:

   • To specify the network provider without changing the cluster network IP address block, enter the following command:

     $ oc patch Network.config.openshift.io cluster \
       --type='merge' --patch '{ "spec": { "networkType": "OVNKubernetes" } }'

   • To specify a different cluster network IP address block, enter the following command:

     $ oc patch Network.config.openshift.io cluster \
       --type='merge' --patch '{
         "spec": {
           "clusterNetwork": [
             {
               "cidr": "<cidr>",
               "hostPrefix": <prefix>
             }
           ],
           "networkType": "OVNKubernetes"
         }
       }'

     where cidr is a CIDR block and prefix is the slice of the CIDR block apportioned to each node in your cluster. You cannot use any CIDR block that overlaps with the 100.64.0.0/16 CIDR block because the OVN-Kubernetes network provider uses this block internally.

     Important

     You cannot change the service network address block during the migration.

10. Verify that the Multus daemon set rollout is complete before continuing with subsequent steps:

    $ oc -n openshift-multus rollout status daemonset/multus

    The name of the Multus pods is in the form of multus-<xxxxx> where <xxxxx> is a random sequence of letters. It might take several moments for the pods to restart.

    Example output

    Waiting for daemon set "multus" rollout to finish: 1 out of 6 new pods have been updated...
    ...
    Waiting for daemon set "multus" rollout to finish: 5 of 6 updated pods are available...
    daemon set "multus" successfully rolled out

11. To complete changing the network plugin, reboot each node in your cluster. You can reboot the nodes in your cluster with either of the following approaches:

    Important

    The following scripts reboot all of the nodes in the cluster at the same time. This can cause your cluster to be unstable. Another option is to reboot your nodes manually one at a time. Rebooting nodes one-by-one causes considerable downtime in a cluster with many nodes.

    Cluster Operators will not work correctly before you reboot the nodes.

    • With the oc rsh command, you can use a bash script similar to the following:

      #!/bin/bash
      readarray -t POD_NODES <<< "$(oc get pod -n openshift-machine-config-operator -o wide| grep daemon|awk '{print $1" "$7}')"
      
      for i in "${POD_NODES[@]}"
      do
        read -r POD NODE <<< "$i"
        until oc rsh -n openshift-machine-config-operator "$POD" chroot /rootfs shutdown -r +1
          do
            echo "cannot reboot node $NODE, retry" && sleep 3
          done
      done

    • With the ssh command, you can use a bash script similar to the following. The script assumes that you have configured sudo to not prompt for a password.

      #!/bin/bash
      
      for ip in $(oc get nodes  -o jsonpath='{.items[*].status.addresses[?(@.type=="InternalIP")].address}')
      do
         echo "reboot node $ip"
         ssh -o StrictHostKeyChecking=no core@$ip sudo shutdown -r -t 3
      done

12. Confirm that the migration succeeded:

    1. To confirm that the network plugin is OVN-Kubernetes, enter the following command. The value of status.networkType must be OVNKubernetes.

       $ oc get network.config/cluster -o jsonpath='{.status.networkType}{"\n"}'

    2. To confirm that the cluster nodes are in the Ready state, enter the following command:

       $ oc get nodes

    3. To confirm that your pods are not in an error state, enter the following command:

       $ oc get pods --all-namespaces -o wide --sort-by='{.spec.nodeName}'

       If pods on a node are in an error state, reboot that node.

    4. To confirm that all of the cluster Operators are not in an abnormal state, enter the following command:

       $ oc get co

       The status of every cluster Operator must be the following: AVAILABLE="True", PROGRESSING="False", DEGRADED="False". If a cluster Operator is not available or degraded, check the logs for the cluster Operator for more information.

13. Complete the following steps only if the migration succeeds and your cluster is in a good state:

    1. To remove the migration configuration from the CNO configuration object, enter the following command:

       $ oc patch Network.operator.openshift.io cluster --type='merge' \
         --patch '{ "spec": { "migration": null } }'

    2. To remove custom configuration for the OpenShift SDN network provider, enter the following command:

       $ oc patch Network.operator.openshift.io cluster --type='merge' \
         --patch '{ "spec": { "defaultNetwork": { "openshiftSDNConfig": null } } }'

    3. To remove the OpenShift SDN network provider namespace, enter the following command:

       $ oc delete namespace openshift-sdn

Procedure

1. Update the NetNamespace object with the egress IP address using the following JSON:

    $ oc patch netnamespace <project_name> --type=merge -p \
     '{
       "egressIPs": [
         "<ip_address>"
       ]
     }'

   where:

   <project_name>

   Specifies the name of the project.

   <ip_address>

   Specifies one or more egress IP addresses for the egressIPs array.

   For example, to assign project1 to an IP address of 192.168.1.100 and project2 to an IP address of 192.168.1.101:

   $ oc patch netnamespace project1 --type=merge -p \
     '{"egressIPs": ["192.168.1.100"]}'
   $ oc patch netnamespace project2 --type=merge -p \
     '{"egressIPs": ["192.168.1.101"]}'

   Note

   Because OpenShift SDN manages the NetNamespace object, you can make changes only by modifying the existing NetNamespace object. Do not create a new NetNamespace object.

2. Indicate which nodes can host egress IP addresses by setting the egressCIDRs parameter for each host using the following JSON:

   $ oc patch hostsubnet <node_name> --type=merge -p \
     '{
       "egressCIDRs": [
         "<ip_address_range>", "<ip_address_range>"
       ]
     }'

   where:

   <node_name>

   Specifies a node name.

   <ip_address_range>

   Specifies an IP address range in CIDR format. You can specify more than one address range for the egressCIDRs array.

   For example, to set node1 and node2 to host egress IP addresses in the range 192.168.1.0 to 192.168.1.255:

   $ oc patch hostsubnet node1 --type=merge -p \
     '{"egressCIDRs": ["192.168.1.0/24"]}'
   $ oc patch hostsubnet node2 --type=merge -p \
     '{"egressCIDRs": ["192.168.1.0/24"]}'

   OpenShift Container Platform automatically assigns specific egress IP addresses to available nodes in a balanced way. In this case, it assigns the egress IP address 192.168.1.100 to node1 and the egress IP address 192.168.1.101 to node2 or vice versa.

Procedure

1. Update the NetNamespace object by specifying the following JSON object with the desired IP addresses:

    $ oc patch netnamespace <project_name> --type=merge -p \
     '{
       "egressIPs": [
         "<ip_address>"
       ]
     }'

   where:

   <project_name>

   Specifies the name of the project.

   <ip_address>

   Specifies one or more egress IP addresses for the egressIPs array.

   For example, to assign the project1 project to the IP addresses 192.168.1.100 and 192.168.1.101:

   $ oc patch netnamespace project1 --type=merge \
     -p '{"egressIPs": ["192.168.1.100","192.168.1.101"]}'

   To provide high availability, set the egressIPs value to two or more IP addresses on different nodes. If multiple egress IP addresses are set, then pods use all egress IP addresses roughly equally.

   Note

   Because OpenShift SDN manages the NetNamespace object, you can make changes only by modifying the existing NetNamespace object. Do not create a new NetNamespace object.

2. Manually assign the egress IP address to the node hosts.

   If your cluster is installed on public cloud infrastructure, you must confirm that the node has available IP address capacity.

   Set the egressIPs parameter on the HostSubnet object on the node host. Using the following JSON, include as many IP addresses as you want to assign to that node host:

   $ oc patch hostsubnet <node_name> --type=merge -p \
     '{
       "egressIPs": [
         "<ip_address>",
         "<ip_address>"
         ]
     }'

   where:

   <node_name>

   Specifies a node name.

   <ip_address>

   Specifies an IP address. You can specify more than one IP address for the egressIPs array.

   For example, to specify that node1 should have the egress IPs 192.168.1.100, 192.168.1.101, and 192.168.1.102:

   $ oc patch hostsubnet node1 --type=merge -p \
     '{"egressIPs": ["192.168.1.100", "192.168.1.101", "192.168.1.102"]}'

   In the previous example, all egress traffic for project1 will be routed to the node hosting the specified egress IP, and then connected through Network Address Translation (NAT) to that IP address.

Procedure

1. Create a policy rule:

   1. Create a <policy_name>.yaml file where <policy_name> describes the egress policy rules.
   2. In the file you created, define an egress policy object.

2. Enter the following command to create the policy object. Replace <policy_name> with the name of the policy and <project> with the project that the rule applies to.

   $ oc create -f <policy_name>.yaml -n <project>

   In the following example, a new EgressNetworkPolicy object is created in a project named project1:

   $ oc create -f default.yaml -n project1

   Example output

   egressnetworkpolicy.network.openshift.io/v1 created

3. Optional: Save the <policy_name>.yaml file so that you can make changes later.

Procedure

1. Optional: To view the names of the EgressNetworkPolicy objects defined in your cluster, enter the following command:

   $ oc get egressnetworkpolicy --all-namespaces

2. To inspect a policy, enter the following command. Replace <policy_name> with the name of the policy to inspect.

   $ oc describe egressnetworkpolicy <policy_name>

   Example output

   Name:		default
   Namespace:	project1
   Created:	20 minutes ago
   Labels:		<none>
   Annotations:	<none>
   Rule:		Allow to 1.2.3.0/24
   Rule:		Allow to www.example.com
   Rule:		Deny to 0.0.0.0/0

Procedure

1. Find the name of the EgressNetworkPolicy object for the project. Replace <project> with the name of the project.

   $ oc get -n <project> egressnetworkpolicy

2. Optional: If you did not save a copy of the EgressNetworkPolicy object when you created the egress network firewall, enter the following command to create a copy.

   $ oc get -n <project> egressnetworkpolicy <name> -o yaml > <filename>.yaml

   Replace <project> with the name of the project. Replace <name> with the name of the object. Replace <filename> with the name of the file to save the YAML to.

3. After making changes to the policy rules, enter the following command to replace the EgressNetworkPolicy object. Replace <filename> with the name of the file containing the updated EgressNetworkPolicy object.

   $ oc replace -f <filename>.yaml

Procedure

1. Find the name of the EgressNetworkPolicy object for the project. Replace <project> with the name of the project.

   $ oc get -n <project> egressnetworkpolicy

2. Enter the following command to delete the EgressNetworkPolicy object. Replace <project> with the name of the project and <name> with the name of the object.

   $ oc delete -n <project> egressnetworkpolicy <name>

Procedure

1. Create an egress router pod.

2. To ensure that other pods can find the IP address of the egress router pod, create a service to point to the egress router pod, as in the following example:

   apiVersion: v1
   kind: Service
   metadata:
     name: egress-1
   spec:
     ports:
     - name: http
       port: 80
     - name: https
       port: 443
     type: ClusterIP
     selector:
       name: egress-1

   Your pods can now connect to this service. Their connections are redirected to the corresponding ports on the external server, using the reserved egress IP address.

Procedure

1. Create an egress router pod.

2. To ensure that other pods can find the IP address of the egress router pod, create a service to point to the egress router pod, as in the following example:

   apiVersion: v1
   kind: Service
   metadata:
     name: egress-1
   spec:
     ports:
     - name: http-proxy
       port: 8080 1
     type: ClusterIP
     selector:
       name: egress-1

   1

   Ensure the http port is set to 8080.

3. To configure the client pod (not the egress proxy pod) to use the HTTP proxy, set the http_proxy or https_proxy variables:

   apiVersion: v1
   kind: Pod
   metadata:
     name: app-1
     labels:
       name: app-1
   spec:
     containers:
       env:
       - name: http_proxy
         value: http://egress-1:8080/ 1
       - name: https_proxy
         value: http://egress-1:8080/
       ...

   1

   The service created in the previous step.

   Note

   Using the http_proxy and https_proxy environment variables is not necessary for all setups. If the above does not create a working setup, then consult the documentation for the tool or software you are running in the pod.

Procedure

1. Create an egress router pod.

2. Create a service for the egress router pod:

   1. Create a file named egress-router-service.yaml that contains the following YAML. Set spec.ports to the list of ports that you defined previously for the EGRESS_DNS_PROXY_DESTINATION environment variable.

      apiVersion: v1
      kind: Service
      metadata:
        name: egress-dns-svc
      spec:
        ports:
          ...
        type: ClusterIP
        selector:
          name: egress-dns-proxy

      For example:

      apiVersion: v1
      kind: Service
      metadata:
        name: egress-dns-svc
      spec:
        ports:
        - name: con1
          protocol: TCP
          port: 80
          targetPort: 80
        - name: con2
          protocol: TCP
          port: 100
          targetPort: 100
        type: ClusterIP
        selector:
          name: egress-dns-proxy

   2. To create the service, enter the following command:

      $ oc create -f egress-router-service.yaml

      Pods can now connect to this service. The connections are proxied to the corresponding ports on the external server, using the reserved egress IP address.

Procedure

1. Create a file containing the mapping data for the egress router pod, as in the following example:

   # Egress routes for Project "Test", version 3
   
   80   tcp 203.0.113.25
   
   8080 tcp 203.0.113.26 80
   8443 tcp 203.0.113.26 443
   
   # Fallback
   203.0.113.27

   You can put blank lines and comments into this file.

2. Create a ConfigMap object from the file:

   $ oc delete configmap egress-routes --ignore-not-found

   $ oc create configmap egress-routes \
     --from-file=destination=my-egress-destination.txt

   In the previous command, the egress-routes value is the name of the ConfigMap object to create and my-egress-destination.txt is the name of the file that the data is read from.

   Tip

   You can alternatively apply the following YAML to create the config map:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: egress-routes
   data:
     destination: |
       # Egress routes for Project "Test", version 3
   
       80   tcp 203.0.113.25
   
       8080 tcp 203.0.113.26 80
       8443 tcp 203.0.113.26 443
   
       # Fallback
       203.0.113.27

3. Create an egress router pod definition and specify the configMapKeyRef stanza for the EGRESS_DESTINATION field in the environment stanza:

   ...
   env:
   - name: EGRESS_DESTINATION
     valueFrom:
       configMapKeyRef:
         name: egress-routes
         key: destination
   ...

1. Change your current project to the project that you enabled multicast for. Replace <project> with the project name.

   $ oc project <project>

2. Create a pod to act as a multicast receiver:

   $ cat <<EOF| oc create -f -
   apiVersion: v1
   kind: Pod
   metadata:
     name: mlistener
     labels:
       app: multicast-verify
   spec:
     containers:
       - name: mlistener
         image: registry.access.redhat.com/ubi9
         command: ["/bin/sh", "-c"]
         args:
           ["dnf -y install socat hostname && sleep inf"]
         ports:
           - containerPort: 30102
             name: mlistener
             protocol: UDP
   EOF

3. Create a pod to act as a multicast sender:

   $ cat <<EOF| oc create -f -
   apiVersion: v1
   kind: Pod
   metadata:
     name: msender
     labels:
       app: multicast-verify
   spec:
     containers:
       - name: msender
         image: registry.access.redhat.com/ubi9
         command: ["/bin/sh", "-c"]
         args:
           ["dnf -y install socat && sleep inf"]
   EOF

4. In a new terminal window or tab, start the multicast listener.

   1. Get the IP address for the Pod:

      $ POD_IP=$(oc get pods mlistener -o jsonpath='{.status.podIP}')

   2. Start the multicast listener by entering the following command:

      $ oc exec mlistener -i -t -- \
          socat UDP4-RECVFROM:30102,ip-add-membership=224.1.0.1:$POD_IP,fork EXEC:hostname

5. Start the multicast transmitter.

   1. Get the pod network IP address range:

      $ CIDR=$(oc get Network.config.openshift.io cluster \
          -o jsonpath='{.status.clusterNetwork[0].cidr}')

   2. To send a multicast message, enter the following command:

      $ oc exec msender -i -t -- \
          /bin/bash -c "echo | socat STDIO UDP4-DATAGRAM:224.1.0.1:30102,range=$CIDR,ip-multicast-ttl=64"

      If multicast is working, the previous command returns the following output:

      mlistener

Procedure

1. Use the following command to join projects to an existing project network:

   $ oc adm pod-network join-projects --to=<project1> <project2> <project3>

   Alternatively, instead of specifying specific project names, you can use the --selector=<project_selector> option to specify projects based upon an associated label.

2. Optional: Run the following command to view the pod networks that you have joined together:

   $ oc get netnamespaces

   Projects in the same pod-network have the same network ID in the NETID column.

Procedure

1. Edit the Network.operator.openshift.io custom resource (CR) by running the following command:

   $ oc edit network.operator.openshift.io cluster

2. Modify the kubeProxyConfig parameter in the CR with your changes to the kube-proxy configuration, such as in the following example CR:

   apiVersion: operator.openshift.io/v1
   kind: Network
   metadata:
     name: cluster
   spec:
     kubeProxyConfig:
       iptablesSyncPeriod: 30s
       proxyArguments:
         iptables-min-sync-period: ["30s"]

3. Save the file and exit the text editor.

   The syntax is validated by the oc command when you save the file and exit the editor. If your modifications contain a syntax error, the editor opens the file and displays an error message.

4. Enter the following command to confirm the configuration update:

   $ oc get networks.operator.openshift.io -o yaml

   Example output

   apiVersion: v1
   items:
   - apiVersion: operator.openshift.io/v1
     kind: Network
     metadata:
       name: cluster
     spec:
       clusterNetwork:
       - cidr: 10.128.0.0/14
         hostPrefix: 23
       defaultNetwork:
         type: OpenShiftSDN
       kubeProxyConfig:
         iptablesSyncPeriod: 30s
         proxyArguments:
           iptables-min-sync-period:
           - 30s
       serviceNetwork:
       - 172.30.0.0/16
     status: {}
   kind: List

5. Optional: Enter the following command to confirm that the Cluster Network Operator accepted the configuration change:

   $ oc get clusteroperator network

   Example output

   NAME      VERSION     AVAILABLE   PROGRESSING   DEGRADED   SINCE
   network   4.1.0-0.9   True        False         False      1m

   The AVAILABLE field is True when the configuration update is applied successfully.