Networking

Procedure

1. Create a security group that allows SSH access into the virtual private cloud (VPC) created by the openshift-install command.
2. Create an Amazon EC2 instance on one of the public subnets the installer created.

3. Associate a public IP address with the Amazon EC2 instance that you created.

   Unlike with the OpenShift Container Platform installation, you should associate the Amazon EC2 instance you created with an SSH keypair. It does not matter what operating system you choose for this instance, as it will simply serve as an SSH bastion to bridge the internet into your OpenShift Container Platform cluster’s VPC. The Amazon Machine Image (AMI) you use does matter. With Red Hat Enterprise Linux CoreOS (RHCOS), for example, you can provide keys via Ignition, like the installer does.

4. After you provisioned your Amazon EC2 instance and can SSH into it, you must add the SSH key that you associated with your OpenShift Container Platform installation. This key can be different from the key for the bastion instance, but does not have to be.

   Note

   Direct SSH access is only recommended for disaster recovery. When the Kubernetes API is responsive, run privileged pods instead.

5. Run oc get nodes, inspect the output, and choose one of the nodes that is a master. The hostname looks similar to ip-10-0-1-163.ec2.internal.

6. From the bastion SSH host you manually deployed into Amazon EC2, SSH into that control plane host. Ensure that you use the same SSH key you specified during the installation:

   $ ssh -i <ssh-key-path> core@<master-hostname>

1. Run the following command to view the Deployment status:

   $ oc get -n openshift-network-operator deployment/network-operator

   Example output

   NAME               READY   UP-TO-DATE   AVAILABLE   AGE
   network-operator   1/1     1            1           56m

2. Run the following command to view the state of the Cluster Network Operator:

   $ oc get clusteroperator/network

   Example output

   NAME      VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
   network   4.5.4     True        False         False      50m

   The following fields provide information about the status of the operator: AVAILABLE, PROGRESSING, and DEGRADED. The AVAILABLE field is True when the Cluster Network Operator reports an available status condition.

1. Use the oc get command to view the deployment status:

   $ oc get -n openshift-dns-operator deployment/dns-operator

   Example output

   NAME           READY     UP-TO-DATE   AVAILABLE   AGE
   dns-operator   1/1       1            1           23h

2. Use the oc get command to view the state of the DNS Operator:

   $ oc get clusteroperator/dns

   Example output

   NAME      VERSION     AVAILABLE   PROGRESSING   DEGRADED   SINCE
   dns       4.1.0-0.11  True        False         False      92m

   AVAILABLE, PROGRESSING and DEGRADED provide information about the status of the operator. AVAILABLE is True when at least 1 pod from the CoreDNS daemon set reports an Available status condition.

Procedure

1. Use the oc describe command to view the default dns:

   $ oc describe dns.operator/default

   Example output

   Name:         default
   Namespace:
   Labels:       <none>
   Annotations:  <none>
   API Version:  operator.openshift.io/v1
   Kind:         DNS
   ...
   Status:
     Cluster Domain:  cluster.local 1
     Cluster IP:      172.30.0.10 2
   ...

   1

   The Cluster Domain field is the base DNS domain used to construct fully qualified pod and service domain names.

   2

   The Cluster IP is the address pods query for name resolution. The IP is defined as the 10th address in the service CIDR range.

2. To find the service CIDR of your cluster, use the oc get command:

   $ oc get networks.config/cluster -o jsonpath='{$.status.serviceNetwork}'

Procedure

1. Modify the DNS Operator object named default:

   $ oc edit dns.operator/default

   After you issue the previous command, the Operator creates and updates the config map named dns-default with additional server configuration blocks based on Server. If none of the servers have a zone that matches the query, then name resolution falls back to the upstream DNS servers.

   Configuring DNS forwarding

   apiVersion: operator.openshift.io/v1
   kind: DNS
   metadata:
     name: default
   spec:
     servers:
     - name: example-server 1
       zones: 2
       - example.com
       forwardPlugin:
         policy: Random 3
         upstreams: 4
         - 1.1.1.1
         - 2.2.2.2:5353
     upstreamResolvers: 5
       policy: Random 6
       upstreams: 7
       - type: SystemResolvConf 8
       - type: Network
         address: 1.2.3.4 9
         port: 53 10

   1

   Must comply with the rfc6335 service name syntax.

   2

   Must conform to the definition of a subdomain in the rfc1123 service name syntax. The cluster domain, cluster.local, is an invalid subdomain for the zones field.

   3

   Defines the policy to select upstream resolvers. Default value is Random. You can also use the values RoundRobin, and Sequential.

   4

   A maximum of 15 upstreams is allowed per forwardPlugin.

   5

   Optional. You can use it to override the default policy and forward DNS resolution to the specified DNS resolvers (upstream resolvers) for the default domain. If you do not provide any upstream resolvers, the DNS name queries go to the servers in /etc/resolv.conf.

   6

   Determines the order in which upstream servers are selected for querying. You can specify one of these values: Random, RoundRobin, or Sequential. The default value is Sequential.

   7

   Optional. You can use it to provide upstream resolvers.

   8

   You can specify two types of upstreams - SystemResolvConf and Network. SystemResolvConf configures the upstream to use /etc/resolv.conf and Network defines a Networkresolver. You can specify one or both.

   9

   If the specified type is Network, you must provide an IP address. The address field must be a valid IPv4 or IPv6 address.

   10

   If the specified type is Network, you can optionally provide a port. The port field must have a value between 1 and 65535. If you do not specify a port for the upstream, by default port 853 is tried.

2. Optional: When working in a highly regulated environment, you might need the ability to secure DNS traffic when forwarding requests to upstream resolvers so that you can ensure additional DNS traffic and data privacy. Cluster administrators can configure transport layer security (TLS) for forwarded DNS queries.

   Configuring DNS forwarding with TLS

   apiVersion: operator.openshift.io/v1
   kind: DNS
   metadata:
     name: default
   spec:
     servers:
     - name: example-server 1
       zones: 2
       - example.com
       forwardPlugin:
         transportConfig:
           transport: TLS 3
           tls:
             caBundle:
               name: mycacert
             serverName: dnstls.example.com  4
         policy: Random 5
         upstreams: 6
         - 1.1.1.1
         - 2.2.2.2:5353
     upstreamResolvers: 7
       transportConfig:
         transport: TLS
         tls:
           caBundle:
             name: mycacert
           serverName: dnstls.example.com
       upstreams:
       - type: Network 8
         address: 1.2.3.4 9
         port: 53 10

   1

   Must comply with the rfc6335 service name syntax.

   2

   Must conform to the definition of a subdomain in the rfc1123 service name syntax. The cluster domain, cluster.local, is an invalid subdomain for the zones field. The cluster domain, cluster.local, is an invalid subdomain for zones.

   3

   When configuring TLS for forwarded DNS queries, set the transport field to have the value TLS. By default, CoreDNS caches forwarded connections for 10 seconds. CoreDNS will hold a TCP connection open for those 10 seconds if no request is issued. With large clusters, ensure that your DNS server is aware that it might get many new connections to hold open because you can initiate a connection per node. Set up your DNS hierarchy accordingly to avoid performance issues.

   4

   When configuring TLS for forwarded DNS queries, this is a mandatory server name used as part of the server name indication (SNI) to validate the upstream TLS server certificate.

   5

   Defines the policy to select upstream resolvers. Default value is Random. You can also use the values RoundRobin, and Sequential.

   6

   Required. You can use it to provide upstream resolvers. A maximum of 15 upstreams entries are allowed per forwardPlugin entry.

   7

   Optional. You can use it to override the default policy and forward DNS resolution to the specified DNS resolvers (upstream resolvers) for the default domain. If you do not provide any upstream resolvers, the DNS name queries go to the servers in /etc/resolv.conf.

   8

   Network type indicates that this upstream resolver should handle forwarded requests separately from the upstream resolvers listed in /etc/resolv.conf. Only the Network type is allowed when using TLS and you must provide an IP address.

   9

   The address field must be a valid IPv4 or IPv6 address.

   10

   You can optionally provide a port. The port must have a value between 1 and 65535. If you do not specify a port for the upstream, by default port 853 is tried.

   Note

   If servers is undefined or invalid, the config map only contains the default server.

Verification

1. View the config map:

   $ oc get configmap/dns-default -n openshift-dns -o yaml

   Sample DNS ConfigMap based on previous sample DNS

   apiVersion: v1
   data:
     Corefile: |
       example.com:5353 {
           forward . 1.1.1.1 2.2.2.2:5353
       }
       bar.com:5353 example.com:5353 {
           forward . 3.3.3.3 4.4.4.4:5454 1
       }
       .:5353 {
           errors
           health
           kubernetes cluster.local in-addr.arpa ip6.arpa {
               pods insecure
               upstream
               fallthrough in-addr.arpa ip6.arpa
           }
           prometheus :9153
           forward . /etc/resolv.conf 1.2.3.4:53 {
               policy Random
           }
           cache 30
           reload
       }
   kind: ConfigMap
   metadata:
     labels:
       dns.operator.openshift.io/owning-dns: default
     name: dns-default
     namespace: openshift-dns

   1

   Changes to the forwardPlugin triggers a rolling update of the CoreDNS daemon set.

Procedure

1. Edit the DNS Operator object named default by running the following command:

   $ oc edit dns.operator.openshift.io/default

2. Modify the time-to-live (TTL) caching values:

   Configuring DNS caching

   apiVersion: operator.openshift.io/v1
   kind: DNS
   metadata:
     name: default
   spec:
     cache:
       positiveTTL: 1h 1
       negativeTTL: 0.5h10m 2

   1

   The string value 1h is converted to its respective number of seconds by CoreDNS. If this field is omitted, the value is assumed to be 0s and the cluster uses the internal default value of 900s as a fallback.

   2

   The string value can be a combination of units such as 0.5h10m and is converted to its respective number of seconds by CoreDNS. If this field is omitted, the value is assumed to be 0s and the cluster uses the internal default value of 30s as a fallback.

   Warning

   Setting TTL fields to low values could lead to an increased load on the cluster, any upstream resolvers, or both.

1. Create the IngressNodeFirewallConfig inside the openshift-ingress-node-firewall namespace named ingressnodefirewallconfig.

2. Run the following command to deploy Ingress Node Firewall Operator rules:

   $ oc apply -f rule.yaml

Procedure

1. Run the following command to view all current rules :

   $ oc get ingressnodefirewall

2. Choose one of the returned <resource> names and run the following command to view the rules or configs:

   $ oc get <resource> <name> -o yaml

Procedure

1. Create a custom resource (CR) file named sample-ingress.yaml containing the following:

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     namespace: openshift-ingress-operator
     name: <name> 1
   spec:
     domain: <domain> 2
     endpointPublishingStrategy:
       type: LoadBalancerService
       loadBalancer:
         scope: External 3
         dnsManagementPolicy: Unmanaged 4

   1

   Specify the <name> with a name for the IngressController object.

   2

   Specify the domain based on the DNS record that was created as a prerequisite.

   3

   Specify the scope as External to expose the load balancer externally.

   4

   dnsManagementPolicy indicates if the Ingress Controller is managing the lifecycle of the wildcard DNS record associated with the load balancer. The valid values are Managed and Unmanaged. The default value is Managed.

2. Save the file to apply the changes.

   oc apply -f <name>.yaml 1

Procedure

1. Modify the chosen IngressController to set dnsManagementPolicy:

   SCOPE=$(oc -n openshift-ingress-operator get ingresscontroller <name> -o=jsonpath="{.status.endpointPublishingStrategy.loadBalancer.scope}")
   
   oc -n openshift-ingress-operator patch ingresscontrollers/<name> --type=merge --patch='{"spec":{"endpointPublishingStrategy":{"type":"LoadBalancerService","loadBalancer":{"dnsManagementPolicy":"Unmanaged", "scope":"${SCOPE}"}}}}'

2. Optional: You can delete the associated DNS record in the cloud provider.

Procedure

1. To list the current PodNetworkConnectivityCheck objects, enter the following command:

   $ oc get podnetworkconnectivitycheck -n openshift-network-diagnostics

   Example output

   NAME                                                                                                                                AGE
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0   75m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-1   73m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-2   75m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-kubernetes-apiserver-service-cluster                               75m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-kubernetes-default-service-cluster                                 75m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-load-balancer-api-external                                         75m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-load-balancer-api-internal                                         75m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-network-check-target-ci-ln-x5sv9rb-f76d1-4rzrp-master-0            75m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-network-check-target-ci-ln-x5sv9rb-f76d1-4rzrp-master-1            75m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-network-check-target-ci-ln-x5sv9rb-f76d1-4rzrp-master-2            75m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-network-check-target-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh      74m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-network-check-target-ci-ln-x5sv9rb-f76d1-4rzrp-worker-c-n8mbf      74m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-network-check-target-ci-ln-x5sv9rb-f76d1-4rzrp-worker-d-4hnrz      74m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-network-check-target-service-cluster                               75m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-openshift-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0    75m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-openshift-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-1    75m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-openshift-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-2    74m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-openshift-apiserver-service-cluster                                75m

2. View the connection test logs:

   1. From the output of the previous command, identify the endpoint that you want to review the connectivity logs for.

   2. To view the object, enter the following command:

      $ oc get podnetworkconnectivitycheck <name> \
        -n openshift-network-diagnostics -o yaml

      where <name> specifies the name of the PodNetworkConnectivityCheck object.

      Example output

      apiVersion: controlplane.operator.openshift.io/v1alpha1
      kind: PodNetworkConnectivityCheck
      metadata:
        name: network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0
        namespace: openshift-network-diagnostics
        ...
      spec:
        sourcePod: network-check-source-7c88f6d9f-hmg2f
        targetEndpoint: 10.0.0.4:6443
        tlsClientCert:
          name: ""
      status:
        conditions:
        - lastTransitionTime: "2021-01-13T20:11:34Z"
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: tcp
            connection to 10.0.0.4:6443 succeeded'
          reason: TCPConnectSuccess
          status: "True"
          type: Reachable
        failures:
        - latency: 2.241775ms
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: failed
            to establish a TCP connection to 10.0.0.4:6443: dial tcp 10.0.0.4:6443: connect:
            connection refused'
          reason: TCPConnectError
          success: false
          time: "2021-01-13T20:10:34Z"
        - latency: 2.582129ms
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: failed
            to establish a TCP connection to 10.0.0.4:6443: dial tcp 10.0.0.4:6443: connect:
            connection refused'
          reason: TCPConnectError
          success: false
          time: "2021-01-13T20:09:34Z"
        - latency: 3.483578ms
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: failed
            to establish a TCP connection to 10.0.0.4:6443: dial tcp 10.0.0.4:6443: connect:
            connection refused'
          reason: TCPConnectError
          success: false
          time: "2021-01-13T20:08:34Z"
        outages:
        - end: "2021-01-13T20:11:34Z"
          endLogs:
          - latency: 2.032018ms
            message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0:
              tcp connection to 10.0.0.4:6443 succeeded'
            reason: TCPConnect
            success: true
            time: "2021-01-13T20:11:34Z"
          - latency: 2.241775ms
            message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0:
              failed to establish a TCP connection to 10.0.0.4:6443: dial tcp 10.0.0.4:6443:
              connect: connection refused'
            reason: TCPConnectError
            success: false
            time: "2021-01-13T20:10:34Z"
          - latency: 2.582129ms
            message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0:
              failed to establish a TCP connection to 10.0.0.4:6443: dial tcp 10.0.0.4:6443:
              connect: connection refused'
            reason: TCPConnectError
            success: false
            time: "2021-01-13T20:09:34Z"
          - latency: 3.483578ms
            message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0:
              failed to establish a TCP connection to 10.0.0.4:6443: dial tcp 10.0.0.4:6443:
              connect: connection refused'
            reason: TCPConnectError
            success: false
            time: "2021-01-13T20:08:34Z"
          message: Connectivity restored after 2m59.999789186s
          start: "2021-01-13T20:08:34Z"
          startLogs:
          - latency: 3.483578ms
            message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0:
              failed to establish a TCP connection to 10.0.0.4:6443: dial tcp 10.0.0.4:6443:
              connect: connection refused'
            reason: TCPConnectError
            success: false
            time: "2021-01-13T20:08:34Z"
        successes:
        - latency: 2.845865ms
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: tcp
            connection to 10.0.0.4:6443 succeeded'
          reason: TCPConnect
          success: true
          time: "2021-01-13T21:14:34Z"
        - latency: 2.926345ms
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: tcp
            connection to 10.0.0.4:6443 succeeded'
          reason: TCPConnect
          success: true
          time: "2021-01-13T21:13:34Z"
        - latency: 2.895796ms
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: tcp
            connection to 10.0.0.4:6443 succeeded'
          reason: TCPConnect
          success: true
          time: "2021-01-13T21:12:34Z"
        - latency: 2.696844ms
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: tcp
            connection to 10.0.0.4:6443 succeeded'
          reason: TCPConnect
          success: true
          time: "2021-01-13T21:11:34Z"
        - latency: 1.502064ms
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: tcp
            connection to 10.0.0.4:6443 succeeded'
          reason: TCPConnect
          success: true
          time: "2021-01-13T21:10:34Z"
        - latency: 1.388857ms
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: tcp
            connection to 10.0.0.4:6443 succeeded'
          reason: TCPConnect
          success: true
          time: "2021-01-13T21:09:34Z"
        - latency: 1.906383ms
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: tcp
            connection to 10.0.0.4:6443 succeeded'
          reason: TCPConnect
          success: true
          time: "2021-01-13T21:08:34Z"
        - latency: 2.089073ms
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: tcp
            connection to 10.0.0.4:6443 succeeded'
          reason: TCPConnect
          success: true
          time: "2021-01-13T21:07:34Z"
        - latency: 2.156994ms
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: tcp
            connection to 10.0.0.4:6443 succeeded'
          reason: TCPConnect
          success: true
          time: "2021-01-13T21:06:34Z"
        - latency: 1.777043ms
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: tcp
            connection to 10.0.0.4:6443 succeeded'
          reason: TCPConnect
          success: true
          time: "2021-01-13T21:05:34Z"

1. To obtain the current MTU for the cluster network, enter the following command:

   $ oc describe network.config cluster

   Example output

   ...
   Status:
     Cluster Network:
       Cidr:               10.217.0.0/22
       Host Prefix:        23
     Cluster Network MTU:  1400
     Network Type:         OpenShiftSDN
     Service Network:
       10.217.4.0/23
   ...

2. Prepare your configuration for the hardware MTU:

   • If your hardware MTU is specified with DHCP, update your DHCP configuration such as with the following dnsmasq configuration:

     dhcp-option-force=26,<mtu>

     where:

     <mtu>

     Specifies the hardware MTU for the DHCP server to advertise.

   • If your hardware MTU is specified with a kernel command line with PXE, update that configuration accordingly.

   • If your hardware MTU is specified in a NetworkManager connection configuration, complete the following steps. This approach is the default for OpenShift Container Platform if you do not explicitly specify your network configuration with DHCP, a kernel command line, or some other method. Your cluster nodes must all use the same underlying network configuration for the following procedure to work unmodified.

     1. Find the primary network interface:

        • If you are using the OpenShift SDN network plugin, enter the following command:

          $ oc debug node/<node_name> -- chroot /host ip route list match 0.0.0.0/0 | awk '{print $5 }'

          where:

          <node_name>

          Specifies the name of a node in your cluster.

        • If you are using the OVN-Kubernetes network plugin, enter the following command:

          $ oc debug node/<node_name> -- chroot /host nmcli -g connection.interface-name c show ovs-if-phys0

          where:

          <node_name>

          Specifies the name of a node in your cluster.

     2. Create the following NetworkManager configuration in the <interface>-mtu.conf file:

        Example NetworkManager connection configuration

        [connection-<interface>-mtu]
        match-device=interface-name:<interface>
        ethernet.mtu=<mtu>

        where:

        <mtu>

        Specifies the new hardware MTU value.

        <interface>

        Specifies the primary network interface name.

     3. Create two MachineConfig objects, one for the control plane nodes and another for the worker nodes in your cluster:

        1. Create the following Butane config in the control-plane-interface.bu file:

           variant: openshift
           version: 4.13.0
           metadata:
             name: 01-control-plane-interface
             labels:
               machineconfiguration.openshift.io/role: master
           storage:
             files:
               - path: /etc/NetworkManager/conf.d/99-<interface>-mtu.conf 1
                 contents:
                   local: <interface>-mtu.conf 2
                 mode: 0600

           1 1

           Specify the NetworkManager connection name for the primary network interface.

           2

           Specify the local filename for the updated NetworkManager configuration file from the previous step.

        2. Create the following Butane config in the worker-interface.bu file:

           variant: openshift
           version: 4.13.0
           metadata:
             name: 01-worker-interface
             labels:
               machineconfiguration.openshift.io/role: worker
           storage:
             files:
               - path: /etc/NetworkManager/conf.d/99-<interface>-mtu.conf 1
                 contents:
                   local: <interface>-mtu.conf 2
                 mode: 0600

           1

           Specify the NetworkManager connection name for the primary network interface.

           2

           Specify the local filename for the updated NetworkManager configuration file from the previous step.

        3. Create MachineConfig objects from the Butane configs by running the following command:

           $ for manifest in control-plane-interface worker-interface; do
               butane --files-dir . $manifest.bu > $manifest.yaml
             done

3. To begin the MTU migration, specify the migration configuration by entering the following command. The Machine Config Operator performs a rolling reboot of the nodes in the cluster in preparation for the MTU change.

   $ oc patch Network.operator.openshift.io cluster --type=merge --patch \
     '{"spec": { "migration": { "mtu": { "network": { "from": <overlay_from>, "to": <overlay_to> } , "machine": { "to" : <machine_to> } } } } }'

   where:

   <overlay_from>

   Specifies the current cluster network MTU value.

   <overlay_to>

   Specifies the target MTU for the cluster network. This value is set relative to the value for <machine_to> and for OVN-Kubernetes must be 100 less and for OpenShift SDN must be 50 less.

   <machine_to>

   Specifies the MTU for the primary network interface on the underlying host network.

   Example that increases the cluster MTU

   $ oc patch Network.operator.openshift.io cluster --type=merge --patch \
     '{"spec": { "migration": { "mtu": { "network": { "from": 1400, "to": 9000 } , "machine": { "to" : 9100} } } } }'

4. As the MCO updates machines in each machine config pool, it reboots each node one by one. You must wait until all the nodes are updated. Check the machine config pool status by entering the following command:

   $ oc get mcp

   A successfully updated node has the following status: UPDATED=true, UPDATING=false, DEGRADED=false.

   Note

   By default, the MCO updates one machine per pool at a time, causing the total time the migration takes to increase with the size of the cluster.

5. Confirm the status of the new machine configuration on the hosts:

   1. To list the machine configuration state and the name of the applied machine configuration, enter the following command:

      $ oc describe node | egrep "hostname|machineconfig"

      Example output

      kubernetes.io/hostname=master-0
      machineconfiguration.openshift.io/currentConfig: rendered-master-c53e221d9d24e1c8bb6ee89dd3d8ad7b
      machineconfiguration.openshift.io/desiredConfig: rendered-master-c53e221d9d24e1c8bb6ee89dd3d8ad7b
      machineconfiguration.openshift.io/reason:
      machineconfiguration.openshift.io/state: Done

      Verify that the following statements are true:

      • The value of machineconfiguration.openshift.io/state field is Done.
      • The value of the machineconfiguration.openshift.io/currentConfig field is equal to the value of the machineconfiguration.openshift.io/desiredConfig field.

   2. To confirm that the machine config is correct, enter the following command:

      $ oc get machineconfig <config_name> -o yaml | grep ExecStart

      where <config_name> is the name of the machine config from the machineconfiguration.openshift.io/currentConfig field.

      The machine config must include the following update to the systemd configuration:

      ExecStart=/usr/local/bin/mtu-migration.sh

6. Update the underlying network interface MTU value:

   • If you are specifying the new MTU with a NetworkManager connection configuration, enter the following command. The MachineConfig Operator automatically performs a rolling reboot of the nodes in your cluster.

     $ for manifest in control-plane-interface worker-interface; do
         oc create -f $manifest.yaml
       done

   • If you are specifying the new MTU with a DHCP server option or a kernel command line and PXE, make the necessary changes for your infrastructure.

7. As the MCO updates machines in each machine config pool, it reboots each node one by one. You must wait until all the nodes are updated. Check the machine config pool status by entering the following command:

   $ oc get mcp

   A successfully updated node has the following status: UPDATED=true, UPDATING=false, DEGRADED=false.

   Note

   By default, the MCO updates one machine per pool at a time, causing the total time the migration takes to increase with the size of the cluster.

8. Confirm the status of the new machine configuration on the hosts:

   1. To list the machine configuration state and the name of the applied machine configuration, enter the following command:

      $ oc describe node | egrep "hostname|machineconfig"

      Example output

      kubernetes.io/hostname=master-0
      machineconfiguration.openshift.io/currentConfig: rendered-master-c53e221d9d24e1c8bb6ee89dd3d8ad7b
      machineconfiguration.openshift.io/desiredConfig: rendered-master-c53e221d9d24e1c8bb6ee89dd3d8ad7b
      machineconfiguration.openshift.io/reason:
      machineconfiguration.openshift.io/state: Done

      Verify that the following statements are true:

      • The value of machineconfiguration.openshift.io/state field is Done.
      • The value of the machineconfiguration.openshift.io/currentConfig field is equal to the value of the machineconfiguration.openshift.io/desiredConfig field.

   2. To confirm that the machine config is correct, enter the following command:

      $ oc get machineconfig <config_name> -o yaml | grep path:

      where <config_name> is the name of the machine config from the machineconfiguration.openshift.io/currentConfig field.

      If the machine config is successfully deployed, the previous output contains the /etc/NetworkManager/conf.d/99-<interface>-mtu.conf file path and the ExecStart=/usr/local/bin/mtu-migration.sh line.

9. To finalize the MTU migration, enter one of the following commands:

   • If you are using the OVN-Kubernetes network plugin:

     $ oc patch Network.operator.openshift.io cluster --type=merge --patch \
       '{"spec": { "migration": null, "defaultNetwork":{ "ovnKubernetesConfig": { "mtu": <mtu> }}}}'

     where:

     <mtu>

     Specifies the new cluster network MTU that you specified with <overlay_to>.

   • If you are using the OpenShift SDN network plugin:

     $ oc patch Network.operator.openshift.io cluster --type=merge --patch \
       '{"spec": { "migration": null, "defaultNetwork":{ "openshiftSDNConfig": { "mtu": <mtu> }}}}'

     where:

     <mtu>

     Specifies the new cluster network MTU that you specified with <overlay_to>.

10. After finalizing the MTU migration, each MCP node is rebooted one by one. You must wait until all the nodes are updated. Check the machine config pool status by entering the following command:

    $ oc get mcp

    A successfully updated node has the following status: UPDATED=true, UPDATING=false, DEGRADED=false.

1. To get the current MTU for the cluster network, enter the following command:

   $ oc describe network.config cluster

2. Get the current MTU for the primary network interface of a node.

   1. To list the nodes in your cluster, enter the following command:

      $ oc get nodes

   2. To obtain the current MTU setting for the primary network interface on a node, enter the following command:

      $ oc debug node/<node> -- chroot /host ip address show <interface>

      where:

      <node>

      Specifies a node from the output from the previous step.

      <interface>

      Specifies the primary network interface name for the node.

      Example output

      ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8051

Procedure

1. To expand the node port range, enter the following command. Replace <port> with the largest port number in the new range.

   $ oc patch network.config.openshift.io cluster --type=merge -p \
     '{
       "spec":
         { "serviceNodePortRange": "30000-<port>" }
     }'

   Tip

   You can alternatively apply the following YAML to update the node port range:

   apiVersion: config.openshift.io/v1
   kind: Network
   metadata:
     name: cluster
   spec:
     serviceNodePortRange: "30000-<port>"

   Example output

   network.config.openshift.io/cluster patched

2. To confirm that the configuration is active, enter the following command. It can take several minutes for the update to apply.

   $ oc get configmaps -n openshift-kube-apiserver config \
     -o jsonpath="{.data['config\.yaml']}" | \
     grep -Eo '"service-node-port-range":["[[:digit:]]+-[[:digit:]]+"]'

   Example output

   "service-node-port-range":["30000-33000"]

Procedure

1. To obtain the cluster network range and host prefix for your cluster, enter the following command:

   $ oc get network.operator.openshift.io \
     -o jsonpath="{.items[0].spec.clusterNetwork}"

   Example output

   [{"cidr":"10.217.0.0/22","hostPrefix":23}]

2. To expand the cluster network IP address range, enter the following command. Use the CIDR IP address range and host prefix returned from the output of the previous command.

   $ oc patch Network.config.openshift.io cluster --type='merge' --patch \
     '{
       "spec":{
         "clusterNetwork": [ {"cidr":"<network>/<cidr>","hostPrefix":<prefix>} ],
         "networkType": "OVNKubernetes"
       }
     }'

   where:

   <network>

   Specifies the network part of the cidr field that you obtained from the previous step. You cannot change this value.

   <cidr>

   Specifies the network prefix length. For example, 14. Change this value to a smaller number than the value from the output in the previous step to expand the cluster network range.

   <prefix>

   Specifies the current host prefix for your cluster. This value must be the same value for the hostPrefix field that you obtained from the previous step.

   Example command

   $ oc patch Network.config.openshift.io cluster --type='merge' --patch \
     '{
       "spec":{
         "clusterNetwork": [ {"cidr":"10.217.0.0/14","hostPrefix": 23} ],
         "networkType": "OVNKubernetes"
       }
     }'

   Example output

   network.config.openshift.io/cluster patched

3. To confirm that the configuration is active, enter the following command. It can take up to 30 minutes for this change to take effect.

   $ oc get network.operator.openshift.io \
     -o jsonpath="{.items[0].spec.clusterNetwork}"

   Example output

   [{"cidr":"10.217.0.0/14","hostPrefix":23}]

Procedure

1. Create an IP failover service account:

   $ oc create sa ipfailover

2. Update security context constraints (SCC) for hostNetwork:

   $ oc adm policy add-scc-to-user privileged -z ipfailover

   $ oc adm policy add-scc-to-user hostnetwork -z ipfailover

3. Red Hat OpenStack Platform (RHOSP) only: Complete the following steps to make a failover VIP address reachable on RHOSP ports.

   1. Use the RHOSP CLI to show the default RHOSP API and VIP addresses in the allowed_address_pairs parameter of your RHOSP cluster:

      $ openstack port show <cluster_name> -c allowed_address_pairs

      Output example

      *Field*                  *Value*
      allowed_address_pairs    ip_address='192.168.0.5', mac_address='fa:16:3e:31:f9:cb'
                               ip_address='192.168.0.7', mac_address='fa:16:3e:31:f9:cb'

   2. Set a different VIP address for the IP failover deployment and make the address reachable on RHOSP ports by entering the following command in the RHOSP CLI. Do not set any default RHOSP API and VIP addresses as the failover VIP address for the IP failover deployment.

      Example of adding the 1.1.1.1 failover IP address as an allowed address on RHOSP ports.

      $ openstack port set <cluster_name> --allowed-address ip-address=1.1.1.1,mac-address=fa:fa:16:3e:31:f9:cb

   3. Create a deployment YAML file to configure IP failover for your deployment. See "Example deployment YAML for IP failover configuration" in a later step.

   4. Specify the following specification in the IP failover deployment so that you pass the failover VIP address to the OPENSHIFT_HA_VIRTUAL_IPS environment variable:

      Example of adding the 1.1.1.1 VIP address to OPENSHIFT_HA_VIRTUAL_IPS

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: ipfailover-keepalived
      # ...
            spec:
                env:
                - name: OPENSHIFT_HA_VIRTUAL_IPS
                value: "1.1.1.1"
      # ...

4. Create a deployment YAML file to configure IP failover.

   Note

   For Red Hat OpenStack Platform (RHOSP), you do not need to re-create the deployment YAML file. You already created this file as part of the earlier instructions.

   Example deployment YAML for IP failover configuration

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: ipfailover-keepalived 1
     labels:
       ipfailover: hello-openshift
   spec:
     strategy:
       type: Recreate
     replicas: 2
     selector:
       matchLabels:
         ipfailover: hello-openshift
     template:
       metadata:
         labels:
           ipfailover: hello-openshift
       spec:
         serviceAccountName: ipfailover
         privileged: true
         hostNetwork: true
         nodeSelector:
           node-role.kubernetes.io/worker: ""
         containers:
         - name: openshift-ipfailover
           image: registry.redhat.io/openshift4/ose-keepalived-ipfailover:v4.13
           ports:
           - containerPort: 63000
             hostPort: 63000
           imagePullPolicy: IfNotPresent
           securityContext:
             privileged: true
           volumeMounts:
           - name: lib-modules
             mountPath: /lib/modules
             readOnly: true
           - name: host-slash
             mountPath: /host
             readOnly: true
             mountPropagation: HostToContainer
           - name: etc-sysconfig
             mountPath: /etc/sysconfig
             readOnly: true
           - name: config-volume
             mountPath: /etc/keepalive
           env:
           - name: OPENSHIFT_HA_CONFIG_NAME
             value: "ipfailover"
           - name: OPENSHIFT_HA_VIRTUAL_IPS 2
             value: "1.1.1.1-2"
           - name: OPENSHIFT_HA_VIP_GROUPS 3
             value: "10"
           - name: OPENSHIFT_HA_NETWORK_INTERFACE 4
             value: "ens3" #The host interface to assign the VIPs
           - name: OPENSHIFT_HA_MONITOR_PORT 5
             value: "30060"
           - name: OPENSHIFT_HA_VRRP_ID_OFFSET 6
             value: "0"
           - name: OPENSHIFT_HA_REPLICA_COUNT 7
             value: "2" #Must match the number of replicas in the deployment
           - name: OPENSHIFT_HA_USE_UNICAST
             value: "false"
           #- name: OPENSHIFT_HA_UNICAST_PEERS
             #value: "10.0.148.40,10.0.160.234,10.0.199.110"
           - name: OPENSHIFT_HA_IPTABLES_CHAIN 8
             value: "INPUT"
           #- name: OPENSHIFT_HA_NOTIFY_SCRIPT 9
           #  value: /etc/keepalive/mynotifyscript.sh
           - name: OPENSHIFT_HA_CHECK_SCRIPT 10
             value: "/etc/keepalive/mycheckscript.sh"
           - name: OPENSHIFT_HA_PREEMPTION 11
             value: "preempt_delay 300"
           - name: OPENSHIFT_HA_CHECK_INTERVAL 12
             value: "2"
           livenessProbe:
             initialDelaySeconds: 10
             exec:
               command:
               - pgrep
               - keepalived
         volumes:
         - name: lib-modules
           hostPath:
             path: /lib/modules
         - name: host-slash
           hostPath:
             path: /
         - name: etc-sysconfig
           hostPath:
             path: /etc/sysconfig
         # config-volume contains the check script
         # created with `oc create configmap keepalived-checkscript --from-file=mycheckscript.sh`
         - configMap:
             defaultMode: 0755
             name: keepalived-checkscript
           name: config-volume
         imagePullSecrets:
           - name: openshift-pull-secret 13

   1

   The name of the IP failover deployment.

   2

   The list of IP address ranges to replicate. This must be provided. For example, 1.2.3.4-6,1.2.3.9.

   3

   The number of groups to create for VRRP. If not set, a group is created for each virtual IP range specified with the OPENSHIFT_HA_VIP_GROUPS variable.

   4

   The interface name that IP failover uses to send VRRP traffic. By default, eth0 is used.

   5

   The IP failover pod tries to open a TCP connection to this port on each VIP. If connection is established, the service is considered to be running. If this port is set to 0, the test always passes. The default value is 80.

   6

   The offset value used to set the virtual router IDs. Using different offset values allows multiple IP failover configurations to exist within the same cluster. The default offset is 0, and the allowed range is 0 through 255.

   7

   The number of replicas to create. This must match spec.replicas value in IP failover deployment configuration. The default value is 2.

   8

   The name of the iptables chain to automatically add an iptables rule to allow the VRRP traffic on. If the value is not set, an iptables rule is not added. If the chain does not exist, it is not created, and Keepalived operates in unicast mode. The default is INPUT.

   9

   The full path name in the pod file system of a script that is run whenever the state changes.

   10

   The full path name in the pod file system of a script that is periodically run to verify the application is operating.

   11

   The strategy for handling a new higher priority host. The default value is preempt_delay 300, which causes a Keepalived instance to take over a VIP after 5 minutes if a lower-priority master is holding the VIP.

   12

   The period, in seconds, that the check script is run. The default value is 2.

   13

   Create the pull secret before creating the deployment, otherwise you will get an error when creating the deployment.

Procedure

1. Create the desired script and create a ConfigMap object to hold it. The script has no input arguments and must return 0 for OK and 1 for fail.

   The check script, mycheckscript.sh:

   #!/bin/bash
       # Whatever tests are needed
       # E.g., send request and verify response
   exit 0

2. Create the ConfigMap object :

   $ oc create configmap mycustomcheck --from-file=mycheckscript.sh

3. Add the script to the pod. The defaultMode for the mounted ConfigMap object files must able to run by using oc commands or by editing the deployment configuration. A value of 0755, 493 decimal, is typical:

   $ oc set env deploy/ipfailover-keepalived \
       OPENSHIFT_HA_CHECK_SCRIPT=/etc/keepalive/mycheckscript.sh

   $ oc set volume deploy/ipfailover-keepalived --add --overwrite \
       --name=config-volume \
       --mount-path=/etc/keepalive \
       --source='{"configMap": { "name": "mycustomcheck", "defaultMode": 493}}'

   Note

   The oc set env command is whitespace sensitive. There must be no whitespace on either side of the = sign.

   Tip

   You can alternatively edit the ipfailover-keepalived deployment configuration:

   $ oc edit deploy ipfailover-keepalived

       spec:
         containers:
         - env:
           - name: OPENSHIFT_HA_CHECK_SCRIPT  1
             value: /etc/keepalive/mycheckscript.sh
   ...
           volumeMounts: 2
           - mountPath: /etc/keepalive
             name: config-volume
         dnsPolicy: ClusterFirst
   ...
         volumes: 3
         - configMap:
             defaultMode: 0755 4
             name: customrouter
           name: config-volume
   ...

   1

   In the spec.container.env field, add the OPENSHIFT_HA_CHECK_SCRIPT environment variable to point to the mounted script file.

   2

   Add the spec.container.volumeMounts field to create the mount point.

   3

   Add a new spec.volumes field to mention the config map.

   4

   This sets run permission on the files. When read back, it is displayed in decimal, 493.

   Save the changes and exit the editor. This restarts ipfailover-keepalived.

Procedure

1. Optional: Identify and delete any check and notify scripts that are stored as config maps:

   1. Identify whether any pods for IP failover use a config map as a volume:

      $ oc get pod -l ipfailover \
        -o jsonpath="\
      {range .items[?(@.spec.volumes[*].configMap)]}
      {'Namespace: '}{.metadata.namespace}
      {'Pod:       '}{.metadata.name}
      {'Volumes that use config maps:'}
      {range .spec.volumes[?(@.configMap)]}  {'volume:    '}{.name}
        {'configMap: '}{.configMap.name}{'\n'}{end}
      {end}"

      Example output

      Namespace: default
      Pod:       keepalived-worker-59df45db9c-2x9mn
      Volumes that use config maps:
        volume:    config-volume
        configMap: mycustomcheck

   2. If the preceding step provided the names of config maps that are used as volumes, delete the config maps:

      $ oc delete configmap <configmap_name>

2. Identify an existing deployment for IP failover:

   $ oc get deployment -l ipfailover

   Example output

   NAMESPACE   NAME         READY   UP-TO-DATE   AVAILABLE   AGE
   default     ipfailover   2/2     2            2           105d

3. Delete the deployment:

   $ oc delete deployment <ipfailover_deployment_name>

4. Remove the ipfailover service account:

   $ oc delete sa ipfailover

5. Run a job that removes the IP tables rule that was added when IP failover was initially configured:

   1. Create a file such as remove-ipfailover-job.yaml with contents that are similar to the following example:

      apiVersion: batch/v1
      kind: Job
      metadata:
        generateName: remove-ipfailover-
        labels:
          app: remove-ipfailover
      spec:
        template:
          metadata:
            name: remove-ipfailover
          spec:
            containers:
            - name: remove-ipfailover
              image: registry.redhat.io/openshift4/ose-keepalived-ipfailover:v4.13
              command: ["/var/lib/ipfailover/keepalived/remove-failover.sh"]
            nodeSelector: 1
              kubernetes.io/hostname: <host_name>  2
            restartPolicy: Never

      1

      The nodeSelector is likely the same as the selector used in the old IP failover deployment.

      2

      Run the job for each node in your cluster that was configured for IP failover and replace the hostname each time.

   2. Run the job:

      $ oc create -f remove-ipfailover-job.yaml

      Example output

      job.batch/remove-ipfailover-2h8dm created

Procedure

1. Create a network attachment definition, such as tuning-example.yaml, with the following content:

   apiVersion: "k8s.cni.cncf.io/v1"
   kind: NetworkAttachmentDefinition
   metadata:
     name: <name> 1
     namespace: default 2
   spec:
     config: '{
       "cniVersion": "0.4.0", 3
       "name": "<name>", 4
       "plugins": [{
          "type": "<main_CNI_plugin>" 5
         },
         {
          "type": "tuning", 6
          "sysctl": {
               "net.ipv4.conf.IFNAME.accept_redirects": "1" 7
           }
         }
        ]
   }

   1

   Specifies the name for the additional network attachment to create. The name must be unique within the specified namespace.

   2

   Specifies the namespace that the object is associated with.

   3

   Specifies the CNI specification version.

   4

   Specifies the name for the configuration. It is recommended to match the configuration name to the name value of the network attachment definition.

   5

   Specifies the name of the main CNI plugin to configure.

   6

   Specifies the name of the CNI meta plugin.

   7

   Specifies the sysctl to set.

   An example yaml file is shown here:

   apiVersion: "k8s.cni.cncf.io/v1"
   kind: NetworkAttachmentDefinition
   metadata:
     name: tuningnad
     namespace: default
   spec:
     config: '{
       "cniVersion": "0.4.0",
       "name": "tuningnad",
       "plugins": [{
         "type": "bridge"
         },
         {
         "type": "tuning",
         "sysctl": {
            "net.ipv4.conf.IFNAME.accept_redirects": "1"
           }
       }
     ]
   }'

2. Apply the yaml by running the following command:

   $ oc apply -f tuning-example.yaml

   Example output

   networkattachmentdefinition.k8.cni.cncf.io/tuningnad created

3. Create a pod such as examplepod.yaml with the network attachment definition similar to the following:

   apiVersion: v1
   kind: Pod
   metadata:
     name: tunepod
     namespace: default
     annotations:
       k8s.v1.cni.cncf.io/networks: tuningnad 1
   spec:
     containers:
     - name: podexample
       image: centos
       command: ["/bin/bash", "-c", "sleep INF"]
       securityContext:
         runAsUser: 2000 2
         runAsGroup: 3000 3
         allowPrivilegeEscalation: false 4
         capabilities: 5
           drop: ["ALL"]
     securityContext:
       runAsNonRoot: true 6
       seccompProfile: 7
         type: RuntimeDefault

   1

   Specify the name of the configured NetworkAttachmentDefinition.

   2

   runAsUser controls which user ID the container is run with.

   3

   runAsGroup controls which primary group ID the containers is run with.

   4

   allowPrivilegeEscalation determines if a pod can request to allow privilege escalation. If unspecified, it defaults to true. This boolean directly controls whether the no_new_privs flag gets set on the container process.

   5

   capabilities permit privileged actions without giving full root access. This policy ensures all capabilities are dropped from the pod.

   6

   runAsNonRoot: true requires that the container will run with a user with any UID other than 0.

   7

   RuntimeDefault enables the default seccomp profile for a pod or container workload.

4. Apply the yaml by running the following command:

   $ oc apply -f examplepod.yaml

5. Verify that the pod is created by running the following command:

   $ oc get pod

   Example output

   NAME      READY   STATUS    RESTARTS   AGE
   tunepod   1/1     Running   0          47s

6. Log in to the pod by running the following command:

   $ oc rsh tunepod

7. Verify the values of the configured sysctl flags. For example, find the value net.ipv4.conf.net1.accept_redirects by running the following command:

   sh-4.4# sysctl net.ipv4.conf.net1.accept_redirects

   Expected output

   net.ipv4.conf.net1.accept_redirects = 1

Procedure

1. Create a file named load-sctp-module.yaml that contains the following YAML definition:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     name: load-sctp-module
     labels:
       machineconfiguration.openshift.io/role: worker
   spec:
     config:
       ignition:
         version: 3.2.0
       storage:
         files:
           - path: /etc/modprobe.d/sctp-blacklist.conf
             mode: 0644
             overwrite: true
             contents:
               source: data:,
           - path: /etc/modules-load.d/sctp-load.conf
             mode: 0644
             overwrite: true
             contents:
               source: data:,sctp

2. To create the MachineConfig object, enter the following command:

   $ oc create -f load-sctp-module.yaml

3. Optional: To watch the status of the nodes while the MachineConfig Operator applies the configuration change, enter the following command. When the status of a node transitions to Ready, the configuration update is applied.

   $ oc get nodes

Procedure

1. Create a pod starts an SCTP listener:

   1. Create a file named sctp-server.yaml that defines a pod with the following YAML:

      apiVersion: v1
      kind: Pod
      metadata:
        name: sctpserver
        labels:
          app: sctpserver
      spec:
        containers:
          - name: sctpserver
            image: registry.access.redhat.com/ubi9/ubi
            command: ["/bin/sh", "-c"]
            args:
              ["dnf install -y nc && sleep inf"]
            ports:
              - containerPort: 30102
                name: sctpserver
                protocol: SCTP

   2. Create the pod by entering the following command:

      $ oc create -f sctp-server.yaml

2. Create a service for the SCTP listener pod.

   1. Create a file named sctp-service.yaml that defines a service with the following YAML:

      apiVersion: v1
      kind: Service
      metadata:
        name: sctpservice
        labels:
          app: sctpserver
      spec:
        type: NodePort
        selector:
          app: sctpserver
        ports:
          - name: sctpserver
            protocol: SCTP
            port: 30102
            targetPort: 30102

   2. To create the service, enter the following command:

      $ oc create -f sctp-service.yaml

3. Create a pod for the SCTP client.

   1. Create a file named sctp-client.yaml with the following YAML:

      apiVersion: v1
      kind: Pod
      metadata:
        name: sctpclient
        labels:
          app: sctpclient
      spec:
        containers:
          - name: sctpclient
            image: registry.access.redhat.com/ubi9/ubi
            command: ["/bin/sh", "-c"]
            args:
              ["dnf install -y nc && sleep inf"]

   2. To create the Pod object, enter the following command:

      $ oc apply -f sctp-client.yaml

4. Run an SCTP listener on the server.

   1. To connect to the server pod, enter the following command:

      $ oc rsh sctpserver

   2. To start the SCTP listener, enter the following command:

      $ nc -l 30102 --sctp

5. Connect to the SCTP listener on the server.

   1. Open a new terminal window or tab in your terminal program.

   2. Obtain the IP address of the sctpservice service. Enter the following command:

      $ oc get services sctpservice -o go-template='{{.spec.clusterIP}}{{"\n"}}'

   3. To connect to the client pod, enter the following command:

      $ oc rsh sctpclient

   4. To start the SCTP client, enter the following command. Replace <cluster_IP> with the cluster IP address of the sctpservice service.

      # nc <cluster_IP> 30102 --sctp

Procedure

1. Create a namespace for the PTP Operator.

   1. Save the following YAML in the ptp-namespace.yaml file:

      apiVersion: v1
      kind: Namespace
      metadata:
        name: openshift-ptp
        annotations:
          workload.openshift.io/allowed: management
        labels:
          name: openshift-ptp
          openshift.io/cluster-monitoring: "true"

   2. Create the Namespace CR:

      $ oc create -f ptp-namespace.yaml

2. Create an Operator group for the PTP Operator.

   1. Save the following YAML in the ptp-operatorgroup.yaml file:

      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: ptp-operators
        namespace: openshift-ptp
      spec:
        targetNamespaces:
        - openshift-ptp

   2. Create the OperatorGroup CR:

      $ oc create -f ptp-operatorgroup.yaml

3. Subscribe to the PTP Operator.

   1. Save the following YAML in the ptp-sub.yaml file:

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: ptp-operator-subscription
        namespace: openshift-ptp
      spec:
        channel: "stable"
        name: ptp-operator
        source: redhat-operators
        sourceNamespace: openshift-marketplace

   2. Create the Subscription CR:

      $ oc create -f ptp-sub.yaml

4. To verify that the Operator is installed, enter the following command:

   $ oc get csv -n openshift-ptp -o custom-columns=Name:.metadata.name,Phase:.status.phase

   Example output

   Name                         Phase
   4.13.0-202301261535          Succeeded

Procedure

1. Install the PTP Operator using the OpenShift Container Platform web console:

   1. In the OpenShift Container Platform web console, click Operators → OperatorHub.
   2. Choose PTP Operator from the list of available Operators, and then click Install.
   3. On the Install Operator page, under A specific namespace on the cluster select openshift-ptp. Then, click Install.

2. Optional: Verify that the PTP Operator installed successfully:

   1. Switch to the Operators → Installed Operators page.

   2. Ensure that PTP Operator is listed in the openshift-ptp project with a Status of InstallSucceeded.

      Note

      During installation an Operator might display a Failed status. If the installation later succeeds with an InstallSucceeded message, you can ignore the Failed message.

      If the Operator does not appear as installed, to troubleshoot further:

      • Go to the Operators → Installed Operators page and inspect the Operator Subscriptions and Install Plans tabs for any failure or errors under Status.
      • Go to the Workloads → Pods page and check the logs for pods in the openshift-ptp project.

Procedure

1. Check the Operator and operands are successfully deployed in the cluster for the configured nodes.

   $ oc get pods -n openshift-ptp -o wide

   Example output

   NAME                            READY   STATUS    RESTARTS   AGE     IP            NODE
   linuxptp-daemon-lmvgn           3/3     Running   0          4d17h   10.1.196.24   compute-0.example.com
   linuxptp-daemon-qhfg7           3/3     Running   0          4d17h   10.1.196.25   compute-1.example.com
   ptp-operator-6b8dcbf7f4-zndk7   1/1     Running   0          5d7h    10.129.0.61   control-plane-1.example.com

   Note

   When the PTP fast event bus is enabled, the number of ready linuxptp-daemon pods is 3/3. If the PTP fast event bus is not enabled, 2/2 is displayed.

2. Check that supported hardware is found in the cluster.

   $ oc -n openshift-ptp get nodeptpdevices.ptp.openshift.io

   Example output

   NAME                                  AGE
   control-plane-0.example.com           10d
   control-plane-1.example.com           10d
   compute-0.example.com                 10d
   compute-1.example.com                 10d
   compute-2.example.com                 10d

3. Check the available PTP network interfaces for a node:

   $ oc -n openshift-ptp get nodeptpdevices.ptp.openshift.io <node_name> -o yaml

   where:

   <node_name>

   Specifies the node you want to query, for example, compute-0.example.com.

   Example output

   apiVersion: ptp.openshift.io/v1
   kind: NodePtpDevice
   metadata:
     creationTimestamp: "2021-09-14T16:52:33Z"
     generation: 1
     name: compute-0.example.com
     namespace: openshift-ptp
     resourceVersion: "177400"
     uid: 30413db0-4d8d-46da-9bef-737bacd548fd
   spec: {}
   status:
     devices:
     - name: eno1
     - name: eno2
     - name: eno3
     - name: eno4
     - name: enp5s0f0
     - name: enp5s0f1

4. Check that the PTP interface is successfully synchronized to the primary clock by accessing the linuxptp-daemon pod for the corresponding node.

   1. Get the name of the linuxptp-daemon pod and corresponding node you want to troubleshoot by running the following command:

      $ oc get pods -n openshift-ptp -o wide

      Example output

      NAME                            READY   STATUS    RESTARTS   AGE     IP            NODE
      linuxptp-daemon-lmvgn           3/3     Running   0          4d17h   10.1.196.24   compute-0.example.com
      linuxptp-daemon-qhfg7           3/3     Running   0          4d17h   10.1.196.25   compute-1.example.com
      ptp-operator-6b8dcbf7f4-zndk7   1/1     Running   0          5d7h    10.129.0.61   control-plane-1.example.com

   2. Remote shell into the required linuxptp-daemon container:

      $ oc rsh -n openshift-ptp -c linuxptp-daemon-container <linux_daemon_container>

      where:

      <linux_daemon_container>

      is the container you want to diagnose, for example linuxptp-daemon-lmvgn.

   3. In the remote shell connection to the linuxptp-daemon container, use the PTP Management Client (pmc) tool to diagnose the network interface. Run the following pmc command to check the sync status of the PTP device, for example ptp4l.

      # pmc -u -f /var/run/ptp4l.0.config -b 0 'GET PORT_DATA_SET'

      Example output when the node is successfully synced to the primary clock

      sending: GET PORT_DATA_SET
          40a6b7.fffe.166ef0-1 seq 0 RESPONSE MANAGEMENT PORT_DATA_SET
              portIdentity            40a6b7.fffe.166ef0-1
              portState               SLAVE
              logMinDelayReqInterval  -4
              peerMeanPathDelay       0
              logAnnounceInterval     -3
              announceReceiptTimeout  3
              logSyncInterval         -4
              delayMechanism          1
              logMinPdelayReqInterval -4
              versionNumber           2