Networking

Procedure

1. Create a security group that allows SSH access into the virtual private cloud (VPC) created by the openshift-install command.
2. Create an Amazon EC2 instance on one of the public subnets the installer created.

3. Associate a public IP address with the Amazon EC2 instance that you created.

   Unlike with the OpenShift Container Platform installation, you should associate the Amazon EC2 instance you created with an SSH keypair. It does not matter what operating system you choose for this instance, as it will simply serve as an SSH bastion to bridge the internet into your OpenShift Container Platform cluster’s VPC. The Amazon Machine Image (AMI) you use does matter. With Red Hat Enterprise Linux CoreOS (RHCOS), for example, you can provide keys via Ignition, like the installer does.

4. After you provisioned your Amazon EC2 instance and can SSH into it, you must add the SSH key that you associated with your OpenShift Container Platform installation. This key can be different from the key for the bastion instance, but does not have to be.

   Note

   Direct SSH access is only recommended for disaster recovery. When the Kubernetes API is responsive, run privileged pods instead.

5. Run oc get nodes, inspect the output, and choose one of the nodes that is a master. The hostname looks similar to ip-10-0-1-163.ec2.internal.

6. From the bastion SSH host you manually deployed into Amazon EC2, SSH into that control plane host. Ensure that you use the same SSH key you specified during the installation:

   $ ssh -i <ssh-key-path> core@<master-hostname>

1. Run the following command to view the Deployment status:

   $ oc get -n openshift-network-operator deployment/network-operator

   Example output

   NAME               READY   UP-TO-DATE   AVAILABLE   AGE
   network-operator   1/1     1            1           56m

2. Run the following command to view the state of the Cluster Network Operator:

   $ oc get clusteroperator/network

   Example output

   NAME      VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
   network   4.5.4     True        False         False      50m

   The following fields provide information about the status of the operator: AVAILABLE, PROGRESSING, and DEGRADED. The AVAILABLE field is True when the Cluster Network Operator reports an available status condition.

Procedure

1. Backup the existing network configuration by running the following command:

   $ oc get network.operator cluster -o yaml > network-config-backup.yaml

2. Run the following command to modify the existing network configuration:

   $ oc edit network.operator cluster

   1. Add or update the following block under spec as illustrated in the following example:

      spec:
        clusterNetwork:
        - cidr: 10.128.0.0/14
          hostPrefix: 23
        serviceNetwork:
        - 172.30.0.0/16
        networkType: OVNKubernetes
        clusterNetworkMTU: 8900
        defaultNetwork:
          ovnKubernetesConfig:
            gatewayConfig:
              ipForwarding: Global

   2. Save and close the file.

3. After applying the changes, the OpenShift Cluster Network Operator (CNO) applies the update across the cluster. You can monitor the progress by using the following command:

   $ oc get clusteroperators network

   The status should eventually report as Available, Progressing=False, and Degraded=False.

4. Alternatively, you can enable IP forwarding globally by running the following command:

   $ oc patch network.operator cluster -p '{"spec":{"defaultNetwork":{"ovnKubernetesConfig":{"gatewayConfig":{"ipForwarding": "Global"}}}}}

   Note

   The other valid option for this parameter is Restricted in case you want to revert this change. Restricted is the default and with that setting global IP address forwarding is disabled.

1. Use the oc get command to view the deployment status:

   $ oc get -n openshift-dns-operator deployment/dns-operator

   Example output

   NAME           READY     UP-TO-DATE   AVAILABLE   AGE
   dns-operator   1/1       1            1           23h

2. Use the oc get command to view the state of the DNS Operator:

   $ oc get clusteroperator/dns

   Example output

   NAME      VERSION     AVAILABLE   PROGRESSING   DEGRADED   SINCE
   dns       4.1.0-0.11  True        False         False      92m

   AVAILABLE, PROGRESSING and DEGRADED provide information about the status of the operator. AVAILABLE is True when at least 1 pod from the CoreDNS daemon set reports an Available status condition.

Procedure

1. Use the oc describe command to view the default dns:

   $ oc describe dns.operator/default

   Example output

   Name:         default
   Namespace:
   Labels:       <none>
   Annotations:  <none>
   API Version:  operator.openshift.io/v1
   Kind:         DNS
   ...
   Status:
     Cluster Domain:  cluster.local 1
     Cluster IP:      172.30.0.10 2
   ...

   1

   The Cluster Domain field is the base DNS domain used to construct fully qualified pod and service domain names.

   2

   The Cluster IP is the address pods query for name resolution. The IP is defined as the 10th address in the service CIDR range.

2. To find the service CIDR of your cluster, use the oc get command:

   $ oc get networks.config/cluster -o jsonpath='{$.status.serviceNetwork}'

Procedure

1. Modify the DNS Operator object named default:

   $ oc edit dns.operator/default

   After you issue the previous command, the Operator creates and updates the config map named dns-default with additional server configuration blocks based on Server. If none of the servers have a zone that matches the query, then name resolution falls back to the upstream DNS servers.

   Configuring DNS forwarding

   apiVersion: operator.openshift.io/v1
   kind: DNS
   metadata:
     name: default
   spec:
     servers:
     - name: example-server 1
       zones: 2
       - example.com
       forwardPlugin:
         policy: Random 3
         upstreams: 4
         - 1.1.1.1
         - 2.2.2.2:5353
     upstreamResolvers: 5
       policy: Random 6
       upstreams: 7
       - type: SystemResolvConf 8
       - type: Network
         address: 1.2.3.4 9
         port: 53 10

   1

   Must comply with the rfc6335 service name syntax.

   2

   Must conform to the definition of a subdomain in the rfc1123 service name syntax. The cluster domain, cluster.local, is an invalid subdomain for the zones field.

   3

   Defines the policy to select upstream resolvers. Default value is Random. You can also use the values RoundRobin, and Sequential.

   4

   A maximum of 15 upstreams is allowed per forwardPlugin.

   5

   Optional. You can use it to override the default policy and forward DNS resolution to the specified DNS resolvers (upstream resolvers) for the default domain. If you do not provide any upstream resolvers, the DNS name queries go to the servers in /etc/resolv.conf.

   6

   Determines the order in which upstream servers are selected for querying. You can specify one of these values: Random, RoundRobin, or Sequential. The default value is Sequential.

   7

   Optional. You can use it to provide upstream resolvers.

   8

   You can specify two types of upstreams - SystemResolvConf and Network. SystemResolvConf configures the upstream to use /etc/resolv.conf and Network defines a Networkresolver. You can specify one or both.

   9

   If the specified type is Network, you must provide an IP address. The address field must be a valid IPv4 or IPv6 address.

   10

   If the specified type is Network, you can optionally provide a port. The port field must have a value between 1 and 65535. If you do not specify a port for the upstream, by default port 853 is tried.

2. Optional: When working in a highly regulated environment, you might need the ability to secure DNS traffic when forwarding requests to upstream resolvers so that you can ensure additional DNS traffic and data privacy. Cluster administrators can configure transport layer security (TLS) for forwarded DNS queries.

   Configuring DNS forwarding with TLS

   apiVersion: operator.openshift.io/v1
   kind: DNS
   metadata:
     name: default
   spec:
     servers:
     - name: example-server 1
       zones: 2
       - example.com
       forwardPlugin:
         transportConfig:
           transport: TLS 3
           tls:
             caBundle:
               name: mycacert
             serverName: dnstls.example.com  4
         policy: Random 5
         upstreams: 6
         - 1.1.1.1
         - 2.2.2.2:5353
     upstreamResolvers: 7
       transportConfig:
         transport: TLS
         tls:
           caBundle:
             name: mycacert
           serverName: dnstls.example.com
       upstreams:
       - type: Network 8
         address: 1.2.3.4 9
         port: 53 10

   1

   Must comply with the rfc6335 service name syntax.

   2

   Must conform to the definition of a subdomain in the rfc1123 service name syntax. The cluster domain, cluster.local, is an invalid subdomain for the zones field. The cluster domain, cluster.local, is an invalid subdomain for zones.

   3

   When configuring TLS for forwarded DNS queries, set the transport field to have the value TLS. By default, CoreDNS caches forwarded connections for 10 seconds. CoreDNS will hold a TCP connection open for those 10 seconds if no request is issued. With large clusters, ensure that your DNS server is aware that it might get many new connections to hold open because you can initiate a connection per node. Set up your DNS hierarchy accordingly to avoid performance issues.

   4

   When configuring TLS for forwarded DNS queries, this is a mandatory server name used as part of the server name indication (SNI) to validate the upstream TLS server certificate.

   5

   Defines the policy to select upstream resolvers. Default value is Random. You can also use the values RoundRobin, and Sequential.

   6

   Required. You can use it to provide upstream resolvers. A maximum of 15 upstreams entries are allowed per forwardPlugin entry.

   7

   Optional. You can use it to override the default policy and forward DNS resolution to the specified DNS resolvers (upstream resolvers) for the default domain. If you do not provide any upstream resolvers, the DNS name queries go to the servers in /etc/resolv.conf.

   8

   Network type indicates that this upstream resolver should handle forwarded requests separately from the upstream resolvers listed in /etc/resolv.conf. Only the Network type is allowed when using TLS and you must provide an IP address.

   9

   The address field must be a valid IPv4 or IPv6 address.

   10

   You can optionally provide a port. The port must have a value between 1 and 65535. If you do not specify a port for the upstream, by default port 853 is tried.

   Note

   If servers is undefined or invalid, the config map only contains the default server.

Verification

1. View the config map:

   $ oc get configmap/dns-default -n openshift-dns -o yaml

   Sample DNS ConfigMap based on previous sample DNS

   apiVersion: v1
   data:
     Corefile: |
       example.com:5353 {
           forward . 1.1.1.1 2.2.2.2:5353
       }
       bar.com:5353 example.com:5353 {
           forward . 3.3.3.3 4.4.4.4:5454 1
       }
       .:5353 {
           errors
           health
           kubernetes cluster.local in-addr.arpa ip6.arpa {
               pods insecure
               upstream
               fallthrough in-addr.arpa ip6.arpa
           }
           prometheus :9153
           forward . /etc/resolv.conf 1.2.3.4:53 {
               policy Random
           }
           cache 30
           reload
       }
   kind: ConfigMap
   metadata:
     labels:
       dns.operator.openshift.io/owning-dns: default
     name: dns-default
     namespace: openshift-dns

   1

   Changes to the forwardPlugin triggers a rolling update of the CoreDNS daemon set.

Procedure

1. Edit the DNS Operator object named default by running the following command:

   $ oc edit dns.operator.openshift.io/default

2. Modify the time-to-live (TTL) caching values:

   Configuring DNS caching

   apiVersion: operator.openshift.io/v1
   kind: DNS
   metadata:
     name: default
   spec:
     cache:
       positiveTTL: 1h 1
       negativeTTL: 0.5h10m 2

   1

   The string value 1h is converted to its respective number of seconds by CoreDNS. If this field is omitted, the value is assumed to be 0s and the cluster uses the internal default value of 900s as a fallback.

   2

   The string value can be a combination of units such as 0.5h10m and is converted to its respective number of seconds by CoreDNS. If this field is omitted, the value is assumed to be 0s and the cluster uses the internal default value of 30s as a fallback.

   Warning

   Setting TTL fields to low values could lead to an increased load on the cluster, any upstream resolvers, or both.

Procedure

1. Create a YAML file that defines the custom IngressController object:

   Example custom-ingress-controller.yaml file

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
       name: <custom_name> 1
       namespace: openshift-ingress-operator
   spec:
       defaultCertificate:
           name: <custom-ingress-custom-certs> 2
       replicas: 1 3
       domain: <custom_domain> 4

   1

   Specify the a custom name for the IngressController object.

   2

   Specify the name of the secret with the custom wildcard certificate.

   3

   Minimum replica needs to be ONE

   4

   Specify the domain to your domain name. The domain specified on the IngressController object and the domain used for the certificate must match. For example, if the domain value is "custom_domain.mycompany.com", then the certificate must have SAN *.custom_domain.mycompany.com (with the *. added to the domain).

2. Create the object by running the following command:

   $ oc create -f custom-ingress-controller.yaml

1. Create the IngressNodeFirewallConfig inside the openshift-ingress-node-firewall namespace named ingressnodefirewallconfig.

2. Run the following command to deploy Ingress Node Firewall Operator rules:

   $ oc apply -f rule.yaml

Procedure

1. Run the following command to view all current rules :

   $ oc get ingressnodefirewall

2. Choose one of the returned <resource> names and run the following command to view the rules or configs:

   $ oc get <resource> <name> -o yaml

Procedure

1. Create a custom resource (CR) file named sample-ingress.yaml containing the following:

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     namespace: openshift-ingress-operator
     name: <name> 1
   spec:
     domain: <domain> 2
     endpointPublishingStrategy:
       type: LoadBalancerService
       loadBalancer:
         scope: External 3
         dnsManagementPolicy: Unmanaged 4

   1

   Specify the <name> with a name for the IngressController object.

   2

   Specify the domain based on the DNS record that was created as a prerequisite.

   3

   Specify the scope as External to expose the load balancer externally.

   4

   dnsManagementPolicy indicates if the Ingress Controller is managing the lifecycle of the wildcard DNS record associated with the load balancer. The valid values are Managed and Unmanaged. The default value is Managed.

2. Save the file to apply the changes.

   oc apply -f <name>.yaml 1

Procedure

1. Modify the chosen IngressController to set dnsManagementPolicy:

   SCOPE=$(oc -n openshift-ingress-operator get ingresscontroller <name> -o=jsonpath="{.status.endpointPublishingStrategy.loadBalancer.scope}")
   
   oc -n openshift-ingress-operator patch ingresscontrollers/<name> --type=merge --patch='{"spec":{"endpointPublishingStrategy":{"type":"LoadBalancerService","loadBalancer":{"dnsManagementPolicy":"Unmanaged", "scope":"${SCOPE}"}}}}'

2. Optional: You can delete the associated DNS record in the cloud provider.

Procedure

1. To list the current PodNetworkConnectivityCheck objects, enter the following command:

   $ oc get podnetworkconnectivitycheck -n openshift-network-diagnostics

   Example output

   NAME                                                                                                                                AGE
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0   75m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-1   73m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-2   75m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-kubernetes-apiserver-service-cluster                               75m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-kubernetes-default-service-cluster                                 75m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-load-balancer-api-external                                         75m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-load-balancer-api-internal                                         75m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-network-check-target-ci-ln-x5sv9rb-f76d1-4rzrp-master-0            75m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-network-check-target-ci-ln-x5sv9rb-f76d1-4rzrp-master-1            75m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-network-check-target-ci-ln-x5sv9rb-f76d1-4rzrp-master-2            75m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-network-check-target-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh      74m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-network-check-target-ci-ln-x5sv9rb-f76d1-4rzrp-worker-c-n8mbf      74m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-network-check-target-ci-ln-x5sv9rb-f76d1-4rzrp-worker-d-4hnrz      74m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-network-check-target-service-cluster                               75m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-openshift-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0    75m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-openshift-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-1    75m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-openshift-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-2    74m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-openshift-apiserver-service-cluster                                75m

2. View the connection test logs:

   1. From the output of the previous command, identify the endpoint that you want to review the connectivity logs for.

   2. To view the object, enter the following command:

      $ oc get podnetworkconnectivitycheck <name> \
        -n openshift-network-diagnostics -o yaml

      where <name> specifies the name of the PodNetworkConnectivityCheck object.

      Example output

      apiVersion: controlplane.operator.openshift.io/v1alpha1
      kind: PodNetworkConnectivityCheck
      metadata:
        name: network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0
        namespace: openshift-network-diagnostics
        ...
      spec:
        sourcePod: network-check-source-7c88f6d9f-hmg2f
        targetEndpoint: 10.0.0.4:6443
        tlsClientCert:
          name: ""
      status:
        conditions:
        - lastTransitionTime: "2021-01-13T20:11:34Z"
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: tcp
            connection to 10.0.0.4:6443 succeeded'
          reason: TCPConnectSuccess
          status: "True"
          type: Reachable
        failures:
        - latency: 2.241775ms
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: failed
            to establish a TCP connection to 10.0.0.4:6443: dial tcp 10.0.0.4:6443: connect:
            connection refused'
          reason: TCPConnectError
          success: false
          time: "2021-01-13T20:10:34Z"
        - latency: 2.582129ms
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: failed
            to establish a TCP connection to 10.0.0.4:6443: dial tcp 10.0.0.4:6443: connect:
            connection refused'
          reason: TCPConnectError
          success: false
          time: "2021-01-13T20:09:34Z"
        - latency: 3.483578ms
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: failed
            to establish a TCP connection to 10.0.0.4:6443: dial tcp 10.0.0.4:6443: connect:
            connection refused'
          reason: TCPConnectError
          success: false
          time: "2021-01-13T20:08:34Z"
        outages:
        - end: "2021-01-13T20:11:34Z"
          endLogs:
          - latency: 2.032018ms
            message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0:
              tcp connection to 10.0.0.4:6443 succeeded'
            reason: TCPConnect
            success: true
            time: "2021-01-13T20:11:34Z"
          - latency: 2.241775ms
            message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0:
              failed to establish a TCP connection to 10.0.0.4:6443: dial tcp 10.0.0.4:6443:
              connect: connection refused'
            reason: TCPConnectError
            success: false
            time: "2021-01-13T20:10:34Z"
          - latency: 2.582129ms
            message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0:
              failed to establish a TCP connection to 10.0.0.4:6443: dial tcp 10.0.0.4:6443:
              connect: connection refused'
            reason: TCPConnectError
            success: false
            time: "2021-01-13T20:09:34Z"
          - latency: 3.483578ms
            message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0:
              failed to establish a TCP connection to 10.0.0.4:6443: dial tcp 10.0.0.4:6443:
              connect: connection refused'
            reason: TCPConnectError
            success: false
            time: "2021-01-13T20:08:34Z"
          message: Connectivity restored after 2m59.999789186s
          start: "2021-01-13T20:08:34Z"
          startLogs:
          - latency: 3.483578ms
            message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0:
              failed to establish a TCP connection to 10.0.0.4:6443: dial tcp 10.0.0.4:6443:
              connect: connection refused'
            reason: TCPConnectError
            success: false
            time: "2021-01-13T20:08:34Z"
        successes:
        - latency: 2.845865ms
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: tcp
            connection to 10.0.0.4:6443 succeeded'
          reason: TCPConnect
          success: true
          time: "2021-01-13T21:14:34Z"
        - latency: 2.926345ms
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: tcp
            connection to 10.0.0.4:6443 succeeded'
          reason: TCPConnect
          success: true
          time: "2021-01-13T21:13:34Z"
        - latency: 2.895796ms
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: tcp
            connection to 10.0.0.4:6443 succeeded'
          reason: TCPConnect
          success: true
          time: "2021-01-13T21:12:34Z"
        - latency: 2.696844ms
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: tcp
            connection to 10.0.0.4:6443 succeeded'
          reason: TCPConnect
          success: true
          time: "2021-01-13T21:11:34Z"
        - latency: 1.502064ms
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: tcp
            connection to 10.0.0.4:6443 succeeded'
          reason: TCPConnect
          success: true
          time: "2021-01-13T21:10:34Z"
        - latency: 1.388857ms
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: tcp
            connection to 10.0.0.4:6443 succeeded'
          reason: TCPConnect
          success: true
          time: "2021-01-13T21:09:34Z"
        - latency: 1.906383ms
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: tcp
            connection to 10.0.0.4:6443 succeeded'
          reason: TCPConnect
          success: true
          time: "2021-01-13T21:08:34Z"
        - latency: 2.089073ms
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: tcp
            connection to 10.0.0.4:6443 succeeded'
          reason: TCPConnect
          success: true
          time: "2021-01-13T21:07:34Z"
        - latency: 2.156994ms
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: tcp
            connection to 10.0.0.4:6443 succeeded'
          reason: TCPConnect
          success: true
          time: "2021-01-13T21:06:34Z"
        - latency: 1.777043ms
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: tcp
            connection to 10.0.0.4:6443 succeeded'
          reason: TCPConnect
          success: true
          time: "2021-01-13T21:05:34Z"

1. To obtain the current MTU for the cluster network, enter the following command:

   $ oc describe network.config cluster

   Example output

   ...
   Status:
     Cluster Network:
       Cidr:               10.217.0.0/22
       Host Prefix:        23
     Cluster Network MTU:  1400
     Network Type:         OpenShiftSDN
     Service Network:
       10.217.4.0/23
   ...

2. Prepare your configuration for the hardware MTU:

   • If your hardware MTU is specified with DHCP, update your DHCP configuration such as with the following dnsmasq configuration:

     dhcp-option-force=26,<mtu>

     where:

     <mtu>

     Specifies the hardware MTU for the DHCP server to advertise.

   • If your hardware MTU is specified with a kernel command line with PXE, update that configuration accordingly.

   • If your hardware MTU is specified in a NetworkManager connection configuration, complete the following steps. This approach is the default for OpenShift Container Platform if you do not explicitly specify your network configuration with DHCP, a kernel command line, or some other method. Your cluster nodes must all use the same underlying network configuration for the following procedure to work unmodified.

     1. Find the primary network interface:

        • If you are using the OpenShift SDN network plugin, enter the following command:

          $ oc debug node/<node_name> -- chroot /host ip route list match 0.0.0.0/0 | awk '{print $5 }'

          where:

          <node_name>

          Specifies the name of a node in your cluster.

        • If you are using the OVN-Kubernetes network plugin, enter the following command:

          $ oc debug node/<node_name> -- chroot /host nmcli -g connection.interface-name c show ovs-if-phys0

          where:

          <node_name>

          Specifies the name of a node in your cluster.

     2. Create the following NetworkManager configuration in the <interface>-mtu.conf file:

        Example NetworkManager connection configuration

        [connection-<interface>-mtu]
        match-device=interface-name:<interface>
        ethernet.mtu=<mtu>

        where:

        <mtu>

        Specifies the new hardware MTU value.

        <interface>

        Specifies the primary network interface name.

     3. Create two MachineConfig objects, one for the control plane nodes and another for the worker nodes in your cluster:

        1. Create the following Butane config in the control-plane-interface.bu file:

           variant: openshift
           version: 4.14.0
           metadata:
             name: 01-control-plane-interface
             labels:
               machineconfiguration.openshift.io/role: master
           storage:
             files:
               - path: /etc/NetworkManager/conf.d/99-<interface>-mtu.conf 1
                 contents:
                   local: <interface>-mtu.conf 2
                 mode: 0600

           1 1

           Specify the NetworkManager connection name for the primary network interface.

           2

           Specify the local filename for the updated NetworkManager configuration file from the previous step.

        2. Create the following Butane config in the worker-interface.bu file:

           variant: openshift
           version: 4.14.0
           metadata:
             name: 01-worker-interface
             labels:
               machineconfiguration.openshift.io/role: worker
           storage:
             files:
               - path: /etc/NetworkManager/conf.d/99-<interface>-mtu.conf 1
                 contents:
                   local: <interface>-mtu.conf 2
                 mode: 0600

           1

           Specify the NetworkManager connection name for the primary network interface.

           2

           Specify the local filename for the updated NetworkManager configuration file from the previous step.

        3. Create MachineConfig objects from the Butane configs by running the following command:

           $ for manifest in control-plane-interface worker-interface; do
               butane --files-dir . $manifest.bu > $manifest.yaml
             done

           Warning

           Do not apply these machine configs until explicitly instructed later in this procedure. Applying these machine configs now causes a loss of stability for the cluster.

3. To begin the MTU migration, specify the migration configuration by entering the following command. The Machine Config Operator performs a rolling reboot of the nodes in the cluster in preparation for the MTU change.

   $ oc patch Network.operator.openshift.io cluster --type=merge --patch \
     '{"spec": { "migration": { "mtu": { "network": { "from": <overlay_from>, "to": <overlay_to> } , "machine": { "to" : <machine_to> } } } } }'

   where:

   <overlay_from>

   Specifies the current cluster network MTU value.

   <overlay_to>

   Specifies the target MTU for the cluster network. This value is set relative to the value for <machine_to> and for OVN-Kubernetes must be 100 less and for OpenShift SDN must be 50 less.

   <machine_to>

   Specifies the MTU for the primary network interface on the underlying host network.

   Example that increases the cluster MTU

   $ oc patch Network.operator.openshift.io cluster --type=merge --patch \
     '{"spec": { "migration": { "mtu": { "network": { "from": 1400, "to": 9000 } , "machine": { "to" : 9100} } } } }'

4. As the MCO updates machines in each machine config pool, it reboots each node one by one. You must wait until all the nodes are updated. Check the machine config pool status by entering the following command:

   $ oc get mcp

   A successfully updated node has the following status: UPDATED=true, UPDATING=false, DEGRADED=false.

   Note

   By default, the MCO updates one machine per pool at a time, causing the total time the migration takes to increase with the size of the cluster.

5. Confirm the status of the new machine configuration on the hosts:

   1. To list the machine configuration state and the name of the applied machine configuration, enter the following command:

      $ oc describe node | egrep "hostname|machineconfig"

      Example output

      kubernetes.io/hostname=master-0
      machineconfiguration.openshift.io/currentConfig: rendered-master-c53e221d9d24e1c8bb6ee89dd3d8ad7b
      machineconfiguration.openshift.io/desiredConfig: rendered-master-c53e221d9d24e1c8bb6ee89dd3d8ad7b
      machineconfiguration.openshift.io/reason:
      machineconfiguration.openshift.io/state: Done

      Verify that the following statements are true:

      • The value of machineconfiguration.openshift.io/state field is Done.
      • The value of the machineconfiguration.openshift.io/currentConfig field is equal to the value of the machineconfiguration.openshift.io/desiredConfig field.

   2. To confirm that the machine config is correct, enter the following command:

      $ oc get machineconfig <config_name> -o yaml | grep ExecStart

      where <config_name> is the name of the machine config from the machineconfiguration.openshift.io/currentConfig field.

      The machine config must include the following update to the systemd configuration:

      ExecStart=/usr/local/bin/mtu-migration.sh

6. Update the underlying network interface MTU value:

   • If you are specifying the new MTU with a NetworkManager connection configuration, enter the following command. The MachineConfig Operator automatically performs a rolling reboot of the nodes in your cluster.

     $ for manifest in control-plane-interface worker-interface; do
         oc create -f $manifest.yaml
       done

   • If you are specifying the new MTU with a DHCP server option or a kernel command line and PXE, make the necessary changes for your infrastructure.

7. As the MCO updates machines in each machine config pool, it reboots each node one by one. You must wait until all the nodes are updated. Check the machine config pool status by entering the following command:

   $ oc get mcp

   A successfully updated node has the following status: UPDATED=true, UPDATING=false, DEGRADED=false.

   Note

   By default, the MCO updates one machine per pool at a time, causing the total time the migration takes to increase with the size of the cluster.

8. Confirm the status of the new machine configuration on the hosts:

   1. To list the machine configuration state and the name of the applied machine configuration, enter the following command:

      $ oc describe node | egrep "hostname|machineconfig"

      Example output

      kubernetes.io/hostname=master-0
      machineconfiguration.openshift.io/currentConfig: rendered-master-c53e221d9d24e1c8bb6ee89dd3d8ad7b
      machineconfiguration.openshift.io/desiredConfig: rendered-master-c53e221d9d24e1c8bb6ee89dd3d8ad7b
      machineconfiguration.openshift.io/reason:
      machineconfiguration.openshift.io/state: Done

      Verify that the following statements are true:

      • The value of machineconfiguration.openshift.io/state field is Done.
      • The value of the machineconfiguration.openshift.io/currentConfig field is equal to the value of the machineconfiguration.openshift.io/desiredConfig field.

   2. To confirm that the machine config is correct, enter the following command:

      $ oc get machineconfig <config_name> -o yaml | grep path:

      If the machine config is successfully deployed, the previous output contains the /etc/NetworkManager/conf.d/99-<interface>-mtu.conf file path and the ExecStart=/usr/local/bin/mtu-migration.sh line.

9. To finalize the MTU migration, enter one of the following commands:

   • If you are using the OVN-Kubernetes network plugin:

     $ oc patch Network.operator.openshift.io cluster --type=merge --patch \
       '{"spec": { "migration": null, "defaultNetwork":{ "ovnKubernetesConfig": { "mtu": <mtu> }}}}'

     where:

     <mtu>

     Specifies the new cluster network MTU that you specified with <overlay_to>.

   • If you are using the OpenShift SDN network plugin:

     $ oc patch Network.operator.openshift.io cluster --type=merge --patch \
       '{"spec": { "migration": null, "defaultNetwork":{ "openshiftSDNConfig": { "mtu": <mtu> }}}}'

     where:

     <mtu>

     Specifies the new cluster network MTU that you specified with <overlay_to>.

10. After finalizing the MTU migration, each MCP node is rebooted one by one. You must wait until all the nodes are updated. Check the machine config pool status by entering the following command:

    $ oc get mcp

    A successfully updated node has the following status: UPDATED=true, UPDATING=false, DEGRADED=false.

1. To get the current MTU for the cluster network, enter the following command:

   $ oc describe network.config cluster

2. Get the current MTU for the primary network interface of a node.

   1. To list the nodes in your cluster, enter the following command:

      $ oc get nodes

   2. To obtain the current MTU setting for the primary network interface on a node, enter the following command:

      $ oc debug node/<node> -- chroot /host ip address show <interface>

      where:

      <node>

      Specifies a node from the output from the previous step.

      <interface>

      Specifies the primary network interface name for the node.

      Example output

      ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8051

Procedure

1. To expand the node port range, enter the following command. Replace <port> with the largest port number in the new range.

   $ oc patch network.config.openshift.io cluster --type=merge -p \
     '{
       "spec":
         { "serviceNodePortRange": "30000-<port>" }
     }'

   Tip

   You can alternatively apply the following YAML to update the node port range:

   apiVersion: config.openshift.io/v1
   kind: Network
   metadata:
     name: cluster
   spec:
     serviceNodePortRange: "30000-<port>"

   Example output

   network.config.openshift.io/cluster patched

2. To confirm that the configuration is active, enter the following command. It can take several minutes for the update to apply.

   $ oc get configmaps -n openshift-kube-apiserver config \
     -o jsonpath="{.data['config\.yaml']}" | \
     grep -Eo '"service-node-port-range":["[[:digit:]]+-[[:digit:]]+"]'

   Example output

   "service-node-port-range":["30000-33000"]

Procedure

1. To obtain the cluster network range and host prefix for your cluster, enter the following command:

   $ oc get network.operator.openshift.io \
     -o jsonpath="{.items[0].spec.clusterNetwork}"

   Example output

   [{"cidr":"10.217.0.0/22","hostPrefix":23}]

2. To expand the cluster network IP address range, enter the following command. Use the CIDR IP address range and host prefix returned from the output of the previous command.

   $ oc patch Network.config.openshift.io cluster --type='merge' --patch \
     '{
       "spec":{
         "clusterNetwork": [ {"cidr":"<network>/<cidr>","hostPrefix":<prefix>} ],
         "networkType": "OVNKubernetes"
       }
     }'

   where:

   <network>

   Specifies the network part of the cidr field that you obtained from the previous step. You cannot change this value.

   <cidr>

   Specifies the network prefix length. For example, 14. Change this value to a smaller number than the value from the output in the previous step to expand the cluster network range.

   <prefix>

   Specifies the current host prefix for your cluster. This value must be the same value for the hostPrefix field that you obtained from the previous step.

   Example command

   $ oc patch Network.config.openshift.io cluster --type='merge' --patch \
     '{
       "spec":{
         "clusterNetwork": [ {"cidr":"10.217.0.0/14","hostPrefix": 23} ],
         "networkType": "OVNKubernetes"
       }
     }'

   Example output

   network.config.openshift.io/cluster patched

3. To confirm that the configuration is active, enter the following command. It can take up to 30 minutes for this change to take effect.

   $ oc get network.operator.openshift.io \
     -o jsonpath="{.items[0].spec.clusterNetwork}"

   Example output

   [{"cidr":"10.217.0.0/14","hostPrefix":23}]

Procedure

1. Create an IP failover service account:

   $ oc create sa ipfailover

2. Update security context constraints (SCC) for hostNetwork:

   $ oc adm policy add-scc-to-user privileged -z ipfailover

   $ oc adm policy add-scc-to-user hostnetwork -z ipfailover

3. Red Hat OpenStack Platform (RHOSP) only: Complete the following steps to make a failover VIP address reachable on RHOSP ports.

   1. Use the RHOSP CLI to show the default RHOSP API and VIP addresses in the allowed_address_pairs parameter of your RHOSP cluster:

      $ openstack port show <cluster_name> -c allowed_address_pairs

      Output example

      *Field*                  *Value*
      allowed_address_pairs    ip_address='192.168.0.5', mac_address='fa:16:3e:31:f9:cb'
                               ip_address='192.168.0.7', mac_address='fa:16:3e:31:f9:cb'

   2. Set a different VIP address for the IP failover deployment and make the address reachable on RHOSP ports by entering the following command in the RHOSP CLI. Do not set any default RHOSP API and VIP addresses as the failover VIP address for the IP failover deployment.

      Example of adding the 1.1.1.1 failover IP address as an allowed address on RHOSP ports.

      $ openstack port set <cluster_name> --allowed-address ip-address=1.1.1.1,mac-address=fa:fa:16:3e:31:f9:cb

   3. Create a deployment YAML file to configure IP failover for your deployment. See "Example deployment YAML for IP failover configuration" in a later step.

   4. Specify the following specification in the IP failover deployment so that you pass the failover VIP address to the OPENSHIFT_HA_VIRTUAL_IPS environment variable:

      Example of adding the 1.1.1.1 VIP address to OPENSHIFT_HA_VIRTUAL_IPS

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: ipfailover-keepalived
      # ...
            spec:
                env:
                - name: OPENSHIFT_HA_VIRTUAL_IPS
                value: "1.1.1.1"
      # ...

4. Create a deployment YAML file to configure IP failover.

   Note

   For Red Hat OpenStack Platform (RHOSP), you do not need to re-create the deployment YAML file. You already created this file as part of the earlier instructions.

   Example deployment YAML for IP failover configuration

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: ipfailover-keepalived 1
     labels:
       ipfailover: hello-openshift
   spec:
     strategy:
       type: Recreate
     replicas: 2
     selector:
       matchLabels:
         ipfailover: hello-openshift
     template:
       metadata:
         labels:
           ipfailover: hello-openshift
       spec:
         serviceAccountName: ipfailover
         privileged: true
         hostNetwork: true
         nodeSelector:
           node-role.kubernetes.io/worker: ""
         containers:
         - name: openshift-ipfailover
           image: registry.redhat.io/openshift4/ose-keepalived-ipfailover:4.14
           ports:
           - containerPort: 63000
             hostPort: 63000
           imagePullPolicy: IfNotPresent
           securityContext:
             privileged: true
           volumeMounts:
           - name: lib-modules
             mountPath: /lib/modules
             readOnly: true
           - name: host-slash
             mountPath: /host
             readOnly: true
             mountPropagation: HostToContainer
           - name: etc-sysconfig
             mountPath: /etc/sysconfig
             readOnly: true
           - name: config-volume
             mountPath: /etc/keepalive
           env:
           - name: OPENSHIFT_HA_CONFIG_NAME
             value: "ipfailover"
           - name: OPENSHIFT_HA_VIRTUAL_IPS 2
             value: "1.1.1.1-2"
           - name: OPENSHIFT_HA_VIP_GROUPS 3
             value: "10"
           - name: OPENSHIFT_HA_NETWORK_INTERFACE 4
             value: "ens3" #The host interface to assign the VIPs
           - name: OPENSHIFT_HA_MONITOR_PORT 5
             value: "30060"
           - name: OPENSHIFT_HA_VRRP_ID_OFFSET 6
             value: "0"
           - name: OPENSHIFT_HA_REPLICA_COUNT 7
             value: "2" #Must match the number of replicas in the deployment
           - name: OPENSHIFT_HA_USE_UNICAST
             value: "false"
           #- name: OPENSHIFT_HA_UNICAST_PEERS
             #value: "10.0.148.40,10.0.160.234,10.0.199.110"
           - name: OPENSHIFT_HA_IPTABLES_CHAIN 8
             value: "INPUT"
           #- name: OPENSHIFT_HA_NOTIFY_SCRIPT 9
           #  value: /etc/keepalive/mynotifyscript.sh
           - name: OPENSHIFT_HA_CHECK_SCRIPT 10
             value: "/etc/keepalive/mycheckscript.sh"
           - name: OPENSHIFT_HA_PREEMPTION 11
             value: "preempt_delay 300"
           - name: OPENSHIFT_HA_CHECK_INTERVAL 12
             value: "2"
           livenessProbe:
             initialDelaySeconds: 10
             exec:
               command:
               - pgrep
               - keepalived
         volumes:
         - name: lib-modules
           hostPath:
             path: /lib/modules
         - name: host-slash
           hostPath:
             path: /
         - name: etc-sysconfig
           hostPath:
             path: /etc/sysconfig
         # config-volume contains the check script
         # created with `oc create configmap keepalived-checkscript --from-file=mycheckscript.sh`
         - configMap:
             defaultMode: 0755
             name: keepalived-checkscript
           name: config-volume
         imagePullSecrets:
           - name: openshift-pull-secret 13

   1

   The name of the IP failover deployment.

   2

   The list of IP address ranges to replicate. This must be provided. For example, 1.2.3.4-6,1.2.3.9.

   3

   The number of groups to create for VRRP. If not set, a group is created for each virtual IP range specified with the OPENSHIFT_HA_VIP_GROUPS variable.

   4

   The interface name that IP failover uses to send VRRP traffic. By default, eth0 is used.

   5

   The IP failover pod tries to open a TCP connection to this port on each VIP. If connection is established, the service is considered to be running. If this port is set to 0, the test always passes. The default value is 80.

   6

   The offset value used to set the virtual router IDs. Using different offset values allows multiple IP failover configurations to exist within the same cluster. The default offset is 0, and the allowed range is 0 through 255.

   7

   The number of replicas to create. This must match spec.replicas value in IP failover deployment configuration. The default value is 2.

   8

   The name of the iptables chain to automatically add an iptables rule to allow the VRRP traffic on. If the value is not set, an iptables rule is not added. If the chain does not exist, it is not created, and Keepalived operates in unicast mode. The default is INPUT.

   9

   The full path name in the pod file system of a script that is run whenever the state changes.

   10

   The full path name in the pod file system of a script that is periodically run to verify the application is operating.

   11

   The strategy for handling a new higher priority host. The default value is preempt_delay 300, which causes a Keepalived instance to take over a VIP after 5 minutes if a lower-priority master is holding the VIP.

   12

   The period, in seconds, that the check script is run. The default value is 2.

   13

   Create the pull secret before creating the deployment, otherwise you will get an error when creating the deployment.

Procedure

1. Create the desired script and create a ConfigMap object to hold it. The script has no input arguments and must return 0 for OK and 1 for fail.

   The check script, mycheckscript.sh:

   #!/bin/bash
       # Whatever tests are needed
       # E.g., send request and verify response
   exit 0

2. Create the ConfigMap object :

   $ oc create configmap mycustomcheck --from-file=mycheckscript.sh

3. Add the script to the pod. The defaultMode for the mounted ConfigMap object files must able to run by using oc commands or by editing the deployment configuration. A value of 0755, 493 decimal, is typical:

   $ oc set env deploy/ipfailover-keepalived \
       OPENSHIFT_HA_CHECK_SCRIPT=/etc/keepalive/mycheckscript.sh

   $ oc set volume deploy/ipfailover-keepalived --add --overwrite \
       --name=config-volume \
       --mount-path=/etc/keepalive \
       --source='{"configMap": { "name": "mycustomcheck", "defaultMode": 493}}'

   Note

   The oc set env command is whitespace sensitive. There must be no whitespace on either side of the = sign.

   Tip

   You can alternatively edit the ipfailover-keepalived deployment configuration:

   $ oc edit deploy ipfailover-keepalived

       spec:
         containers:
         - env:
           - name: OPENSHIFT_HA_CHECK_SCRIPT  1
             value: /etc/keepalive/mycheckscript.sh
   ...
           volumeMounts: 2
           - mountPath: /etc/keepalive
             name: config-volume
         dnsPolicy: ClusterFirst
   ...
         volumes: 3
         - configMap:
             defaultMode: 0755 4
             name: customrouter
           name: config-volume
   ...

   1

   In the spec.container.env field, add the OPENSHIFT_HA_CHECK_SCRIPT environment variable to point to the mounted script file.

   2

   Add the spec.container.volumeMounts field to create the mount point.

   3

   Add a new spec.volumes field to mention the config map.

   4

   This sets run permission on the files. When read back, it is displayed in decimal, 493.

   Save the changes and exit the editor. This restarts ipfailover-keepalived.

Procedure

1. Optional: Identify and delete any check and notify scripts that are stored as config maps:

   1. Identify whether any pods for IP failover use a config map as a volume:

      $ oc get pod -l ipfailover \
        -o jsonpath="\
      {range .items[?(@.spec.volumes[*].configMap)]}
      {'Namespace: '}{.metadata.namespace}
      {'Pod:       '}{.metadata.name}
      {'Volumes that use config maps:'}
      {range .spec.volumes[?(@.configMap)]}  {'volume:    '}{.name}
        {'configMap: '}{.configMap.name}{'\n'}{end}
      {end}"

      Example output

      Namespace: default
      Pod:       keepalived-worker-59df45db9c-2x9mn
      Volumes that use config maps:
        volume:    config-volume
        configMap: mycustomcheck

   2. If the preceding step provided the names of config maps that are used as volumes, delete the config maps:

      $ oc delete configmap <configmap_name>

2. Identify an existing deployment for IP failover:

   $ oc get deployment -l ipfailover

   Example output

   NAMESPACE   NAME         READY   UP-TO-DATE   AVAILABLE   AGE
   default     ipfailover   2/2     2            2           105d

3. Delete the deployment:

   $ oc delete deployment <ipfailover_deployment_name>

4. Remove the ipfailover service account:

   $ oc delete sa ipfailover

5. Run a job that removes the IP tables rule that was added when IP failover was initially configured:

   1. Create a file such as remove-ipfailover-job.yaml with contents that are similar to the following example:

      apiVersion: batch/v1
      kind: Job
      metadata:
        generateName: remove-ipfailover-
        labels:
          app: remove-ipfailover
      spec:
        template:
          metadata:
            name: remove-ipfailover
          spec:
            containers:
            - name: remove-ipfailover
              image: registry.redhat.io/openshift4/ose-keepalived-ipfailover:4.14
              command: ["/var/lib/ipfailover/keepalived/remove-failover.sh"]
            nodeSelector: 1
              kubernetes.io/hostname: <host_name>  2
            restartPolicy: Never

      1

      The nodeSelector is likely the same as the selector used in the old IP failover deployment.

      2

      Run the job for each node in your cluster that was configured for IP failover and replace the hostname each time.

   2. Run the job:

      $ oc create -f remove-ipfailover-job.yaml

      Example output

      job.batch/remove-ipfailover-2h8dm created

Procedure

1. Create a network attachment definition, such as tuning-example.yaml, with the following content:

   apiVersion: "k8s.cni.cncf.io/v1"
   kind: NetworkAttachmentDefinition
   metadata:
     name: <name> 1
     namespace: default 2
   spec:
     config: '{
       "cniVersion": "0.4.0", 3
       "name": "<name>", 4
       "plugins": [{
          "type": "<main_CNI_plugin>" 5
         },
         {
          "type": "tuning", 6
          "sysctl": {
               "net.ipv4.conf.IFNAME.accept_redirects": "1" 7
           }
         }
        ]
   }

   1

   Specifies the name for the additional network attachment to create. The name must be unique within the specified namespace.

   2

   Specifies the namespace that the object is associated with.

   3

   Specifies the CNI specification version.

   4

   Specifies the name for the configuration. It is recommended to match the configuration name to the name value of the network attachment definition.

   5

   Specifies the name of the main CNI plugin to configure.

   6

   Specifies the name of the CNI meta plugin.

   7

   Specifies the sysctl to set. The interface name is represented by the IFNAME token and is replaced with the actual name of the interface at runtime.

   An example YAML file is shown here:

   apiVersion: "k8s.cni.cncf.io/v1"
   kind: NetworkAttachmentDefinition
   metadata:
     name: tuningnad
     namespace: default
   spec:
     config: '{
       "cniVersion": "0.4.0",
       "name": "tuningnad",
       "plugins": [{
         "type": "bridge"
         },
         {
         "type": "tuning",
         "sysctl": {
            "net.ipv4.conf.IFNAME.accept_redirects": "1"
           }
       }
     ]
   }'

2. Apply the YAML by running the following command:

   $ oc apply -f tuning-example.yaml

   Example output

   networkattachmentdefinition.k8.cni.cncf.io/tuningnad created

3. Create a pod such as examplepod.yaml with the network attachment definition similar to the following:

   apiVersion: v1
   kind: Pod
   metadata:
     name: tunepod
     namespace: default
     annotations:
       k8s.v1.cni.cncf.io/networks: tuningnad 1
   spec:
     containers:
     - name: podexample
       image: centos
       command: ["/bin/bash", "-c", "sleep INF"]
       securityContext:
         runAsUser: 2000 2
         runAsGroup: 3000 3
         allowPrivilegeEscalation: false 4
         capabilities: 5
           drop: ["ALL"]
     securityContext:
       runAsNonRoot: true 6
       seccompProfile: 7
         type: RuntimeDefault

   1

   Specify the name of the configured NetworkAttachmentDefinition.

   2

   runAsUser controls which user ID the container is run with.

   3

   runAsGroup controls which primary group ID the containers is run with.

   4

   allowPrivilegeEscalation determines if a pod can request to allow privilege escalation. If unspecified, it defaults to true. This boolean directly controls whether the no_new_privs flag gets set on the container process.

   5

   capabilities permit privileged actions without giving full root access. This policy ensures all capabilities are dropped from the pod.

   6

   runAsNonRoot: true requires that the container will run with a user with any UID other than 0.

   7

   RuntimeDefault enables the default seccomp profile for a pod or container workload.

4. Apply the yaml by running the following command:

   $ oc apply -f examplepod.yaml

5. Verify that the pod is created by running the following command:

   $ oc get pod

   Example output

   NAME      READY   STATUS    RESTARTS   AGE
   tunepod   1/1     Running   0          47s

6. Log in to the pod by running the following command:

   $ oc rsh tunepod

7. Verify the values of the configured sysctl flags. For example, find the value net.ipv4.conf.net1.accept_redirects by running the following command:

   sh-4.4# sysctl net.ipv4.conf.net1.accept_redirects

   Expected output

   net.ipv4.conf.net1.accept_redirects = 1

Procedure

1. Create a network attachment definition, such as tuning-example.yaml, with the following content:

   apiVersion: "k8s.cni.cncf.io/v1"
   kind: NetworkAttachmentDefinition
   metadata:
     name: <name> 1
     namespace: default 2
   spec:
     config: '{
       "cniVersion": "0.4.0", 3
       "name": "<name>", 4
       "plugins": [{
          "type": "<main_CNI_plugin>" 5
         },
         {
          "type": "tuning", 6
          "allmulti": true 7
           }
         }
        ]
   }

   1

   Specifies the name for the additional network attachment to create. The name must be unique within the specified namespace.

   2

   Specifies the namespace that the object is associated with.

   3

   Specifies the CNI specification version.

   4

   Specifies the name for the configuration. Match the configuration name to the name value of the network attachment definition.

   5

   Specifies the name of the main CNI plugin to configure.

   6

   Specifies the name of the CNI meta plugin.

   7

   Changes the all-multicast mode of interface. If enabled, all multicast packets on the network will be received by the interface.

   An example YAML file is shown here:

   apiVersion: "k8s.cni.cncf.io/v1"
   kind: NetworkAttachmentDefinition
   metadata:
     name: setallmulti
     namespace: default
   spec:
     config: '{
       "cniVersion": "0.4.0",
       "name": "setallmulti",
       "plugins": [
         {
           "type": "bridge"
         },
         {
           "type": "tuning",
           "allmulti": true
         }
       ]
     }'

2. Apply the settings specified in the YAML file by running the following command:

   $ oc apply -f tuning-allmulti.yaml

   Example output

   networkattachmentdefinition.k8s.cni.cncf.io/setallmulti created

3. Create a pod with a network attachment definition similar to that specified in the following examplepod.yaml sample file:

   apiVersion: v1
   kind: Pod
   metadata:
     name: allmultipod
     namespace: default
     annotations:
       k8s.v1.cni.cncf.io/networks: setallmulti 1
   spec:
     containers:
     - name: podexample
       image: centos
       command: ["/bin/bash", "-c", "sleep INF"]
       securityContext:
         runAsUser: 2000 2
         runAsGroup: 3000 3
         allowPrivilegeEscalation: false 4
         capabilities: 5
           drop: ["ALL"]
     securityContext:
       runAsNonRoot: true 6
       seccompProfile: 7
         type: RuntimeDefault

   1

   Specifies the name of the configured NetworkAttachmentDefinition.

   2

   Specifies the user ID the container is run with.

   3

   Specifies which primary group ID the containers is run with.

   4

   Specifies if a pod can request privilege escalation. If unspecified, it defaults to true. This boolean directly controls whether the no_new_privs flag gets set on the container process.

   5

   Specifies the container capabilities. The drop: ["ALL"] statement indicates that all Linux capabilities are dropped from the pod, providing a more restrictive security profile.

   6

   Specifies that the container will run with a user with any UID other than 0.

   7

   Specifies the container’s seccomp profile. In this case, the type is set to RuntimeDefault. Seccomp is a Linux kernel feature that restricts the system calls available to a process, enhancing security by minimizing the attack surface.

4. Apply the settings specified in the YAML file by running the following command:

   $ oc apply -f examplepod.yaml

5. Verify that the pod is created by running the following command:

   $ oc get pod

   Example output

   NAME          READY   STATUS    RESTARTS   AGE
   allmultipod   1/1     Running   0          23s

6. Log in to the pod by running the following command:

   $ oc rsh allmultipod

7. List all the interfaces associated with the pod by running the following command:

   sh-4.4# ip link

   Example output

   1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
       link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
   2: eth0@if22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8901 qdisc noqueue state UP mode DEFAULT group default
       link/ether 0a:58:0a:83:00:10 brd ff:ff:ff:ff:ff:ff link-netnsid 0 1
   3: net1@if24: <BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
       link/ether ee:9b:66:a4:ec:1d brd ff:ff:ff:ff:ff:ff link-netnsid 0 2

   1

   eth0@if22 is the primary interface.

   2

   net1@if24 is the secondary interface configured with the network-attachment-definition that supports the all-multicast mode (ALLMULTI flag).

Procedure

1. Create a file named load-sctp-module.yaml that contains the following YAML definition:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     name: load-sctp-module
     labels:
       machineconfiguration.openshift.io/role: worker
   spec:
     config:
       ignition:
         version: 3.2.0
       storage:
         files:
           - path: /etc/modprobe.d/sctp-blacklist.conf
             mode: 0644
             overwrite: true
             contents:
               source: data:,
           - path: /etc/modules-load.d/sctp-load.conf
             mode: 0644
             overwrite: true
             contents:
               source: data:,sctp

2. To create the MachineConfig object, enter the following command:

   $ oc create -f load-sctp-module.yaml

3. Optional: To watch the status of the nodes while the MachineConfig Operator applies the configuration change, enter the following command. When the status of a node transitions to Ready, the configuration update is applied.

   $ oc get nodes

Procedure

1. Create a pod starts an SCTP listener:

   1. Create a file named sctp-server.yaml that defines a pod with the following YAML:

      apiVersion: v1
      kind: Pod
      metadata:
        name: sctpserver
        labels:
          app: sctpserver
      spec:
        containers:
          - name: sctpserver
            image: registry.access.redhat.com/ubi9/ubi
            command: ["/bin/sh", "-c"]
            args:
              ["dnf install -y nc && sleep inf"]
            ports:
              - containerPort: 30102
                name: sctpserver
                protocol: SCTP

   2. Create the pod by entering the following command:

      $ oc create -f sctp-server.yaml

2. Create a service for the SCTP listener pod.

   1. Create a file named sctp-service.yaml that defines a service with the following YAML:

      apiVersion: v1
      kind: Service
      metadata:
        name: sctpservice
        labels:
          app: sctpserver
      spec:
        type: NodePort
        selector:
          app: sctpserver
        ports:
          - name: sctpserver
            protocol: SCTP
            port: 30102
            targetPort: 30102

   2. To create the service, enter the following command:

      $ oc create -f sctp-service.yaml

3. Create a pod for the SCTP client.

   1. Create a file named sctp-client.yaml with the following YAML:

      apiVersion: v1
      kind: Pod
      metadata:
        name: sctpclient
        labels:
          app: sctpclient
      spec:
        containers:
          - name: sctpclient
            image: registry.access.redhat.com/ubi9/ubi
            command: ["/bin/sh", "-c"]
            args:
              ["dnf install -y nc && sleep inf"]

   2. To create the Pod object, enter the following command:

      $ oc apply -f sctp-client.yaml

4. Run an SCTP listener on the server.

   1. To connect to the server pod, enter the following command:

      $ oc rsh sctpserver

   2. To start the SCTP listener, enter the following command:

      $ nc -l 30102 --sctp

5. Connect to the SCTP listener on the server.

   1. Open a new terminal window or tab in your terminal program.

   2. Obtain the IP address of the sctpservice service. Enter the following command:

      $ oc get services sctpservice -o go-template='{{.spec.clusterIP}}{{"\n"}}'

   3. To connect to the client pod, enter the following command:

      $ oc rsh sctpclient

   4. To start the SCTP client, enter the following command. Replace <cluster_IP> with the cluster IP address of the sctpservice service.

      # nc <cluster_IP> 30102 --sctp