Networking

Procedure

1. Create a security group that allows SSH access into the virtual private cloud (VPC) created by the openshift-install command.
2. Create an Amazon EC2 instance on one of the public subnets the installer created.

3. Associate a public IP address with the Amazon EC2 instance that you created.

   Unlike with the OpenShift Container Platform installation, you should associate the Amazon EC2 instance you created with an SSH keypair. It does not matter what operating system you choose for this instance, as it will simply serve as an SSH bastion to bridge the internet into your OpenShift Container Platform cluster’s VPC. The Amazon Machine Image (AMI) you use does matter. With Red Hat Enterprise Linux CoreOS (RHCOS), for example, you can provide keys via Ignition, like the installer does.

4. After you provisioned your Amazon EC2 instance and can SSH into it, you must add the SSH key that you associated with your OpenShift Container Platform installation. This key can be different from the key for the bastion instance, but does not have to be.

   Note

   Direct SSH access is only recommended for disaster recovery. When the Kubernetes API is responsive, run privileged pods instead.

5. Run oc get nodes, inspect the output, and choose one of the nodes that is a master. The hostname looks similar to ip-10-0-1-163.ec2.internal.

6. From the bastion SSH host you manually deployed into Amazon EC2, SSH into that control plane host. Ensure that you use the same SSH key you specified during the installation:

   $ ssh -i <ssh-key-path> core@<master-hostname>

1. Run the following command to view the Deployment status:

   $ oc get -n openshift-network-operator deployment/network-operator

   Example output

   NAME               READY   UP-TO-DATE   AVAILABLE   AGE
   network-operator   1/1     1            1           56m

2. Run the following command to view the state of the Cluster Network Operator:

   $ oc get clusteroperator/network

   Example output

   NAME      VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
   network   4.5.4     True        False         False      50m

   The following fields provide information about the status of the operator: AVAILABLE, PROGRESSING, and DEGRADED. The AVAILABLE field is True when the Cluster Network Operator reports an available status condition.

1. Use the oc get command to view the deployment status:

   $ oc get -n openshift-dns-operator deployment/dns-operator

   Example output

   NAME           READY     UP-TO-DATE   AVAILABLE   AGE
   dns-operator   1/1       1            1           23h

2. Use the oc get command to view the state of the DNS Operator:

   $ oc get clusteroperator/dns

   Example output

   NAME      VERSION     AVAILABLE   PROGRESSING   DEGRADED   SINCE
   dns       4.1.0-0.11  True        False         False      92m

   AVAILABLE, PROGRESSING and DEGRADED provide information about the status of the operator. AVAILABLE is True when at least 1 pod from the CoreDNS daemon set reports an Available status condition.

Procedure

1. Use the oc describe command to view the default dns:

   $ oc describe dns.operator/default

   Example output

   Name:         default
   Namespace:
   Labels:       <none>
   Annotations:  <none>
   API Version:  operator.openshift.io/v1
   Kind:         DNS
   ...
   Status:
     Cluster Domain:  cluster.local 1
     Cluster IP:      172.30.0.10 2
   ...

   1

   The Cluster Domain field is the base DNS domain used to construct fully qualified pod and service domain names.

   2

   The Cluster IP is the address pods query for name resolution. The IP is defined as the 10th address in the service CIDR range.

2. To find the service CIDR of your cluster, use the oc get command:

   $ oc get networks.config/cluster -o jsonpath='{$.status.serviceNetwork}'

Procedure

1. Modify the DNS Operator object named default:

   $ oc edit dns.operator/default

   This allows the Operator to create and update the ConfigMap named dns-default with additional server configuration blocks based on Server. If none of the servers has a zone that matches the query, then name resolution falls back to the name servers that are specified in /etc/resolv.conf.

   Sample DNS

   apiVersion: operator.openshift.io/v1
   kind: DNS
   metadata:
     name: default
   spec:
     servers:
     - name: foo-server 1
       zones: 2
         - example.com
       forwardPlugin:
         upstreams: 3
           - 1.1.1.1
           - 2.2.2.2:5353
     - name: bar-server
       zones:
         - bar.com
         - example.com
       forwardPlugin:
         upstreams:
           - 3.3.3.3
           - 4.4.4.4:5454

   1

   name must comply with the rfc6335 service name syntax.

   2

   zones must conform to the definition of a subdomain in rfc1123. The cluster domain, cluster.local, is an invalid subdomain for zones.

   3

   A maximum of 15 upstreams is allowed per forwardPlugin.

   Note

   If servers is undefined or invalid, the ConfigMap only contains the default server.

2. View the ConfigMap:

   $ oc get configmap/dns-default -n openshift-dns -o yaml

   Sample DNS ConfigMap based on previous sample DNS

   apiVersion: v1
   data:
     Corefile: |
       example.com:5353 {
           forward . 1.1.1.1 2.2.2.2:5353
       }
       bar.com:5353 example.com:5353 {
           forward . 3.3.3.3 4.4.4.4:5454 1
       }
       .:5353 {
           errors
           health
           kubernetes cluster.local in-addr.arpa ip6.arpa {
               pods insecure
               upstream
               fallthrough in-addr.arpa ip6.arpa
           }
           prometheus :9153
           forward . /etc/resolv.conf {
               policy sequential
           }
           cache 30
           reload
       }
   kind: ConfigMap
   metadata:
     labels:
       dns.operator.openshift.io/owning-dns: default
     name: dns-default
     namespace: openshift-dns

   1

   Changes to the forwardPlugin triggers a rolling update of the CoreDNS daemon set.

Procedure

1. To list the current PodNetworkConnectivityCheck objects, enter the following command:

   $ oc get podnetworkconnectivitycheck -n openshift-network-diagnostics

   Example output

   NAME                                                                                                                                AGE
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0   75m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-1   73m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-2   75m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-kubernetes-apiserver-service-cluster                               75m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-kubernetes-default-service-cluster                                 75m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-load-balancer-api-external                                         75m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-load-balancer-api-internal                                         75m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-network-check-target-ci-ln-x5sv9rb-f76d1-4rzrp-master-0            75m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-network-check-target-ci-ln-x5sv9rb-f76d1-4rzrp-master-1            75m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-network-check-target-ci-ln-x5sv9rb-f76d1-4rzrp-master-2            75m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-network-check-target-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh      74m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-network-check-target-ci-ln-x5sv9rb-f76d1-4rzrp-worker-c-n8mbf      74m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-network-check-target-ci-ln-x5sv9rb-f76d1-4rzrp-worker-d-4hnrz      74m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-network-check-target-service-cluster                               75m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-openshift-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0    75m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-openshift-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-1    75m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-openshift-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-2    74m
   network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-openshift-apiserver-service-cluster                                75m

2. View the connection test logs:

   1. From the output of the previous command, identify the endpoint that you want to review the connectivity logs for.

   2. To view the object, enter the following command:

      $ oc get podnetworkconnectivitycheck <name> \
        -n openshift-network-diagnostics -o yaml

      where <name> specifies the name of the PodNetworkConnectivityCheck object.

      Example output

      apiVersion: controlplane.operator.openshift.io/v1alpha1
      kind: PodNetworkConnectivityCheck
      metadata:
        name: network-check-source-ci-ln-x5sv9rb-f76d1-4rzrp-worker-b-6xdmh-to-kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0
        namespace: openshift-network-diagnostics
        ...
      spec:
        sourcePod: network-check-source-7c88f6d9f-hmg2f
        targetEndpoint: 10.0.0.4:6443
        tlsClientCert:
          name: ""
      status:
        conditions:
        - lastTransitionTime: "2021-01-13T20:11:34Z"
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: tcp
            connection to 10.0.0.4:6443 succeeded'
          reason: TCPConnectSuccess
          status: "True"
          type: Reachable
        failures:
        - latency: 2.241775ms
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: failed
            to establish a TCP connection to 10.0.0.4:6443: dial tcp 10.0.0.4:6443: connect:
            connection refused'
          reason: TCPConnectError
          success: false
          time: "2021-01-13T20:10:34Z"
        - latency: 2.582129ms
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: failed
            to establish a TCP connection to 10.0.0.4:6443: dial tcp 10.0.0.4:6443: connect:
            connection refused'
          reason: TCPConnectError
          success: false
          time: "2021-01-13T20:09:34Z"
        - latency: 3.483578ms
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: failed
            to establish a TCP connection to 10.0.0.4:6443: dial tcp 10.0.0.4:6443: connect:
            connection refused'
          reason: TCPConnectError
          success: false
          time: "2021-01-13T20:08:34Z"
        outages:
        - end: "2021-01-13T20:11:34Z"
          endLogs:
          - latency: 2.032018ms
            message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0:
              tcp connection to 10.0.0.4:6443 succeeded'
            reason: TCPConnect
            success: true
            time: "2021-01-13T20:11:34Z"
          - latency: 2.241775ms
            message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0:
              failed to establish a TCP connection to 10.0.0.4:6443: dial tcp 10.0.0.4:6443:
              connect: connection refused'
            reason: TCPConnectError
            success: false
            time: "2021-01-13T20:10:34Z"
          - latency: 2.582129ms
            message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0:
              failed to establish a TCP connection to 10.0.0.4:6443: dial tcp 10.0.0.4:6443:
              connect: connection refused'
            reason: TCPConnectError
            success: false
            time: "2021-01-13T20:09:34Z"
          - latency: 3.483578ms
            message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0:
              failed to establish a TCP connection to 10.0.0.4:6443: dial tcp 10.0.0.4:6443:
              connect: connection refused'
            reason: TCPConnectError
            success: false
            time: "2021-01-13T20:08:34Z"
          message: Connectivity restored after 2m59.999789186s
          start: "2021-01-13T20:08:34Z"
          startLogs:
          - latency: 3.483578ms
            message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0:
              failed to establish a TCP connection to 10.0.0.4:6443: dial tcp 10.0.0.4:6443:
              connect: connection refused'
            reason: TCPConnectError
            success: false
            time: "2021-01-13T20:08:34Z"
        successes:
        - latency: 2.845865ms
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: tcp
            connection to 10.0.0.4:6443 succeeded'
          reason: TCPConnect
          success: true
          time: "2021-01-13T21:14:34Z"
        - latency: 2.926345ms
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: tcp
            connection to 10.0.0.4:6443 succeeded'
          reason: TCPConnect
          success: true
          time: "2021-01-13T21:13:34Z"
        - latency: 2.895796ms
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: tcp
            connection to 10.0.0.4:6443 succeeded'
          reason: TCPConnect
          success: true
          time: "2021-01-13T21:12:34Z"
        - latency: 2.696844ms
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: tcp
            connection to 10.0.0.4:6443 succeeded'
          reason: TCPConnect
          success: true
          time: "2021-01-13T21:11:34Z"
        - latency: 1.502064ms
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: tcp
            connection to 10.0.0.4:6443 succeeded'
          reason: TCPConnect
          success: true
          time: "2021-01-13T21:10:34Z"
        - latency: 1.388857ms
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: tcp
            connection to 10.0.0.4:6443 succeeded'
          reason: TCPConnect
          success: true
          time: "2021-01-13T21:09:34Z"
        - latency: 1.906383ms
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: tcp
            connection to 10.0.0.4:6443 succeeded'
          reason: TCPConnect
          success: true
          time: "2021-01-13T21:08:34Z"
        - latency: 2.089073ms
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: tcp
            connection to 10.0.0.4:6443 succeeded'
          reason: TCPConnect
          success: true
          time: "2021-01-13T21:07:34Z"
        - latency: 2.156994ms
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: tcp
            connection to 10.0.0.4:6443 succeeded'
          reason: TCPConnect
          success: true
          time: "2021-01-13T21:06:34Z"
        - latency: 1.777043ms
          message: 'kubernetes-apiserver-endpoint-ci-ln-x5sv9rb-f76d1-4rzrp-master-0: tcp
            connection to 10.0.0.4:6443 succeeded'
          reason: TCPConnect
          success: true
          time: "2021-01-13T21:05:34Z"

Procedure

1. To expand the node port range, enter the following command. Replace <port> with the largest port number in the new range.

   $ oc patch network.config.openshift.io cluster --type=merge -p \
     '{
       "spec":
         { "serviceNodePortRange": "30000-<port>" }
     }'

   Tip

   You can alternatively apply the following YAML to update the node port range:

   apiVersion: config.openshift.io/v1
   kind: Network
   metadata:
     name: cluster
   spec:
     serviceNodePortRange: "30000-<port>"

   Example output

   network.config.openshift.io/cluster patched

2. To confirm that the configuration is active, enter the following command. It can take several minutes for the update to apply.

   $ oc get configmaps -n openshift-kube-apiserver config \
     -o jsonpath="{.data['config\.yaml']}" | \
     grep -Eo '"service-node-port-range":["[[:digit:]]+-[[:digit:]]+"]'

   Example output

   "service-node-port-range":["30000-33000"]

Procedure

1. Create an IP failover service account:

   $ oc create sa ipfailover

2. Update security context constraints (SCC) for hostNetwork:

   $ oc adm policy add-scc-to-user privileged -z ipfailover
   $ oc adm policy add-scc-to-user hostnetwork -z ipfailover

3. Create a deployment YAML file to configure IP failover:

   Example deployment YAML for IP failover configuration

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: ipfailover-keepalived 1
     labels:
       ipfailover: hello-openshift
   spec:
     strategy:
       type: Recreate
     replicas: 2
     selector:
       matchLabels:
         ipfailover: hello-openshift
     template:
       metadata:
         labels:
           ipfailover: hello-openshift
       spec:
         serviceAccountName: ipfailover
         privileged: true
         hostNetwork: true
         nodeSelector:
           node-role.kubernetes.io/worker: ""
         containers:
         - name: openshift-ipfailover
           image: quay.io/openshift/origin-keepalived-ipfailover
           ports:
           - containerPort: 63000
             hostPort: 63000
           imagePullPolicy: IfNotPresent
           securityContext:
             privileged: true
           volumeMounts:
           - name: lib-modules
             mountPath: /lib/modules
             readOnly: true
           - name: host-slash
             mountPath: /host
             readOnly: true
             mountPropagation: HostToContainer
           - name: etc-sysconfig
             mountPath: /etc/sysconfig
             readOnly: true
           - name: config-volume
             mountPath: /etc/keepalive
           env:
           - name: OPENSHIFT_HA_CONFIG_NAME
             value: "ipfailover"
           - name: OPENSHIFT_HA_VIRTUAL_IPS 2
             value: "1.1.1.1-2"
           - name: OPENSHIFT_HA_VIP_GROUPS 3
             value: "10"
           - name: OPENSHIFT_HA_NETWORK_INTERFACE 4
             value: "ens3" #The host interface to assign the VIPs
           - name: OPENSHIFT_HA_MONITOR_PORT 5
             value: "30060"
           - name: OPENSHIFT_HA_VRRP_ID_OFFSET 6
             value: "0"
           - name: OPENSHIFT_HA_REPLICA_COUNT 7
             value: "2" #Must match the number of replicas in the deployment
           - name: OPENSHIFT_HA_USE_UNICAST
             value: "false"
           #- name: OPENSHIFT_HA_UNICAST_PEERS
             #value: "10.0.148.40,10.0.160.234,10.0.199.110"
           - name: OPENSHIFT_HA_IPTABLES_CHAIN 8
             value: "INPUT"
           #- name: OPENSHIFT_HA_NOTIFY_SCRIPT 9
           #  value: /etc/keepalive/mynotifyscript.sh
           - name: OPENSHIFT_HA_CHECK_SCRIPT 10
             value: "/etc/keepalive/mycheckscript.sh"
           - name: OPENSHIFT_HA_PREEMPTION 11
             value: "preempt_delay 300"
           - name: OPENSHIFT_HA_CHECK_INTERVAL 12
             value: "2"
           livenessProbe:
             initialDelaySeconds: 10
             exec:
               command:
               - pgrep
               - keepalived
         volumes:
         - name: lib-modules
           hostPath:
             path: /lib/modules
         - name: host-slash
           hostPath:
             path: /
         - name: etc-sysconfig
           hostPath:
             path: /etc/sysconfig
         # config-volume contains the check script
         # created with `oc create configmap keepalived-checkscript --from-file=mycheckscript.sh`
         - configMap:
             defaultMode: 0755
             name: keepalived-checkscript
           name: config-volume
         imagePullSecrets:
           - name: openshift-pull-secret 13

   1

   The name of the IP failover deployment.

   2

   The list of IP address ranges to replicate. This must be provided. For example, 1.2.3.4-6,1.2.3.9.

   3

   The number of groups to create for VRRP. If not set, a group is created for each virtual IP range specified with the OPENSHIFT_HA_VIP_GROUPS variable.

   4

   The interface name that IP failover uses to send VRRP traffic. By default, eth0 is used.

   5

   The IP failover pod tries to open a TCP connection to this port on each VIP. If connection is established, the service is considered to be running. If this port is set to 0, the test always passes. The default value is 80.

   6

   The offset value used to set the virtual router IDs. Using different offset values allows multiple IP failover configurations to exist within the same cluster. The default offset is 0, and the allowed range is 0 through 255.

   7

   The number of replicas to create. This must match spec.replicas value in IP failover deployment configuration. The default value is 2.

   8

   The name of the iptables chain to automatically add an iptables rule to allow the VRRP traffic on. If the value is not set, an iptables rule is not added. If the chain does not exist, it is not created, and Keepalived operates in unicast mode. The default is INPUT.

   9

   The full path name in the pod file system of a script that is run whenever the state changes.

   10

   The full path name in the pod file system of a script that is periodically run to verify the application is operating.

   11

   The strategy for handling a new higher priority host. The default value is preempt_delay 300, which causes a Keepalived instance to take over a VIP after 5 minutes if a lower-priority master is holding the VIP.

   12

   The period, in seconds, that the check script is run. The default value is 2.

   13

   Create the pull secret before creating the deployment, otherwise you will get an error when creating the deployment.

Procedure

1. Create the desired script and create a config map to hold it. The script has no input arguments and must return 0 for OK and 1 for fail.

   The check script, mycheckscript.sh:

   #!/bin/bash
       # Whatever tests are needed
       # E.g., send request and verify response
   exit 0

2. Create the config map:

   $ oc create configmap mycustomcheck --from-file=mycheckscript.sh

3. Add the script to the pod. The defaultMode for the mounted config map files must able to run by using oc commands or by editing the deployment configuration. A value of 0755, 493 decimal, is typical:

   $ oc set env deploy/ipfailover-keepalived \
       OPENSHIFT_HA_CHECK_SCRIPT=/etc/keepalive/mycheckscript.sh

   $ oc set volume deploy/ipfailover-keepalived --add --overwrite \
       --name=config-volume \
       --mount-path=/etc/keepalive \
       --source='{"configMap": { "name": "mycustomcheck", "defaultMode": 493}}'

   Note

   The oc set env command is whitespace sensitive. There must be no whitespace on either side of the = sign.

   Tip

   You can alternatively edit the ipfailover-keepalived deployment configuration:

   $ oc edit deploy ipfailover-keepalived

       spec:
         containers:
         - env:
           - name: OPENSHIFT_HA_CHECK_SCRIPT  1
             value: /etc/keepalive/mycheckscript.sh
   ...
           volumeMounts: 2
           - mountPath: /etc/keepalive
             name: config-volume
         dnsPolicy: ClusterFirst
   ...
         volumes: 3
         - configMap:
             defaultMode: 0755 4
             name: customrouter
           name: config-volume
   ...

   1

   In the spec.container.env field, add the OPENSHIFT_HA_CHECK_SCRIPT environment variable to point to the mounted script file.

   2

   Add the spec.container.volumeMounts field to create the mount point.

   3

   Add a new spec.volumes field to mention the config map.

   4

   This sets run permission on the files. When read back, it is displayed in decimal, 493.

   Save the changes and exit the editor. This restarts ipfailover-keepalived.

Procedure

1. Optional: Identify and delete any check and notify scripts that are stored as config maps:

   1. Identify whether any pods for IP failover use a config map as a volume:

      $ oc get pod -l ipfailover \
        -o jsonpath="\
      {range .items[?(@.spec.volumes[*].configMap)]}
      {'Namespace: '}{.metadata.namespace}
      {'Pod:       '}{.metadata.name}
      {'Volumes that use config maps:'}
      {range .spec.volumes[?(@.configMap)]}  {'volume:    '}{.name}
        {'configMap: '}{.configMap.name}{'\n'}{end}
      {end}"

      Example output

      Namespace: default
      Pod:       keepalived-worker-59df45db9c-2x9mn
      Volumes that use config maps:
        volume:    config-volume
        configMap: mycustomcheck

   2. If the preceding step provided the names of config maps that are used as volumes, delete the config maps:

      $ oc delete configmap <configmap_name>

2. Identify an existing deployment for IP failover:

   $ oc get deployment -l ipfailover

   Example output

   NAMESPACE   NAME         READY   UP-TO-DATE   AVAILABLE   AGE
   default     ipfailover   2/2     2            2           105d

3. Delete the deployment:

   $ oc delete deployment <ipfailover_deployment_name>

4. Remove the ipfailover service account:

   $ oc delete sa ipfailover

5. Run a job that removes the IP tables rule that was added when IP failover was initially configured:

   1. Create a file such as remove-ipfailover-job.yaml with contents that are similar to the following example:

      apiVersion: batch/v1
      kind: Job
      metadata:
        generateName: remove-ipfailover-
        labels:
          app: remove-ipfailover
      spec:
        template:
          metadata:
            name: remove-ipfailover
          spec:
            containers:
            - name: remove-ipfailover
              image: quay.io/openshift/origin-keepalived-ipfailover:4.9
              command: ["/var/lib/ipfailover/keepalived/remove-failover.sh"]
            nodeSelector:
              kubernetes.io/hostname: <host_name>  <.>
            restartPolicy: Never

      <.> Run the job for each node in your cluster that was configured for IP failover and replace the hostname each time.

   2. Run the job:

      $ oc create -f remove-ipfailover-job.yaml

      Example output

      job.batch/remove-ipfailover-2h8dm created

Procedure

1. Create a file named load-sctp-module.yaml that contains the following YAML definition:

   apiVersion: machineconfiguration.openshift.io/v1
   kind: MachineConfig
   metadata:
     name: load-sctp-module
     labels:
       machineconfiguration.openshift.io/role: worker
   spec:
     config:
       ignition:
         version: 3.2.0
       storage:
         files:
           - path: /etc/modprobe.d/sctp-blacklist.conf
             mode: 0644
             overwrite: true
             contents:
               source: data:,
           - path: /etc/modules-load.d/sctp-load.conf
             mode: 0644
             overwrite: true
             contents:
               source: data:,sctp

2. To create the MachineConfig object, enter the following command:

   $ oc create -f load-sctp-module.yaml

3. Optional: To watch the status of the nodes while the MachineConfig Operator applies the configuration change, enter the following command. When the status of a node transitions to Ready, the configuration update is applied.

   $ oc get nodes

Procedure

1. Create a pod starts an SCTP listener:

   1. Create a file named sctp-server.yaml that defines a pod with the following YAML:

      apiVersion: v1
      kind: Pod
      metadata:
        name: sctpserver
        labels:
          app: sctpserver
      spec:
        containers:
          - name: sctpserver
            image: registry.access.redhat.com/ubi8/ubi
            command: ["/bin/sh", "-c"]
            args:
              ["dnf install -y nc && sleep inf"]
            ports:
              - containerPort: 30102
                name: sctpserver
                protocol: SCTP

   2. Create the pod by entering the following command:

      $ oc create -f sctp-server.yaml

2. Create a service for the SCTP listener pod.

   1. Create a file named sctp-service.yaml that defines a service with the following YAML:

      apiVersion: v1
      kind: Service
      metadata:
        name: sctpservice
        labels:
          app: sctpserver
      spec:
        type: NodePort
        selector:
          app: sctpserver
        ports:
          - name: sctpserver
            protocol: SCTP
            port: 30102
            targetPort: 30102

   2. To create the service, enter the following command:

      $ oc create -f sctp-service.yaml

3. Create a pod for the SCTP client.

   1. Create a file named sctp-client.yaml with the following YAML:

      apiVersion: v1
      kind: Pod
      metadata:
        name: sctpclient
        labels:
          app: sctpclient
      spec:
        containers:
          - name: sctpclient
            image: registry.access.redhat.com/ubi8/ubi
            command: ["/bin/sh", "-c"]
            args:
              ["dnf install -y nc && sleep inf"]

   2. To create the Pod object, enter the following command:

      $ oc apply -f sctp-client.yaml

4. Run an SCTP listener on the server.

   1. To connect to the server pod, enter the following command:

      $ oc rsh sctpserver

   2. To start the SCTP listener, enter the following command:

      $ nc -l 30102 --sctp

5. Connect to the SCTP listener on the server.

   1. Open a new terminal window or tab in your terminal program.

   2. Obtain the IP address of the sctpservice service. Enter the following command:

      $ oc get services sctpservice -o go-template='{{.spec.clusterIP}}{{"\n"}}'

   3. To connect to the client pod, enter the following command:

      $ oc rsh sctpclient

   4. To start the SCTP client, enter the following command. Replace <cluster_IP> with the cluster IP address of the sctpservice service.

      # nc <cluster_IP> 30102 --sctp

Procedure

1. To create a namespace for the PTP Operator, enter the following command:

   $ cat << EOF| oc create -f -
   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-ptp
     annotations:
       workload.openshift.io/allowed: management
     labels:
       name: openshift-ptp
       openshift.io/cluster-monitoring: "true"
   EOF

2. To create an Operator group for the Operator, enter the following command:

   $ cat << EOF| oc create -f -
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: ptp-operators
     namespace: openshift-ptp
   spec:
     targetNamespaces:
     - openshift-ptp
   EOF

3. Subscribe to the PTP Operator.

   1. Run the following command to set the OpenShift Container Platform major and minor version as an environment variable, which is used as the channel value in the next step.

      $ OC_VERSION=$(oc version -o yaml | grep openshiftVersion | \
          grep -o '[0-9]*[.][0-9]*' | head -1)

   2. To create a subscription for the PTP Operator, enter the following command:

      $ cat << EOF| oc create -f -
      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: ptp-operator-subscription
        namespace: openshift-ptp
      spec:
        channel: "${OC_VERSION}"
        name: ptp-operator
        source: redhat-operators
        sourceNamespace: openshift-marketplace
      EOF

4. To verify that the Operator is installed, enter the following command:

   $ oc get csv -n openshift-ptp \
     -o custom-columns=Name:.metadata.name,Phase:.status.phase

   Example output

   Name                                        Phase
   ptp-operator.4.4.0-202006160135             Succeeded

Procedure

1. Install the PTP Operator using the OpenShift Container Platform web console:

   1. In the OpenShift Container Platform web console, click Operators → OperatorHub.
   2. Choose PTP Operator from the list of available Operators, and then click Install.
   3. On the Install Operator page, under A specific namespace on the cluster select openshift-ptp. Then, click Install.

2. Optional: Verify that the PTP Operator installed successfully:

   1. Switch to the Operators → Installed Operators page.

   2. Ensure that PTP Operator is listed in the openshift-ptp project with a Status of InstallSucceeded.

      Note

      During installation an Operator might display a Failed status. If the installation later succeeds with an InstallSucceeded message, you can ignore the Failed message.

      If the Operator does not appear as installed, to troubleshoot further:

      • Go to the Operators → Installed Operators page and inspect the Operator Subscriptions and Install Plans tabs for any failure or errors under Status.
      • Go to the Workloads → Pods page and check the logs for pods in the openshift-ptp project.

Procedure

1. Create the following PtpConfig CR, and then save the YAML in the ordinary-clock-ptp-config.yaml file.

   apiVersion: ptp.openshift.io/v1
   kind: PtpConfig
   metadata:
     name: ordinary-clock-ptp-config 1
     namespace: openshift-ptp
   spec:
     profile: 2
     - name: "profile1" 3
       interface: "ens787f1" 4
       ptp4lOpts: "-s -2" 5
       phc2sysOpts: "-a -r" 6
       ptp4lConf: "" 7
       ptpSchedulingPolicy: SCHED_OTHER 8
       ptpSchedulingPriority: 10 9
     recommend: 10
     - profile: "profile1" 11
       priority: 10 12
       match: 13
       - nodeLabel: "node-role.kubernetes.io/worker" 14
         nodeName: "compute-0.example.com" 15

   1

   The name of the PtpConfig CR.

   2

   Specify an array of one or more profile objects.

   3

   Specify the name of a profile object that uniquely identifies a profile object.

   4

   Specify the network interface name to use by the ptp4l service, for example ens787f1.

   5

   Specify system config options for the ptp4l service, for example -2 to select the IEEE 802.3 network transport. The options should not include the network interface name -i <interface> and service config file -f /etc/ptp4l.conf because the network interface name and the service config file are automatically appended.

   6

   Specify system config options for the phc2sys service, for example -a -r. If this field is empty the PTP Operator does not start the phc2sys service.

   7

   Specify a string that contains the configuration to replace the default /etc/ptp4l.conf file. To use the default configuration, leave the field empty.

   8

   Scheduling policy for ptp4l and phc2sys processes. Default value is SCHED_OTHER. Use SCHED_FIFO on systems that support FIFO scheduling.

   9

   Integer value from 1-65 used to set FIFO priority for ptp4l and phc2sys processes when ptpSchedulingPolicy is set to SCHED_FIFO. The ptpSchedulingPriority field is not used when ptpSchedulingPolicy is set to SCHED_OTHER.

   10

   Specify an array of one or more recommend objects that define rules on how the profile should be applied to nodes.

   11

   Specify the profile object name defined in the profile section.

   12

   Specify the priority with an integer value between 0 and 99. A larger number gets lower priority, so a priority of 99 is lower than a priority of 10. If a node can be matched with multiple profiles according to rules defined in the match field, the profile with the higher priority is applied to that node.

   13

   Specify match rules with nodeLabel or nodeName.

   14

   Specify nodeLabel with the key of node.Labels from the node object by using the oc get nodes --show-labels command.

   15

   Specify nodeName with node.Name from the node object by using the oc get nodes command.

2. Create the CR by running the following command:

   $ oc create -f ordinary-clock-ptp-config.yaml

Verification steps

1. Check that the PtpConfig profile is applied to the node.

   1. Get the list of pods in the openshift-ptp namespace by running the following command:

      $ oc get pods -n openshift-ptp -o wide

      Example output

      NAME                            READY   STATUS    RESTARTS   AGE   IP               NODE
      linuxptp-daemon-4xkbb           1/1     Running   0          43m   10.1.196.24      compute-0.example.com
      linuxptp-daemon-tdspf           1/1     Running   0          43m   10.1.196.25      compute-1.example.com
      ptp-operator-657bbb64c8-2f8sj   1/1     Running   0          43m   10.129.0.61      control-plane-1.example.com

   2. Check that the profile is correct. Examine the logs of the linuxptp daemon that corresponds to the node you specified in the PtpConfig profile. Run the following command:

      $ oc logs linuxptp-daemon-4xkbb -n openshift-ptp -c linuxptp-daemon-container

      Example output

      I1115 09:41:17.117596 4143292 daemon.go:107] in applyNodePTPProfile
      I1115 09:41:17.117604 4143292 daemon.go:109] updating NodePTPProfile to:
      I1115 09:41:17.117607 4143292 daemon.go:110] ------------------------------------
      I1115 09:41:17.117612 4143292 daemon.go:102] Profile Name: profile1
      I1115 09:41:17.117616 4143292 daemon.go:102] Interface: ens787f1
      I1115 09:41:17.117620 4143292 daemon.go:102] Ptp4lOpts: -s -2
      I1115 09:41:17.117623 4143292 daemon.go:102] Phc2sysOpts: -a -r
      I1115 09:41:17.117626 4143292 daemon.go:116] ------------------------------------

Procedure

1. Create the following PtpConfig CR, and then save the YAML in the boundary-clock-ptp-config.yaml file.

   apiVersion: ptp.openshift.io/v1
   kind: PtpConfig
   metadata:
     name: boundary-clock-ptp-config 1
     namespace: openshift-ptp
   spec:
     profile: 2
     - name: "profile1" 3
       interface: "" 4
       ptp4lOpts: "-2" 5
       ptp4lConf: | 6
         [ens1f0] 7
         masterOnly 0
         [ens1f3] 8
         masterOnly 1
         [global]
         #
         # Default Data Set
         #
         twoStepFlag                       1
         #slaveOnly                        1
         priority1                         128
         priority2                         128
         domainNumber                      24
         #utc_offset                       37
         clockClass                        248
         clockAccuracy                     0xFE
         offsetScaledLogVariance         0xFFFF
         free_running                      0
         freq_est_interval               1
         dscp_event                        0
         dscp_general                      0
         dataset_comparison              G.8275.x
         G.8275.defaultDS.localPriority  128
         #
         # Port Data Set
         #
         logAnnounceInterval          -3
         logSyncInterval                -4
         logMinDelayReqInterval       -4
         logMinPdelayReqInterval      -4
         announceReceiptTimeout       3
         syncReceiptTimeout           0
         delayAsymmetry                 0
         fault_reset_interval         4
         neighborPropDelayThresh      20000000
         masterOnly                     0
         G.8275.portDS.localPriority  128
         #
         # Run time options
         #
         assume_two_step              0
         logging_level                6
         path_trace_enabled         0
         follow_up_info               0
         hybrid_e2e                   0
         inhibit_multicast_service  0
         net_sync_monitor           0
         tc_spanning_tree           0
         tx_timestamp_timeout       10
         #was 1 (default !)
         unicast_listen          0
         unicast_master_table  0
         unicast_req_duration  3600
         use_syslog              1
         verbose                   0
         summary_interval      -4
         kernel_leap             1
         check_fup_sync          0
         #
         # Servo Options
         #
         pi_proportional_const     0.0
         pi_integral_const         0.0
         pi_proportional_scale     0.0
         pi_proportional_exponent  -0.3
         pi_proportional_norm_max  0.7
         pi_integral_scale         0.0
         pi_integral_exponent      0.4
         pi_integral_norm_max      0.3
         step_threshold            2.0
         first_step_threshold      0.00002
         max_frequency               900000000
         clock_servo                 pi
         sanity_freq_limit         200000000
         ntpshm_segment              0
         #
         # Transport options
         #
         transportSpecific   0x0
         ptp_dst_mac          01:1B:19:00:00:00
         p2p_dst_mac          01:80:C2:00:00:0E
         udp_ttl                1
         udp6_scope           0x0E
         uds_address          /var/run/ptp4l
         #
         # Default interface options
         #
         clock_type             BC
         network_transport    UDPv4
         delay_mechanism        E2E
         time_stamping          hardware
         tsproc_mode            filter
         delay_filter           moving_median
         delay_filter_length  10
         egressLatency          0
         ingressLatency         0
         boundary_clock_jbod  0 9
         #
         # Clock description
         #
         productDescription    ;;
         revisionData            ;;
         manufacturerIdentity  00:00:00
         userDescription         ;
         timeSource              0xA0
       phc2sysOpts: "-a -r" 10
       ptpSchedulingPolicy: SCHED_OTHER 11
       ptpSchedulingPriority: 10 12
     recommend: 13
     - profile: "profile1" 14
       priority: 10 15
       match: 16
       - nodeLabel: "node-role.kubernetes.io/worker" 17
         nodeName: "compute-0.example.com" 18

   1

   The name of the PtpConfig CR.

   2

   Specify an array of one or more profile objects.

   3

   Specify the name of a profile object which uniquely identifies a profile object.

   4

   This field should remain empty for boundary clock.

   5

   Specify system config options for the ptp4l service, for example -2. The options should not include the network interface name -i <interface> and service config file -f /etc/ptp4l.conf because the network interface name and the service config file are automatically appended.

   6

   Specify the needed configuration to start ptp4l as boundary clock. For example, ens1f0 synchronizes from a grandmaster clock and ens1f3 synchronizes connected devices.

   7

   The interface name to synchronize from.

   8

   The interface to synchronize devices connected to the interface.

   9

   For Intel Columbiaville 800 Series NICs, ensure boundary_clock_jbod is set to 0. For Intel Fortville X710 Series NICs, ensure boundary_clock_jbod is set to 1.

   10

   Specify system config options for the phc2sys service, for example -a -r. If this field is empty the PTP Operator does not start the phc2sys service.

   11

   Scheduling policy for ptp4l and phc2sys processes. Default value is SCHED_OTHER. Use SCHED_FIFO on systems that support FIFO scheduling.

   12

   Integer value from 1-65 used to set FIFO priority for ptp4l and phc2sys processes when ptpSchedulingPolicy is set to SCHED_FIFO. The ptpSchedulingPriority field is not used when ptpSchedulingPolicy is set to SCHED_OTHER.

   13

   Specify an array of one or more recommend objects that define rules on how the profile should be applied to nodes.

   14

   Specify the profile object name defined in the profile section.

   15

   Specify the priority with an integer value between 0 and 99. A larger number gets lower priority, so a priority of 99 is lower than a priority of 10. If a node can be matched with multiple profiles according to rules defined in the match field, the profile with the higher priority is applied to that node.

   16

   Specify match rules with nodeLabel or nodeName.

   17

   Specify nodeLabel with the key of node.Labels from the node object by using the oc get nodes --show-labels command.

   18

   Specify nodeName with node.Name from the node object by using the oc get nodes command.

2. Create the CR by running the following command:

   $ oc create -f boundary-clock-ptp-config.yaml

Verification steps

1. Check that the PtpConfig profile is applied to the node.

   1. Get the list of pods in the openshift-ptp namespace by running the following command:

      $ oc get pods -n openshift-ptp -o wide

      Example output

      NAME                            READY   STATUS    RESTARTS   AGE   IP               NODE
      linuxptp-daemon-4xkbb           1/1     Running   0          43m   10.1.196.24      compute-0.example.com
      linuxptp-daemon-tdspf           1/1     Running   0          43m   10.1.196.25      compute-1.example.com
      ptp-operator-657bbb64c8-2f8sj   1/1     Running   0          43m   10.129.0.61      control-plane-1.example.com

   2. Check that the profile is correct. Examine the logs of the linuxptp daemon that corresponds to the node you specified in the PtpConfig profile. Run the following command:

      $ oc logs linuxptp-daemon-4xkbb -n openshift-ptp -c linuxptp-daemon-container

      Example output

      I1115 09:41:17.117596 4143292 daemon.go:107] in applyNodePTPProfile
      I1115 09:41:17.117604 4143292 daemon.go:109] updating NodePTPProfile to:
      I1115 09:41:17.117607 4143292 daemon.go:110] ------------------------------------
      I1115 09:41:17.117612 4143292 daemon.go:102] Profile Name: profile1
      I1115 09:41:17.117616 4143292 daemon.go:102] Interface:
      I1115 09:41:17.117620 4143292 daemon.go:102] Ptp4lOpts: -2
      I1115 09:41:17.117623 4143292 daemon.go:102] Phc2sysOpts: -a -r
      I1115 09:41:17.117626 4143292 daemon.go:116] ------------------------------------

Procedure

1. Edit the PtpConfig CR profile:

   $ oc edit PtpConfig -n openshift-ptp

2. Change the ptpSchedulingPolicy and ptpSchedulingPriority fields:

   apiVersion: ptp.openshift.io/v1
   kind: PtpConfig
   metadata:
     name: <ptp_config_name>
     namespace: openshift-ptp
   ...
   spec:
     profile:
     - name: "profile1"
   ...
       ptpSchedulingPolicy: SCHED_FIFO 1
       ptpSchedulingPriority: 10 2

   1

   Scheduling policy for ptp4l and phc2sys processes. Use SCHED_FIFO on systems that support FIFO scheduling.

   2

   Required. Sets the integer value 1-65 used to configure FIFO priority for ptp4l and phc2sys processes.

3. Save and exit to apply the changes to the PtpConfig CR.

Verification

1. Get the name of the linuxptp-daemon pod and corresponding node where the PtpConfig CR has been applied:

   $ oc get pods -n openshift-ptp -o wide

   Example output

   NAME                            READY   STATUS    RESTARTS   AGE     IP            NODE
   linuxptp-daemon-gmv2n           3/3     Running   0          1d17h   10.1.196.24   compute-0.example.com
   linuxptp-daemon-lgm55           3/3     Running   0          1d17h   10.1.196.25   compute-1.example.com
   ptp-operator-3r4dcvf7f4-zndk7   1/1     Running   0          1d7h    10.129.0.61   control-plane-1.example.com

2. Check that the ptp4l process is running with the updated chrt FIFO priority:

   $ oc -n openshift-ptp logs linuxptp-daemon-lgm55 -c linuxptp-daemon-container|grep chrt

   Example output

   I1216 19:24:57.091872 1600715 daemon.go:285] /bin/chrt -f 65 /usr/sbin/ptp4l -f /var/run/ptp4l.0.config -2  --summary_interval -4 -m

Procedure

1. Check the Operator and operands are successfully deployed in the cluster for the configured nodes.

   $ oc get pods -n openshift-ptp -o wide

   Example output

   NAME                            READY   STATUS    RESTARTS   AGE     IP            NODE
   linuxptp-daemon-lmvgn           3/3     Running   0          4d17h   10.1.196.24   compute-0.example.com
   linuxptp-daemon-qhfg7           3/3     Running   0          4d17h   10.1.196.25   compute-1.example.com
   ptp-operator-6b8dcbf7f4-zndk7   1/1     Running   0          5d7h    10.129.0.61   control-plane-1.example.com

   Note

   When the PTP fast event bus is enabled, the number of ready linuxptp-daemon pods is 3/3. If the PTP fast event bus is not enabled, 2/2 is displayed.

2. Check that supported hardware is found in the cluster.

   $ oc -n openshift-ptp get nodeptpdevices.ptp.openshift.io

   Example output

   NAME                                  AGE
   control-plane-0.example.com           10d
   control-plane-1.example.com           10d
   compute-0.example.com                 10d
   compute-1.example.com                 10d
   compute-2.example.com                 10d

3. Check the available PTP network interfaces for a node:

   $ oc -n openshift-ptp get nodeptpdevices.ptp.openshift.io <node_name> -o yaml

   where:

   <node_name>

   Specifies the node you want to query, for example, compute-0.example.com.

   Example output

   apiVersion: ptp.openshift.io/v1
   kind: NodePtpDevice
   metadata:
     creationTimestamp: "2021-09-14T16:52:33Z"
     generation: 1
     name: compute-0.example.com
     namespace: openshift-ptp
     resourceVersion: "177400"
     uid: 30413db0-4d8d-46da-9bef-737bacd548fd
   spec: {}
   status:
     devices:
     - name: eno1
     - name: eno2
     - name: eno3
     - name: eno4
     - name: enp5s0f0
     - name: enp5s0f1

4. Check that the PTP interface is successfully synchronized to the primary clock by accessing the linuxptp-daemon pod for the corresponding node.

   1. Get the name of the linuxptp-daemon pod and corresponding node you want to troubleshoot by running the following command:

      $ oc get pods -n openshift-ptp -o wide

      Example output

      NAME                            READY   STATUS    RESTARTS   AGE     IP            NODE
      linuxptp-daemon-lmvgn           3/3     Running   0          4d17h   10.1.196.24   compute-0.example.com
      linuxptp-daemon-qhfg7           3/3     Running   0          4d17h   10.1.196.25   compute-1.example.com
      ptp-operator-6b8dcbf7f4-zndk7   1/1     Running   0          5d7h    10.129.0.61   control-plane-1.example.com

   2. Remote shell into the required linuxptp-daemon container:

      $ oc rsh -n openshift-ptp -c linuxptp-daemon-container <linux_daemon_container>

      where:

      <linux_daemon_container>

      is the container you want to diagnose, for example linuxptp-daemon-lmvgn.

   3. In the remote shell connection to the linuxptp-daemon container, use the PTP Management Client (pmc) tool to diagnose the network interface. Run the following pmc command to check the sync status of the PTP device, for example ptp4l.

      # pmc -u -f /var/run/ptp4l.0.config -b 0 'GET PORT_DATA_SET'

      Example output when the node is successfully synced to the primary clock

      sending: GET PORT_DATA_SET
          40a6b7.fffe.166ef0-1 seq 0 RESPONSE MANAGEMENT PORT_DATA_SET
              portIdentity            40a6b7.fffe.166ef0-1
              portState               SLAVE
              logMinDelayReqInterval  -4
              peerMeanPathDelay       0
              logAnnounceInterval     -3
              announceReceiptTimeout  3
              logSyncInterval         -4
              delayMechanism          1
              logMinPdelayReqInterval -4
              versionNumber           2