Chapter 12. Configuring IP failover

Procedure

1. Create an IP failover service account:

   $ oc create sa ipfailover

   Copy to Clipboard Toggle word wrap

2. Update security context constraints (SCC) for hostNetwork:

   $ oc adm policy add-scc-to-user privileged -z ipfailover

   Copy to Clipboard Toggle word wrap

   $ oc adm policy add-scc-to-user hostnetwork -z ipfailover

   Copy to Clipboard Toggle word wrap

3. Red Hat OpenStack Platform (RHOSP) only: Complete the following steps to make a failover VIP address reachable on RHOSP ports.

   1. Use the RHOSP CLI to show the default RHOSP API and VIP addresses in the allowed_address_pairs parameter of your RHOSP cluster:

      $ openstack port show <cluster_name> -c allowed_address_pairs

      Copy to Clipboard Toggle word wrap

      Output example

      *Field*                  *Value*
      allowed_address_pairs    ip_address='192.168.0.5', mac_address='fa:16:3e:31:f9:cb'
                               ip_address='192.168.0.7', mac_address='fa:16:3e:31:f9:cb'

      Copy to Clipboard Toggle word wrap

   2. Set a different VIP address for the IP failover deployment and make the address reachable on RHOSP ports by entering the following command in the RHOSP CLI. Do not set any default RHOSP API and VIP addresses as the failover VIP address for the IP failover deployment.

      Example of adding the 1.1.1.1 failover IP address as an allowed address on RHOSP ports.

      $ openstack port set <cluster_name> --allowed-address ip-address=1.1.1.1,mac-address=fa:fa:16:3e:31:f9:cb

      Copy to Clipboard Toggle word wrap

   3. Create a deployment YAML file to configure IP failover for your deployment. See "Example deployment YAML for IP failover configuration" in a later step.

   4. Specify the following specification in the IP failover deployment so that you pass the failover VIP address to the OPENSHIFT_HA_VIRTUAL_IPS environment variable:

      Example of adding the 1.1.1.1 VIP address to OPENSHIFT_HA_VIRTUAL_IPS

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: ipfailover-keepalived
      # ...
            spec:
                env:
                - name: OPENSHIFT_HA_VIRTUAL_IPS
                value: "1.1.1.1"
      # ...

      Copy to Clipboard Toggle word wrap

4. Create a deployment YAML file to configure IP failover.

   Note

   For Red Hat OpenStack Platform (RHOSP), you do not need to re-create the deployment YAML file. You already created this file as part of the earlier instructions.

   Example deployment YAML for IP failover configuration

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: ipfailover-keepalived
     labels:
       ipfailover: hello-openshift
   spec:
     strategy:
       type: Recreate
     replicas: 2
     selector:
       matchLabels:
         ipfailover: hello-openshift
     template:
       metadata:
         labels:
           ipfailover: hello-openshift
       spec:
         serviceAccountName: ipfailover
         privileged: true
         hostNetwork: true
         nodeSelector:
           node-role.kubernetes.io/worker: ""
         containers:
         - name: openshift-ipfailover
           image: registry.redhat.io/openshift4/ose-keepalived-ipfailover-rhel9:v4.16
           ports:
           - containerPort: 63000
             hostPort: 63000
           imagePullPolicy: IfNotPresent
           securityContext:
             privileged: true
           volumeMounts:
           - name: lib-modules
             mountPath: /lib/modules
             readOnly: true
           - name: host-slash
             mountPath: /host
             readOnly: true
             mountPropagation: HostToContainer
           - name: etc-sysconfig
             mountPath: /etc/sysconfig
             readOnly: true
           - name: config-volume
             mountPath: /etc/keepalive
           env:
           - name: OPENSHIFT_HA_CONFIG_NAME
             value: "ipfailover"
           - name: OPENSHIFT_HA_VIRTUAL_IPS
             value: "1.1.1.1-2"
           - name: OPENSHIFT_HA_VIP_GROUPS
             value: "10"
           - name: OPENSHIFT_HA_NETWORK_INTERFACE
             value: "ens3" #The host interface to assign the VIPs
           - name: OPENSHIFT_HA_MONITOR_PORT
             value: "30060"
           - name: OPENSHIFT_HA_VRRP_ID_OFFSET
             value: "0"
           - name: OPENSHIFT_HA_REPLICA_COUNT
             value: "2" #Must match the number of replicas in the deployment
           - name: OPENSHIFT_HA_USE_UNICAST
             value: "false"
           #- name: OPENSHIFT_HA_UNICAST_PEERS
             #value: "10.0.148.40,10.0.160.234,10.0.199.110"
           - name: OPENSHIFT_HA_IPTABLES_CHAIN
             value: "INPUT"
           #- name: OPENSHIFT_HA_NOTIFY_SCRIPT
           #  value: /etc/keepalive/mynotifyscript.sh
           - name: OPENSHIFT_HA_CHECK_SCRIPT
             value: "/etc/keepalive/mycheckscript.sh"
           - name: OPENSHIFT_HA_PREEMPTION
             value: "preempt_delay 300"
           - name: OPENSHIFT_HA_CHECK_INTERVAL
             value: "2"
           livenessProbe:
             initialDelaySeconds: 10
             exec:
               command:
               - pgrep
               - keepalived
         volumes:
         - name: lib-modules
           hostPath:
             path: /lib/modules
         - name: host-slash
           hostPath:
             path: /
         - name: etc-sysconfig
           hostPath:
             path: /etc/sysconfig
         # config-volume contains the check script
         # created with `oc create configmap keepalived-checkscript --from-file=mycheckscript.sh`
         - configMap:
             defaultMode: 0755
             name: keepalived-checkscript
           name: config-volume
         imagePullSecrets:
           - name: openshift-pull-secret

   Copy to Clipboard Toggle word wrap

   where:

   ipfailover-keepalived

   Specifies the name of the IP failover deployment.

   OPENSHIFT_HA_VIRTUAL_IPS

   Specifies the lis t of IP address ranges to replicate. This must be provided. For example, 1.2.3.4-6,1.2.3.9.

   OPENSHIFT_HA_VIP_GROUPS

   Specifies the number of groups to create for VRRP. If not set, a group is created for each virtual IP range specified with the OPENSHIFT_HA_VIP_GROUPS variable.

   OPENSHIFT_HA_NETWORK_INTERFACE

   Specifies the interface name that IP failover uses to send VRRP traffic. By default, eth0 is used.

   OPENSHIFT_HA_MONITOR_PORT

   Specifies the IP failover pod tries to open a TCP connection to this port on each VIP. If connection is established, the service is considered to be running. If this port is set to 0, the test always passes. The default value is 80.

   OPENSHIFT_HA_VRRP_ID_OFFSET

   Specifies the offset value used to set the virtual router IDs. Using different offset values allows multiple IP failover configurations to exist within the same cluster. The default offset is 10, and the allowed range is 0 through 255.

   OPENSHIFT_HA_REPLICA_COUNT

   Specifies the number of replicas to create. This must match spec.replicas value in IP failover configuration. The default value is 2.

   OPENSHIFT_HA_USE_UNICAST

   Specifies whether to use unicast mode for VRRP. The default value is false.

   OPENSHIFT_HA_UNICAST_PEERS

   Specifies the list of IP addresses of the unicast peers. This must be provided if OPENSHIFT_HA_USE_UNICAST is set to true.

   OPENSHIFT_HA_IPTABLES_CHAIN

   Specifies the name of the iptables chain to automatically add an iptables rule to allow the VRRP traffic on. If the value is not set, an iptables rule is not added. If the chain does not exist, it is not created, and Keepalived operates in unicast mode. The default is INPUT.

   OPENSHIFT_HA_NOTIFY_SCRIPT

   Specifies the full path name in the pod file system of a script that is run whenever the state changes.

   OPENSHIFT_HA_CHECK_SCRIPT

   Specifies the full path name in the pod file system of a script that is periodically run to verify the application is operating.

   OPENSHIFT_HA_PREEMPTION

   Specifies the strategy for handling a new higher priority host. The default value is preempt_delay 300, which causes a Keepalived instance to take over a VIP after 5 minutes if a lower-priority master is holding the VIP.

   OPENSHIFT_HA_CHECK_INTERVAL

   Specifies the period, in seconds, that the check script is run. The default value is 2.

   openshift-pull-secret

   Specifies the name of the pull secret to use for the IP failover deployment. Create the pull secret before creating the deployment, otherwise you will get an error when creating the deployment.

Procedure

1. Create the desired script and create a ConfigMap object to hold it. The script has no input arguments and must return 0 for OK and 1 for fail.

   The check script, mycheckscript.sh:

   #!/bin/bash
       # Whatever tests are needed
       # E.g., send request and verify response
   exit 0

   Copy to Clipboard Toggle word wrap

2. Create the ConfigMap object by running the following command:

   $ oc create configmap mycustomcheck --from-file=mycheckscript.sh

   Copy to Clipboard Toggle word wrap

3. Add the script to the pod. The defaultMode for the mounted ConfigMap object files must able to run by using oc commands or by editing the IP failover configuration.

   1. Add the script to the pod by running the following command. A value of 0755, 493 decimal, is typical. For example:

      $ oc set env deploy/ipfailover-keepalived \
          OPENSHIFT_HA_CHECK_SCRIPT=/etc/keepalive/mycheckscript.sh

      Copy to Clipboard Toggle word wrap

      $ oc set volume deploy/ipfailover-keepalived --add --overwrite \
          --name=config-volume \
          --mount-path=/etc/keepalive \
          --source='{"configMap": { "name": "mycustomcheck", "defaultMode": 493}}'

      Copy to Clipboard Toggle word wrap
      Note

      The oc set env command is whitespace sensitive. There must be no whitespace on either side of the = sign.

   2. Alternatively, edit the ipfailover-keepalived configuration by running the following command:

      $ oc edit deploy ipfailover-keepalived

      Copy to Clipboard Toggle word wrap

      Example ipfailover-keepalived configuration

          spec:
            containers:
            - env:
              - name: OPENSHIFT_HA_CHECK_SCRIPT
                value: /etc/keepalive/mycheckscript.sh
      ...
              volumeMounts:
              - mountPath: /etc/keepalive
                name: config-volume
            dnsPolicy: ClusterFirst
      ...
            volumes:
            - configMap:
                defaultMode: 0755
                name: customrouter
              name: config-volume
      ...

      Copy to Clipboard Toggle word wrap

      where:

      spec.container.env.name

      Specifies the OPENSHIFT_HA_CHECK_SCRIPT environment variable to point to the mounted script file.

      spec.container.volumeMounts

      Specifies the spec.container.volumeMounts field to create the mount point.

      spec.volumes

      Specifies a new spec.volumes field to mention the config map.

      spec.volumes.configMap.defaultMode

      Specifies run permission on the files. When read back, it is displayed in decimal, 493.

   3. Save the changes and exit the editor. This restarts the ipfailover-keepalived configuration.

Procedure

1. Optional: Identify and delete any check and notify scripts that are stored as config maps:

   1. Identify whether any pods for IP failover use a config map as a volume:

      $ oc get pod -l ipfailover \
        -o jsonpath="\
      {range .items[?(@.spec.volumes[*].configMap)]}
      {'Namespace: '}{.metadata.namespace}
      {'Pod:       '}{.metadata.name}
      {'Volumes that use config maps:'}
      {range .spec.volumes[?(@.configMap)]}  {'volume:    '}{.name}
        {'configMap: '}{.configMap.name}{'\n'}{end}
      {end}"

      Copy to Clipboard Toggle word wrap

      Example output

      Namespace: default
      Pod:       keepalived-worker-59df45db9c-2x9mn
      Volumes that use config maps:
        volume:    config-volume
        configMap: mycustomcheck

      Copy to Clipboard Toggle word wrap

   2. If the preceding step provided the names of config maps that are used as volumes, delete the config maps:

      $ oc delete configmap <configmap_name>

      Copy to Clipboard Toggle word wrap

2. Identify an existing deployment for IP failover:

   $ oc get deployment -l ipfailover

   Copy to Clipboard Toggle word wrap

   Example output

   NAMESPACE   NAME         READY   UP-TO-DATE   AVAILABLE   AGE
   default     ipfailover   2/2     2            2           105d

   Copy to Clipboard Toggle word wrap

3. Delete the deployment:

   $ oc delete deployment <ipfailover_deployment_name>

   Copy to Clipboard Toggle word wrap

4. Remove the ipfailover service account:

   $ oc delete sa ipfailover

   Copy to Clipboard Toggle word wrap

5. Run a job that removes the IP tables rule that was added when IP failover was initially configured:

   1. Create a file such as remove-ipfailover-job.yaml with contents that are similar to the following example:

      apiVersion: batch/v1
      kind: Job
      metadata:
        generateName: remove-ipfailover-
        labels:
          app: remove-ipfailover
      spec:
        template:
          metadata:
            name: remove-ipfailover
          spec:
            containers:
            - name: remove-ipfailover
              image: registry.redhat.io/openshift4/ose-keepalived-ipfailover-rhel9:v4.16
              command: ["/var/lib/ipfailover/keepalived/remove-failover.sh"]
            nodeSelector:
              kubernetes.io/hostname: <host_name>
            restartPolicy: Never

      Copy to Clipboard Toggle word wrap
      • The nodeSelector is likely the same as the selector used in the old IP failover deployment.
      • Run the job for each node in your cluster that was configured for IP failover and replace the hostname each time.

   2. Run the job:

      $ oc create -f remove-ipfailover-job.yaml

      Copy to Clipboard Toggle word wrap

      Example output

      job.batch/remove-ipfailover-2h8dm created

      Copy to Clipboard Toggle word wrap