Chapter 17. High Availability

Label the nodes for the service. This step can be optional if you run the service on any of the nodes in your Kubernetes cluster and use VIPs that can float within those nodes. This process may already exist within a complex cluster, in that nodes may be filtered by any constraints or requirements specified (e.g., nodes with SSD drives, or higher CPU, memory, or disk requirements, etc.).

The following example defines a label as router instances that are servicing traffic in the US west geography ha-router=geo-us-west:

Create the ConfigMap:

$ oc create configmap mycustomcheck --from-file=mycheckscript.sh

There are two approaches to adding the script to the pod: use oc commands or edit the deployment configuration.

OpenShift Enterprise’s ipfailover internally uses keepalived, so ensure that multicast is enabled on the nodes labeled above and that the nodes can accept network traffic for 224.0.0.18 (the VRRP multicast IP address). Depending on your environment’s multicast configuration, you may need to add an iptables rule to each of the above labeled nodes. If you do need to add the iptables rules, please also ensure that the rules persist after a system restart:

$ for node in openshift-node-{5,6,7,8,9}; do   ssh $node <<EOF

export interface=${interface:-"eth0"}
echo "Check multicast enabled ... ";
ip addr show $interface | grep -i MULTICAST

echo "Check multicast groups ... "
ip maddr show $interface | grep 224.0.0 | grep $interface

echo "Optionally, add accept rule and persist it ... "
sudo /sbin/iptables -I INPUT -i $interface -d 224.0.0.18/32 -j ACCEPT

echo "Please ensure the above rule is added on system restarts."

EOF
done;

Depending on your environment policies, you can either reuse the router service account created previously or create a new ipfailover service account.

Ensure that either the router service account exists as described in Deploying a Router or create a new ipfailover service account. The example below creates a new service account with the name ipfailover in the default namespace:

$ oc create serviceaccount ipfailover -n default

Add the ipfailover service account in the default namespace to the privileged SCC:

$ oadm policy add-scc-to-user privileged system:serviceaccount:default:ipfailover

Start the router with at least two replicas on nodes matching the labels used in the first step. The following example runs three instances using the ipfailover service account:

$ oadm router ha-router-us-west --replicas=5 \
    --selector="ha-svc-nodes=geo-us-west" \
    --labels="ha-svc-nodes=geo-us-west" \
    --service-account=ipfailover

Finally, configure the VIPs and failover for the nodes labeled with ha-router=geo-us-west in the first step. Ensure the number of replicas match the number of nodes and that they satisfy the label setup in the first step. The name of the ipfailover configuration (ipf-ha-router-us-west in the example below) should be different from the name of the router configuration (ha-router-us-west) as both the router and ipfailover create deployment configuration with those names. Specify the VIPs addresses and the port number that ipfailover should monitor on the desired instances:

$ oadm ipfailover ipf-ha-router-us-west \
    --replicas=5 --watch-port=80 \
    --selector="ha-router=geo-us-west" \
    --virtual-ips="10.245.2.101-105" \
    --iptables-chain="INPUT" \
    --service-account=ipfailover --create

Label the nodes for the service. This step can be optional if you run the service on any of the nodes in your Kubernetes cluster and use VIPs that can float within those nodes. This process may already exist within a complex cluster, in that the nodes may be filtered by any constraints or requirements specified (e.g., nodes with SSD drives, or higher CPU, memory, or disk requirements, etc.).

The following example labels a highly-available cache service that is listening on port 9736 as ha-cache=geo:

$ oc label nodes openshift-node-{6,3,7,9} "ha-cache=geo"

OpenShift Enterprise’s ipfailover internally uses keepalived, so ensure that multicast is enabled on the nodes labeled above and that the nodes can accept network traffic for 224.0.0.18 (the VRRP multicast IP address). Depending on your environment’s multicast configuration, you may need to add an iptables rule to each of the above labeled nodes. If you do need to add the iptables rules, please also ensure that the rules persist after a system restart:

$ for node in openshift-node-{6,3,7,9}; do   ssh $node <<EOF
export interface=${interface:-"eth0"}
echo "Check multicast enabled ... ";
ip addr show $interface | grep -i MULTICAST

echo "Check multicast groups ... "
ip maddr show $interface | grep 224.0.0 | grep $interface

echo "Optionally, add accept rule and persist it ... "
sudo /sbin/iptables -I INPUT -i $interface -d 224.0.0.18/32 -j ACCEPT

echo "Please ensure the above rule is added on system restarts."

EOF
done;

Create a new ipfailover service account in the default namespace:

$ oc create serviceaccount ipfailover -n default

Add the ipfailover service account in the default namespace to the privileged SCC:

$ oadm policy add-scc-to-user privileged system:serviceaccount:default:ipfailover

Run a geo-cache service with two or more replicas. An example configuration for running a geo-cache service is provided here.

$ oc create -n <namespace> -f ./examples/geo-cache.json

Finally, configure the VIPs and failover for the nodes labeled with ha-cache=geo in the first step. Ensure the number of replicas match the number of nodes and that they satisfy the label setup in first step. Specify the VIP addresses and the port number that ipfailover should monitor for the desired instances:

$ oadm ipfailover ipf-ha-geo-cache \\
    --replicas=5 --watch-port=9736 \
    --selector="ha-svc-nodes=geo-us-west" \
    --virtual-ips=10.245.3.101-105 \
    --vrrp-id-offset=10 \
    --service-account=ipfailover --create

Consider an IP failover configuration that was created using the following:

$ oadm ipfailover ipf-ha-router-us-west \
    --replicas=5 --watch-port=80 \
    --selector="ha-router=geo-us-west" \
    --virtual-ips="10.245.2.101-105" \
    --service-account=ipfailover --create

Edit the deployment configuration:

$ oc edit dc/ipf-ha-router-us-west

Update the spec.strategy.type field from Recreate to Rolling:

spec:
  replicas: 5
  selector:
    ha-router: geo-us-west
  strategy:
    recreateParams:
      timeoutSeconds: 600
    resources: {}
    type: Rolling 1

Update the OPENSHIFT_HA_VIRTUAL_IPS environment variable to contain the additional virtual IP addresses:

- name: OPENSHIFT_HA_VIRTUAL_IPS
  value: 10.245.2.101-105,10.245.2.110,10.245.2.201-205 1