9.5. Pacemaker Support for Docker Containers (Technology Preview)
Important
Pacemaker support for Docker containers is provided for technology preview only. For details on what "technology preview" means, see Technology Preview Features Support Scope.
There is one exception to this feature being Technology Preview: As of Red Hat Enterprise Linux 7.4, Red Hat fully supports the usage of Pacemaker bundles for Red Hat Openstack Platform (RHOSP) deployments.
Pacemaker supports a special syntax for launching a Docker container with any infrastructure it requires: the bundle. After you have created a Pacemaker bundle, you can create a Pacemaker resource that the bundle encapsulates.
- Section 9.5.1, “Configuring a Pacemaker Bundle Resource” describes the syntax for the command to create a Pacemaker bundle and provides tables summarizing the parameters you can define for each bundle parameter.
- Section 9.5.2, “Configuring a Pacemaker Resource in a Bundle” provides information on configuring a resource contained in a Pacemaker bundle.
- Section 9.5.3, “Limitations of Pacemaker Bundles” notes the limitations of Pacemaker bundles.
- Section 9.5.4, “Pacemaker Bundle Configuration Example” provides a Pacemaker bundle configuration example.
9.5.1. Configuring a Pacemaker Bundle Resource
The syntax for the command to create a Pacemaker bundle for a Docker container is as follows. This command creates a bundle that encapsulates no other resources. For information on creating a cluster resource in a bundle see Section 9.5.2, “Configuring a Pacemaker Resource in a Bundle”.
pcs resource bundle create bundle_id container docker [container_options] [network network_options] [port-map port_options]... [storage-map storage_options]... [meta meta_options] [--disabled] [--wait[=n]]
The required bundle_id parameter must be a unique name for the bundle. If the
--disabled option is specified, the bundle is not started automatically. If the --wait option is specified, Pacemaker will wait up to n seconds for the bundle to start and then return 0 on success or 1 on error. If n is not specified it defaults to 60 minutes.
The following sections describe the parameters you can configure for each element of a Pacemaker bundle.
9.5.1.1. Docker Parameters
Table 9.6, “Docker Container Parameters” describes the
docker container options you can set for a bundle.
Note
Before configuring a
docker bundle in Pacemaker, you must install Docker and supply a fully configured Docker image on every node allowed to run the bundle.
| Field | Default | Description |
|---|---|---|
image
|
Docker image tag (required)
| |
replicas
|
Value of
promoted-max if that is positive, otherwise 1.
|
A positive integer specifying the number of container instances to launch
|
replicas-per-host
|
1
|
A positive integer specifying the number of container instances allowed to run on a single node
|
promoted-max
|
0
|
A non-negative integer that, if positive, indicates that the containerized service should be treated as a multistate service, with this many replicas allowed to run the service in the master role
|
network
| |
If specified, this will be passed to the
docker run command as the network setting for the Docker container.
|
run-command
| /usr/sbin/pacemaker_remoted if the bundle contains a resource, otherwise none
|
This command will be run inside the container when launching it ("PID 1"). If the bundle contains a resource, this command must start the
pacemaker_remoted daemon (but it could, for example, be a script that performs others tasks as well).
|
options
| |
Extra command-line options to pass to the
docker run command
|
9.5.1.2. Bundle Network Parameters
Table 9.7, “Bundle Resource Network Parameters” describes the
network options you can set for a bundle.
| Field | Default | Description |
|---|---|---|
add-host
|
TRUE
|
If TRUE, and
ip-range-start is used, Pacemaker will automatically ensure that the /etc/hosts file inside the containers has entries for each replica name and its assigned IP.
|
ip-range-start
| |
If specified, Pacemaker will create an implicit
ocf:heartbeat:IPaddr2 resource for each container instance, starting with this IP address, using as many sequential addresses as were specified as the replicas parameter for the Docker element. These addresses can be used from the host’s network to reach the service inside the container, although it is not visible within the container itself. Only IPv4 addresses are currently supported.
|
host-netmask
|
32
|
If
ip-range-start is specified, the IP addresses are created with this CIDR netmask (as a number of bits).
|
host-interface
| |
If
ip-range-start is specified, the IP addresses are created on this host interface (by default, it will be determined from the IP address).
|
control-port
|
3121
|
If the bundle contains a Pacemaker resource, the cluster will use this integer TCP port for communication with Pacemaker Remote inside the container. Changing this is useful when the container is unable to listen on the default port, which could happen when the container uses the host’s network rather than
ip-range-start (in which case replicas-per-host must be 1), or when the bundle may run on a Pacemaker Remote node that is already listening on the default port. Any PCMK_remote_port environment variable set on the host or in the container is ignored for bundle connections.
When a Pacemaker bundle configuration uses the
control-port parameter, then if the bundle has its own IP address the port needs to be open on that IP address on and from all full cluster nodes running corosync. If, instead, the bundle has set the network="host" container parameter, the port needs to be open on each cluster node's IP address from all cluster nodes.
|
Note
Replicas are named by the bundle ID plus a dash and an integer counter starting with zero. For example, if a bundle named
httpd-bundle has configured replicas=2, its containers will be named httpd-bundle-0 and httpd-bundle-1.
In addition to the network parameters, you can optionally specify
port-map parameters for a bundle. Table 9.8, “Bundle Resource port-map Parameters” describes these port-map parameters.
| Field | Default | Description |
|---|---|---|
id
| |
A unique name for the port mapping (required)
|
port
| |
If this is specified, connections to this TCP port number on the host network (on the container’s assigned IP address, if
ip-range-start is specified) will be forwarded to the container network. Exactly one of port or range must be specified in a port-mapping.
|
internal-port
|
Value of
port
|
If
port and internal-port are specified, connections to port on the host’s network will be forwarded to this port on the container network.
|
range
| |
If
range is specified, connections to these TCP port numbers (expressed as first_port-last_port) on the host network (on the container’s assigned IP address, if ip-range-start is specified) will be forwarded to the same ports in the container network. Exactly one of port or range must be specified in a port mapping.
|
Note
If the bundle contains a resource, Pacemaker will automatically map the
control-port, so it is not necessary to specify that port in a port mapping.
9.5.1.3. Bundle Storage Parameters
You can optionally configure
storage-map parameters for a bundle. Table 9.9, “Bundle Resource Storage Mapping Parameters” describes these parameters.
| Field | Default | Description |
|---|---|---|
id
| |
A unique name for the storage mapping (required)
|
source-dir
| |
The absolute path on the host’s filesystem that will be mapped into the container. Exactly one of
source-dir and source-dir-root parameter must be specified when configuring a storage-map parameter.
|
source-dir-root
| |
The start of a path on the host’s filesystem that will be mapped into the container, using a different subdirectory on the host for each container instance. The subdirectory will be named with the same name as the bundle name, plus a dash and an integer counter starting with 0. Exactly one
source-dir and source-dir-root parameter must be specified when configuring a storage-map parameter.
|
target-dir
| |
The path name within the container where the host storage will be mapped (required)
|
options
| |
File system mount options to use when mapping the storage
|
As an example of how subdirectories on a host are named using the
source-dir-root parameter, if source-dir-root=/path/to/my/directory, target-dir=/srv/appdata, and the bundle is named mybundle with replicas=2, then the cluster will create two container instances with host names mybundle-0 and mybundle-1 and create two directories on the host running the containers: /path/to/my/directory/mybundle-0 and /path/to/my/directory/mybundle-1. Each container will be given one of those directories, and any application running inside the container will see the directory as /srv/appdata.
Note
Pacemaker does not define the behavior if the source directory does not already exist on the host. However, it is expected that the container technology or its resource agent will create the source directory in that case.
Note
If the bundle contains a Pacemaker resource, Pacemaker will automatically map the equivalent of
source-dir=/etc/pacemaker/authkeytarget-dir=/etc/pacemaker/authkey and source-dir-root=/var/log/pacemaker/bundlestarget-dir=/var/log into the container, so it is not necessary to specify those paths in when configuring storage-map parameters.
Important
The
PCMK_authkey_location environment variable must not be set to anything other than the default of /etc/pacemaker/authkey on any node in the cluster.
9.5.2. Configuring a Pacemaker Resource in a Bundle
A bundle may optionally contain one Pacemaker cluster resource. As with a resource that is not contained in a bundle, the cluster resource may have operations, instance attributes, and metadata attributes defined. If a bundle contains a resource, the container image must include the Pacemaker Remote daemon, and
ip-range-start or control-port must be configured in the bundle. Pacemaker will create an implicit ocf:pacemaker:remote resource for the connection, launch Pacemaker Remote within the container, and monitor and manage the resource by means of Pacemaker Remote. If the bundle has more than one container instance (replica), the Pacemaker resource will function as an implicit clone, which will be a multistate clone if the bundle has configured the promoted-max option as greater than zero.
You create a resource in a Pacemaker bundle with the
pcs resource create command by specifying the bundle parameter for the command and the bundle ID in which to include the resource. For an example of creating a Pacemaker bundle that contains a resource, see Section 9.5.4, “Pacemaker Bundle Configuration Example”.
Important
Containers in bundles that contain a resource must have an accessible networking environment, so that Pacemaker on the cluster nodes can contact Pacemaker Remote inside the container. For example, the
docker option --net=none should not be used with a resource. The default (using a distinct network space inside the container) works in combination with the ip-range-start parameter. If the docker option --net=host is used (making the container share the host’s network space), a unique control-port parameter should be specified for each bundle. Any firewall must allow access to the control-port.
9.5.2.1. Node Attributes and Bundle Resources
If the bundle contains a cluster resource, the resource agent may want to set node attributes such as master scores. However, with containers, it is not apparent which node should get the attribute.
If the container uses shared storage that is the same no matter which node the container is hosted on, then it is appropriate to use the master score on the bundle node itself. On the other hand, if the container uses storage exported from the underlying host, then it may be more appropriate to use the master score on the underlying host. Since this depends on the particular situation, the
container-attribute-target resource metadata attribute allows the user to specify which approach to use. If it is set to host, then user-defined node attributes will be checked on the underlying host. If it is anything else, the local node (in this case the bundle node) is used. This behavior applies only to user-defined attributes; the cluster will always check the local node for cluster-defined attributes such as #uname.
If
container-attribute-target is set to host, the cluster will pass additional environment variables to the resource agent that allow it to set node attributes appropriately.
9.5.2.2. Metadata Attributes and Bundle Resources
Any metadata attribute set on a bundle will be inherited by the resource contained in a bundle and any resources implicitly created by Pacemaker for the bundle. This includes options such as
priority, target-role, and is-managed.
9.5.3. Limitations of Pacemaker Bundles
Pacemaker bundles operate with the following limitations:
- Bundles may not be included in groups or explicitly cloned with a
pcscommand. This includes a resource that the bundle contains, and any resources implicitly created by Pacemaker for the bundle. Note, however, that if a bundle is configured with a value ofreplicasgreater than one, the bundle behaves as if it were a clone. - Restarting Pacemaker while a bundle is unmanaged or the cluster is in maintenance mode may cause the bundle to fail.
- Bundles do not have instance attributes, utilization attributes, or operations, although a resource contained in a bundle may have them.
- A bundle that contains a resource can run on a Pacemaker Remote node only if the bundle uses a distinct
control-port.
9.5.4. Pacemaker Bundle Configuration Example
The following example creates a Pacemaker
bundle resource with a bundle ID of httpd-bundle that contains an ocf:heartbeat:apache resource with a resource ID of httpd.
This procedure requires the following prerequisite configuration:
- Docker has been installed and enabled on every node in the cluster.
- There is an existing Docker image, named
pcmktest:http - The container image includes the Pacemaker Remote daemon.
- The container image includes a configured Apache web server.
- Every node in the cluster has directories
/var/local/containers/httpd-bundle-0,/var/local/containers/httpd-bundle-1, and/var/local/containers/httpd-bundle-2, containing anindex.htmlfile for the web server root. In production, a single, shared document root would be more likely, but for the example this configuration allows you to make theindex.htmlfile on each host different so that you can connect to the web server and verify whichindex.htmlfile is being served.
This procedure configures the following parameters for the Pacemaker bundle:
- The bundle ID is
httpd-bundle. - The previously-configured Docker container image is
pcmktest:http. - This example will launch three container instances.
- This example will pass the command-line option
--log-driver=journaldto thedocker runcommand. This parameter is not required, but is included to show how to pass an extra option to thedockercommand. A value of--log-driver=journaldmeans that the system logs inside the container will be logged in the underlying hosts'ssystemdjournal. - Pacemaker will create three sequential implicit
ocf:heartbeat:IPaddr2resources, one for each container image, starting with the IP address 192.168.122.131. - The IP addresses are created on the host interface eth0.
- The IP addresses are created with a CIDR netmask of 24.
- This example creates a port map ID of
http-port; connections to port 80 on the container's assigned IP address will be forwarded to the container network. - This example creates a storage map ID of
httpd-root. For this storage mapping:- The value of
source-dir-rootis/var/local/containers, which specifies the start of the path on the host's file system that will be mapped into the container, using a different subdirectory on the host for each container instance. - The value of
target-diris/var/www/html, which specifies the path name within the container where the host storage will be mapped. - The file system
rwmount option will be used when mapping the storage. - Since this example container includes a resource, Pacemaker will automatically map the equivalent of
source-dir=/etc/pacemaker/authkeyin the container, so you do not need to specify that path in the storage mapping.
In this example, the existing cluster configuration is put into a temporary file named
temp-cib.xml, which is then copied to a file named temp-cib.xml.deltasrc. All modifications to the cluster configuration are made to the tmp-cib.xml file. When the udpates are complete, this procedure uses the diff-against option of the pcs cluster cib-push command so that only the updates to the configuration file are pushed to the active configuration file.
#pcs cluster cib tmp-cib.xml#cp tmp-cib.xml tmp-cib.xml.deltasrc#pcs -f tmp.cib.xml resource bundle create httpd-bundle\container docker image=pcmktest:http replicas=3\options=--log-driver=journald\network ip-range-start=192.168.122.131 host-interface=eth0\host-netmask=24 port-map id=httpd-port port=80\storage-map id=httpd-root source-dir-root=/var/local/containers\target-dir=/var/www/html options=rw\ #pcs -f tmp-cib.xml resource create httpd ocf:heartbeat:apache\statusurl=http://localhost/server-status bundle httpd-bundle#pcs cluster cib-push tmp-cib.xml diff-against=tmp-cib.xml.deltasrc
