7.2. Administering a Fabric Ensemble
Recommendations for an ensemble in production
To assure high availability of the Fabric registry in a production environment, it is recommended that you observe the following guidelines for a Fabric ensemble:
- Deploy a minimum of five Fabric servers in production (if one server is taken down for maintenance, one other server can fail, and the Fabric registry will still be available).
- Fabric servers should be deployed on separate host machines.
- Each Fabric server should only have a Fabric registry agent deployed inside it. No other profiles should be deployed in it.
- The size of the ensemble should be fixed at the outset, and not changed later (if you subsequently add or remove containers from the ensemble, the ZooKeeper IP ports would be re-assigned).
Creating an ensemble
A Fabric ensemble is created in two stages, as follows:
- Create an initial ensemble, consisting of one Fabric server.
- Expand the ensemble, by adding an even number of containers.
Creating an initial ensemble
An initial ensemble is usually created by invoking the
fabric:create console command (which converts the current container into a Fabric server, which is a sole member of the newly created ensemble). Alternatively, when creating a new container with the fabric:container-create-ssh or fabric:container-create-cloud commands, you can pass the --ensemble-server option.
For details of how to create an initial ensemble using the
fabric:create command, see Chapter 2, Creating a New Fabric.
Expanding the ensemble
Once you have an initial ensemble, consisting of one Fabric server, you can expand the ensemble by invoking the
fabric:ensemble-add command. To expand the ensemble, perform the following steps:
- Create some new managed containers in the current fabric, which you can then add to the ensemble. Use the default profile for these new containers. For a production environment, it is recommended that you create at least four new managed containers (must be an even number), each running on their own host.
- While logged on to a container in the fabric, use the
fabric:ensemble-addcommand to add the managed containers to the ensemble. For example, given the four managed containers,container1,container2,container3, andcontainer4, you would enter the following command:fabric:ensemble-add container1 container2 container3 container4
NoteBecause the total number of containers in an ensemble must always be odd, you must specify an even number of containers to thefabric:ensemble-addcommand. - To check that the ensemble has been successfully created, invoke the
fabric:container-listcommand.
Important
Do not attempt to expand (or shrink) a Fabric ensemble in a production environment. When you add containers to (or remove containers from) an ensemble, the ZooKeeper IP ports are all re-assigned, which typically causes the containers in the fabric to lose connectivity with the ensemble.
Taking a Fabric server down for maintenance
If you need to perform any maintenance on the host where a Fabric server is running, you can do this while maintaining availability of the Fabric registry, so long as a quorum (more than half) of the Fabric servers are still running. To stop a Fabric server, simply invoke the
fabric:container-stop command, specifying the name of the Fabric server.
In general, it is recommended to have at least five Fabric servers in the ensemble. Three is not an adequate number. For example, with three servers in the ensemble consider what happens when you take a Fabric server down for maintenance. The two remaining Fabric servers form a quorum, but there is now no tolerance for failure. If one of the remaining Fabric servers fails, the whole fabric fails. In order to maintain high availability during maintenance, it is therefore essential to have at least five Fabric servers in the ensemble.
