Chapter 1. Pre-Installation Requirements

Red Hat Ceph Storage v1.2.3 and beyond requires a homogeneous operating system distribution and version e.g. CentOS 6 on x86_64 architecture for all Ceph nodes, including the Calamari cluster. We do not support clusters with heterogeneous operating systems and versions.

Ceph nodes must be able to resolve short host names, not just fully qualified domain names. Set up a default search domain to resolve short host names. To retrieve a Ceph node’s short host name, execute:

hostname -s

Each Ceph node MUST be able to ping every other Ceph node in the cluster by its short host name.

All Ceph clusters require a public network. You MUST have a network interface card configured to a public network where Ceph clients can reach Ceph Monitors and Ceph OSDs. You SHOULD have a network interface card for a cluster network so that Ceph can conduct heart-beating, peering, replication and recovery on a network separate from the public network.

We DO NOT RECOMMEND using a single NIC for both a public and private network.

Ensure that you configure your network interfaces and make them persistent so that the settings are identical on reboot. For example:

Navigate to /etc/sysconfig/network-scripts and ensure that the ifcfg-<iface> settings for your public and cluster interfaces (assuming you will use a cluster network too [RECOMMENDED]) are properly configured.

For details on configuring network interface scripts for CentOS 6, see Ethernet Interfaces.

The default firewall configuration for CentOS is fairly strict. You MUST adjust your firewall settings on the Calamari node to allow inbound requests on port 80 so that clients in your network can access the Calamari web user interface.

Calamari also communicates with Ceph nodes via ports 2003, 4505 and 4506. You MUST open ports 80, 2003, and 4505-4506 on your Calamari node.

sudo iptables -I INPUT 1 -i <iface> -p tcp -s <ip-address>/<netmask> --dport 80 -j ACCEPT
sudo iptables -I INPUT 1 -i <iface> -p tcp -s <ip-address>/<netmask> --dport 2003 -j ACCEPT
sudo iptables -I INPUT 1 -i <iface> -m multiport -p tcp -s <ip-address>/<netmask> --dports 4505:4506 -j ACCEPT

You MUST open port 6789 on your public network on ALL Ceph monitor nodes.

sudo iptables -I INPUT 1 -i <iface> -p tcp -s <ip-address>/<netmask> --dport 6789 -j ACCEPT

Finally, you MUST also open ports for OSD traffic (e.g., 6800-7100). Each OSD on each Ceph node needs three ports: one for talking to clients and monitors (public network); one for sending data to other OSDs (cluster network, if available; otherwise, public network); and, one for heartbeating (cluster network, if available; otherwise, public network). For example, if you have 4 OSDs, open 4 x 3 ports (12).

sudo iptables -I INPUT 1 -i <iface> -m multiport -p tcp -s <ip-address>/<netmask> --dports 6800:6811 -j ACCEPT

Once you have finished configuring iptables, ensure that you make the changes persistent on each node so that they will be in effect when your nodes reboot. For example:

/sbin/service iptables save

You MUST install Network Time Protocol (NTP) on all Ceph monitor hosts and ensure that monitor hosts are NTP peers. You SHOULD consider installing NTP on Ceph OSD nodes, but it is not required. NTP helps preempt issues that arise from clock drift.

For additional details on NTP for CentOS 6, see Network Time Protocol Setup.

For ALL Ceph Nodes perform the following steps:

For additional details on OpenSSH for CentOS 6, see OpenSSH.

The ceph-deploy utility must login to a Ceph node as a user that has passwordless sudo privileges, because it needs to install software and configuration files without prompting for passwords.

ceph-deploy supports a --username option so you can specify any user that has password-less sudo (including root, although this is NOT recommended). To use ceph-deploy --username <username>, the user you specify must have password-less SSH access to the Ceph node, because ceph-deploy will not prompt you for a password.

We recommend creating a Ceph user on ALL Ceph nodes in the cluster. A uniform user name across the cluster may improve ease of use (not required), but you should avoid obvious user names, because hackers typically use them with brute force hacks (e.g., root, admin, <productname>). The following procedure, substituting <username> for the user name you define, describes how to create a user with passwordless sudo on a node called ceph-server.

Since ceph-deploy will not prompt for a password, you must generate SSH keys on the admin node and distribute the public key to each Ceph node. ceph-deploy will attempt to generate the SSH keys for initial monitors.

If you have RAID (not recommended), configure your RAID controllers to RAID 0 (JBOD).

Hosts with high numbers of OSDs (e.g., > 20) may spawn a lot of threads, especially during recovery and re-balancing. Many Linux kernels default to a relatively small maximum number of threads (e.g., 32768). Check your default settings to see if they are suitable.

cat /proc/sys/kernel/pid_max

Consider setting kernel.pid_max to a higher number of threads. The theoretical maximum is 4,194,303 threads. For example, you could add the following to the /etc/sysctl.conf file to set it to the maximum:

kernel.pid_max = 4194303

To see the changes you made without a reboot, execute:

sudo sysctl -p

To verify the changes, execute:

sudo sysctl -a | grep kernel.pid_max

Ceph aims for data safety, which means that when the Ceph Client receives notice that data was written to a storage drive, that data was actually written to the storage drive (i.e., it’s not in a journal or drive cache, but yet to be written). On CentOS 6, disable the write cache if the journal is on a raw drive.

Use hdparm to disable write caching on OSD storage drives:

sudo hdparm -W 0 /<path-to>/<disk> 0

You may receive an error while trying to execute ceph-deploy commands. If requiretty is set by default on your Ceph hosts, disable it by executing sudo visudo and locate the Defaults requiretty setting. Change it to Defaults:ceph !requiretty where ceph is the user name from the step of Create a Ceph User to ensure that ceph-deploy can connect using the ceph user and execute commands as root.

SELinux is set to Enforcing by default. For Ceph Storage v1.2.3, set SELinux to Permissive or disable it entirely and ensure that your installation and cluster is working properly. To set SELinux to Permissive, execute the following:

sudo setenforce 0

To configure SELinux persistently, modify the configuration file at /etc/selinux/config.

Some Ceph package dependencies require versions that differ from the package versions from EPEL. Disable EPEL to ensure that you install the packages required for use with Ceph.