Chapter 1. Overview
Designed for cloud infrastructures and web-scale object storage, Red Hat® Ceph Storage is a massively scalable, open, software-defined storage platform that combines the most stable version of Ceph with a Ceph management platform, deployment tools and support services. Providing the tools to flexibly and cost-effectively manage petabyte-scale data deployments in the enterprise, Red Hat Ceph Storage manages cloud data so enterprises can focus on managing their businesses.
This document provides procedures for installing Red Hat Ceph Storage v1.3 for x86_64 architecture on Ubuntu Trusty
14.04.
Red Hat® Ceph Storage clusters consist of the following types of nodes:
- Administration node: We expect that you will have a dedicated administration node that will host the Calamari monitoring and administration server, your cluster’s configuration files and keys, and optionally local repositories for installing Ceph on nodes that cannot access the internet for security reasons.
- Monitor nodes: Ceph can run with one monitor; however, for high availability, we expect that you will have at least three monitor nodes to ensure high availability in a production cluster.
- OSD nodes: Ceph can run with very few OSDs (3, by default), but production clusters realize better performance beginning at modest scales (e.g., 50 OSDs). Ideally, a Ceph cluster will have multiple OSD nodes, allowing you to create a CRUSH map to isolate failure domains.
For minimum recommended hardware, see the Hardware Guide.
1.1. Prerequisites
Before installing Red Hat Ceph Storage, review the following prerequisites first and prepare the cluster nodes.
1.1.1. Operating System
Red Hat Ceph Storage 1.3 and later requires Ubuntu 16.04 with a homogeneous version running on AMD64 and Intel 64 architectures for all Ceph nodes, including the Red Hat Ceph Storage node.
Red Hat does not support clusters with heterogeneous operating systems and versions.
1.1.2. Kernel
Red Hat Ceph Storage v1.3 is supported on Ubuntu 14.04 configurations including the original 14.04 LTS kernel (3.13). Configurations including the latest HWE kernel (3.19) are also supported. We recommend customers default their choice to the LTS kernel where the HWE is not required by their hardware in light of its longer support lifecycle.
1.1.3. DNS Name Resolution
Ceph nodes must be able to resolve short host names, not just fully qualified domain names. Set up a default search domain to resolve short host names. To retrieve a Ceph node’s short host name, execute:
hostname -s
Each Ceph node MUST be able to ping every other Ceph node in the cluster by its short host name.
1.1.4. Network Interface Cards
All Ceph clusters require a public network. You MUST have a network interface card configured to a public network where Ceph clients can reach Ceph Monitors and Ceph OSDs. You SHOULD have a network interface card for a cluster network so that Ceph can conduct heart-beating, peering, replication and recovery on a network separate from the public network.
We DO NOT RECOMMEND using a single NIC for both a public and private network.
1.1.5. Firewall
You MUST adjust your firewall settings on the Calamari node to allow inbound requests on port 80
so that clients in your network can access the Calamari web user interface.
Calamari also communicates with Ceph nodes via ports 2003
, 4505
and 4506
. You MUST open ports 80
, 2003
, and 4505-4506
on your Calamari node.
sudo iptables -I INPUT 1 -i <iface> -p tcp -s <ip-address>/<netmask> --dport 80 -j ACCEPT sudo iptables -I INPUT 1 -i <iface> -p tcp -s <ip-address>/<netmask> --dport 2003 -j ACCEPT sudo iptables -I INPUT 1 -i <iface> -m multiport -p tcp -s <ip-address>/<netmask> --dports 4505:4506 -j ACCEPT
You MUST open port 6789
on your public network on ALL Ceph monitor nodes.
sudo iptables -I INPUT 1 -i <iface> -p tcp -s <ip-address>/<netmask> --dport 6789 -j ACCEPT
Finally, you MUST also open ports for OSD traffic (e.g., 6800-7300
). Each OSD on each Ceph node needs three ports: one for talking to clients and monitors (public network); one for sending data to other OSDs (cluster network, if available; otherwise, public network); and, one for heartbeating (cluster network, if available; otherwise, public network). The OSDs will bind to the next available port if they are restarted in certain scenarios, so open the entire port range 6800 through 7300 for completeness.
sudo iptables -I INPUT 1 -i <iface> -m multiport -p tcp -s <ip-address>/<netmask> --dports 6800:7300 -j ACCEPT
Once you have finished configuring iptables
, ensure that you make the changes persistent on each node so that they will be in effect when your nodes reboot.
Execute:
sudo apt-get install iptables-persistent
A terminal UI will open up. Select yes
for the prompts to save current IPv4
iptables rules to /etc/iptables/rules.v4
and current IPv6
iptables rules to /etc/iptables/rules.v6
.
The IPv4
iptables rules that you set in the earlier steps will be loaded in /etc/iptables/rules.v4
and will be persistent across reboots.
If you add a new IPv4
iptables rule after installing iptables-persistent
you will have to add it to the rule file. In such case, execute the following as a root
user:
iptables-save > /etc/iptables/rules.v4
1.1.6. Network Time Protocol
You MUST install Network Time Protocol (NTP) on all Ceph monitor hosts and ensure that monitor hosts are NTP peers. You SHOULD consider installing NTP on Ceph OSD nodes, but it is not required. NTP helps preempt issues that arise from clock drift.
Install NTP
sudo apt-get install ntp
Start the NTP service and ensure it’s running.
sudo service ntp start sudo service ntp status
Ensure that NTP is synchronizing Ceph monitor node clocks properly.
ntpq -p
1.1.7. Install SSH Server
For ALL Ceph Nodes perform the following steps:
Install an SSH server (if necessary) on each Ceph Node:
sudo apt-get install openssh-server
- Ensure the SSH server is running on ALL Ceph Nodes.
1.1.8. Create a Ceph Deploy User
The ceph-deploy
utility must login to a Ceph node as a user that has passwordless sudo
privileges, because it needs to install software and configuration files without prompting for passwords.
ceph-deploy
supports a --username
option so you can specify any user that has password-less sudo
(including root
, although this is NOT recommended). To use ceph-deploy --username <username>
, the user you specify must have password-less SSH access to the Ceph node, because ceph-deploy
will not prompt you for a password.
We recommend creating a Ceph Deploy user on ALL Ceph nodes in the cluster. Please do NOT use "ceph" as the user name. A uniform user name across the cluster may improve ease of use (not required), but you should avoid obvious user names, because hackers typically use them with brute force hacks (for example, root
, admin
, or <productname>
). The following procedure, substituting <username>
for the user name you define, describes how to create a Ceph Deploy user with passwordless sudo
on a Ceph node.
In a future Ceph release, the "ceph" user name will be reserved for the Ceph daemons. If the "ceph" user already exists on the Ceph nodes, removing this user must be done before attempting an upgrade to future Ceph releases.
Create a Ceph Deploy user on each Ceph Node.
ssh root@<hostname> adduser <username>
Replace
<hostname>
with the hostname of your Ceph node.For the Ceph Deploy user you added to each Ceph node, ensure that the user has
sudo
privileges and disablerequiretty
.cat << EOF >/etc/sudoers.d/<username> <username> ALL = (root) NOPASSWD:ALL Defaults:<username> !requiretty EOF
Ensure the correct file permissions.
# chmod 0440 /etc/sudoers.d/<username>
1.1.9. Enable Password-less SSH
Since ceph-deploy
will not prompt for a password, you must generate SSH keys on the admin node and distribute the public key to each Ceph node. ceph-deploy
will attempt to generate the SSH keys for initial monitors.
Generate the SSH keys, but do not use
sudo
or theroot
user. Leave the passphrase empty:ssh-keygen Generating public/private key pair. Enter file in which to save the key (/ceph-admin/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /ceph-admin/.ssh/id_rsa. Your public key has been saved in /ceph-admin/.ssh/id_rsa.pub.
Copy the key to each Ceph Node, replacing
<username>
with the user name you created with Create a Ceph Deploy User.ssh-copy-id <username>@node1 ssh-copy-id <username>@node2 ssh-copy-id <username>@node3
(Recommended) Modify the
~/.ssh/config
file of yourceph-deploy
admin node so thatceph-deploy
can log in to Ceph nodes as the user you created without requiring you to specify--username <username>
each time you executeceph-deploy
. This has the added benefit of streamliningssh
andscp
usage. Replace<username>
with the user name you created:Host node1 Hostname node1 User <username> Host node2 Hostname node2 User <username> Host node3 Hostname node3 User <username>
After editing the
~/.ssh/config
file on theadmin node
, execute the following to ensure the permissions are correct:chmod 600 ~/.ssh/config
1.1.10. Configuring RAID Controllers
If a RAID controller with 1-2 GB of cache is installed on a host, then enabling write-back caches might result in increased small I/O write throughput. In order for this to be done safely, the cache must be non-volatile.
Modern RAID controllers usually have super capacitors that provide enough power to drain volatile memory to non-volatile NAND memory during a power loss event. It is important to understand how a particular controller and firmware behave after power is restored.
Some of them might require manual intervention. Hard drives typically advertise to the operating system whether their disk caches should be enabled or disabled by default. However, certain RAID controllers or some firmware might not provide such information, so verify that disk level caches are disabled to avoid file system corruption.
Create a single RAID 0 volume with write-back for each OSD data drive with write-back cache enabled.
If SAS or SATA connected SSDs are also present on the controller, it is worth investigating whether your controller and firmware support passthrough mode. This will avoid the caching logic, and generally result in much lower latencies for fast media.
1.1.11. Adjust PID Count
Hosts with high numbers of OSDs (e.g., > 20) may spawn a lot of threads, especially during recovery and re-balancing. Ubuntu 14.04 kernels (3.13 and 3.16) default to a relatively small maximum number of threads (e.g., 32768
). Check your default settings to see if they are suitable.
cat /proc/sys/kernel/pid_max
Consider setting kernel.pid_max
to a higher number of threads. The theoretical maximum is 4,194,303 threads. For example, you could add the following to the /etc/sysctl.conf
file to set it to the maximum:
kernel.pid_max = 4194303
To see the changes you made without a reboot, execute:
# sysctl -p
To verify the changes, execute:
# sysctl -a | grep kernel.pid_max
1.1.12. Enable Ceph Repositories
Starting from v1.3.1, Red Hat Ceph Storage supports two installation methods for Ubuntu Trusty:
Online Repositories: For Ceph Storage clusters with Ceph nodes that can connect directly to the internet, you can use online repositories for
Calamari
,Installer
,Monitors
,OSDs
andRGW
from https://rhcs.download.redhat.com/ubuntu. You will need yourCustomer Name
andCustomer Password
received from https://rhcs.download.redhat.com to be able to use the repos.ImportantPlease contact your account manager to obtain credentials for
https://rhcs.download.redhat.com
.-
Local Repository: For Ceph Storage clusters where security measures preclude nodes from accessing the internet, you may install Red Hat Ceph Storage v1.3 from a single software build delivered as an ISO with the
ice_setup
package, which installs theice_setup
program. When you execute theice_setup
program, it will install local repositories, the Calamari monitoring and administration server and the Ceph installation scripts, including a hidden.cephdeploy.conf
file pointingceph-deploy
to the local repositories.
For installation via online repositories, configure the Installer repository on your administration node in order to install ceph-deploy
, then use ceph-deploy
to configure all other RHCS repositories.
For your administration node, set the Installer (
ceph-deploy
) repository then useceph-deploy
to enable the Calamari and Tools repositories.sudo bash -c 'umask 0077; echo deb https://customername:customerpasswd@rhcs.download.redhat.com/ubuntu/1.3-updates/Installer $(lsb_release -sc) main | tee /etc/apt/sources.list.d/Installer.list' sudo bash -c 'wget -O - https://www.redhat.com/security/fd431d51.txt | apt-key add -' sudo apt-get update sudo apt-get install ceph-deploy ceph-deploy repo --repo-url 'https://customername:customerpasswd@rhcs.download.redhat.com/ubuntu/1.3-updates/Calamari' Calamari `hostname -f` ceph-deploy repo --repo-url 'https://customername:customerpasswd@rhcs.download.redhat.com/ubuntu/1.3-updates/Tools' Tools `hostname -f` sudo apt-get update
For Ceph monitor nodes, install the MON repository by running the following from the Admin node:
ceph-deploy repo --repo-url 'https://customername:customerpasswd@rhcs.download.redhat.com/ubuntu/1.3-updates/MON' --gpg-url https://www.redhat.com/security/fd431d51.txt ceph-mon <MONHOST1> [<MONHOST2> [<MONHOST3> ...]]
Update monitor nodes:
sudo apt-get update
For Ceph OSD nodes, install the OSD repository by running the following from the Admin node:
ceph-deploy repo --repo-url 'https://customername:customerpasswd@rhcs.download.redhat.com/ubuntu/1.3-updates/OSD' --gpg-url https://www.redhat.com/security/fd431d51.txt ceph-osd <OSDHOST1> [<OSDHOST2> [<OSDHOST3> ...]]
Update OSD nodes:
sudo apt-get update
For gateway nodes, install the Tools repository by running the following from the Admin node:
ceph-deploy repo --repo-url 'https://customername:customerpasswd@rhcs.download.redhat.com/ubuntu/1.3-updates/Tools' --gpg-url https://www.redhat.com/security/fd431d51.txt Tools <RGWNODE1> [RGWNODE2 [RGWNODE3 ...]]
Update gateway nodes:
sudo apt-get update
1.2. Setting Up the Administration Server
You are supposed to have a dedicated administration node that hosts the Calamari monitoring and administration server.
The administration server hardware requirements vary with the size of the cluster. A minimum recommended hardware configuration for a Calamari server includes:
- at least 4 GB of RAM,
- a dual core CPU on AMD64 or Intel 64 architecture,
- enough network throughput to handle communication with Ceph hosts.
The hardware requirements scale linearly with the number of Ceph servers, so if you intend to run a fairly large cluster, ensure that you have enough RAM, processing power, and network throughput for the administration node.
Red Hat Ceph Storage uses an administration server for the Calamari monitoring and administration server, and the cluster’s Ceph configuration file and authentication keys. If you install Red Hat Ceph Storage from an ISO image and require local repositories for the Ceph packages, the administration server will also contain the Red Hat Ceph Storage repositories.
To use the HTTPS protocol with Calamari, set up the Apache web server first. See the Setting Up an SSL Server chapter in the System Administrator’s Guide for Red Hat Enterprise Linux 7.
1.2.1. Create a Working Directory
Create a working directory for the Ceph cluster configuration files and keys. Then, navigate to that directory. For example:
mkdir ~/ceph-config cd ~/ceph-config
The ice-setup
and ceph-deploy
utilities must be executed within this working directory. See the Installation by ISO and Executing ceph-deploy sections for details.
1.2.2. Installation by CDN
If you have correctly set the online repositories for Calamari
and Installer
in section Set Online Ceph Repositories, execute the following to install Calamari:
sudo apt-get install calamari-server calamari-clients
1.2.3. Installation by ISO
To install the ceph-deploy
utility and the Calamari server by using the ISO image, execute following steps:
- Visit the Red Hat Ceph Storage for Ubuntu page on the Customer Portal to obtain the Red Hat Ceph Storage installation ISO image files.
Using
sudo
, mount the downloaded ISO image to the/mnt/
directory, for example:$ sudo mount <path_to_iso>/rhceph-1.3.2-ubuntu-x86_64-dvd.iso /mnt
Using
sudo
, install theice_setup
program:$ sudo dpkg -i /mnt/ice-setup_*.deb
NoteIf you receive an error that the
python-pkg-resources
package is missing, run thesudo apt-get -f install
command to install the missingpython-pkg-resources
package.Navigate to the working directory that you created in the Create a Working Directory section:
$ cd ~/ceph-config
Within this directory, run
ice_setup
usingsudo
:$ sudo ice_setup -d /mnt
Follow the instructions in the interactive shell.
The
ice_setup
program performs the following operations:-
creates a local repository for the
ceph-deploy
andcalamari
packages - installs the Calamari server packages on the administration node
-
installs the
ceph-deploy
package on the administration node -
creates the
/opt/ICE/
and/opt/calamari/
directories -
writes the
.cephdeploy.conf
file to the/root/
directory and to the current working directory, for example,~/ceph-config
-
creates a local repository for the
1.2.4. Initialize Calamari
Once you have installed the Calamari package by using either the Content Delivery Network or the ISO image, initialize the Calamari monitoring and administration server:
# calamari-ctl initialize
As root
, update existing cluster nodes that report to Calamari.
# salt '*' state.highstate
At this point, you should be able to access the Calamari web server using a web browser. Proceed to the Storage Cluster Quick Start.
The initialization program implies that you can only execute ceph-deploy
when pointing to a remote site. You may also direct ceph-deploy
to your Calamari administration node for example, ceph-deploy admin <admin-hostname>
. You can also use the Calamari administration node to run a Ceph daemon, although this is not recommended.