Container Guide

1. As the Ansible user, create the ceph-ansible-keys directory where Ansible stores temporary values generated by the ceph-ansible playbook.

   [user@admin ~]$ mkdir ~/ceph-ansible-keys

2. As root, create a symbolic link to the /usr/share/ceph-ansible/group_vars directory in the /etc/ansible/ directory:

   [root@admin ~]# ln -s /usr/share/ceph-ansible/group_vars /etc/ansible/group_vars

3. Navigate to the /usr/share/ceph-ansible/ directory:

   [root@admin ~]$ cd /usr/share/ceph-ansible

4. Create new copies of the yml.sample files:

   [root@admin ceph-ansible]# cp group_vars/all.yml.sample group_vars/all.yml
   [root@admin ceph-ansible]# cp group_vars/osds.yml.sample group_vars/osds.yml
   [root@admin ceph-ansible]# cp site-docker.yml.sample site-docker.yml

5. Edit the copied files.

   1. Edit the group_vars/all.yml file. See the table below for the most common required and optional parameters to uncomment. Note that the table does not include all parameters.

      Important

      Do not set the cluster: ceph parameter to any value other than ceph because using custom cluster names is not supported.

      Table 1.1. General Ansible Settings

      Option	Value	Required	Notes
      monitor_interface	The interface that the Monitor nodes listen to	monitor_interface, monitor_address, or monitor_address_block is required
      monitor_address	The address that the Monitor nodes listen to
      monitor_address_block	The subnet of the Ceph public network		Use when the IP addresses of the nodes are unknown, but the subnet is known
      ip_version	ipv6	Yes if using IPv6 addressing
      journal_size	The required size of the journal in MB	No
      public_network	The IP address and netmask of the Ceph public network	Yes	The Verifying the Network Configuration for Red Hat Ceph Storage section in the Installation Guide for Red Hat Enterprise Linux
      cluster_network	The IP address and netmask of the Ceph cluster network	No
      ceph_docker_image	rhceph/rhceph-3-rhel7, or cephimageinlocalreg if using a local Docker registry	Yes
      containerized_deployment	true	Yes
      ceph_docker_registry	registry.access.redhat.com, or <local-host-fqdn> if using a local Docker registry	Yes

      An example of the all.yml file can look like:

      monitor_interface: eth0
      journal_size: 5120
      public_network: 192.168.0.0/24
      ceph_docker_image: rhceph/rhceph-3-rhel7
      containerized_deployment: true
      ceph_docker_registry: registry.access.redhat.com

      For additional details, see the all.yml file.

   2. Edit the group_vars/osds.yml file. See the table below for the most common required and optional parameters to uncomment. Note that the table does not include all parameters.

      Important

      Use a different physical device to install an OSD than the device where the operating system is installed. Sharing the same device between the operating system and OSDs causes performance issues.

      Table 1.2. OSD Ansible Settings

      Option	Value	Required	Notes
      osd_scenario	collocated to use the same device for write-ahead logging and key/value data (BlueStore) or journal (FileStore) and OSD data non-collocated to use a dedicated device, such as SSD or NVMe media to store write-ahead log and key/value data (BlueStore) or journal data (FileStore) lvm to use the Logical Volume Manager to store OSD data	Yes	When using osd_scenario: non-collocated, ceph-ansible expects the numbers of variables in devices and dedicated_devices to match. For example, if you specify 10 disks in devices, you must specify 10 entries in dedicated_devices.
      osd_auto_discovery	true to automatically discover OSDs	Yes if using osd_scenario: collocated	Cannot be used when devices setting is used
      devices	List of devices where ceph data is stored	Yes to specify the list of devices	Cannot be used when osd_auto_discovery setting is used. When using lvm as the osd_scenario and setting the devices option, ceph-volume lvm batch mode creates the optimized OSD configuration.
      dedicated_devices	List of dedicated devices for non-collocated OSDs where ceph journal is stored	Yes if osd_scenario: non-collocated	Should be nonpartitioned devices
      dmcrypt	true to encrypt OSDs	No	Defaults to false
      lvm_volumes	A list of FileStore or BlueStore dictionaries	Yes if using osd_scenario: lvm and storage devices are not defined using devices	Each dictionary must contain a data, journal and data_vg keys. Any logical volume or volume group must be the name and not the full path. The data, and journal keys can be a logical volume (LV) or partition, but do not use one journal for multiple data LVs. The data_vg key must be the volume group containing the data LV. Optionally, the journal_vg key can be used to specify the volume group containing the journal LV, if applicable. See the examples below for various supported configurations.
      osds_per_device	The number of OSDs to create per device.	No	Defaults to 1
      osd_objectstore	The Ceph object store type for the OSDs.	No	Defaults to bluestore. The other option is filestore. Required for upgrades.

      The following are examples of the osds.yml file when using the three OSD scenarios: collocated, non-collocated, and lvm. The default OSD object store format is BlueStore, if not specified.

      Collocated

      osd_objectstore: filestore
      osd_scenario: collocated
      devices:
        - /dev/sda
        - /dev/sdb

      Non-collocated - BlueStore

      osd_objectstore: bluestore
      osd_scenario: non-collocated
      devices:
       - /dev/sda
       - /dev/sdb
       - /dev/sdc
       - /dev/sdd
      dedicated_devices:
       - /dev/nvme0n1
       - /dev/nvme0n1
       - /dev/nvme1n1
       - /dev/nvme1n1

      This non-collocated example will create four BlueStore OSDs, one per device. In this example, the traditional hard drives (sda, sdb, sdc, sdd) are used for object data, and the solid state drives (SSDs) (/dev/nvme0n1, /dev/nvme1n1) are used for the BlueStore databases and write-ahead logs. This configuration pairs the /dev/sda and /dev/sdb devices with the /dev/nvme0n1 device, and pairs the /dev/sdc and /dev/sdd devices with the /dev/nvme1n1 device.

      Non-collocated - FileStore

      osd_objectstore: filestore
      osd_scenario: non-collocated
      devices:
        - /dev/sda
        - /dev/sdb
        - /dev/sdc
        - /dev/sdd
      dedicated_devices:
         - /dev/nvme0n1
         - /dev/nvme0n1
         - /dev/nvme1n1
         - /dev/nvme1n1

      LVM simple

      osd_objectstore: bluestore
      osd_scenario: lvm
      devices:
        - /dev/sda
        - /dev/sdb

      or

      osd_objectstore: bluestore
      osd_scenario: lvm
      devices:
        - /dev/sda
        - /dev/sdb
        - /dev/nvme0n1

      With these simple configurations ceph-ansible uses batch mode (ceph-volume lvm batch) to create the OSDs.

      In the first scenario, if the devices are traditional hard drives or SSDs, then one OSD per device is created.

      In the second scenario, when there is a mix of traditional hard drives and SSDs, the data is placed on the traditional hard drives (sda, sdb) and the BlueStore database (block.db) is created as large as possible on the SSD (nvme0n1).

      LVM advance

      osd_objectstore: filestore
      osd_scenario: lvm
      lvm_volumes:
         - data: data-lv1
           data_vg: vg1
           journal: journal-lv1
           journal_vg: vg2
         - data: data-lv2
           journal: /dev/sda
           data_vg: vg1

      or

      osd_objectstore: bluestore
      osd_scenario: lvm
      lvm_volumes:
        - data: data-lv1
          data_vg: data-vg1
          db: db-lv1
          db_vg: db-vg1
          wal: wal-lv1
          wal_vg: wal-vg1
        - data: data-lv2
          data_vg: data-vg2
          db: db-lv2
          db_vg: db-vg2
          wal: wal-lv2
          wal_vg: wal-vg2

      With these advance scenario examples, the volume groups and logical volumes must be created beforehand. They will not be created by ceph-ansible.

      Note

      If using all NVMe SSDs set the osd_scenario: lvm and osds_per_device: 4 options. For more information, see the Configuring OSD Ansible settings for all NVMe Storage section in the Red Hat Ceph Storage Container Guide.

      For additional details, see the comments in the osds.yml file.

6. Edit the Ansible inventory file located by default at /etc/ansible/hosts. Remember to comment out example hosts.

   1. Add the Monitor nodes under the [mons] section:

      [mons]
      <monitor-host-name>
      <monitor-host-name>
      <monitor-host-name>

   2. Add OSD nodes under the [osds] section. If the nodes have sequential naming, consider using a range:

      [osds]
      <osd-host-name[1:10]>

      Note

      For OSDs in a new installation, the default object store format is BlueStore.

      Alternatively, you can colocate Monitors with the OSD daemons on one node by adding the same node under the [mons] and [osds] sections. See Chapter 2, Colocation of Containerized Ceph Daemons for details.

7. Optionally, for all deployments, bare-metal or in containers, you can create a custom CRUSH hierarchy using ansible-playbook:

   1. Setup your Ansible inventory file. Specify where you want the OSD hosts to be in the CRUSH map’s hierarchy by using the osd_crush_location parameter. You must specify at least two CRUSH bucket types to specify the location of the OSD, and one bucket type must be host. By default, these include root, datacenter, room, row, pod, pdu, rack, chassis and host.

      Syntax

      [osds]
      CEPH_OSD_NAME osd_crush_location="{ 'root': ROOT_BUCKET_', 'rack': 'RACK_BUCKET', 'pod': 'POD_BUCKET', 'host': 'CEPH_HOST_NAME' }"

      Example

      [osds]
      ceph-osd-01 osd_crush_location="{ 'root': 'default', 'rack': 'rack1', 'pod': 'monpod', 'host': 'ceph-osd-01' }"

   2. Set the crush_rule_config and create_crush_tree parameters to True, and create at least one CRUSH rule if you do not want to use the default CRUSH rules. For example, if you are using HDD devices, edit the paramters as follows:

      crush_rule_config: True
      crush_rule_hdd:
          name: replicated_hdd_rule
          root: root-hdd
          type: host
          class: hdd
          default: True
      crush_rules:
        - "{{ crush_rule_hdd }}"
      create_crush_tree: True

      If you are using SSD devices, then edit the parameters as follows:

      crush_rule_config: True
      crush_rule_ssd:
          name: replicated_ssd_rule
          root: root-ssd
          type: host
          class: ssd
          default: True
      crush_rules:
        - "{{ crush_rule_ssd }}"
      create_crush_tree: True

      Note

      The default CRUSH rules fail if both ssd and hdd OSDs are not deployed because the default rules now include the class parameter, which must be defined.

      Note

      Add the custom CRUSH hierarchy to the OSD files in the host_vars directory as described in a step below to make this configuration work.

   3. Create pools, with created crush_rules in group_vars/clients.yml file.

      Example

      copy_admin_key: True
      user_config: True
      pool1:
        name: "pool1"
        pg_num: 128
        pgp_num: 128
        rule_name: "HDD"
        type: "replicated"
        device_class: "hdd"
      pools:
        - "{{ pool1 }}"

   4. View the tree.

      [root@mon ~]# ceph osd tree

   5. Validate the pools.

      # for i in $(rados lspools);do echo "pool: $i"; ceph osd pool get $i crush_rule;done
      
      pool: pool1
      crush_rule: HDD

8. For all deployments, bare-metal or in containers, open for editing the Ansible inventory file, by default the /etc/ansible/hosts file. Comment out the example hosts.

   1. Add the Ceph Manager (ceph-mgr) nodes under the [mgrs] section. Colocate the Ceph Manager daemon with Monitor nodes.

      [mgrs]
      <monitor-host-name>
      <monitor-host-name>
      <monitor-host-name>

9. As the Ansible user, ensure that Ansible can reach the Ceph hosts:

   [user@admin ~]$ ansible all -m ping

10. As root, create the /var/log/ansible/ directory and assign the appropriate permissions for the ansible user:

    [root@admin ~]# mkdir /var/log/ansible
    [root@admin ~]# chown ansible:ansible  /var/log/ansible
    [root@admin ~]# chmod 755 /var/log/ansible

    1. Edit the /usr/share/ceph-ansible/ansible.cfg file, updating the log_path value as follows:

       log_path = /var/log/ansible/ansible.log

11. As the Ansible user, change to the /usr/share/ceph-ansible/ directory:

    [user@admin ~]$ cd /usr/share/ceph-ansible/

12. Run the ceph-ansible playbook:

    [user@admin ceph-ansible]$ ansible-playbook site-docker.yml

    Note

    If you deploy Red Hat Ceph Storage to Red Hat Enterprise Linux Atomic Host hosts, use the --skip-tags=with_pkg option:

    [user@admin ceph-ansible]$ ansible-playbook site-docker.yml --skip-tags=with_pkg

    Note

    To increase the deployment speed, use the --forks option to ansible-playbook. By default, ceph-ansible sets forks to 20. With this setting, up to twenty nodes will be installed at the same time. To install up to thirty nodes at a time, run ansible-playbook --forks 30 PLAYBOOK FILE. The resources on the admin node must be monitored to ensure they are not overused. If they are, lower the number passed to --forks.

13. Using the root account on a Monitor node, verify the status of the Ceph cluster:

    docker exec ceph-<mon|mgr>-<id> ceph health

    Replace:

    • <id> with the host name of the Monitor node:

    For example:

    [root@monitor ~]# docker exec ceph-mon-mon0 ceph health
    HEALTH_OK

Procedure

1. Set osd_scenario: lvm and osds_per_device: 4 in group_vars/osds.yml:

   osd_scenario: lvm
   osds_per_device: 4

2. List the NVMe devices under devices:

   devices:
     - /dev/nvme0n1
     - /dev/nvme1n1
     - /dev/nvme2n1
     - /dev/nvme3n1

3. The settings in group_vars/osds.yml will look similar to this example:

   osd_scenario: lvm
   osds_per_device: 4
   devices:
     - /dev/nvme0n1
     - /dev/nvme1n1
     - /dev/nvme2n1
     - /dev/nvme3n1

1. As the root user, navigate to the /usr/share/ceph-ansible/ directory.

   [root@admin ~]# cd /usr/share/ceph-ansible/

2. Uncomment the radosgw_interface parameter in the group_vars/all.yml file.

   radosgw_interface: interface

   Replace interface with the interface that the Ceph Object Gateway nodes listen to.

3. Optional. Change the default variables.

   1. Create a new copy of the rgws.yml.sample file located in the group_vars directory.

      [root@admin ceph-ansible]# cp group_vars/rgws.yml.sample group_vars/rgws.yml

   2. Edit the group_vars/rgws.yml file. For additional details, see the rgws.yml file.

4. Add the host name of the Ceph Object Gateway node to the [rgws] section of the Ansible inventory file located by default at /etc/ansible/hosts.

   [rgws]
   gateway01

   Alternatively, you can colocate the Ceph Object Gateway with the OSD daemon on one node by adding the same node under the [osds] and [rgws] sections. See Colocation of containerized Ceph daemons for details.

5. As the Ansible user, run the ceph-ansible playbook.

   [user@admin ceph-ansible]$ ansible-playbook site-docker.yml --limit rgws

   Note

   If you deploy Red Hat Ceph Storage to Red Hat Enterprise Linux Atomic Host hosts, use the --skip-tags=with_pkg option:

   [user@admin ceph-ansible]$ ansible-playbook site-docker.yml --skip-tags=with_pkg

6. Verify that the Ceph Object Gateway node was deployed successfully.

   1. Connect to a Monitor node as the root user:

      ssh hostname

      Replace hostname with the host name of the Monitor node, for example:

      [user@admin ~]$ ssh root@monitor

   2. Verify that the Ceph Object Gateway pools were created properly:

      [root@monitor ~]# docker exec ceph-mon-mon1 rados lspools
      rbd
      cephfs_data
      cephfs_metadata
      .rgw.root
      default.rgw.control
      default.rgw.data.root
      default.rgw.gc
      default.rgw.log
      default.rgw.users.uid

   3. From any client on the same network as the Ceph cluster, for example the Monitor node, use the curl command to send an HTTP request on port 8080 using the IP address of the Ceph Object Gateway host:

      curl http://IP-address:8080

      Replace IP-address with the IP address of the Ceph Object Gateway node. To determine the IP address of the Ceph Object Gateway host, use the ifconfig or ip commands:

      [root@client ~]# curl http://192.168.122.199:8080
      <?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>

   4. List buckets:

      [root@monitor ~]# docker exec ceph-mon-mon1 radosgw-admin bucket list

1. Add a new section [mdss] to the /etc/ansible/hosts file:

   [mdss]
   hostname
   hostname
   hostname

   Replace hostname with the host names of the nodes where you want to install the Ceph Metadata Servers.

   Alternatively, you can colocate the Metadata Server with the OSD daemon on one node by adding the same node under the [osds] and [mdss] sections. See Colocation of containerized Ceph daemons for details.

2. Navigate to the /usr/share/ceph-ansible directory:

   [root@admin ~]# cd /usr/share/ceph-ansible

3. Optional. Change the default variables.

   1. Create a copy of the group_vars/mdss.yml.sample file named mdss.yml:

      [root@admin ceph-ansible]# cp group_vars/mdss.yml.sample group_vars/mdss.yml

   2. Optionally, edit parameters in mdss.yml. See mdss.yml for details.

4. As the Ansible user, run the Ansible playbook:

   [user@admin ceph-ansible]$ ansible-playbook site-docker.yml --limit mdss

5. After installing Metadata Servers, configure them. For details, see the Configuring Metadata Server Daemons chapter in the Ceph File System Guide for Red Hat Ceph Storage 3.

1. Create the nfss file from the sample file:

   [root@ansible ~]# cd /usr/share/ceph-ansible/group_vars
   [root@ansible ~]# cp nfss.yml.sample nfss.yml

2. Add gateway hosts to the /etc/ansible/hosts file under an [nfss] group to identify their group membership to Ansible. If the hosts have sequential naming, use a range. For example:

   [nfss]
   <nfs_host_name_1>
   <nfs_host_name_2>
   <nfs_host_name[3..10]>

3. Navigate to the Ansible configuration directory, /etc/ansible/:

   [root@ansible ~]# cd /usr/share/ceph-ansible

4. To copy the administrator key to the Ceph Object Gateway node, uncomment the copy_admin_key setting in the /usr/share/ceph-ansible/group_vars/nfss.yml file:

   copy_admin_key: true

5. Configure the FSAL (File System Abstraction Layer) sections of the /usr/share/ceph-ansible/group_vars/nfss.yml file. Provide an ID, S3 user ID, S3 access key and secret. For NFSv4, it should look something like this:

   ###################
   # FSAL RGW Config #
   ###################
   #ceph_nfs_rgw_export_id: <replace-w-numeric-export-id>
   #ceph_nfs_rgw_pseudo_path: "/"
   #ceph_nfs_rgw_protocols: "3,4"
   #ceph_nfs_rgw_access_type: "RW"
   #ceph_nfs_rgw_user: "cephnfs"
   # Note: keys are optional and can be generated, but not on containerized, where
   # they must be configered.
   #ceph_nfs_rgw_access_key: "<replace-w-access-key>"
   #ceph_nfs_rgw_secret_key: "<replace-w-secret-key>"

   Warning

   Access and secret keys are optional, and can be generated.

6. Run the Ansible playbook:

   [user@admin ceph-ansible]$ ansible-playbook site-docker.yml --limit nfss

Procedure

1. As the root user, open and edit the /etc/ansible/hosts file. Add a node name entry in the iSCSI gateway group:

   Example

   [iscsigws]
   ceph-igw-1
   ceph-igw-2

2. Navigate to the /usr/share/ceph-ansible directory:

   [root@admin ~]# cd /usr/share/ceph-ansible/

3. Create a copy of the iscsigws.yml.sample file and name it iscsigws.yml:

   [root@admin ceph-ansible]# cp group_vars/iscsigws.yml.sample group_vars/iscsigws.yml

   Important

   The new file name (iscsigws.yml) and the new section heading ([iscsigws]) are only applicable to Red Hat Ceph Storage 3.1 or higher. Upgrading from previous versions of Red Hat Ceph Storage to 3.1 will still use the old file name (iscsi-gws.yml) and the old section heading ([iscsi-gws]).

   Important

   Currently, Red Hat does not support the following options to be installed using ceph-ansible for container-based deployments:

   • gateway_iqn
   • rbd_devices
   • client_connections

   See the Configuring the Ceph iSCSI gateway in a container section for instructions on configuring these options manually.

4. Open the iscsigws.yml file for editing.

5. Configure the gateway_ip_list option by adding the iSCSI gateway IP addresses, using IPv4 or IPv6 addresses:

   Example

   gateway_ip_list: 192.168.1.1,192.168.1.2

   Important

   You cannot use a mix of IPv4 and IPv6 addresses.

6. Optionally, uncomment the trusted_ip_list option and add the IPv4 or IPv6 addresses accordingly, if you want to use SSL. You will need root access to the iSCSI gateway containers to configure SSL. To configure SSL, do the following steps:

   1. If needed, install the openssl package within all the iSCSI gateway containers.

   2. On the primary iSCSI gateway container, create a directory to hold the SSL keys:

      # mkdir ~/ssl-keys
      # cd ~/ssl-keys

   3. On the primary iSCSI gateway container, create the certificate and key files:

      # openssl req -newkey rsa:2048 -nodes -keyout iscsi-gateway.key -x509 -days 365 -out iscsi-gateway.crt

      Note

      You will be prompted to enter the environmental information.

   4. On the primary iSCSI gateway container, create a PEM file:

      # cat iscsi-gateway.crt iscsi-gateway.key > iscsi-gateway.pem

   5. On the primary iSCSI gateway container, create a public key:

      # openssl x509 -inform pem -in iscsi-gateway.pem -pubkey -noout > iscsi-gateway-pub.key

   6. From the primary iSCSI gateway container, copy the iscsi-gateway.crt, iscsi-gateway.pem, iscsi-gateway-pub.key, and iscsi-gateway.key files to the /etc/ceph/ directory on the other iSCSI gateway containers.

7. Optionally, review and uncomment any of the following iSCSI target API service options accordingly:

   #api_user: admin
   #api_password: admin
   #api_port: 5000
   #api_secure: false
   #loop_delay: 1
   #trusted_ip_list: 192.168.122.1,192.168.122.2

8. Optionally, review and uncomment any of the following resource options, updating them according to the workload needs:

   # TCMU_RUNNER resource limitation
   #ceph_tcmu_runner_docker_memory_limit: 1g
   #ceph_tcmu_runner_docker_cpu_limit: 1
   
   # RBD_TARGET_GW resource limitation
   #ceph_rbd_target_gw_docker_memory_limit: 1g
   #ceph_rbd_target_gw_docker_cpu_limit: 1
   
   # RBD_TARGET_API resource limitation
   #ceph_rbd_target_api_docker_memory_limit: 1g
   #ceph_rbd_target_api_docker_cpu_limit: 1

9. As the Ansible user, run the Ansible playbook:

   [user@admin ceph-ansible]$ ansible-playbook site-docker.yml --limit iscsigws

   For Red Hat Enterprise Linux Atomic, add the --skip-tags=with_pkg option:

   [user@admin ceph-ansible]$ ansible-playbook site-docker.yml --limit iscsigws --skip-tags=with_pkg

10. Once the Ansible playbook has finished, open TCP ports 3260 and the api_port specified in the iscsigws.yml file on each node listed in the trusted_ip_list option.

    Note

    If the api_port option is not specified, the default port is 5000.

Procedure

1. To change the default CPU limit for a daemon, set the ceph_daemon-type_docker_cpu_limit parameter in the appropriate .yml configuration file when deploying the daemon. See the following table for details.

   Daemon	Parameter	Configuration file
   OSD	ceph_osd_docker_cpu_limit	osds.yml
   MDS	ceph_mds_docker_cpu_limit	mdss.yml
   RGW	ceph_rgw_docker_cpu_limit	rgws.yml

   For example, to change the default CPU limit to 2 for the Ceph Object Gateway, edit the /usr/share/ceph-ansible/group_vars/rgws.yml file as follows:

   ceph_rgw_docker_cpu_limit: 2

2. To change the default RAM for OSD daemons, set the osd_memory_target in the /usr/share/ceph-ansible/group_vars/all.yml file when deploying the daemon. For example, to limit the OSD RAM to 6 GB:

   ceph_conf_overrides:
     osd:
       osd_memory_target=6000000000

   Important

   In an hyperconverged infrastructure (HCI) configuration, using the osd_memory_target parameter is the recommended way to limit memory of OSDs. The ceph_osd_docker_memory_limit parameter should not be necessary, but if you wish to use it, then set ceph_osd_docker_memory_limit to 50% higher than osd_memory_target, so that the CGroup limit is more constraining than it is by default for an HCI configuration. For example, if osd_memory_target is set to 6 GB, set ceph_osd_docker_memory_limit to 9 GB:

   ceph_osd_docker_memory_limit: 9g

Procedure

1. To start, stop, or restart a Ceph daemon running in a container, run a systemctl command as root composed in the following format:

   systemctl action ceph-daemon@ID

   Where:

   • action is the action to perform; start, stop, or restart
   • daemon is the daemon; osd, mon, mds, or rgw

   • ID is either

     • The short host name where the ceph-mon, ceph-mds, or ceph-rgw daemons are running
     • The ID of the ceph-osd daemon if it was deployed the osd_scenario parameter set to lvm
     • The device name that the ceph-osd daemon uses if it was deployed with the osd_scenario parameter set to collocated or non-collocated

   For example, to restart a ceph-osd daemon with the ID osd01:

   # systemctl restart ceph-osd@osd01

   To start a ceph-mon demon that runs on the ceph-monitor01 host:

   # systemctl start ceph-mon@ceph-monitor01

   To stop a ceph-rgw daemon that runs on the ceph-rgw01 host:

   # systemctl stop ceph-radosgw@ceph-rgw01

2. Verify that the action was completed successfully.

   systemctl status ceph-daemon@_ID

   For example:

   # systemctl status ceph-mon@ceph-monitor01

Procedure

1. To view the entire Ceph log file, run a journalctl command as root composed in the following format:

   journalctl -u ceph-daemon@ID

   Where:

   • daemon is the Ceph daemon; osd, mon, or rgw

   • ID is either

     • The short host name where the ceph-mon, ceph-mds, or ceph-rgw daemons are running
     • The ID of the ceph-osd daemon if it was deployed the osd_scenario parameter set to lvm
     • The device name that the ceph-osd daemon uses if it was deployed with the osd_scenario parameter set to collocated or non-collocated

   For example, to view the entire log for the ceph-osd daemon with the ID osd01:

   # journalctl -u ceph-osd@osd01

2. To show only the recent journal entries, use the -f option.

   journalctl -fu ceph-daemon@ID

   For example, to view only recent journal entries for the ceph-mon daemon that runs on the ceph-monitor01 host:

   # journalctl -fu ceph-mon@ceph-monitor01

1. Install the ceph-osd package and create a new OSD instance
2. Prepare and mount the OSD data and journal drives
3. Add the new OSD node to the CRUSH map
4. Update the owner and group permissions
5. Enable and start the ceph-osd daemon

Procedure

1. Enable the Red Hat Ceph Storage 3 OSD software repository.

   Red Hat Enterprise Linux

   [root@osd ~]# subscription-manager repos --enable=rhel-7-server-rhceph-3-osd- els-rpms

2. Create the /etc/ceph/ directory:
3. On the new OSD node, copy the Ceph administration keyring and configuration files from one of the Ceph Monitor nodes:

4. Install the ceph-osd package on the new Ceph OSD node:

   Red Hat Enterprise Linux

   [root@osd ~]# yum install ceph-osd

5. Decide if you want to collocate a journal or use a dedicated journal for the new OSDs.

   Note

   The --filestore option is required.

   1. For OSDs with a collocated journal:

      Syntax

      [root@osd ~]# docker exec $CONTAINER_ID ceph-disk --setuser ceph --setgroup ceph prepare  --filestore /dev/$DEVICE_NAME

      Example:

      [root@osd ~]# docker exec ceph-osd-osd1 ceph-disk --setuser ceph --setgroup ceph prepare  --filestore /dev/sda

   2. For OSDs with a dedicated journal:

      Syntax

      [root@osd ~]# docker exec $CONTAINER_ID ceph-disk --setuser ceph --setgroup ceph prepare  --filestore /dev/$DEVICE_NAME /dev/$JOURNAL_DEVICE_NAME

      or

      [root@osd ~]# docker exec $CONTAINER_ID ceph-volume lvm prepare  --filestore --data /dev/$DEVICE_NAME --journal /dev/$JOURNAL_DEVICE_NAME

      Examples

      [root@osd ~]# docker exec ceph-osd-osd1 ceph-disk --setuser ceph --setgroup ceph prepare  --filestore /dev/sda /dev/sdb

      [root@osd ~]# docker exec ceph-osd-osd1 ceph-volume lvm prepare  --filestore --data /dev/vg00/lvol1 --journal /dev/sdb

6. Set the noup option:

   [root@osd ~]# ceph osd set noup

7. Activate the new OSD:

   Syntax

   [root@osd ~]# docker exec $CONTAINER_ID ceph-disk activate /dev/$DEVICE_NAME

   or

   [root@osd ~]# docker exec $CONTAINER_ID ceph-volume lvm activate --filestore $OSD_ID $OSD_FSID

   Example

   [root@osd ~]# docker exec ceph-osd-osd1 ceph-disk activate /dev/sda

   [root@osd ~]# docker exec ceph-osd-osd1 ceph-volume lvm activate --filestore 0 6cc43680-4f6e-4feb-92ff-9c7ba204120e

8. Add the OSD to the CRUSH map:

   Syntax

   ceph osd crush add $OSD_ID $WEIGHT [$BUCKET_TYPE=$BUCKET_NAME ...]

   Example

   [root@osd ~]# ceph osd crush add 4 1 host=node4

   Note

   If you specify more than one bucket, the command places the OSD into the most specific bucket out of those you specified, and it moves the bucket underneath any other buckets you specified.

   Note

   You can also edit the CRUSH map manually. See the Editing a CRUSH map section in the Storage Strategies guide for Red Hat Ceph Storage 3.

   Important

   If you specify only the root bucket, then the OSD attaches directly to the root, but the CRUSH rules expect OSDs to be inside of the host bucket.

9. Unset the noup option:

   [root@osd ~]# ceph osd unset noup

10. Update the owner and group permissions for the newly created directories:

    Syntax

    chown -R $OWNER:$GROUP $PATH_TO_DIRECTORY

    Example

    [root@osd ~]# chown -R ceph:ceph /var/lib/ceph/osd
    [root@osd ~]# chown -R ceph:ceph /var/log/ceph
    [root@osd ~]# chown -R ceph:ceph /var/run/ceph
    [root@osd ~]# chown -R ceph:ceph /etc/ceph

11. If you use clusters with custom names, then add the following line to the appropriate file:

    Red Hat Enterprise Linux

    [root@osd ~]# echo "CLUSTER=$CLUSTER_NAME" >> /etc/sysconfig/ceph

    Replace $CLUSTER_NAME with the custom cluster name.

12. To ensure that the new OSD is up and ready to receive data, enable and start the OSD service:

    Syntax

    systemctl enable ceph-osd@$OSD_ID
    systemctl start ceph-osd@$OSD_ID

    Example

    [root@osd ~]# systemctl enable ceph-osd@4
    [root@osd ~]# systemctl start ceph-osd@4

Procedure

1. Disable and stop the OSD service:

   Syntax

   systemctl disable ceph-osd@$DEVICE_NAME
   systemctl stop ceph-osd@$DEVICE_NAME

   Example

   [root@osd ~]# systemctl disable ceph-osd@sdb
   [root@osd ~]# systemctl stop ceph-osd@sdb

   Once the OSD is stopped, it is down.

2. Remove the OSD from the storage cluster:

   Syntax

   ceph osd out $DEVICE_NAME

   Example

   [root@osd ~]# ceph osd out sdb

   Important

   Once the OSD is out, Ceph will start rebalancing and copying data to other OSDs in the storage cluster. Red Hat recommends waiting until the storage cluster becomes active+clean before proceeding to the next step. To observe the data migration, run the following command:

   [root@monitor ~]# ceph -w

3. Remove the OSD from the CRUSH map so that it no longer receives data.

   Syntax

   ceph osd crush remove $OSD_NAME

   Example

   [root@osd ~]# ceph osd crush remove osd.4

   Note

   You can also decompile the CRUSH map, remove the OSD from the device list, remove the device as an item in the host bucket or remove the host bucket. If it is in the CRUSH map and you intend to remove the host, recompile the map and set it. See the Storage Strategies Guide for details.

4. Remove the OSD authentication key:

   Syntax

   ceph auth del osd.$DEVICE_NAME

   Example

   [root@osd ~]# ceph auth del osd.sdb

5. Remove the OSD:

   Syntax

   ceph osd rm $DEVICE_NAME

   Example

   [root@osd ~]# ceph osd rm sdb

6. Edit the storage cluster’s configuration file, by default /etc/ceph.conf, and remove the OSD entry, if it exists:

   Example

   [osd.4]
   host = $HOST_NAME

7. Remove the reference to the OSD in the /etc/fstab file, if the OSD was added manually.

8. Copy the updated configuration file to the /etc/ceph/ directory of all other nodes in the storage cluster.

   Syntax

   scp /etc/ceph/$CLUSTER_NAME.conf $USER_NAME@$HOST_NAME:/etc/ceph/

   Example

   [root@osd ~]# scp /etc/ceph/ceph.conf root@node4:/etc/ceph/

Procedure

1. Destroy the OSD:

   ceph osd destroy $OSD_ID --yes-i-really-mean-it

   Example

   $ ceph osd destroy 1 --yes-i-really-mean-it

2. Optionally, if the replacement disk was used previously, then you need to zap the disk:

   docker exec $CONTAINER_ID ceph-volume lvm zap $DEVICE

   Example

   $ docker exec ceph-osd-osd1 ceph-volume lvm zap /dev/sdb

3. Create the new OSD with the existing OSD ID:

   docker exec $CONTAINER_ID ceph-volume lvm create --osd-id $OSD_ID --data $DEVICE

   Example

   $ docker exec ceph-osd-osd1 ceph-volume lvm create --osd-id 1 --data /dev/sdb

1. As the root user, navigate to the /usr/share/ceph-ansible/ directory.

   [root@admin ~]# cd /usr/share/ceph-ansible

2. Copy the purge-docker-cluster.yml playbook from the /usr/share/infrastructure-playbooks/ directory to the current directory:

   [root@admin ceph-ansible]# cp infrastructure-playbooks/purge-docker-cluster.yml .

3. As the Ansible user, use the purge-docker-cluster.yml playbook to purge the Ceph cluster.

   1. To remove all packages, containers, configuration files, and all the data created by the ceph-ansible playbook:

      [user@admin ceph-ansible]$ ansible-playbook purge-docker-cluster.yml

   2. To specify a different inventory file than the default one (/etc/ansible/hosts), use -i parameter:

      ansible-playbook purge-docker-cluster.yml -i inventory-file

      Replace inventory-file with the path to the inventory file.

      For example:

      [user@admin ceph-ansible]$ ansible-playbook purge-docker-cluster.yml -i ~/ansible/hosts

   3. To skip the removal of the Ceph container image, use the --skip-tags=”remove_img” option:

      [user@admin ceph-ansible]$ ansible-playbook --skip-tags="remove_img" purge-docker-cluster.yml

   4. To skip the removal of the packages that were installed during the installation, use the --skip-tags=”with_pkg” option:

      [user@admin ceph-ansible]$ ansible-playbook --skip-tags="with_pkg" purge-docker-cluster.yml

1. As the root user, navigate to the /usr/share/ceph-ansible/ directory:

   [root@admin ~]# cd /usr/share/ceph-ansible/

2. Skip this step when upgrading from Red Hat Ceph Storage version 3.x to the latest version. Back up the group_vars/all.yml and group_vars/osds.yml files.

   [root@admin ceph-ansible]# cp group_vars/all.yml group_vars/all_old.yml
   [root@admin ceph-ansible]# cp group_vars/osds.yml group_vars/osds_old.yml
   [root@admin ceph-ansible]# cp group_vars/clients.yml group_vars/clients_old.yml

3. Skip this step when upgrading from Red Hat Ceph Storage version 3.x to the latest version. When upgrading from Red Hat Ceph Storage 2.x to 3.x, create new copies of the group_vars/all.yml.sample, group_vars/osds.yml.sample and group_vars/clients.yml.sample files, and rename them to group_vars/all.yml, group_vars/osds.yml, and group_vars/clients.yml respectively. Open and edit them accordingly. For details, see Appendix A, Changes in Ansible Variables Between Version 2 and 3 and Section 1.2, “Installing a Red Hat Ceph Storage Cluster in Containers” .

   [root@admin ceph-ansible]# cp group_vars/all.yml.sample group_vars/all.yml
   [root@admin ceph-ansible]# cp group_vars/osds.yml.sample group_vars/osds.yml
   [root@admin ceph-ansible]# cp group_vars/clients.yml.sample group_vars/clients.yml

4. Skip this step when upgrading from Red Hat Ceph Storage version 3.x to the latest version. When upgrading from Red Hat Ceph Storage 2.x to 3.x, open the group_vars/clients.yml file, and uncomment the following lines:

   keys:
     - { name: client.test, caps: { mon: "allow r", osd: "allow class-read object_prefix rbd_children, allow rwx pool=test" },  mode: "{{ ceph_keyring_permissions }}" }

   1. Replace client.test with the real client name, and add the client key to the client definition line, for example:

      key: "ADD-KEYRING-HERE=="

      Now the whole line example would look similar to this:

      - { name: client.test, key: "AQAin8tUMICVFBAALRHNrV0Z4MXupRw4v9JQ6Q==", caps: { mon: "allow r", osd: "allow class-read object_prefix rbd_children, allow rwx pool=test" },  mode: "{{ ceph_keyring_permissions }}" }

      Note

      To get the client key, run the ceph auth get-or-create command to view the key for the named client.

5. When upgrading from 2.x to 3.x, in the group_vars/all.yml file change the ceph_docker_image parameter to point to the Ceph 3 container version.

   ceph_docker_image: rhceph/rhceph-3-rhel7

6. Add the fetch_directory parameter to the group_vars/all.yml file.

   fetch_directory: <full_directory_path>

   Replace:

   • <full_directory_path> with a writable location, such as the Ansible user’s home directory. Provide the existing path that was used for the initial storage cluster installation.

   If the existing path is lost or missing, then do the following first:

   1. Add the following options to the existing group_vars/all.yml file:

      fsid: <add_the_fsid>
      generate_fsid: false

   2. Run the take-over-existing-cluster.yml Ansible playbook:

      [user@admin ceph-ansible]$ cp infrastructure-playbooks/take-over-existing-cluster.yml .
      [user@admin ceph-ansible]$ ansible-playbook take-over-existing-cluster.yml

7. If the cluster you want to upgrade contains any Ceph Object Gateway nodes, add the radosgw_interface parameter to the group_vars/all.yml file.

   radosgw_interface: <interface>

   Replace:

   • <interface> with the interface that the Ceph Object Gateway nodes listen to.

8. Starting with Red Hat Ceph Storage 3.2, the default OSD object store is BlueStore. To keep the traditional OSD object store, you must explicitly set the osd_objectstore option to filestore in the group_vars/all.yml file.

   osd_objectstore: filestore

   Note

   With the osd_objectstore option set to filestore, replacing an OSD will use FileStore, instead of BlueStore.

9. In the Ansible inventory file located at /etc/ansible/hosts, add the Ceph Manager (ceph-mgr) nodes under the [mgrs] section. Colocate the Ceph Manager daemon with Monitor nodes. Skip this step when upgrading from version 3.x to the latest version.

   [mgrs]
   <monitor-host-name>
   <monitor-host-name>
   <monitor-host-name>

10. Copy rolling_update.yml from the infrastructure-playbooks directory to the current directory.

    [root@admin ceph-ansible]# cp infrastructure-playbooks/rolling_update.yml .

11. Create the /var/log/ansible/ directory and assign the appropriate permissions for the ansible user:

    [root@admin ceph-ansible]# mkdir /var/log/ansible
    [root@admin ceph-ansible]# chown ansible:ansible  /var/log/ansible
    [root@admin ceph-ansible]# chmod 755 /var/log/ansible

    1. Edit the /usr/share/ceph-ansible/ansible.cfg file, updating the log_path value as follows:

       log_path = /var/log/ansible/ansible.log

12. As the Ansible user, run the playbook:

    [user@admin ceph-ansible]$ ansible-playbook rolling_update.yml

    To use the playbook only for a particular group of nodes on the Ansible inventory file, use the --limit option. For details, see Section 1.8, “Understanding the limit option”.

13. While logged in as the root user on the RBD mirroring daemon node, upgrade rbd-mirror manually:

    # yum upgrade rbd-mirror

    Restart the daemon:

    # systemctl restart  ceph-rbd-mirror@<client-id>

14. Verify that the cluster health is OK.

    1. Log into a monitor node as the root user and list all running containers.

       [root@monitor ~]# docker ps

    2. Verify the cluster health is OK.

       [root@monitor ~]# docker exec ceph-mon-<mon-id> ceph -s

       Replace:

       • <mon-id> with the name of the Monitor container found in the first step.

       For example:

       [root@monitor ~]# docker exec ceph-mon-monitor ceph -s

15. If working in an OpenStack environment, update all the cephx users to use the RBD profile for pools. The following commands must be run as the root user:

    • Glance users

      ceph auth caps client.glance mon 'profile rbd' osd 'profile rbd pool=<glance-pool-name>'

      Example

      [root@monitor ~]# ceph auth caps client.glance mon 'profile rbd' osd 'profile rbd pool=images'

    • Cinder users

      ceph auth caps client.cinder mon 'profile rbd' osd 'profile rbd pool=<cinder-volume-pool-name>, profile rbd pool=<nova-pool-name>, profile rbd-read-only pool=<glance-pool-name>'

      Example

      [root@monitor ~]# ceph auth caps client.cinder mon 'profile rbd' osd 'profile rbd pool=volumes, profile rbd pool=vms, profile rbd-read-only pool=images'

    • OpenStack general users

      ceph auth caps client.openstack mon 'profile rbd' osd 'profile rbd-read-only pool=<cinder-volume-pool-name>, profile rbd pool=<nova-pool-name>, profile rbd-read-only pool=<glance-pool-name>'

      Example

      [root@monitor ~]# ceph auth caps client.openstack mon 'profile rbd' osd 'profile rbd-read-only pool=volumes, profile rbd pool=vms, profile rbd-read-only pool=images'

      Important

      Do these CAPS updates before performing any live client migrations. This allows clients to use the new libraries running in memory, causing the old CAPS settings to drop from cache and applying the new RBD profile settings.

Procedure

1. As the root user, update the cephmetrics-ansible package from the Ansible administration node:

   [root@admin ~]# yum update cephmetrics-ansible

2. Change to the /usr/share/cephmetrics-ansible directory:

   [root@admin ~]# cd /usr/share/cephmetrics-ansible

3. Install the updated Red Hat Ceph Storage Dashboard:

   [root@admin cephmetrics-ansible]# ansible-playbook -v playbook.yml

1. Install the cephmetrics-ansible package.

   [root@admin ~]# yum install cephmetrics-ansible

2. Using the Ceph Ansible inventory as a base, add the Red Hat Ceph Storage Dashboard node under the [ceph-grafana] section of the Ansible inventory file, by default located at /etc/ansible/hosts.

   [ceph-grafana]
   $HOST_NAME

   Replace:

   • $HOST_NAME with the name of the Red Hat Ceph Storage Dashboard node

   For example:

   [ceph-grafana]
   node0

3. Change to the /usr/share/cephmetrics-ansible/ directory.

   [root@admin ~]# cd /usr/share/cephmetrics-ansible

4. Run the Ansible playbook.

   [root@admin cephmetrics-ansible]# ansible-playbook -v playbook.yml

   Important

   Every time you update the cluster configuration, for example, you add or remove a MON or OSD node, you must re-run the cephmetrics Ansible playbook.

   Note

   The cephmetrics Ansible playbook does the following actions:

   • Updates the ceph-mgr instance to enable the prometheus plugin and opens TCP port 9283.

   • Deploys the Prometheus node-exporter daemon to each node in the storage cluster.

     • Opens TCP port 9100.
     • Starts the node-exporter daemon.

   • Deploys Grafana and Prometheus containers under Docker/systemd on the Red Hat Ceph Storage Dashboard node.

     • Prometheus is configured to gather data from the ceph-mgr nodes and the node-exporters running on each ceph host
     • Opens TCP port 3000.
     • The dashboards, themes and user accounts are all created in Grafana.
     • Outputs the URL of Grafana for the administrator.

1. Enter the following URL to a web browser:

   http://$HOST_NAME:3000

   Replace:

   • $HOST_NAME with the name of the Red Hat Ceph Storage Dashboard node

   For example:

   http://cephmetrics:3000

2. Enter the password for the admin user. If you did not set the password during the installation, use admin, which is the default password.

   Once logged in, you are automatically placed on the Ceph At a Glance dashboard. The Ceph At a Glance dashboard provides a high-level overviews of capacity, performance, and node-level performance information.

   Example

   

Procedure

1. Click the Grafana icon in the upper-left corner.
2. Hover over the user name you want to modify the password for. In this case admin.
3. Click Profile.
4. Click Change Password.
5. Enter the new password twice and click Change Password.