Chapter 7. The Ceph iSCSI Gateway (Limited Availability)

Procedure

1. On the iSCSI gateway nodes, enable the Red Hat Ceph Storage 4 Tools repository. For details, see the Enabling the Red Hat Ceph Storage Repositories section in the Red Hat Ceph Storage Installation Guide.

2. On the Ansible administration node, add an entry in /etc/ansible/hosts file for the gateway group. If you colocate the iSCSI gateway with an OSD node, add the OSD node to the [iscsigws] section.

   [iscsigws]
   ceph-igw-1
   ceph-igw-2

   Copy to Clipboard Toggle word wrap
3. Ansible places a file in the /usr/share/ceph-ansible/group_vars/ directory called iscsigws.yml.sample. Create a copy of the iscsigws.yml.sample file named it iscsigws.yml.
4. Open the iscsigws.yml file for editing.

5. Uncomment the trusted_ip_list option and update the values accordingly, using IPv4 or IPv6 addresses.

   Example

   Adding two gateways with the IPv4 addresses of 10.172.19.21 and 10.172.19.22, configure trusted_ip_list like this:

   trusted_ip_list: 10.172.19.21,10.172.19.22

   Copy to Clipboard Toggle word wrap

6. Optionally, review the Ansible variables and descriptions in the iSCSI Gateway Variables section and update iscsigws.yml as needed.

   Warning

   Gateway configuration changes are only supported from one gateway at a time. Attempting to run changes concurrently through multiple gateways might lead to configuration instability and inconsistency.

   Warning

   Ansible installs the ceph-iscsi package, creates, and updates the /etc/ceph/iscsi-gateway.cfg file based on settings in the group_vars/iscsigws.yml file when the ansible-playbook command is used. If you have previously installed the ceph-iscsi package using the command-line interface described in Installing the iSCSI gateway using the command-line interface, copy the existing settings from the iscsi-gateway.cfg file to the group_vars/iscsigws.yml file.

7. On the Ansible administration node, execute the Ansible playbook.

   • Bare-metal deployments:

     [admin@ansible ~]$ cd /usr/share/ceph-ansible
     [admin@ansible ceph-ansible]$ ansible-playbook site.yml -i hosts

     Copy to Clipboard Toggle word wrap

   • Container deployments:

     [admin@ansible ~]$ cd /usr/share/ceph-ansible
     [admin@ansible ceph-ansible]$ ansible-playbook site-container.yml -i hosts

     Copy to Clipboard Toggle word wrap
     Warning

     On stand-alone iSCSI gateway nodes, verify that the correct Red Hat Ceph Storage 4 software repositories are enabled. If they are unavailable, Ansible might install incorrect packages.

8. To create targets, LUNs, and clients, use the gwcli utility or the Red Hat Ceph Storage Dashboard.

   Important

   Do not use the targetcli utility to change the configuration, this will result in the following issues: ALUA misconfiguration and path failover problems. There is the potential to corrupt data, to have mismatched configuration across iSCSI gateways, and to have mismatched WWN information, which will lead to client pathing problems.

Procedure

1. On all Ceph iSCSI gateway nodes, install the ceph-iscsi and tcmu-runner packages:

   [root@iscsigw ~]# yum install ceph-iscsi tcmu-runner

   Copy to Clipboard Toggle word wrap
   Important

   If previous versions of these packages exist, remove them before installing the newer versions. You must install these newer versions from a Red Hat Ceph Storage repository.

2. Optionally, on all Ceph iSCSI gateway nodes, install and configure the OpenSSL utility, if needed.

   1. Install the openssl package:

      [root@iscsigw ~]# yum install openssl

      Copy to Clipboard Toggle word wrap

   2. On the primary iSCSI gateway node, create a directory to hold the SSL keys:

      [root@iscsigw ~]# mkdir ~/ssl-keys
      [root@iscsigw ~]# cd ~/ssl-keys

      Copy to Clipboard Toggle word wrap

   3. On the primary iSCSI gateway node, create the certificate and key files. Enter the environmental information when prompted.

      [root@iscsigw ~]# openssl req -newkey rsa:2048 -nodes -keyout iscsi-gateway.key -x509 -days 365 -out iscsi-gateway.crt

      Copy to Clipboard Toggle word wrap

   4. On the primary iSCSI gateway node, create a PEM file:

      [root@iscsigw ~]# cat iscsi-gateway.crt iscsi-gateway.key > iscsi-gateway.pem

      Copy to Clipboard Toggle word wrap

   5. On the primary iSCSI gateway node, create a public key:

      [root@iscsigw ~]# openssl x509 -inform pem -in iscsi-gateway.pem -pubkey -noout > iscsi-gateway-pub.key

      Copy to Clipboard Toggle word wrap
   6. From the primary iSCSI gateway node, copy the iscsi-gateway.crt, iscsi-gateway.pem, iscsi-gateway-pub.key, and iscsi-gateway.key files to the /etc/ceph/ directory on the other iSCSI gateway nodes.

3. Create a configuration file on a Ceph iSCSI gateway node, and then copy it to all iSCSI gateway nodes.

   1. Create a file named iscsi-gateway.cfg in the /etc/ceph/ directory:

      [root@iscsigw ~]# touch /etc/ceph/iscsi-gateway.cfg

      Copy to Clipboard Toggle word wrap

   2. Edit the iscsi-gateway.cfg file and add the following lines:

      Syntax

      [config]
      cluster_name = CLUSTER_NAME
      gateway_keyring = CLIENT_KEYRING
      api_secure = false
      trusted_ip_list = IP_ADDR,IP_ADDR

      Copy to Clipboard Toggle word wrap

      Example

      [config]
      cluster_name = ceph
      gateway_keyring = ceph.client.admin.keyring
      api_secure = false
      trusted_ip_list = 192.168.0.10,192.168.0.11

      Copy to Clipboard Toggle word wrap

   3. Copy the iscsi-gateway.cfg file to all iSCSI gateway nodes. Note that the file must be identical on all iSCSI gateway nodes.

4. On all Ceph iSCSI gateway nodes, enable and start the API services:

   [root@iscsigw ~]# systemctl enable rbd-target-api
   [root@iscsigw ~]# systemctl start rbd-target-api
   [root@iscsigw ~]# systemctl enable rbd-target-gw
   [root@iscsigw ~]# systemctl start rbd-target-gw

   Copy to Clipboard Toggle word wrap
5. Next, configure targets, LUNs, and clients. See the Configuring the iSCSI target using the command-line interface section for details.

Procedure

1. Start the iSCSI gateway command-line interface:

   [root@iscsigw ~]# gwcli

   Copy to Clipboard Toggle word wrap

2. Create the iSCSI gateways using either IPv4 or IPv6 addresses:

   Syntax

   >/iscsi-targets create iqn.2003-01.com.redhat.iscsi-gw:_target_name_
   > goto gateways
   > create ISCSI_GW_NAME IP_ADDR_OF_GW
   > create ISCSI_GW_NAME IP_ADDR_OF_GW

   Copy to Clipboard Toggle word wrap

   Example

   >/iscsi-targets create iqn.2003-01.com.redhat.iscsi-gw:ceph-igw
   > goto gateways
   > create ceph-gw-1 10.172.19.21
   > create ceph-gw-2 10.172.19.22

   Copy to Clipboard Toggle word wrap

   Note

   You cannot use a mix of IPv4 and IPv6 addresses.

3. Add a Ceph block device:

   Syntax

   > cd /disks
   >/disks/ create POOL_NAME image=IMAGE_NAME size=IMAGE_SIZE_m|g|t

   Copy to Clipboard Toggle word wrap

   Example

   > cd /disks
   >/disks/ create rbd image=disk_1 size=50g

   Copy to Clipboard Toggle word wrap

   Note

   Do not use any periods (.) in the pool or image name.

4. Create a client:

   Syntax

   > goto hosts
   > create iqn.1994-05.com.redhat:_client_name_
   > auth use username=USER_NAME password=PASSWORD

   Copy to Clipboard Toggle word wrap

   Example

   > goto hosts
   > create iqn.1994-05.com.redhat:rh7-client
   > auth username=iscsiuser1 password=temp12345678

   Copy to Clipboard Toggle word wrap

   Important

   Red Hat does not support mixing clients, some with Challenge Handshake Authentication Protocol (CHAP) enabled and some CHAP disabled. All clients must have either CHAP enabled or have CHAP disabled. The default behavior is to only authenticate an initiator by its initiator name.

   If initiators are failing to log into the target, the CHAP authentication might not be configured correctly for some initiators, for example:

   o- hosts ................................ [Hosts: 2: Auth: MISCONFIG]

   Copy to Clipboard Toggle word wrap

   Use the following command at the hosts level to reset all the CHAP authentication:

   /> goto hosts
   /iscsi-target...csi-igw/hosts> auth nochap
   ok
   ok
   /iscsi-target...csi-igw/hosts> ls
   o- hosts ................................ [Hosts: 2: Auth: None]
     o- iqn.2005-03.com.ceph:esx ........... [Auth: None, Disks: 4(310G)]
     o- iqn.1994-05.com.redhat:rh7-client .. [Auth: None, Disks: 0(0.00Y)]

   Copy to Clipboard Toggle word wrap

5. Add disks to a client:

   Syntax

   >/iscsi-target..eph-igw/hosts
   > cd iqn.1994-05.com.redhat:_CLIENT_NAME_
   > disk add POOL_NAME/IMAGE_NAME

   Copy to Clipboard Toggle word wrap

   Example

   >/iscsi-target..eph-igw/hosts
   > cd iqn.1994-05.com.redhat:rh7-client
   > disk add rbd/disk_1

   Copy to Clipboard Toggle word wrap

6. To confirm that the API is using SSL correctly, search the rbd-target-api log file, located at /var/log/rbd-target-api.log or /var/log/rbd-target/rbd-target-api.log, for https, for example:

   Aug 01 17:27:42 test-node.example.com python[1879]:  * Running on https://0.0.0.0:5000/

   Copy to Clipboard Toggle word wrap

7. Verifying that the Ceph ISCSI gateways are working:

   /> goto gateways
   /iscsi-target...-igw/gateways> ls
   o- gateways ............................ [Up: 2/2, Portals: 2]
     o- ceph-gw-1  ........................ [ 10.172.19.21 (UP)]
     o- ceph-gw-2  ........................ [ 10.172.19.22 (UP)]

   Copy to Clipboard Toggle word wrap

   If the status is UNKNOWN, check for network issues and any misconfigurations. If using a firewall, verify that the appropriate TCP port is open. Verify that the iSCSI gateway is listed in the trusted_ip_list option. Verify that the rbd-target-api service is running on the iSCSI gateway node.

8. Optionally, reconfigure the max_data_area_mb option:

   Syntax

   >/disks/ reconfigure POOL_NAME/IMAGE_NAME max_data_area_mb NEW_BUFFER_SIZE

   Copy to Clipboard Toggle word wrap

   Example

   >/disks/ reconfigure rbd/disk_1 max_data_area_mb 64

   Copy to Clipboard Toggle word wrap

   Note

   The max_data_area_mb option controls the amount of memory in megabytes that each image can use to pass SCSI command data between the iSCSI target and the Ceph cluster. If this value is too small, it can result in excessive queue full retries which will affect performance. If the value is too large, it can result in one disk using too much of the system memory, which can cause allocation failures for other subsystems. The default value for the max_data_area_mb option is 8.

9. Configure an iSCSI initiator.

Procedure

1. Configure Ansible to use the new timer settings.

   1. On the Ansible administration node, add a ceph_conf_overrides section in the group_vars/all.yml file that looks like this, or edit any existing ceph_conf_overrides section as follows:

      ceph_conf_overrides:
           osd:
             osd_client_watch_timeout: 15
             osd_heartbeat_grace: 20
             osd_heartbeat_interval: 5

      Copy to Clipboard Toggle word wrap

      The above settings will be added to the ceph.conf configuration files on the OSD nodes when the Ansible playbook runs.

   2. Change to the ceph-ansible directory:

      [admin@ansible ~]$ cd /usr/share/ceph-ansible

      Copy to Clipboard Toggle word wrap

   3. Use Ansible to update the ceph.conf file and restart the OSD daemons on all the OSD nodes. On the Ansible admin node, run the following command:

      Bare-metal Deployments

      [admin@ansible ceph-ansible]$ ansible-playbook site.yml --limit osds

      Copy to Clipboard Toggle word wrap

      Container Deployments

      [admin@ansible ceph-ansible]$ ansible-playbook site-container.yml --limit osds -i hosts

      Copy to Clipboard Toggle word wrap

2. Verify the timer settings are the same as set in ceph_conf_overrides:

   Syntax

   ceph daemon osd.OSD_ID config get osd_client_watch_timeout
   ceph daemon osd.OSD_ID config get osd_heartbeat_grace
   ceph daemon osd.OSD_ID config get osd_heartbeat_interval

   Copy to Clipboard Toggle word wrap

   Example

   [root@osd ~]# ceph daemon osd.0 config get osd_client_watch_timeout
   {
       "osd_client_watch_timeout": "15"
   }
   
   [root@osd ~]#  ceph daemon osd.0 config get osd_heartbeat_grace
   {
       "osd_heartbeat_grace": "20"
   }
   
   [root@osd ~]# ceph daemon osd.0 config get osd_heartbeat_interval
   {
       "osd_heartbeat_interval": "5"
   }

   Copy to Clipboard Toggle word wrap

3. Optional: If you cannot restart the OSD daemons immediately, you can do online updates from Ceph Monitor nodes, or update all Ceph OSD nodes directly. Once you are able to restart the OSD daemons, use Ansible as described above to add the new timer settings into ceph.conf so that the settings persist across reboots.

   1. To do an online update of OSD timer settings from a Ceph Monitor node:

      Syntax

      ceph tell osd.OSD_ID injectargs '--osd_client_watch_timeout 15'
      ceph tell osd.OSD_ID injectargs '--osd_heartbeat_grace 20'
      ceph tell osd.OSD_ID injectargs '--osd_heartbeat_interval 5'

      Copy to Clipboard Toggle word wrap

      Example

      [root@mon ~]# ceph tell osd.0 injectargs '--osd_client_watch_timeout 15'
      [root@mon ~]# ceph tell osd.0 injectargs '--osd_heartbeat_grace 20'
      [root@mon ~]# ceph tell osd.0 injectargs '--osd_heartbeat_interval 5'

      Copy to Clipboard Toggle word wrap

   2. To do an online update of OSD timer settings from an Ceph OSD node:

      Syntax

      ceph daemon osd.OSD_ID config set osd_client_watch_timeout 15
      ceph daemon osd.OSD_ID config set osd_heartbeat_grace 20
      ceph daemon osd.OSD_ID config set osd_heartbeat_interval 5

      Copy to Clipboard Toggle word wrap

      Example

      [root@osd ~]# ceph daemon osd.0 config set osd_client_watch_timeout 15
      [root@osd ~]# ceph daemon osd.0 config set osd_heartbeat_grace 20
      [root@osd ~]# ceph daemon osd.0 config set osd_heartbeat_interval 5

      Copy to Clipboard Toggle word wrap

Procedure

1. Start the iSCSI gateway command-line interface:

   [root@iscsigw ~]# gwcli

   Copy to Clipboard Toggle word wrap

2. Create a new host group:

   Syntax

   cd iscsi-targets/
   cd IQN/host-groups
   create group_name=GROUP_NAME

   Copy to Clipboard Toggle word wrap

   Example

   /> cd iscsi-targets/
   /iscsi-targets> cd iqn.2003-01.com.redhat.iscsi-gw:ceph-igw/host-groups/
   /iscsi-target.../host-groups> create group_name=igw_grp01

   Copy to Clipboard Toggle word wrap

3. Add a host to the host group:

   Syntax

   cd GROUP_NAME
   host add client_iqn=CLIENT_IQN

   Copy to Clipboard Toggle word wrap

   Example

   > cd igw_grp01
   /iscsi-target.../host-groups/igw_grp01> host add client_iqn=iqn.1994-05.com.redhat:rh8-client

   Copy to Clipboard Toggle word wrap

   Repeat this step to add additional hosts to the group.

4. Add a disk to the host group:

   Syntax

   cd /disks/
   /disks> create pool=POOL image=IMAGE_NAME size=SIZE
   cd /IQN/host-groups/GROUP_NAME
   disk add POOL/IMAGE_NAME

   Copy to Clipboard Toggle word wrap

   Example

   > cd /disks/
   /disks> create pool=rbd image=rbdimage size=1G
   /> cd iscsi-targets/iqn.2003-01.com.redhat.iscsi-gw:ceph-igw/host-groups/igw_grp01/
   /iscsi-target...s/igw_grp01> disk add rbd/rbdimage

   Copy to Clipboard Toggle word wrap

   Repeat this step to add additional disks to the group.

Procedure

1. Install the iSCSI initiator and multipath tools:

   [root@rhel ~]# yum install iscsi-initiator-utils
   [root@rhel ~]# yum install device-mapper-multipath

   Copy to Clipboard Toggle word wrap
2. Set the initiator name by editing the /etc/iscsi/initiatorname.iscsi file. Note that the initiator name must match the initiator name that was used during the initial setup using the gwcli command.

3. Configure multipath I/O.

   1. Create the default /etc/multipath.conf file and enable the multipathd service:

      [root@rhel ~]# mpathconf --enable --with_multipathd y

      Copy to Clipboard Toggle word wrap

   2. Update the /etc/multipath.conf file as follows:

      devices {
              device {
                      vendor                 "LIO-ORG"
                      product                "TCMU device"
                      hardware_handler       "1 alua"
                      path_grouping_policy   "failover"
                      path_selector          "queue-length 0"
                      failback               60
                      path_checker           tur
                      prio                   alua
                      prio_args              exclusive_pref_bit
                      fast_io_fail_tmo       25
                      no_path_retry          queue
              }
      }

      Copy to Clipboard Toggle word wrap

   3. Restart the multipathd service:

      [root@rhel ~]# systemctl reload multipathd

      Copy to Clipboard Toggle word wrap

4. Set up CHAP and iSCSI discovery and login.

   1. Provide a CHAP user name and password by updating the /etc/iscsi/iscsid.conf file accordingly, for example:

      node.session.auth.authmethod = CHAP
      node.session.auth.username = user
      node.session.auth.password = password

      Copy to Clipboard Toggle word wrap

   2. Discover the target portals:

      Syntax

      iscsiadm -m discovery -t st -p IP_ADDR

      Copy to Clipboard Toggle word wrap

   3. Log in to target:

      Syntax

      iscsiadm -m node -T TARGET -l

      Copy to Clipboard Toggle word wrap

5. View the multipath I/O configuration. The multipathd daemon sets up devices automatically based on the settings in the multipath.conf file.

   1. Use the multipath command to show devices setup in a failover configuration with a priority group for each path, for example:

      Example

      [root@rhel ~]# multipath -ll
      mpathbt (360014059ca317516a69465c883a29603) dm-1 LIO-ORG,TCMU device
      size=1.0G features='0' hwhandler='1 alua' wp=rw
      |-+- policy='queue-length 0' prio=50 status=active
      | `- 28:0:0:1 sde  8:64  active ready running
      `-+- policy='queue-length 0' prio=10 status=enabled
        `- 29:0:0:1 sdc  8:32  active ready running

      Copy to Clipboard Toggle word wrap

      The multipath -ll output prio value indicates the ALUA state, where prio=50 indicates it is the path to the owning iSCSI gateway in the ALUA Active-Optimized state and prio=10 indicates it is an Active-non-Optimized path. The status field indicates which path is being used, where active indicates the currently used path, and enabled indicates the failover path, if the active fails.

   2. To match the device name, for example, sde in the multipath -ll output, to the iSCSI gateway:

      Example

      [root@rhel ~]# iscsiadm -m session -P 3

      Copy to Clipboard Toggle word wrap

      The Persistent Portal value is the IP address assigned to the iSCSI gateway listed in the gwcli utility.

Procedure

1. Configure multipath I/O.

   1. Update the /etc/multipath/conf.d/DEVICE_NAME.conf file as follows:

      devices {
              device {
                      vendor                 "LIO-ORG"
                      product                "TCMU device"
                      hardware_handler       "1 alua"
                      path_grouping_policy   "failover"
                      path_selector          "queue-length 0"
                      failback               60
                      path_checker           tur
                      prio                   alua
                      prio_args              exclusive_pref_bit
                      fast_io_fail_tmo       25
                      no_path_retry          queue
              }
      }

      Copy to Clipboard Toggle word wrap

   2. Restart the multipathd service:

      [root@rhv ~]# systemctl reload multipathd

      Copy to Clipboard Toggle word wrap
2. Click the Storage resource tab to list the existing storage domains.
3. Click the New Domain button to open the New Domain window.
4. Enter the Name of the new storage domain.
5. Use the Data Center drop-down menu to select an data center.
6. Use the drop-down menus to select the Domain Function and the Storage Type. The storage domain types that are not compatible with the chosen domain function are not available.
7. Select an active host in the Use Host field. If this is not the first data domain in a data center, you must select the data center’s SPM host.

8. The New Domain window automatically displays known targets with unused LUNs when iSCSI is selected as the storage type. If the target that you are adding storage from is not listed then you can use target discovery to find it, otherwise proceed to the next step.

   1. Click Discover Targets to enable target discovery options. When targets have been discovered and logged in to, the New Domain window automatically displays targets with LUNs unused by the environment. Note that LUNs external to the environment are also displayed. You can use the Discover Targets options to add LUNs on many targets, or multiple paths to the same LUNs.
   2. Enter the fully qualified domain name or IP address of the iSCSI host in the Address field.
   3. Enter the port to connect to the host on when browsing for targets in the Port field. The default is 3260.
   4. If the Challenge Handshake Authentication Protocol (CHAP) is being used to secure the storage, select the User Authentication check box. Enter the CHAP user name and CHAP password.
   5. Click the Discover button.

   6. Select the target to use from the discovery results and click the Login button. Alternatively, click the Login All to log in to all of the discovered targets.

      Important

      If more than one path access is required, ensure to discover and log in to the target through all the required paths. Modifying a storage domain to add additional paths is currently not supported.

9. Click the + button next to the desired target. This will expand the entry and display all unused LUNs attached to the target.
10. Select the check box for each LUN that you are using to create the storage domain.

11. Optionally, you can configure the advanced parameters.

    1. Click Advanced Parameters.
    2. Enter a percentage value into the Warning Low Space Indicator field. If the free space available on the storage domain is below this percentage, warning messages are displayed to the user and logged.
    3. Enter a GB value into the Critical Space Action Blocker field. If the free space available on the storage domain is below this value, error messages are displayed to the user and logged, and any new action that consumes space, even temporarily, will be blocked.
    4. Select the Wipe After Delete check box to enable the wipe after delete option. You can edit this option after creating the domain, but doing so does not change the wipe after delete property of disks that already exist.
    5. Select the Discard After Delete check box to enable the discard after delete option. You can edit this option after creating the domain. This option is only available to block storage domains.
12. Click OK to create the storage domain and close the window.

Procedure

1. Install the iSCSI initiator and configure discovery and setup.

   1. Install the iSCSI initiator driver and MPIO tools.
   2. Launch the MPIO program, click the Discover Multi-Paths tab, check the Add support for iSCSI devices box, and click Add.
   3. Reboot the MPIO program.

   4. On the iSCSI Initiator Properties window, on the Discovery tab , add a target portal. Enter the IP address or DNS name and Port of the Ceph iSCSI gateway:

      

      View larger image

      

   5. On the Targets tab , select the target and click Connect :

      

      View larger image

      

   6. On the Connect To Target window, select the Enable multi-path option , and click the Advanced button :

      

      View larger image

      

   7. Under the Connect using section, select a Target portal IP . Select Enable CHAP login on and enter the Name and Target secret values from the Ceph iSCSI client credentials section, and click OK :

      

      View larger image

      
      Important

      Windows Server 2016 does not accept a CHAP secret less than 12 bytes.

   8. Repeat the previous two steps for each target portal defined when setting up the iSCSI gateway.

   9. If the initiator name is different than the initiator name used during the initial setup, rename the initiator name. From iSCSI Initiator Properties window, on the Configuration tab , click the Change button to rename the initiator name.

      

      View larger image

      

2. Set up multipath I/O. In PowerShell, use the PDORemovePeriod command to set the MPIO load balancing policy and the mpclaim command to set the load balancing policy. The iSCSI Initiator Tool configures the remaining options.

   Note

   Red Hat recommends increasing the PDORemovePeriod option to 120 seconds from PowerShell. You might need to adjust this value based on the application. When all paths are down, and 120 seconds expires, the operating system starts failing I/O requests.

   Set-MPIOSetting -NewPDORemovePeriod 120

   Copy to Clipboard Toggle word wrap

   1. Set the failover policy

      mpclaim.exe -l -m 1

      Copy to Clipboard Toggle word wrap

   2. Verify the failover policy

      mpclaim -s -m
      MSDSM-wide Load Balance Policy: Fail Over Only

      Copy to Clipboard Toggle word wrap

   3. Using the iSCSI Initiator tool, from the Targets tab click on the Devices…​ button :

      

      View larger image

      

   4. From the Devices window, select a disk and click the MPIO…​ button :

      

      View larger image

      

   5. The Device Details window displays the paths to each target portal. The Load Balancing Policy Fail Over Only must be selected.

      

      View larger image

      

   6. View the multipath configuration from the PowerShell:

      mpclaim -s -d MPIO_DISK_ID

      Copy to Clipboard Toggle word wrap

      Replace MPIO_DISK_ID with the appropriate disk identifier.

      Note

      There is one Active/Optimized path which is the path to the iSCSI gateway node that owns the LUN, and there is an Active/Unoptimized path for each other iSCSI gateway node.

      

      View larger image

      

3. Optionally, tune the settings. Consider using the following registry settings:

   • Windows Disk Timeout

     Key

     HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Disk

     Copy to Clipboard Toggle word wrap

     Value

     TimeOutValue = 65

     Copy to Clipboard Toggle word wrap

   • Microsoft iSCSI Initiator Driver

     Key

     HKEY_LOCAL_MACHINE\\SYSTEM\CurrentControlSet\Control\Class\{4D36E97B-E325-11CE-BFC1-08002BE10318}\<Instance_Number>\Parameters

     Copy to Clipboard Toggle word wrap

     Values

     LinkDownTime = 25
     SRBTimeoutDelta = 15

     Copy to Clipboard Toggle word wrap

Procedure

1. Disable HardwareAcceleratedMove (XCOPY):

   > esxcli system settings advanced set --int-value 0 --option /DataMover/HardwareAcceleratedMove

   Copy to Clipboard Toggle word wrap

2. Enable the iSCSI software. From the Navigator pane, click Storage . Select the Adapters tab . Click on Configure iSCSI :

   

   View larger image

   

3. Verify the initiator name in the Name & alias section .

   

   View larger image

   

4. If the initiator name is different than the initiator name used when creating the client during the initial setup using gwcli, change the initiator name: From the VMware ESX host, use these esxcli commands.

   1. Get the adapter name for the iSCSI software:

      > esxcli iscsi adapter list
      > Adapter  Driver     State   UID            Description
      > -------  ---------  ------  -------------  ----------------------
      > vmhba64  iscsi_vmk  online  iscsi.vmhba64  iSCSI Software Adapter

      Copy to Clipboard Toggle word wrap

   2. Set the initiator name:

      Syntax

      > esxcli iscsi adapter set -A ADAPTOR_NAME -n INITIATOR_NAME

      Copy to Clipboard Toggle word wrap

      Example

      > esxcli iscsi adapter set -A vmhba64 -n iqn.1994-05.com.redhat:rh7-client

      Copy to Clipboard Toggle word wrap

5. Configure CHAP. Expand the CHAP authentication section . Select “Do not use CHAP unless required by target” . Enter the CHAP Name and Secret credentials that were used in the initial setup. Verify the Mutual CHAP authentication section has “Do not use CHAP” selected.

   

   View larger image

   
   Warning

   Due to a bug in the VMware Host Client, the CHAP settings are not used initially. On the Ceph iSCSI gateway node, the kernel logs include the following errors as an indication of this bug:

   > kernel: CHAP user or password not set for Initiator ACL
   > kernel: Security negotiation failed.
   > kernel: iSCSI Login negotiation failed.

   Copy to Clipboard Toggle word wrap

   To work around this bug, configure the CHAP settings using the esxcli command. The authname argument is the Name in the vSphere Web Client:

   > esxcli iscsi adapter auth chap set --direction=uni --authname=myiscsiusername --secret=myiscsipassword --level=discouraged -A vmhba64

   Copy to Clipboard Toggle word wrap

6. Configure the iSCSI settings. Expand Advanced settings . Set the RecoveryTimeout value to 25 .

   

   View larger image

   

7. Set the discovery address. In the Dynamic targets section , click Add dynamic target . Under Address add an IP addresses for one of the Ceph iSCSI gateways. Only one IP address needs to be added. Finally, click the Save configuration button . From the main interface, on the Devices tab, you will see the RBD image.

   

   View larger image

   
   Note

   LUN is configured automatically, using the ALUA SATP and MRU PSP. Do not use other SATPs and PSPs. You can verify this by the esxcli command:

   Syntax

   esxcli storage nmp path list -d eui.DEVICE_ID

   Copy to Clipboard Toggle word wrap

   Replace DEVICE_ID with the appropriate device identifier.

8. Verify that multipathing has been set up correctly.

   1. List the devices:

      Example

      > esxcli storage nmp device list | grep iSCSI
         Device Display Name: LIO-ORG iSCSI Disk (naa.6001405f8d087846e7b4f0e9e3acd44b)
         Device Display Name: LIO-ORG iSCSI Disk (naa.6001405057360ba9b4c434daa3c6770c)

      Copy to Clipboard Toggle word wrap

   2. Get the multipath information for the Ceph iSCSI disk from the previous step:

      Example

      > esxcli storage nmp path list -d naa.6001405f8d087846e7b4f0e9e3acd44b
      
      iqn.2005-03.com.ceph:esx1-00023d000001,iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw,t,1-naa.6001405f8d087846e7b4f0e9e3acd44b
         Runtime Name: vmhba64:C0:T0:L0
         Device: naa.6001405f8d087846e7b4f0e9e3acd44b
         Device Display Name: LIO-ORG iSCSI Disk (naa.6001405f8d087846e7b4f0e9e3acd44b)
         Group State: active
         Array Priority: 0
         Storage Array Type Path Config: {TPG_id=1,TPG_state=AO,RTP_id=1,RTP_health=UP}
         Path Selection Policy Path Config: {current path; rank: 0}
      
      iqn.2005-03.com.ceph:esx1-00023d000002,iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw,t,2-naa.6001405f8d087846e7b4f0e9e3acd44b
         Runtime Name: vmhba64:C1:T0:L0
         Device: naa.6001405f8d087846e7b4f0e9e3acd44b
         Device Display Name: LIO-ORG iSCSI Disk (naa.6001405f8d087846e7b4f0e9e3acd44b)
         Group State: active unoptimized
         Array Priority: 0
         Storage Array Type Path Config: {TPG_id=2,TPG_state=ANO,RTP_id=2,RTP_health=UP}
         Path Selection Policy Path Config: {non-current path; rank: 0}

      Copy to Clipboard Toggle word wrap

      From the example output, each path has an iSCSI or SCSI name with the following parts:

      Initiator name = iqn.2005-03.com.ceph:esx1 ISID = 00023d000002 Target name = iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw Target port group = 2 Device id = naa.6001405f8d087846e7b4f0e9e3acd44b

      The Group State value of active indicates this is the Active-Optimized path to the iSCSI gateway. The gwcli command lists the active as the iSCSI gateway owner. The rest of the paths have the Group State value of unoptimized and are the failover path, if the active path goes into a dead state.

9. To match all paths to their respective iSCSI gateways:

   Example

   > esxcli iscsi session connection list
   vmhba64,iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw,00023d000001,0
      Adapter: vmhba64
      Target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw
      ISID: 00023d000001
      CID: 0
      DataDigest: NONE
      HeaderDigest: NONE
      IFMarker: false
      IFMarkerInterval: 0
      MaxRecvDataSegmentLength: 131072
      MaxTransmitDataSegmentLength: 262144
      OFMarker: false
      OFMarkerInterval: 0
      ConnectionAddress: 10.172.19.21
      RemoteAddress: 10.172.19.21
      LocalAddress: 10.172.19.11
      SessionCreateTime: 08/16/18 04:20:06
      ConnectionCreateTime: 08/16/18 04:20:06
      ConnectionStartTime: 08/16/18 04:30:45
      State: logged_in
   
   vmhba64,iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw,00023d000002,0
      Adapter: vmhba64
      Target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw
      ISID: 00023d000002
      CID: 0
      DataDigest: NONE
      HeaderDigest: NONE
      IFMarker: false
      IFMarkerInterval: 0
      MaxRecvDataSegmentLength: 131072
      MaxTransmitDataSegmentLength: 262144
      OFMarker: false
      OFMarkerInterval: 0
      ConnectionAddress: 10.172.19.22
      RemoteAddress: 10.172.19.22
      LocalAddress: 10.172.19.12
      SessionCreateTime: 08/16/18 04:20:06
      ConnectionCreateTime: 08/16/18 04:20:06
      ConnectionStartTime: 08/16/18 04:30:41
      State: logged_in

   Copy to Clipboard Toggle word wrap

   Match the path name with the ISID value, and the RemoteAddress value is the IP address of the owning iSCSI gateway.

Procedure

1. On the new iSCSI gateway nodes, enable the Red Hat Ceph Storage Tools repository:

   Red Hat Enterprise Linux 7

   [root@iscsigw ~]# subscription-manager repos --enable=rhel-7-server-rhceph-4-tools-rpms

   Copy to Clipboard Toggle word wrap

   Red Hat Enterprise Linux 8

   [root@iscsigw ~]# subscription-manager repos --enable=rhceph-4-tools-for-rhel-8-x86_64-rpms

   Copy to Clipboard Toggle word wrap

2. Install the ceph-iscsi-config package:

   [root@iscsigw ~]# yum install ceph-iscsi-config

   Copy to Clipboard Toggle word wrap

3. Append to the list in /etc/ansible/hosts file for the gateway group:

   Example

   [iscsigws]
   ...
   ceph-igw-3
   ceph-igw-4

   Copy to Clipboard Toggle word wrap

   Note

   If colocating the iSCSI gateway with an OSD node, add the OSD node to the [iscsigws] section.

4. Change to the ceph-ansible directory:

   [admin@ansible ~]$ cd /usr/share/ceph-ansible

   Copy to Clipboard Toggle word wrap

5. On the Ansible administration node, run the appropriate Ansible playbook:

   • Bare-metal deployments:

     [admin@ansible ceph-ansible]$ ansible-playbook site.yml -i hosts

     Copy to Clipboard Toggle word wrap

   • Container deployments:

     [admin@ansible ceph-ansible]$ ansible-playbook site-container.yml -i hosts

     Copy to Clipboard Toggle word wrap
   Important

   Providing IP addresses for the gateway_ip_list option is required. You cannot use a mix of IPv4 and IPv6 addresses.

6. From the iSCSI initiators, re-login to use the newly added iSCSI gateways.

Procedure

1. If the Ceph iSCSI gateway is not colocated on an OSD node, copy the Ceph configuration files, located in the /etc/ceph/ directory, from a running Ceph node in the storage cluster to the new iSCSI Gateway node. The Ceph configuration files must exist on the iSCSI gateway node under the /etc/ceph/ directory.
2. Install and configure the Ceph command-line interface.

3. On the new iSCSI gateway nodes, enable the Red Hat Ceph Storage Tools repository:

   Red Hat Enterprise Linux 7

   [root@iscsigw ~]# subscription-manager repos --enable=rhel-7-server-rhceph-4-tools-rpms

   Copy to Clipboard Toggle word wrap

   Red Hat Enterprise Linux 8

   [root@iscsigw ~]# subscription-manager repos --enable=rhceph-4-tools-for-rhel-8-x86_64-rpms

   Copy to Clipboard Toggle word wrap

4. Install the ceph-iscsi, and tcmu-runner packages:

   Red Hat Enterprise Linux 7

   [root@iscsigw ~]# yum install ceph-iscsi tcmu-runner

   Copy to Clipboard Toggle word wrap

   Red Hat Enterprise Linux 8

   [root@iscsigw ~]# dnf install ceph-iscsi tcmu-runner

   Copy to Clipboard Toggle word wrap

   1. If needed, install the openssl package:

      Red Hat Enterprise Linux 7

      [root@iscsigw ~]# yum install openssl

      Copy to Clipboard Toggle word wrap

      Red Hat Enterprise Linux 8

      [root@iscsigw ~]# dnf install openssl

      Copy to Clipboard Toggle word wrap

5. On one of the existing iSCSI gateway nodes, edit the /etc/ceph/iscsi-gateway.cfg file and append the trusted_ip_list option with the new IP addresses for the new iSCSI gateway nodes. For example:

   [config]
   ...
   trusted_ip_list = 10.172.19.21,10.172.19.22,10.172.19.23,10.172.19.24

   Copy to Clipboard Toggle word wrap

6. Copy the updated /etc/ceph/iscsi-gateway.cfg file to all the iSCSI gateway nodes.

   Important

   The iscsi-gateway.cfg file must be identical on all iSCSI gateway nodes.

7. Optionally, if using SSL, also copy the ~/ssl-keys/iscsi-gateway.crt, ~/ssl-keys/iscsi-gateway.pem, ~/ssl-keys/iscsi-gateway-pub.key, and ~/ssl-keys/iscsi-gateway.key files from one of the existing iSCSI gateway nodes to the /etc/ceph/ directory on the new iSCSI gateway nodes.

8. Enable and start the API service on the new iSCSI gateway nodes:

   [root@iscsigw ~]# systemctl enable rbd-target-api
   [root@iscsigw ~]# systemctl start rbd-target-api

   Copy to Clipboard Toggle word wrap

9. Start the iSCSI gateway command-line interface:

   [root@iscsigw ~]# gwcli

   Copy to Clipboard Toggle word wrap

10. Creating the iSCSI gateways using either IPv4 or IPv6 addresses:

    Syntax

    >/iscsi-target create iqn.2003-01.com.redhat.iscsi-gw:_TARGET_NAME_
    > goto gateways
    > create ISCSI_GW_NAME IP_ADDR_OF_GW
    > create ISCSI_GW_NAME IP_ADDR_OF_GW

    Copy to Clipboard Toggle word wrap

    Example

    >/iscsi-target create iqn.2003-01.com.redhat.iscsi-gw:ceph-igw
    > goto gateways
    > create ceph-gw-3 10.172.19.23
    > create ceph-gw-4 10.172.19.24

    Copy to Clipboard Toggle word wrap

    Important

    You cannot use a mix of IPv4 and IPv6 addresses.

11. From the iSCSI initiators, re-login to use the newly added iSCSI gateways.