Administration Guide

• Using a node in a trusted storage pool
• Using a machine outside the trusted storage pool

• Using the [backend-setup] module
• Creating Physical Volume (PV), Volume Group (VG), and Logical Volume (LV) individually

• [hosts]
• [devices]
• [disktype]
• [diskcount]
• [stripesize]
• [vgs]
• [pools]
• [lvs]
• [mountpoints]
• [peer]
• [clients]
• [volume]
• [backend-setup]
• [pv]
• [vg]
• [lv]
• [RH-subscription]
• [yum]
• [shell]
• [update-file]
• [service]
• [script]
• [firewalld]

• hosts

  This is a mandatory section which contains the IP address or hostname of the machines in the trusted storage pool. Each hostname or IP address should be listed in a separate line.

  For example:

  [hosts]
  10.0.0.1
  10.0.0.2

• devices

  This is a generic section and is applicable to all the hosts listed in the [hosts] section. However, if sections of hosts such as the [hostname] or [IP-address] is present, then the data in the generic sections like [devices] is ignored. Host specific data take precedence. This is an optional section.

  For example:

  [devices]
  /dev/sda
  /dev/sdb

  Note

  When configuring the backend setup, the devices should be either listed in this section or in the host specific section.
• disktype

  This section specifies the disk configuration that is used while setting up the backend. gdeploy supports RAID 10, RAID 6, RAID 5, and JBOD configurations. This is an optional section and if the field is left empty, JBOD is taken as the default configuration. Valid values for this field are raid10, raid6, raid5, and jbod.

  For example:

  [disktype]
  raid6

• diskcount

  This section specifies the number of data disks in the setup. This is a mandatory field if a RAID disk type is specified under [disktype]. If the [disktype] is JBOD the [diskcount] value is ignored. This parameter is host specific.

  For example:

  [diskcount]
  10

• stripesize

  This section specifies the stripe_unit size in KB.

  Case 1: This field is not necessary if the [disktype] is JBOD, and any given value will be ignored.
  Case 2: This is a mandatory field if [disktype] is specified as RAID 5 or RAID 6.
  For [disktype] RAID 10, the default value is taken as 256KB. Red Hat does not recommend changing this value. If you specify any other value the following warning is displayed:

  "Warning: We recommend a stripe unit size of 256KB for RAID 10"

  Note

  Do not add any suffixes like K, KB, M, etc. This parameter is host specific and can be added in the hosts section.
  For example:

  [stripesize]
  128

• vgs

  This section is deprecated in gdeploy 2.0. Please see [backend-setup] for more details for gdeploy 2.0. This section specifies the volume group names for the devices listed in [devices]. The number of volume groups in the [vgs] section should match the one in [devices]. If the volume group names are missing, the volume groups will be named as GLUSTER_vg{1, 2, 3, ...} as default.

  For example:

  [vgs]
  CUSTOM_vg1
  CUSTOM_vg2

• pools

  This section is deprecated in gdeploy 2.0. Please see [backend-setup] for more details for gdeploy 2.0. This section specifies the pool names for the volume groups specified in the [vgs] section. The number of pools listed in the [pools] section should match the number of volume groups in the [vgs] section. If the pool names are missing, the pools will be named as GLUSTER_pool{1, 2, 3, ...}.

  For example:

  [pools]
  CUSTOM_pool1
  CUSTOM_pool2

• lvs

  This section is deprecated in gdeploy 2.0. Please see [backend-setup] for more details for gdeploy 2.0. This section provides the logical volume names for the volume groups specified in [vgs]. The number of logical volumes listed in the [lvs] section should match the number of volume groups listed in [vgs]. If the logical volume names are missing, it is named as GLUSTER_lv{1, 2, 3, ...}.

  For example:

  [lvs]
  CUSTOM_lv1
  CUSTOM_lv2

• mountpoints

  This section is deprecated in gdeploy 2.0. Please see [backend-setup] for more details for gdeploy 2.0. This section specifies the brick mount points for the logical volumes. The number of mount points should match the number of logical volumes specified in [lvs] If the mount points are missing, the mount points will be names as /gluster/brick{1, 2, 3…}.

  For example:

  [mountpoints]
  /rhgs/brick1
  /rhgs/brick2

• peer

  This section specifies the configurations for the Trusted Storage Pool management (TSP). This section helps in making all the hosts specified in the [hosts] section to either probe each other to create the trusted storage pool or detach all of them from the trusted storage pool. The only option in this section is the option names 'action' which can have it's values to be either probe or detach.

  For example:

  [peer]
  action=probe

• clients

  This section specifies the client hosts and client_mount_points to mount the gluster storage volume created. The 'action' option is to be specified for the framework to determine the action that has to be performed. The options are 'mount' and 'unmount'. The Client hosts field is mandatory. If the mount points are not specified, default will be taken as /mnt/gluster for all the hosts.

  The option fstype specifies how the gluster volume is to be mounted. Default is glusterfs (FUSE mount). The volume can also be mounted as NFS. Each client can have different types of volume mount, which has to be specified with a comma separated. The following fields are included:

  * action
  * hosts
  * fstype
  * client_mount_points

  For example:

  [clients]
  action=mount
  hosts=10.0.0.10
  fstype=nfs
  options=vers=3
  client_mount_points=/mnt/rhs

• volume

  The section specifies the configuration options for the volume. The following fields are included in this section:

  * action
  * volname
  * transport
  * replica
  * replica_count
  * disperse
  * disperse_count
  * redundancy_count
  * force

  • action

    This option specifies what action must be performed in the volume. The choices can be [create, delete, add-brick, remove-brick].

    create: This choice is used to create a volume.
    delete: If the delete choice is used, all the options other than 'volname' will be ignored.
    add-brick or remove-brick: If the add-brick or remove-brick is chosen, extra option bricks with a comma separated list of brick names(in the format <hostname>:<brick path> should be provided. In case of remove-brick, state option should also be provided specifying the state of the volume after brick removal.
  • volname

    This option specifies the volume name. Default name is glustervol

    Note

    • In case of a volume operation, the 'hosts' section can be omitted, provided volname is in the format <hostname>:<volname>, where hostname is the hostname / IP of one of the nodes in the cluster
    • Only single volume creation/deletion/configuration is supported.
  • transport

    This option specifies the transport type. Default is tcp. Options are tcp or rdma or tcp,rdma.

  • replica

    This option will specify if the volume should be of type replica. options are yes and no. Default is no. If 'replica' is provided as yes, the 'replica_count' should be provided.

  • disperse

    This option specifies if the volume should be of type disperse. Options are yes and no. Default is no.

  • disperse_count

    This field is optional even if 'disperse' is yes. If not specified, the number of bricks specified in the command line is taken as the disperse_count value.

  • redundancy_count

    If this value is not specified, and if 'disperse' is yes, it's default value is computed so that it generates an optimal configuration.

  • force

    This is an optional field and can be used during volume creation to forcefully create the volume.

  For example:

  [volname]
  action=create
  volname=glustervol
  transport=tcp,rdma
  replica=yes
  replica_count=3
  force=yes

• backend-setup

  Available in gdeploy 2.0. This section sets up the backend for using with GlusterFS volume. If more than one backend-setup has to be done, they can be done by numbering the section like [backend-setup1], [backend-setup2], ...

  backend-setup section supports the following variables:
  • devices: This replaces the [pvs] section in gdeploy 1.x. devices variable lists the raw disks which should be used for backend setup. For example:

    [backend-setup]
    devices=sda,sdb,sdc

    This is a mandatory field.
  • dalign:
    The Logical Volume Manager can use a portion of the physical volume for storing its metadata while the rest is used as the data portion. Align the I/O at the Logical Volume Manager (LVM) layer using the dalign option while creating the physical volume. For example:

    [backend-setup]
    devices=sdb,sdc,sdd,sde
    dalign=256k

    For JBOD, use an alignment value of 256K. For hardware RAID, the alignment value should be obtained by multiplying the RAID stripe unit size with the number of data disks. If 12 disks are used in a RAID 6 configuration, the number of data disks is 10; on the other hand, if 12 disks are used in a RAID 10 configuration, the number of data disks is 6.
    The following example is appropriate for 12 disks in a RAID 6 configuration with a stripe unit size of 128 KiB:

    [backend-setup]
    devices=sdb,sdc,sdd,sde
    dalign=1280k

    The following example is appropriate for 12 disks in a RAID 10 configuration with a stripe unit size of 256 KiB:

    [backend-setup]
    devices=sdb,sdc,sdd,sde
    dalign=1536k

    To view the previously configured physical volume settings for the dalign option, run the pvs -o +pe_start device command. For example:

    # pvs -o +pe_start disk
    PV         VG   Fmt  Attr PSize PFree 1st PE
    /dev/sdb        lvm2 a--  9.09t 9.09t   1.25m

    You can also set the dalign option in the PV section.
  • vgs: This is an optional variable. This variable replaces the [vgs] section in gdeploy 1.x. vgs variable lists the names to be used while creating volume groups. The number of VG names should match the number of devices or should be left blank. gdeploy will generate names for the VGs. For example:

    [backend-setup]
    devices=sda,sdb,sdc
    vgs=custom_vg1,custom_vg2,custom_vg3

    A pattern can be provided for the vgs like custom_vg{1..3}, this will create three vgs.

    [backend-setup]
    devices=sda,sdb,sdc
    vgs=custom_vg{1..3}

  • pools: This is an optional variable. The variable replaces the [pools] section in gdeploy 1.x. pools lists the thin pool names for the volume.

    [backend-setup]
    devices=sda,sdb,sdc
    vgs=custom_vg1,custom_vg2,custom_vg3
    pools=custom_pool1,custom_pool2,custom_pool3

    Similar to vg, pattern can be provided for thin pool names. For example custom_pool{1..3}
  • lvs: This is an optional variable. This variable replaces the [lvs] section in gdeploy 1.x. lvs lists the logical volume name for the volume.

    [backend-setup]
    devices=sda,sdb,sdc
    vgs=custom_vg1,custom_vg2,custom_vg3
    pools=custom_pool1,custom_pool2,custom_pool3
    lvs=custom_lv1,custom_lv2,custom_lv3

    Patterns for LV can be provided similar to vg. For example custom_lv{1..3}.
  • mountpoints: This variable deprecates the [mountpoints] section in gdeploy 1.x. Mountpoints lists the mount points where the logical volumes should be mounted. Number of mount points should be equal to the number of logical volumes. For example:

    [backend-setup]
    devices=sda,sdb,sdc
    vgs=custom_vg1,custom_vg2,custom_vg3
    pools=custom_pool1,custom_pool2,custom_pool3
    lvs=custom_lv1,custom_lv2,custom_lv3
    mountpoints=/gluster/data1,/gluster/data2,/gluster/data3

  • ssd - This variable is set if caching has to be added. For example, the backed setup with ssd for caching should be:

    [backend-setup]
    ssd=sdc
    vgs=RHS_vg1
    datalv=lv_data
    cachedatalv=lv_cachedata:1G
    cachemetalv=lv_cachemeta:230G

    Note

    Specifying the name of the data LV is necessary while adding SSD. Make sure the datalv is created already. Otherwise ensure to create it in one of the earlier `backend-setup’ sections.
• PV

  Available in gdeploy 2.0. If the user needs to have more control over setting up the backend, and does not want to use backend-setup section, then pv, vg, and lv modules are to be used. The pv module supports the following variables.

  • action: Mandatory. Supports two values, 'create' and 'resize'
    Example: Creating physical volumes

    [pv]
    action=create
    devices=vdb,vdc,vdd

    Example: Creating physical volumes on a specific host

    [pv:10.0.5.2]
    action=create
    devices=vdb,vdc,vdd

  • devices: Mandatory. The list of devices to use for pv creation.
  • expand: Used when action=resize.
    Example: Expanding an already created pv

    [pv]
    action=resize
    devices=vdb
    expand=yes

  • shrink: Used when action=resize.
    Example: Shrinking an already created pv

    [pv]
    action=resize
    devices=vdb
    shrink=100G

  • dalign:
    The Logical Volume Manager can use a portion of the physical volume for storing its metadata while the rest is used as the data portion. Align the I/O at the Logical Volume Manager (LVM) layer using the dalign option while creating the physical volume. For example:

    [pv]
    action=create
    devices=sdb,sdc,sdd,sde
    dalign=256k

    For JBOD, use an alignment value of 256K. For hardware RAID, the alignment value should be obtained by multiplying the RAID stripe unit size with the number of data disks. If 12 disks are used in a RAID 6 configuration, the number of data disks is 10; on the other hand, if 12 disks are used in a RAID 10 configuration, the number of data disks is 6.
    The following example is appropriate for 12 disks in a RAID 6 configuration with a stripe unit size of 128 KiB:

    [pv]
    action=create
    devices=sdb,sdc,sdd,sde
    dalign=1280k

    The following example is appropriate for 12 disks in a RAID 10 configuration with a stripe unit size of 256 KiB:

    [pv]
    action=create
    devices=sdb,sdc,sdd,sde
    dalign=1536k

    To view the previously configured physical volume settings for the dalign option, run the pvs -o +pe_start device command. For example:

    # pvs -o +pe_start disk
    PV         VG   Fmt  Attr PSize PFree 1st PE
    /dev/sdb        lvm2 a--  9.09t 9.09t   1.25m

    You can also set the dalign option in the backend-setup section.
• VG

  Available in gdeploy 2.0. This module is used to create and extend volume groups. The vg module supports the following variables.

  • action - Action can be one of create or extend.
  • pvname - PVs to use to create the volume. For more than one PV use comma separated values.
  • vgname - The name of the vg. If no name is provided GLUSTER_vg will be used as default name.
  • one-to-one - If set to yes, one-to-one mapping will be done between pv and vg.
  If action is set to extend, the vg will be extended to include pv provided.
  Example1: Create a vg named images_vg with two PVs

  [vg]
  action=create
  vgname=images_vg
  pvname=sdb,sdc

  Example2: Create two vgs named rhgs_vg1 and rhgs_vg2 with two PVs

  [vg]
  action=create
  vgname=rhgs_vg
  pvname=sdb,sdc
  one-to-one=yes

  Example3: Extend an existing vg with the given disk.

  [vg]
  action=extend
  vgname=rhgs_images
  pvname=sdc

• LV

  Available in gdeploy 2.0. This module is used to create, setup-cache, and convert logical volumes. The lv module supports the following variables:

  action - The action variable allows three values `create’, `setup-cache’, `convert’, and `change’. If the action is 'create', the following options are supported:
  • lvname: The name of the logical volume, this is an optional field. Default is GLUSTER_lv
  • poolname - Name of the thinpool volume name, this is an optional field. Default is GLUSTER_pool
  • lvtype - Type of the logical volume to be created, allowed values are `thin’ and `thick’. This is an optional field, default is thick.
  • size - Size of the logical volume volume. Default is to take all available space on the vg.
  • extent - Extent size, default is 100%FREE
  • force - Force lv create, do not ask any questions. Allowed values `yes’, `no’. This is an optional field, default is yes.
  • vgname - Name of the volume group to use.
  • pvname - Name of the physical volume to use.
  • chunksize - The size of the chunk unit used for snapshots, cache pools, and thin pools. By default this is specified in kilobytes. For RAID 5 and 6 volumes, gdeploy calculates the default chunksize by multiplying the stripe size and the disk count. For RAID 10, the default chunksize is 256 KB. See Section 20.2, “Brick Configuration” for details.

    Warning

    Red Hat recommends using at least the default chunksize. If the chunksize is too small and your volume runs out of space for metadata, the volume is unable to create data. This includes the data required to increase the size of the metadata pool or to migrate data away from a volume that has run out of metadata space. Red Hat recommends monitoring your logical volumes to ensure that they are expanded or more storage created before metadata volumes become completely full.
  • poolmetadatasize - Sets the size of pool's metadata logical volume. Allocate the maximum chunk size (16 GiB) if possible. If you allocate less than the maximum, allocate at least 0.5% of the pool size to ensure that you do not run out of metadata space.

    Warning

    If your metadata pool runs out of space, you cannot create data. This includes the data required to increase the size of the metadata pool or to migrate data away from a volume that has run out of metadata space. Monitor your metadata pool using the lvs -o+metadata_percent command and ensure that it does not run out of space.
  • virtualsize - Creates a thinly provisioned device or a sparse device of the given size
  • mkfs - Creates a filesystem of the given type. Default is to use xfs.
  • mkfs-opts - mkfs options.
  • mount - Mount the logical volume.
  If the action is setup-cache, the below options are supported:
  • ssd - Name of the ssd device. For example sda/vda/ … to setup cache.
  • vgname - Name of the volume group.
  • poolname - Name of the pool.
  • cache_meta_lv - Due to requirements from dm-cache (the kernel driver), LVM further splits the cache pool LV into two devices - the cache data LV and cache metadata LV. Provide the cache_meta_lv name here.
  • cache_meta_lvsize - Size of the cache meta lv.
  • cache_lv - Name of the cache data lv.
  • cache_lvsize - Size of the cache data.
  • force - Force
  If the action is convert, the below options are supported:
  • lvtype - type of the lv, available options are thin and thick
  • force - Force the lvconvert, default is yes.
  • vgname - Name of the volume group.
  • poolmetadata - Specifies cache or thin pool metadata logical volume.
  • cachemode - Allowed values writeback, writethrough. Default is writethrough.
  • cachepool - This argument is necessary when converting a logical volume to a cache LV. Name of the cachepool.
  • lvname - Name of the logical volume.
  • chunksize - The size of the chunk unit used for snapshots, cache pools, and thin pools. By default this is specified in kilobytes. For RAID 5 and 6 volumes, gdeploy calculates the default chunksize by multiplying the stripe size and the disk count. For RAID 10, the default chunksize is 256 KB. See Section 20.2, “Brick Configuration” for details.

    Warning

    Red Hat recommends using at least the default chunksize. If the chunksize is too small and your volume runs out of space for metadata, the volume is unable to create data. Red Hat recommends monitoring your logical volumes to ensure that they are expanded or more storage created before metadata volumes become completely full.
  • poolmetadataspare - Controls creation and maintanence of pool metadata spare logical volume that will be used for automated pool recovery.
  • thinpool - Specifies or converts logical volume into a thin pool's data volume. Volume’s name or path has to be given.
  If the action is change, the below options are supported:
  • lvname - Name of the logical volume.
  • vgname - Name of the volume group.
  • zero - Set zeroing mode for thin pool.
  Example 1: Create a thin LV

  [lv]
  action=create
  vgname=RHGS_vg1
  poolname=lvthinpool
  lvtype=thinpool
  poolmetadatasize=200MB
  chunksize=1024k
  size=30GB

  Example 2: Create a thick LV

  [lv]
  action=create
  vgname=RHGS_vg1
  lvname=engine_lv
  lvtype=thick
  size=10GB
  mount=/rhgs/brick1

  If there are more than one LVs, then the LVs can be created by numbering the LV sections, like [lv1], [lv2] …
• RH-subscription

  Available in gdeploy 2.0. This module is used to subscribe, unsubscribe, attach, enable repos etc. The RH-subscription module allows the following variables:

  This module is used to subscribe, unsubscribe, attach, enable repos etc. The RH-subscription module allows the following variables:
  If the action is register, the following options are supported:
  • username/activationkey: Username or activationkey.
  • password/activationkey: Password or activation key
  • auto-attach: true/false
  • pool: Name of the pool.
  • repos: Repos to subscribe to.
  • disable-repos: Repo names to disable. Leaving this option blank will disable all the repos.
  • ignore_register_errors: If set to no, gdeploy will exit if system registration fails.
  • If the action is attach-pool the following options are supported:
    pool - Pool name to be attached.
    ignore_attach_pool_errors - If set to no, gdeploy fails if attach-pool fails.
  • If the action is enable-repos the following options are supported:
    repos - List of comma separated repos that are to be subscribed to.
    ignore_enable_errors - If set to no, gdeploy fails if enable-repos fail.
  • If the action is disable-repos the following options are supported:
    repos - List of comma separated repos that are to be subscribed to.
    ignore_disable_errors - If set to no, gdeploy fails if disable-repos fail
  • If the action is unregister the systems will be unregistered.
    ignore_unregister_errors - If set to no, gdeploy fails if unregistering fails.
  Example 1: Subscribe to Red Hat Subscription network:

  [RH-subscription1]
  action=register
  username=qa@redhat.com
  password=<passwd>
  pool=<pool>
  ignore_register_errors=no

  Example 2: Disable all the repos:

  [RH-subscription2]
  action=disable-repos
  repos=

  Example 3: Enable a few repos

  [RH-subscription3]
  action=enable-repos
  repos=rhel-7-server-rpms,rh-gluster-3-for-rhel-7-server-rpms,rhel-7-server-rhev-mgmt-agent-rpms
  ignore_enable_errors=no

• yum

  Available in gdeploy 2.0. This module is used to install or remove rpm packages, with the yum module we can add repos as well during the install time.

  The action variable allows two values `install’ and `remove’.
  If the action is install the following options are supported:
  • packages - Comma separated list of packages that are to be installed.
  • repos - The repositories to be added.
  • gpgcheck - yes/no values have to be provided.
  • update - Whether yum update has to be initiated.
  If the action is remove then only one option has to be provided:
  • remove - The comma separated list of packages to be removed.
  For example

  [yum1]
  action=install
  gpgcheck=no
  # Repos should be an url; eg: http://repo-pointing-glusterfs-builds
  repos=<glusterfs.repo>,<vdsm.repo>
  packages=vdsm,vdsm-gluster,ovirt-hosted-engine-setup,screen,gluster-nagios-addons,xauth
  update=yes

  Install a package on a particular host.

  [yum2:host1]
  action=install
  gpgcheck=no
  packages=rhevm-appliance

• shell

  Available in gdeploy 2.0. This module allows user to run shell commands on the remote nodes.

  Currently shell provides a single action variable with value execute. And a command variable with any valid shell command as value.
  The below command will execute vdsm-tool on all the nodes.

  [shell]
  action=execute
  command=vdsm-tool configure --force

• update-file

  Available in gdeploy 2.0. update-file module allows users to copy a file, edit a line in a file, or add new lines to a file. action variable can be any of copy, edit, or add.

  When the action variable is set to copy, the following variables are supported.
  • src - The source path of the file to be copied from.
  • dest - The destination path on the remote machine to where the file is to be copied to.
  When the action variable is set to edit, the following variables are supported.
  • dest - The destination file name which has to be edited.
  • replace - A regular expression, which will match a line that will be replaced.
  • line - Text that has to be replaced.
  When the action variable is set to add, the following variables are supported.
  • dest - File on the remote machine to which a line has to be added.
  • line - Line which has to be added to the file. Line will be added towards the end of the file.
  Example 1: Copy a file to a remote machine.

  [update-file]
  action=copy
  src=/tmp/foo.cfg
  dest=/etc/nagios/nrpe.cfg

  Example 2: Edit a line in the remote machine, in the below example lines that have allowed_hosts will be replaced with allowed_hosts=host.redhat.com

  [update-file]
  action=edit
  dest=/etc/nagios/nrpe.cfg
  replace=allowed_hosts
  line=allowed_hosts=host.redhat.com

  Example 3: Add a line to the end of a file

  [update-file]
  action=add
  dest=/etc/ntp.conf
  line=server clock.redhat.com iburst

• service

  Available in gdeploy 2.0. The service module allows user to start, stop, restart, reload, enable, or disable a service. The action variable specifies these values.

  When action variable is set to any of start, stop, restart, reload, enable, disable the variable servicename specifies which service to start, stop etc.
  • service - Name of the service to start, stop etc.
  Example: enable and start ntp daemon.

  [service1]
  action=enable
  service=ntpd

  [service2]
  action=restart
  service=ntpd

• script

  Available in gdeploy 2.0. script module enables user to execute a script/binary on the remote machine. action variable is set to execute. Allows user to specify two variables file and args.

  • file - An executable on the local machine.
  • args - Arguments to the above program.
  Example: Execute script disable-multipath.sh on all the remote nodes listed in `hosts’ section.

  [script]
  action=execute
  file=/usr/share/ansible/gdeploy/scripts/disable-multipath.sh

• firewalld

  Available in gdeploy 2.0. firewalld module allows the user to manipulate firewall rules. action variable supports two values `add’ and `delete’. Both add and delete support the following variables:

  • ports/services - The ports or services to add to firewall.
  • permanent - Whether to make the entry permanent. Allowed values are true/false
  • zone - Default zone is public
  For example:

  [firewalld]
  action=add
  ports=111/tcp,2049/tcp,54321/tcp,5900/tcp,5900-6923/tcp,5666/tcp,16514/tcp
  services=glusterfs

• For Red Hat Enterprise Linux 7: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html-single/Resource_Management_Guide/index.html#sec-What_are_Control_Groups
• For Red Hat Enterprise Linux 6: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/ch-Using_Control_Groups.html

• Generate the passphrase-less SSH keys for the nodes which are going to be part of the trusted storage pool by running the following command:

  # ssh-keygen -f /etc/heketi/heketi_key -t rsa -N ''

• Change the owner and the group permissions for the heketi keys using the following command:

  # chown heketi:heketi /etc/heketi/heketi_key*

• Set up key-based SSH authentication access between Heketi and the Red Hat Gluster Storage servers by running the following command:

  # ssh-copy-id -i /etc/heketi/heketi_key.pub root@server

• As a non root user, set up password-less SSH access between Heketi and the Red Hat Gluster Storage servers by running the following command:

  $ ssh-copy-id -i /etc/heketi/heketi_key.pub user@server

• Note

  To run SSH as a non-root, the username mentioned in username>@server for ssh-copy-id must match with the user name provided to Heketi in the Heketi configuration file below.
• Setup the heketi.json configuration file. The file is located in /etc/heketi/heketi.json. The configuration file has the information required to run the Heketi server. The config file must be in JSON format with the following settings:
  • port: string, Heketi REST service port number
  • use_auth: bool, Enable JWT Authentication
  • jwt: map, JWT Authentication settings
    • admin: map, Settings for the Heketi administrator
      • key: string,
      • user: map, Settings for the Heketi volume requests access user
      • key: string, t
  • glusterfs: map, Red Hat Gluster Storage settings
    • executor: string, Determines the type of command executor to use. Possible values are:
      • mock: Does not send any commands out to servers. Can be used for development and tests
      • ssh: Sends commands to real systems over ssh
    • db: string, Location of Heketi database
    • sshexec: map, SSH configuration
      • keyfile: string, File with private ssh key
      • user: string, SSH user
  Following is an example of the JSON file:

  {
    "_port_comment": "Heketi Server Port Number",
    "port": "8080",
  
    "_use_auth": "Enable JWT authorization. Please enable for deployment",
    "use_auth": false,
  
    "_jwt": "Private keys for access",
    "jwt": {
      "_admin": "Admin has access to all APIs",
      "admin": {
        "key": "My Secret"
      },
      "_user": "User only has access to /volumes endpoint",
      "user": {
        "key": "My Secret"
      }
    },
  
    "_glusterfs_comment": "GlusterFS Configuration",
    "glusterfs": {
      "_executor_comment": [
        "Execute plugin. Possible choices: mock, ssh",
        "mock: This setting is used for testing and development.",
        "      It will not send commands to any node.",
        "ssh:  This setting will notify Heketi to ssh to the nodes.",
        "      It will need the values in sshexec to be configured.",
        "kubernetes: Communicate with GlusterFS containers over",
        "            Kubernetes exec api."
      ],
      "executor": "ssh",
  
      "_sshexec_comment": "SSH username and private key file information",
      "sshexec": {
        "keyfile": "path/to/private_key",
        "user": "sshuser",
        "port": "Optional: ssh port.  Default is 22",
        "fstab": "Optional: Specify fstab file on node.  Default is /etc/fstab",
        "sudo": "Optional: set to true if SSH as a non root user. Default is false."
      },
  
      "_kubeexec_comment": "Kubernetes configuration",
      "kubeexec": {
        "host" :"https://kubernetes.host:8443",
        "cert" : "/path/to/crt.file",
        "insecure": false,
        "user": "kubernetes username",
        "password": "password for kubernetes user",
        "namespace": "OpenShift project or Kubernetes namespace",
        "fstab": "Optional: Specify fstab file on node.  Default is /etc/fstab"
      },
  
      "_db_comment": "Database file name",
      "db": "/var/lib/heketi/heketi.db",
  
      "_loglevel_comment": [
        "Set log level. Choices are:",
        "  none, critical, error, warning, info, debug",
        "Default is warning"
      ],
      "loglevel" : "debug"
    }
  }
  

  Note

  The location for the private SSH key that is created must be set in the keyfile setting of the configuration file, and the key should be readable by the heketi user.

• When the /rhgs file system is unavailable, there is no longer/rhgs/brick1 directory available in the system. Hence, there will be no data loss by writing to a different location.
• This does not require any additional file system for nesting.

• Chain multiple arbitrated replicated volumes together, by placing the arbiter brick for one volume on the same node as a data brick for another volume. Chaining is useful for write-heavy workloads when file size is closer to metadata file size (that is, from 32–128 KiB). This avoids all metadata I/O going through a single disk.
  In arbitrated distributed-replicated volumes, you can also place an arbiter brick on the same node as another replica sub-volume's data brick, since these do not share the same data.
• Place the arbiter bricks from multiple volumes on a single dedicated node. A dedicated arbiter node is suited to write-heavy workloads with larger files, and read-heavy workloads.

• Procedure 5.1, “Converting a replica 2 volume to an arbitrated volume”
• Procedure 5.2, “Converting a replica 3 volume to an arbitrated volume”

• For dedicated arbiter nodes, use JBOD for arbiter bricks, and RAID6 for data bricks.
• For chained arbiter volumes, use the same RAID6 drive for both data and arbiter bricks.

• Section 6.1.3.2, “Mounting Volumes Manually”
• Section 6.1.3.3, “Mounting Volumes Automatically”
• Section 6.1.3.4, “Manually Mounting Sub-directories Using Native Client”

• To set nfs.acl ON, run the following command:

  # gluster volume set VOLNAME nfs.acl on

• To set nfs.acl OFF, run the following command:

  # gluster volume set VOLNAME nfs.acl off

• Section 6.2.2.1, “Setting up CTDB for Gluster NFS ”
  • Section 6.2.2.1.1, “Prerequisites”
  • Section 6.2.2.1.2, “Port and Firewall Information for Gluster NFS”
  • Section 6.2.2.1.3, “Configuring CTDB on Red Hat Gluster Storage Server”
• Section 6.2.2.2, “Using Gluster NFS to Mount Red Hat Gluster Storage Volumes”
  • Section 6.2.2.2.1, “Manually Mounting Volumes Using Gluster NFS”
  • Section 6.2.2.2.2, “Automatically Mounting Volumes Using Gluster NFS”
  • Section 6.2.2.2.3, “Automatically Mounting Subdirectories Using NFS”
  • Section 6.2.2.2.4, “Testing Volumes Mounted Using Gluster NFS”
• Section 6.2.2.3, “Troubleshooting Gluster NFS”

• Section 6.2.3.1, “Supported Features of NFS-Ganesha”
• Section 6.2.3.2, “Setting up NFS Ganesha”
  • Section 6.2.3.2.1, “Port and Firewall Information for NFS-Ganesha”
  • Section 6.2.3.2.2, “Prerequisites to run NFS-Ganesha”
  • Section 6.2.3.2.3, “Configuring the Cluster Services”
  • Section 6.2.3.2.4, “Creating the ganesha-ha.conf file”
  • Section 6.2.3.2.5, “Configuring NFS-Ganesha using Gluster CLI”
  • Section 6.2.3.2.6, “Exporting and Unexporting Volumes through NFS-Ganesha”
  • Section 6.2.3.2.7, “Verifying the NFS-Ganesha Status”
• Section 6.2.3.3, “Accessing NFS-Ganesha Exports”
  • Section 6.2.3.3.1, “Mounting exports in NFSv3 Mode”
  • Section 6.2.3.3.2, “Mounting exports in NFSv4 Mode”
  • Section 6.2.3.3.3, “Finding clients of an NFS server using dbus”
• Section 6.2.3.4, “Modifying the NFS-Ganesha HA Setup”
  • Section 6.2.3.4.1, “Adding a Node to the Cluster”
  • Section 6.2.3.4.2, “Deleting a Node in the Cluster”
• Section 6.2.3.5, “Modifying the Default Export Configurations”
  • Section 6.2.3.5.1, “Providing Permissions for Specific Clients”
  • Section 6.2.3.5.2, “Enabling and Disabling NFSv4 ACLs ”
  • Section 6.2.3.5.3, “Providing Pseudo Path for NFSv4 Mount”
  • Section 6.2.3.5.4, “Exporting Subdirectories”
  • Section 6.2.3.5.5, “Configuring Upcall Poll Interval”
• Section 6.2.3.6, “Configuring Kerberized NFS-Ganesha”
  • Section 6.2.3.6.1, “Setting up the NFS-Ganesha Server:”
  • Section 6.2.3.6.2, “Setting up the NFS Client”
• Section 6.2.3.7, “NFS-Ganesha Service Downtime”
  • Section 6.2.3.7.1, “Modifying the Fail-over Time”
• Section 6.2.3.8, “Manually Configuring NFS-Ganesha Exports”
• Section 6.2.3.9, “Troubleshooting”

• Samba is the server software used to export Linux filesystems with the SMB protocol. For exporting Red Hat Gluster Storage volumes with Samba, it is mandatory to have CTDB configured, which is a component of Samba. For information on subscribing to the correct channels for SMB support, see Subscribing to the Red Hat Gluster Storage server channels in the Red Hat Gluster Storage 3.4 Installation Guide.
• Enable the Samba firewall service in the active zones for runtime and permanent mode. The following commands are for systems based on Red Hat Enterprise Linux 7.
  To get a list of active zones, run the following command:

  # firewall-cmd --get-active-zones

  To allow the firewall services in the active zones, run the following commands

  # firewall-cmd --zone=zone_name --add-service=samba
  # firewall-cmd --zone=zone_name --add-service=samba  --permanent

Prerequisites

• If you already have an older version of CTDB (version <= ctdb1.x), then remove CTDB by executing the following command:

  # yum remove ctdb

  After removing the older version, proceed with installing the latest CTDB.

  Note

  Ensure that the system is subscribed to the samba channel to get the latest CTDB packages.
• Install CTDB on all the nodes that are used as Samba servers to the latest version using the following command:

  # yum install ctdb

• In a CTDB based high availability environment of Samba , the locks will not be migrated on failover.
• Enable the CTDB firewall service in the active zones for runtime and permanent mode. The following commands are for systems based on Red Hat Enterprise Linux 7.
  To get a list of active zones, run the following command:

  # firewall-cmd --get-active-zones

  To add ports to the active zones, run the following commands:

  # firewall-cmd --zone=zone_name --add-port=4379/tcp
  # firewall-cmd --zone=zone_name --add-port=4379/tcp  --permanent

• Enabling Metadata Caching to improve the performance of SMB access of Red Hat Gluster Storage volumes.
• Enhancing Directory Listing Performance
• Enhancing File/Directory Create Performance