Chapter 24. Using shared storage with virtual disk images
This chapter covers the use of shared and network storage devices for virtual disks.
24.1. Using iSCSI for storing virtual disk images
This section demonstrates how to set up an iSCSI target on Red Hat Enterprise Linux and how to configure iSCSI on a libvirt KVM host using virsh, and finally how to provision a guest on iSCSI using
virt-install.
Important
Setting up a Red Hat Enterprise Linux server as an iSCSI target is not recommended. The example used in this section should not be used in production, and is provided as an example which should only be referred to for basic shared storage testing and educational purposes.
24.1.1. How to set up an iSCSI target on Red Hat Enterprise Linux
Install and enable the iSCSI target service
Install and enable the iSCSI target service with the following commands:# yum install scsi-target-utils # chkconfig tgtd on # service tgtd start
Important
Thescsi-target-utilspackage required for this example is provided only in the Cluster Storage add-on for Red Hat Enterprise Linux 5, and may not be available for your system. Contact your support agent to activate this add-on if you are currently unable to install this package.Allocate storage for the LUNs
The iSCSI target service is not dependent on a particular type of exported LUN. The LUNs can be plain files, LVM volumes, or block devices. There is however a performance overhead if using the LVM and/or file system layers as compared to block devices. The guest providing the iSCSI service in this example has no spare block or LVM devices, so raw files will be used.The following commands demonstrate the creation of two LUNs; one is 10GB (sparse file), the other is 500MB (fully-allocated). Be aware that your environment will be different and this is provided only as an example:# mkdir -p /var/lib/tgtd/kvmguest # dd if=/dev/zero of=/var/lib/tgtd/kvmguest/rhelx86_64.img bs=1M seek=10240 count=0 # dd if=/dev/zero of=/var/lib/tgtd/kvmguest/shareddata.img bs=1M count=512 # restorecon -R /var/lib/tgtd
Export the iSCSI target and LUNs
For Red Hat Enterprise Linux 5, severaltgtadmcommands are required to create a target and associate the storage volumes created earlier. First, the following command adds a target using an iSCSI Qualified Name (IQN):# tgtadm --lld iscsi --op new --mode target --tid 1 --targetname \ iqn.2004-04.rhel:rhel5:iscsi.kvmguest
Now the storage volumes must be associated with LUNs in the iSCSI target with these two commands:# tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 \ --backing-store /var/lib/tgtd/kvmguest/rhelx86_64.img # tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 2 \ --backing-store /var/lib/tgtd/kvmguest/shareddata.img
Important
Add the previous 3tgtadmcommands to the end of the/etc/rc.localfile to ensure normal operation upon restarting of the system.To confirm the successful operation of the previous commands, query the iSCSI target setup:tgtadm --lld iscsi --op show --mode target Target 1: iqn.2004-04.rhel:rhel5:iscsi.kvmguest System information: Driver: iscsi State: ready I_T nexus information: LUN information: LUN: 0 Type: controller SCSI ID: IET 00010000 SCSI SN: beaf10 Size: 0 MB, Block size: 1 Online: Yes Removable media: No Readonly: No Backing store type: null Backing store path: None Backing store flags: LUN: 1 Type: disk SCSI ID: IET 00010001 SCSI SN: beaf11 Size: 10737 MB, Block size: 1024 Online: Yes Removable media: No Readonly: No Backing store type: rdwr Backing store path: /var/lib/tgtd/kvmguest/rhelx86_64.img Backing store flags: LUN: 2 Type: disk SCSI ID: IET 00010002 SCSI SN: beaf12 Size: 537 MB, Block size: 512 Online: Yes Removable media: No Readonly: No Backing store type: rdwr Backing store path: /var/lib/tgtd/kvmguest/shareddata.img Backing store flags: Account information: ACL information:Allow client access to the target
Finally, this example command allows access to all clients without any authentication:# tgtadm --lld iscsi --op bind --mode target --tid 1 --initiator-address ALL
Note
The two most common problems encountered when configuring the iSCSI target are related to SELinux and iptables. If adding plain files as LUNs in an iSCSI target, ensure the files are labeled
system_u:object_r:tgtd_var_lib_t:s0. TCP port 3260 must be open in your iptables configuration.
24.1.2. How to configure iSCSI on a libvirt KVM host and provision a guest using virt-install
The previous section described how to set up an iSCSI target. This section demonstrates how to configure iSCSI on a libvirt KVM instance using virsh, and then how to provision a guest using
virt-install.
Defining the storage pool
All libvirt storage is managed through storage pools, of which there are many possible types: SCSI, NFS, ext4, plain directory and iSCSI. All libvirt objects are configured via XML description files, and storage pools are no different in this regard. For an iSCSI storage pool there are three important pieces of information to provide:- The target path - this determines how libvirt will expose device paths for the pool. Paths like
/dev/sdaand/dev/sdbare not an ideal choice as they can change between reboots, and can change across machines within a cluster; in other words, the names are assigned on a first come, first served basis by the kernel. It is strongly recommended to use the/dev/disk/by-pathformat. This results in a consistent naming scheme across all machines. - The host name - this is the fully-qualified DNS name of the iSCSI server.
- The source path - this is the iSCSI qualified name (IQN) seen in the previous setup procedure (iqn.2004-04.rhel:rhel5:iscsi.kvmguest).
Although your environment will likely be different, the following text is what an iSCSI configuration will look like:<pool type='iscsi'> <name>kvmguest</name> <source> <host name='myiscsi.example.com'/> <device path='iqn.2004-04.rhel:rhel5:iscsi.kvmguest'/> </source> <target> <path>/dev/disk/by-path</path> </target> </pool>Save this XML code to a file namediscsirhel5.xmland load it into libvirt using thepool-definecommand:# virsh pool-define iscsirhel5.xml Pool kvmguest defined from iscsirhel5.xml # virsh pool-list --all Name State Autostart ----------------------------------------- default active yes kvmguest inactive no
Starting the storage pool
The configuration is saved, but the iSCSI target has not yet been logged into, so no LUNs will be visible on the host at this point. Use thepool-startcommand to make the LUNs visible with thevol-listcommand.# virsh pool-start kvmguest Pool kvmguest started # virsh vol-list kvmguest Name Path ----------------------------------------- 10.0.0.1 /dev/disk/by-path/ip-192.168.122.2:3260-iscsi-iqn.2004-04.rhel:rhel5:iscsi.kvmguest-lun-1 10.0.0.2 /dev/disk/by-path/ip-192.168.122.2:3260-iscsi-iqn.2004-04.rhel:rhel5:iscsi.kvmguest-lun-2
Querying LUN information
Further information about each LUN can be obtained using thevol-infoandvol-dumpxmlcommands:# virsh vol-info /dev/disk/by-path/ip-192.168.122.2:3260-iscsi-iqn.2004-04.rhel:rhel5:iscsi.kvmguest-lun-1 Name: 10.0.0.1 Type: block Capacity: 10.00 GB Allocation: 10.00 GB # virsh vol-dumpxml /dev/disk/by-path/ip-192.168.122.2:3260-iscsi-iqn.2004-04.rhel:rhel5:iscsi.kvmguest-lun-1 <volume> <name>10.0.0.1</name> <key>/dev/disk/by-path/ip-192.168.122.2:3260-iscsi-iqn.2004-04.rhel:rhel5:iscsi.kvmguest-lun-1</key> <source> </source> <capacity>10737418240</capacity> <allocation>10737418240</allocation> <target> <path>/dev/disk/by-path/ip-192.168.122.2:3260-iscsi-iqn.2004-04.rhel:rhel5:iscsi.kvmguest-lun-2</path> <format type='unknown'/> <permissions> <mode>0660</mode> <owner>0</owner> <group>6</group> <label>system_u:object_r:fixed_disk_device_t:s0</label> </permissions> </target> </volume>Activating the storage at boot time
Once correctly configured, the pool can be set to start automatically upon booting of the host:# virsh pool-autostart kvmguest Pool kvmguest marked as autostarted
Provisioning a guest on iSCSI
Thevirt-installcommand can be used to install new guests from the command line. The --disk argument can take the name of a storage pool, followed by the name of any contained volumes. Continuing this example, the following command will begin the installation of a guest with two disks; the first disk is the root file system, the second disk can be shared between multiple guests for common data:# virt-install --accelerate --name rhelx86_64 --ram 800 --vnc --disk \ vol=kvmguest/10.0.0.1 --disk vol=kvmguest/10.0.0.2,perms=sh --pxe
Once this rhelx86_64 guest is installed, the following command and output shows the XML thatvirt-installused to associate the guest with the iSCSI LUNs:# virsh dumpxml rhelx86_64 <domain type='kvm' id='4'> <name>rhelx86_64</name> <uuid>ad8961e9-156f-746f-5a4e-f220bfafd08d</uuid> <memory>819200</memory> <currentMemory>819200</currentMemory> <vcpu>1</vcpu> <os> <type arch='x86_64' machine='rhel'>hvm</type> <boot dev='network'/> </os> <features> <acpi/> <apic/> <pae/> </features> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>destroy</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw'/> <source dev='/dev/disk/by-path/ip-192.168.122.170:3260-iscsi-iqn.2004-04.rhel:rhel5:iscsi.kvmguest-lun-1'/> <target dev='hda' bus='ide'/> <alias name='ide0-0-0'/> <address type='drive' controller='0' bus='0' unit='0'/> </disk> <disk type='block' device='disk'> <driver name='qemu' type='raw'/> <source dev='/dev/disk/by-path/ip-192.168.122.170:3260-iscsi-iqn.2004-04.rhel:rhel5:iscsi.kvmguest-lun-2'/> <target dev='hdb' bus='ide'/> <shareable/> <alias name='ide0-0-1'/> <address type='drive' controller='0' bus='0' unit='1'/> </disk> <controller type='ide' index='0'> <alias name='ide0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <interface type='network'> <mac address='52:54:00:0a:ca:84'/> <source network='default'/> <target dev='vnet1'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/28'/> <target port='0'/> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/28'> <source path='/dev/pts/28'/> <target port='0'/> <alias name='serial0'/> </console> <input type='mouse' bus='ps2'/> <graphics type='vnc' port='5901' autoport='yes' keymap='en-us'/> <video> <model type='cirrus' vram='9216' heads='1'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> </devices> </domain>There are two important items of note in this output:- The guest uses the large
/dev/disk/by-pathpaths to reference the LUNs. As described earlier, this is so that file names and paths will remain constant. - The second disk has the <shareable/> flag set. Critical for the disk to be safely shared between guests, this ensures that the SELinux labelling will be appropriate for multiple guests to access the disk and that all I/O caching is disabled on the host.
For migration of guests between hosts to succeed, some form of shared storage is required. Although NFS is often used for this purpose, the lack of SELinux labelling for NFS means there is limited sVirt protection between guests. This lack of sVirt support could allow one guest to use another guest's disks, which is usually undesirable.
Using iSCSI provides full sVirt isolation between guests to the same degree of non-shared storage.
