Virtualization Administration Guide

Procedure 3.2. Cloning a Virtual Machine with virt-manager

1. Open virt-manager

   Start virt-manager. Launch the Virtual Machine Manager application from the Applications menu and System Tools submenu. Alternatively, run the virt-manager command as root.
   Select the guest virtual machine you want to clone from the list of guest virtual machines in Virtual Machine Manager.
   Right-click the guest virtual machine you want to clone and select Clone. The Clone Virtual Machine window opens.
   

   Figure 3.1. Clone Virtual Machine window

2. Configure the clone

   • To change the name of the clone, enter a new name for the clone.
   • To change the networking configuration, click Details.
     Enter a new MAC address for the clone.
     Click OK.
     

     Figure 3.2. Change MAC Address window

   • For each disk in the cloned guest virtual machine, select one of the following options:
     • Clone this disk - The disk will be cloned for the cloned guest virtual machine
     • Share disk with guest virtual machine name - The disk will be shared by the guest virtual machine that will be cloned and its clone
     • Details - Opens the Change storage path window, which enables selecting a new path for the disk
       

       Figure 3.3. Change storage path window

3. Clone the guest virtual machine

   Click Clone.

1. Open the libvirtd.conf file as described in Procedure 4.1, “Configuring libvirtd.conf”.
2. Look for the Processing controls section.

   #################################################################
   #
   # Processing controls
   #
   
   # The maximum number of concurrent client connections to allow
   # over all sockets combined.
   #max_clients = 20
   
   
   # The minimum limit sets the number of workers to start up
   # initially. If the number of active clients exceeds this,
   # then more threads are spawned, upto max_workers limit.
   # Typically you'd want max_workers to equal maximum number
   # of clients allowed
   #min_workers = 5
   #max_workers = 20
   
   
   # The number of priority workers. If all workers from above
   # pool will stuck, some calls marked as high priority
   # (notably domainDestroy) can be executed in this pool.
   #prio_workers = 5
   
   # Total global limit on concurrent RPC calls. Should be
   # at least as large as max_workers. Beyond this, RPC requests
   # will be read into memory and queued. This directly impact
   # memory usage, currently each request requires 256 KB of
   # memory. So by default upto 5 MB of memory is used
   #
   # XXX this isn't actually enforced yet, only the per-client
   # limit is used so far
   #max_requests = 20
   
   # Limit on concurrent requests from a single client
   # connection. To avoid one client monopolizing the server
   # this should be a small fraction of the global max_requests
   # and max_workers parameter
   #max_client_requests = 5
   
   #################################################################
   

3. Change the max_clients and max_workers parameters settings. It is recommended that the number be the same in both parameters. The max_clients will use 2 clients per migration (one per side) and max_workers will use 1 worker on the source and 0 workers on the destination during the perform phase and 1 worker on the destination during the finish phase.

   Important

   The max_clients and max_workers parameters settings are effected by all guest virtual machine connections to the libvirtd service. This means that any user that is using the same guest virtual machine and is performing a migration at the same time will also beholden to the limits set in the max_clients and max_workers parameters settings. This is why the maximum value needs to be considered carefully before performing a concurrent live migration.
4. Save the file and restart the service.

   Note

   There may be cases where a migration connection drops because there are too many ssh sessions that have been started, but not yet authenticated. By default, sshd allows only 10 sessions to be in a "pre-authenticated state" at any time. This setting is controlled by the MaxStartups parameter in the sshd configuration file (located here: /etc/ssh/sshd_config), which may require some adjustment. Adjusting this parameter should be done with caution as the limitation is put in place to prevent DoS attacks (and over-use of resources in general). Setting this value too high will negate its purpose. To change this parameter, edit the file /etc/ssh/sshd_config, remove the # from the beginning of the MaxStartups line, and change the 10 (default value) to a higher number. Remember to save the file and restart the sshd service. For more information, refer to the sshd_config man page.

Procedure 9.3. Assigning a PCI device to a guest virtual machine with virsh

1. Identify the device

   First, identify the PCI device designated for device assignment to the virtual machine. Use the lspci command to list the available PCI devices. You can refine the output of lspci with grep.
   This example uses the Ethernet controller highlighted in the following output:

   # lspci | grep Ethernet
   00:19.0 Ethernet controller: Intel Corporation 82567LM-2 Gigabit Network Connection
   01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
   01:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

   This Ethernet controller is shown with the short identifier 00:19.0. We need to find out the full identifier used by virsh in order to assign this PCI device to a virtual machine.
   To do so, use the virsh nodedev-list command to list all devices of a particular type (pci) that are attached to the host machine. Then look at the output for the string that maps to the short identifier of the device you wish to use.
   This example highlights the string that maps to the Ethernet controller with the short identifier 00:19.0. In this example, the : and . characters are replaced with underscores in the full identifier.

   # virsh nodedev-list --cap pci
   pci_0000_00_00_0
   pci_0000_00_01_0
   pci_0000_00_03_0
   pci_0000_00_07_0
   pci_0000_00_10_0
   pci_0000_00_10_1
   pci_0000_00_14_0
   pci_0000_00_14_1
   pci_0000_00_14_2
   pci_0000_00_14_3
   pci_0000_00_19_0
   pci_0000_00_1a_0
   pci_0000_00_1a_1
   pci_0000_00_1a_2
   pci_0000_00_1a_7
   pci_0000_00_1b_0
   pci_0000_00_1c_0
   pci_0000_00_1c_1
   pci_0000_00_1c_4
   pci_0000_00_1d_0
   pci_0000_00_1d_1
   pci_0000_00_1d_2
   pci_0000_00_1d_7
   pci_0000_00_1e_0
   pci_0000_00_1f_0
   pci_0000_00_1f_2
   pci_0000_00_1f_3
   pci_0000_01_00_0
   pci_0000_01_00_1
   pci_0000_02_00_0
   pci_0000_02_00_1
   pci_0000_06_00_0
   pci_0000_07_02_0
   pci_0000_07_03_0

   Record the PCI device number that maps to the device you want to use; this is required in other steps.

2. Review device information

   Information on the domain, bus, and function are available from output of the virsh nodedev-dumpxml command:

   virsh nodedev-dumpxml pci_0000_00_19_0
   <device>
     <name>pci_0000_00_19_0</name>
     <parent>computer</parent>
     <driver>
       <name>e1000e</name>
     </driver>
     <capability type='pci'>
       <domain>0</domain>
       <bus>0</bus>
       <slot>25</slot>
       <function>0</function>
       <product id='0x1502'>82579LM Gigabit Network Connection</product>
       <vendor id='0x8086'>Intel Corporation</vendor>
       <iommuGroup number='7'>
         <address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/>
       </iommuGroup>
     </capability>
   </device>

   Note

   An IOMMU group is determined based on the visibility and isolation of devices from the perspective of the IOMMU. Each IOMMU group may contain one or more devices. When multiple devices are present, all endpoints within the IOMMU group must be claimed for any device within the group to be assigned to a guest. This can be accomplished either by also assigning the extra endpoints to the guest or by detaching them from the host driver using virsh nodedev-detach. Devices contained within a single group may not be split between multiple guests or split between host and guest. Non-endpoint devices such as PCIe root ports, switch ports, and bridges should not be detached from the host drivers and will not interfere with assignment of endpoints.

   Devices within an IOMMU group can be determined using the iommuGroup section of the virsh nodedev-dumpxml output. Each member of the group is provided via a separate "address" field. This information may also be found in sysfs using the following:

   $ ls /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices/

   An example of the output from this would be:

   0000:01:00.0  0000:01:00.1

   To assign only 0000.01.00.0 to the guest, the unused endpoint should be detached from the host before starting the guest:

   $ virsh nodedev-detach pci_0000_01_00_1

3. Determine required configuration details

   Refer to the output from the virsh nodedev-dumpxml pci_0000_00_19_0 command for the values required for the configuration file.
   The example device has the following values: bus = 0, slot = 25 and function = 0. The decimal configuration uses those three values:

   bus='0'
   slot='25'
   function='0'

4. Add configuration details

   Run virsh edit, specifying the virtual machine name, and add a device entry in the <source> section to assign the PCI device to the guest virtual machine.

   # virsh edit guest1-rhel6-64
   <hostdev mode='subsystem' type='pci' managed='yes'>
     <source>
        <address domain='0' bus='0' slot='25' function='0'/>
     </source>
   </hostdev>

   Alternately, run virsh attach-device, specifying the virtual machine name and the guest's XML file:

   virsh attach-device guest1-rhel6-64 file.xml

5. Start the virtual machine

   # virsh start guest1-rhel6-64

Procedure 9.4. Assigning a PCI device to a guest virtual machine using virt-manager

1. Open the hardware settings

   Open the guest virtual machine and click the Add Hardware button to add a new device to the virtual machine.
   

   Figure 9.1. The virtual machine hardware information window

2. Select a PCI device

   Select PCI Host Device from the Hardware list on the left.
   Select an unused PCI device. If you select a PCI device that is in use by another guest an error may result. In this example, a spare 82576 network device is used. Click Finish to complete setup.
   

   Figure 9.2. The Add new virtual hardware wizard

3. Add the new device

   The setup is complete and the guest virtual machine now has direct access to the PCI device.
   

   Figure 9.3. The virtual machine hardware information window

Procedure 9.5. Assigning a PCI device to a virtual machine with virt-install

1. Identify the device

   Identify the PCI device designated for device assignment to the guest virtual machine.

   # lspci | grep Ethernet
   00:19.0 Ethernet controller: Intel Corporation 82567LM-2 Gigabit Network Connection
   01:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
   01:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)

   The virsh nodedev-list command lists all devices attached to the system, and identifies each PCI device with a string. To limit output to only PCI devices, run the following command:

   # virsh nodedev-list --cap pci
   pci_0000_00_00_0
   pci_0000_00_01_0
   pci_0000_00_03_0
   pci_0000_00_07_0
   pci_0000_00_10_0
   pci_0000_00_10_1
   pci_0000_00_14_0
   pci_0000_00_14_1
   pci_0000_00_14_2
   pci_0000_00_14_3
   pci_0000_00_19_0
   pci_0000_00_1a_0
   pci_0000_00_1a_1
   pci_0000_00_1a_2
   pci_0000_00_1a_7
   pci_0000_00_1b_0
   pci_0000_00_1c_0
   pci_0000_00_1c_1
   pci_0000_00_1c_4
   pci_0000_00_1d_0
   pci_0000_00_1d_1
   pci_0000_00_1d_2
   pci_0000_00_1d_7
   pci_0000_00_1e_0
   pci_0000_00_1f_0
   pci_0000_00_1f_2
   pci_0000_00_1f_3
   pci_0000_01_00_0
   pci_0000_01_00_1
   pci_0000_02_00_0
   pci_0000_02_00_1
   pci_0000_06_00_0
   pci_0000_07_02_0
   pci_0000_07_03_0

   Record the PCI device number; the number is needed in other steps.
   Information on the domain, bus and function are available from output of the virsh nodedev-dumpxml command:

   # virsh nodedev-dumpxml pci_0000_01_00_0
   <device>
     <name>pci_0000_01_00_0</name>
     <parent>pci_0000_00_01_0</parent>
     <driver>
       <name>igb</name>
     </driver>
     <capability type='pci'>
       <domain>0</domain>
       <bus>1</bus>
       <slot>0</slot>
       <function>0</function>
       <product id='0x10c9'>82576 Gigabit Network Connection</product>
       <vendor id='0x8086'>Intel Corporation</vendor>
       <iommuGroup number='7'>
         <address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/>
       </iommuGroup>
     </capability>
   </device>

   Note

   If there are multiple endpoints in the IOMMU group and not all of them are assigned to the guest, you will need to manually detach the other endpoint(s) from the host by running the following command before you start the guest:

   $ virsh nodedev-detach pci_0000_00_19_1

   Refer to the Note in Section 9.1.1, “Assigning a PCI Device with virsh” for more information on IOMMU groups.

2. Add the device

   Use the PCI identifier output from the virsh nodedev command as the value for the --host-device parameter.

   virt-install \
   --name=guest1-rhel6-64 \
   --disk path=/var/lib/libvirt/images/guest1-rhel6-64.img,size=8 \
   --nonsparse --graphics spice \
   --vcpus=2 --ram=2048 \
   --location=http://example1.com/installation_tree/RHEL6.0-Server-x86_64/os \
   --nonetworks \
   --os-type=linux \
   --os-variant=rhel6
   --host-device=pci_0000_01_00_0

3. Complete the installation

   Complete the guest installation. The PCI device should be attached to the guest.

Procedure 9.6. Detaching a PCI device from a guest with virsh

1. Detach the device

   Use the following command to detach the PCI device from the guest by removing it in the guest's XML file:

   # virsh detach-device name_of_guest file.xml

2. Re-attach the device to the host (optional)

   If the device is in managed mode, skip this step. The device will be returned to the host automatically.
   If the device is not using managed mode, use the following command to re-attach the PCI device to the host machine:

   # virsh nodedev-reattach device

   For example, to re-attach the pci_0000_01_00_0 device to the host:

   virsh nodedev-reattach pci_0000_01_00_0

   The device is now available for host use.

Procedure 9.7. Detaching a PCI Device from a guest with virt-manager

1. Open the virtual hardware details screen

   In virt-manager, double-click on the virtual machine that contains the device. Select the Show virtual hardware details button to display a list of virtual hardware.
   

   Figure 9.4. The virtual hardware details button

2. Select and remove the device

   Select the PCI device to be detached from the list of virtual devices in the left panel.
   

   Figure 9.5. Selecting the PCI device to be detached

   Click the Remove button to confirm. The device is now available for host use.

Procedure 9.8. Configuring MAC addresses, vLAN, and virtual ports for assigning PCI devices on SR-IOV

1. Shutdown the guest virtual machine

   Using virsh shutdown command (refer to Section 14.9.1, “Shutting Down a Guest Virtual Machine”), shutdown the guest virtual machine named guestVM.

   # virsh shutdown guestVM

2. Gather information

   In order to use <interface type='hostdev'>, you must have an SR-IOV-capable network card, host physical machine hardware that supports either the Intel VT-d or AMD IOMMU extensions, and you must know the PCI address of the VF that you wish to assign.

3. Open the XML file for editing

   Run the # virsh save-image-edit command to open the XML file for editing (refer to Section 14.8.10, “Edit Domain XML Configuration Files” for more information). As you would want to restore the guest virtual machine to its former running state, the --running would be used in this case. The name of the configuration file in this example is guestVM.xml, as the name of the guest virtual machine is guestVM.

    # virsh save-image-edit guestVM.xml --running 

   The guestVM.xml opens in your default editor.

4. Edit the XML file

   Update the configuration file (guestVM.xml) to have a <devices> entry similar to the following:

   
    <devices>
      ...
      <interface type='hostdev' managed='yes'>
        <source>
          <address type='pci' domain='0x0' bus='0x00' slot='0x07' function='0x0'/> <!--these values can be decimal as well-->
        </source>
        <mac address='52:54:00:6d:90:02'/>                                         <!--sets the mac address-->
        <virtualport type='802.1Qbh'>                                              <!--sets the virtual port for the 802.1Qbh switch-->
          <parameters profileid='finance'/>
        </virtualport>
        <vlan>                                                                     <!--sets the vlan tag-->
         <tag id='42'/>
        </vlan>
      </interface>
      ...
    </devices>
   
   

   Figure 9.7. Sample domain XML for hostdev interface type

   Note that if you do not provide a MAC address, one will be automatically generated, just as with any other type of interface device. Also, the <virtualport> element is only used if you are connecting to an 802.11Qgh hardware switch (802.11Qbg (a.k.a. "VEPA") switches are currently not supported.

5. Re-start the guest virtual machine

   Run the virsh start command to restart the guest virtual machine you shutdown in the first step (example uses guestVM as the guest virtual machine's domain name). Refer to Section 14.8.1, “Starting a Defined Domain” for more information.

    # virsh start guestVM 

   When the guest virtual machine starts, it sees the network device provided to it by the physical host machine's adapter, with the configured MAC address. This MAC address will remain unchanged across guest virtual machine and host physical machine reboots.

Procedure 9.9. Creating a device pool

1. Shutdown the guest virtual machine

   Using virsh shutdown command (refer to Section 14.9, “Shutting Down, Rebooting, and Forcing Shutdown of a Guest Virtual Machine”), shutdown the guest virtual machine named guestVM.

   # virsh shutdown guestVM

2. Create a configuration file

   Using your editor of choice create an XML file (named passthrough.xml, for example) in the /tmp directory. Make sure to replace pf dev='eth3' with the netdev name of your own SR-IOV device's PF
   The following is an example network definition that will make available a pool of all VFs for the SR-IOV adapter with its physical function (PF) at "eth3' on the host physical machine:

         
   <network>
      <name>passthrough</name>                                                <!--This is the name of the file you created-->
      <forward mode='hostdev' managed='yes'>
        <pf dev='myNetDevName'/>                                              <!--Use the netdev name of your SR-IOV devices PF here-->
      </forward>
   </network>
         
   
   

   Figure 9.8. Sample network definition domain XML

3. Load the new XML file

   Run the following command, replacing /tmp/passthrough.xml, with the name and location of your XML file you created in the previous step:

   # virsh net-define /tmp/passthrough.xml

4. Restarting the guest

   Run the following replacing passthrough.xml, with the name of your XML file you created in the previous step:

    # virsh net-autostart passthrough # virsh net-start passthrough 

5. Re-start the guest virtual machine

   Run the virsh start command to restart the guest virtual machine you shutdown in the first step (example uses guestVM as the guest virtual machine's domain name). Refer to Section 14.8.1, “Starting a Defined Domain” for more information.

    # virsh start guestVM 

6. Initiating passthrough for devices

   Although only a single device is shown, libvirt will automatically derive the list of all VFs associated with that PF the first time a guest virtual machine is started with an interface definition in its domain XML like the following:

            
   <interface type='network'>
      <source network='passthrough'>
   </interface>
         
   
   

   Figure 9.9. Sample domain XML for interface network definition

7. Verification

   You can verify this by running virsh net-dumpxml passthrough command after starting the first guest that uses the network; you will get output similar to the following:

         
   <network connections='1'>
      <name>passthrough</name>
      <uuid>a6b49429-d353-d7ad-3185-4451cc786437</uuid>
      <forward mode='hostdev' managed='yes'>
        <pf dev='eth3'/>
        <address type='pci' domain='0x0000' bus='0x02' slot='0x10' function='0x1'/>
        <address type='pci' domain='0x0000' bus='0x02' slot='0x10' function='0x3'/>
        <address type='pci' domain='0x0000' bus='0x02' slot='0x10' function='0x5'/>
        <address type='pci' domain='0x0000' bus='0x02' slot='0x10' function='0x7'/>
        <address type='pci' domain='0x0000' bus='0x02' slot='0x11' function='0x1'/>
        <address type='pci' domain='0x0000' bus='0x02' slot='0x11' function='0x3'/>
        <address type='pci' domain='0x0000' bus='0x02' slot='0x11' function='0x5'/>
      </forward>
   </network>
         
   
   

   Figure 9.10. XML dump file passthrough contents

Procedure 10.1. Setting up communication between guest agent and host

1. Open the guest XML

   Open the guest XML with the QEMU guest agent configuration. You will need the guest name to open the file. Use the command # virsh list on the host machine to list the guests that it can recognize. In this example, the guest's name is rhel6:

   # virsh edit rhel6

2. Edit the guest XML file

   Add the following elements to the XML file and save the changes.

   <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/rhel6.agent'/>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
   </channel>
   
   

   Figure 10.1. Editing the guest XML to configure the QEMU guest agent

3. Start the QEMU guest agent in the guest

   Download and install the guest agent in the guest virtual machine using yum install qemu-guest-agent if you have not done so already. Once installed, start the service as follows:

   # service start qemu-guest-agent

1. Create a GPT disk label on the disk

   The disk must be relabeled with a GUID Partition Table (GPT) disk label. GPT disk labels allow for creating a large numbers of partitions, up to 128 partitions, on each device. GPT partition tables can store partition data for far more partitions than the MS-DOS partition table.

   # parted /dev/sdb
   GNU Parted 2.1
   Using /dev/sdb
   Welcome to GNU Parted! Type 'help' to view a list of commands.
   (parted) mklabel
   New disk label type? gpt
   (parted) quit
   Information: You may need to update /etc/fstab.
   #
   

2. Create the storage pool configuration file

   Create a temporary XML text file containing the storage pool information required for the new device.
   The file must be in the format shown below, and contain the following fields:

   <name>guest_images_disk</name>

   The name parameter determines the name of the storage pool. This example uses the name guest_images_disk in the example below.

   <device path='/dev/sdb'/>

   The device parameter with the path attribute specifies the device path of the storage device. This example uses the device /dev/sdb.

   <target> <path>/dev</path></target>

   The file system target parameter with the path sub-parameter determines the location on the host physical machine file system to attach volumes created with this storage pool.

   For example, sdb1, sdb2, sdb3. Using /dev/, as in the example below, means volumes created from this storage pool can be accessed as /dev/sdb1, /dev/sdb2, /dev/sdb3.

   <format type='gpt'/>

   The format parameter specifies the partition table type. This example uses the gpt in the example below, to match the GPT disk label type created in the previous step.

   Create the XML file for the storage pool device with a text editor.

   Example 12.2. Disk based storage device storage pool

   <pool type='disk'>
     <name>guest_images_disk</name>
     <source>
       <device path='/dev/sdb'/>
       <format type='gpt'/>
     </source>
     <target>
       <path>/dev</path>
     </target>
   </pool>
   

3. Attach the device

   Add the storage pool definition using the virsh pool-define command with the XML configuration file created in the previous step.

   # virsh pool-define ~/guest_images_disk.xml
   Pool guest_images_disk defined from /root/guest_images_disk.xml
   # virsh pool-list --all
   Name                 State      Autostart
   -----------------------------------------
   default              active     yes
   guest_images_disk    inactive   no
   

4. Start the storage pool

   Start the storage pool with the virsh pool-start command. Verify the pool is started with the virsh pool-list --all command.

   # virsh pool-start guest_images_disk
   Pool guest_images_disk started
   # virsh pool-list --all
   Name                 State      Autostart
   -----------------------------------------
   default              active     yes
   guest_images_disk    active     no
   

5. Turn on autostart

   Turn on autostart for the storage pool. Autostart configures the libvirtd service to start the storage pool when the service starts.

   # virsh pool-autostart guest_images_disk
   Pool guest_images_disk marked as autostarted
   # virsh pool-list --all
   Name                 State      Autostart
   -----------------------------------------
   default              active     yes
   guest_images_disk    active     yes
   

6. Verify the storage pool configuration

   Verify the storage pool was created correctly, the sizes reported correctly, and the state reports as running.

   # virsh pool-info guest_images_disk
   Name:           guest_images_disk
   UUID:           551a67c8-5f2a-012c-3844-df29b167431c
   State:          running
   Capacity:       465.76 GB
   Allocation:     0.00
   Available:      465.76 GB
   # ls -la /dev/sdb
   brw-rw----. 1 root disk 8, 16 May 30 14:08 /dev/sdb
   # virsh vol-list guest_images_disk
   Name                 Path
   -----------------------------------------
   

7. Optional: Remove the temporary configuration file

   Remove the temporary storage pool XML configuration file if it is not needed.

   # rm ~/guest_images_disk.xml

1. To avoid any issues with other guest virtual machines using the same pool, it is best to stop the storage pool and release any resources in use by it.

   # virsh pool-destroy guest_images_disk

2. Remove the storage pool's definition

   # virsh pool-undefine guest_images_disk

Procedure 12.1. Creating a partition-based storage pool with virt-manager

1. Open the storage pool settings

   1. In the virt-manager graphical interface, select the host physical machine from the main window.
      Open the Edit menu and select Connection Details
      

      Figure 12.1. Connection Details

   2. Click on the Storage tab of the Connection Details window.
      

      Figure 12.2. Storage tab

2. Create the new storage pool

   1. Add a new pool (part 1)

      Press the + button (the add pool button). The Add a New Storage Pool wizard appears.
      Choose a Name for the storage pool. This example uses the name guest_images_fs. Change the Type to fs: Pre-Formatted Block Device.
      

      Figure 12.3. Storage pool name and type

      Press the Forward button to continue.

   2. Add a new pool (part 2)

      Change the Target Path, Format, and Source Path fields.
      

      Figure 12.4. Storage pool path and format

      Target Path

      Enter the location to mount the source device for the storage pool in the Target Path field. If the location does not already exist, virt-manager will create the directory.

      Format

      Select a format from the Format list. The device is formatted with the selected format.

      This example uses the ext4 file system, the default Red Hat Enterprise Linux file system.

      Source Path

      Enter the device in the Source Path field.

      This example uses the /dev/sdc1 device.

      Verify the details and press the Finish button to create the storage pool.

3. Verify the new storage pool

   The new storage pool appears in the storage list on the left after a few seconds. Verify the size is reported as expected, 458.20 GB Free in this example. Verify the State field reports the new storage pool as Active.
   Select the storage pool. In the Autostart field, click the On Boot check box. This will make sure the storage device starts whenever the libvirtd service starts.
   

   Figure 12.5. Storage list confirmation

   The storage pool is now created, close the Connection Details window.

1. To avoid any issues with other guest virtual machines using the same pool, it is best to stop the storage pool and release any resources in use by it. To do this, select the storage pool you want to stop and click the red X icon at the bottom of the Storage window.
   

   Figure 12.6. Stop Icon

2. Delete the storage pool by clicking the Trash can icon. This icon is only enabled if you stop the storage pool first.

Procedure 12.2. Creating pre-formatted block device storage pools using virsh

1. Create the storage pool definition

   Use the virsh pool-define-as command to create a new storage pool definition. There are three options that must be provided to define a pre-formatted disk as a storage pool:

   Partition name

   The name parameter determines the name of the storage pool. This example uses the name guest_images_fs in the example below.

   device

   The device parameter with the path attribute specifies the device path of the storage device. This example uses the partition /dev/sdc1.

   mountpoint

   The mountpoint on the local file system where the formatted device will be mounted. If the mount point directory does not exist, the virsh command can create the directory.

   The directory /guest_images is used in this example.

   # virsh pool-define-as guest_images_fs fs - - /dev/sdc1 - "/guest_images"
   Pool guest_images_fs defined
   

   The new pool and mount points are now created.

2. Verify the new pool

   List the present storage pools.

   # virsh pool-list --all
   Name                 State      Autostart
   -----------------------------------------
   default              active     yes
   guest_images_fs      inactive   no
   

3. Create the mount point

   Use the virsh pool-build command to create a mount point for a pre-formatted file system storage pool.

   # virsh pool-build guest_images_fs
   Pool guest_images_fs built
   # ls -la /guest_images
   total 8
   drwx------.  2 root root 4096 May 31 19:38 .
   dr-xr-xr-x. 25 root root 4096 May 31 19:38 ..
   # virsh pool-list --all
   Name                 State      Autostart
   -----------------------------------------
   default              active     yes
   guest_images_fs      inactive   no
   

4. Start the storage pool

   Use the virsh pool-start command to mount the file system onto the mount point and make the pool available for use.

   # virsh pool-start guest_images_fs
   Pool guest_images_fs started
   # virsh pool-list --all
   Name                 State      Autostart
   -----------------------------------------
   default              active     yes
   guest_images_fs      active     no
   

5. Turn on autostart

   By default, a storage pool defined with virsh, is not set to automatically start each time libvirtd starts. To remedy this, enable the automatic start with the virsh pool-autostart command. The storage pool is now automatically started each time libvirtd starts.

   # virsh pool-autostart guest_images_fs
   Pool guest_images_fs marked as autostarted
   
   # virsh pool-list --all
   Name                 State      Autostart
   -----------------------------------------
   default              active     yes
   guest_images_fs      active     yes
   

6. Verify the storage pool

   Verify the storage pool was created correctly, the sizes reported are as expected, and the state is reported as running. Verify there is a "lost+found" directory in the mount point on the file system, indicating the device is mounted.

   # virsh pool-info guest_images_fs
   Name:           guest_images_fs
   UUID:           c7466869-e82a-a66c-2187-dc9d6f0877d0
   State:          running
   Persistent:     yes
   Autostart:      yes
   Capacity:       458.39 GB
   Allocation:     197.91 MB
   Available:      458.20 GB
   # mount | grep /guest_images
   /dev/sdc1 on /guest_images type ext4 (rw)
   # ls -la /guest_images
   total 24
   drwxr-xr-x.  3 root root  4096 May 31 19:47 .
   dr-xr-xr-x. 25 root root  4096 May 31 19:38 ..
   drwx------.  2 root root 16384 May 31 14:18 lost+found
   

1. To avoid any issues with other guest virtual machines using the same pool, it is best to stop the storage pool and release any resources in use by it.

   # virsh pool-destroy guest_images_disk

2. Optionally, if you want to remove the directory where the storage pool resides use the following command:

   # virsh pool-delete guest_images_disk

3. Remove the storage pool's definition

   # virsh pool-undefine guest_images_disk

1. Create the local directory

   1. Optional: Create a new directory for the storage pool

      Create the directory on the host physical machine for the storage pool. This example uses a directory named /guest virtual machine_images.

      # mkdir /guest_images

   2. Set directory ownership

      Change the user and group ownership of the directory. The directory must be owned by the root user.

      # chown root:root /guest_images

   3. Set directory permissions

      Change the file permissions of the directory.

      # chmod 700 /guest_images

   4. Verify the changes

      Verify the permissions were modified. The output shows a correctly configured empty directory.

      # ls -la /guest_images
      total 8
      drwx------.  2 root root 4096 May 28 13:57 .
      dr-xr-xr-x. 26 root root 4096 May 28 13:57 ..
      

2. Configure SELinux file contexts

   Configure the correct SELinux context for the new directory. Note that the name of the pool and the directory do not have to match. However, when you shutdown the guest virtual machine, libvirt has to set the context back to a default value. The context of the directory determines what this default value is. It is worth explicitly labeling the directory virt_image_t, so that when the guest virtual machine is shutdown, the images get labeled 'virt_image_t' and are thus isolated from other processes running on the host physical machine.

   # semanage fcontext -a -t virt_image_t '/guest_images(/.*)?'
   # restorecon -R /guest_images
   

3. Open the storage pool settings

   1. In the virt-manager graphical interface, select the host physical machine from the main window.
      Open the Edit menu and select Connection Details
      

      Figure 12.7. Connection details window

   2. Click on the Storage tab of the Connection Details window.
      

      Figure 12.8. Storage tab

4. Create the new storage pool

   1. Add a new pool (part 1)

      Press the + button (the add pool button). The Add a New Storage Pool wizard appears.
      Choose a Name for the storage pool. This example uses the name guest_images. Change the Type to dir: Filesystem Directory.
      

      Figure 12.9. Name the storage pool

      Press the Forward button to continue.

   2. Add a new pool (part 2)

      Change the Target Path field. For example, /guest_images.
      Verify the details and press the Finish button to create the storage pool.

5. Verify the new storage pool

   The new storage pool appears in the storage list on the left after a few seconds. Verify the size is reported as expected, 36.41 GB Free in this example. Verify the State field reports the new storage pool as Active.
   Select the storage pool. In the Autostart field, confirm that the On Boot check box is checked. This will make sure the storage pool starts whenever the libvirtd service starts.
   

   Figure 12.10. Verify the storage pool information

   The storage pool is now created, close the Connection Details window.

1. To avoid any issues with other guest virtual machines using the same pool, it is best to stop the storage pool and release any resources in use by it. To do this, select the storage pool you want to stop and click the red X icon at the bottom of the Storage window.
   

   Figure 12.11. Stop Icon

2. Delete the storage pool by clicking the Trash can icon. This icon is only enabled if you stop the storage pool first.

1. Create the storage pool definition

   Use the virsh pool-define-as command to define a new storage pool. There are two options required for creating directory-based storage pools:
   • The name of the storage pool.
     This example uses the name guest_images. All further virsh commands used in this example use this name.
   • The path to a file system directory for storing guest image files. If this directory does not exist, virsh will create it.
     This example uses the /guest_images directory.

    # virsh pool-define-as guest_images dir - - - - "/guest_images"
   Pool guest_images defined

2. Verify the storage pool is listed

   Verify the storage pool object is created correctly and the state reports it as inactive.

   # virsh pool-list --all
   Name                 State      Autostart
   -----------------------------------------
   default              active     yes
   guest_images     inactive   no

3. Create the local directory

   Use the virsh pool-build command to build the directory-based storage pool for the directory guest_images (for example), as shown:

   # virsh pool-build guest_images
   Pool guest_images built
   # ls -la /guest_images
   total 8
   drwx------.  2 root root 4096 May 30 02:44 .
   dr-xr-xr-x. 26 root root 4096 May 30 02:44 ..
   # virsh pool-list --all
   Name                 State      Autostart
   -----------------------------------------
   default              active     yes
   guest_images     inactive   no

4. Start the storage pool

   Use the virsh command pool-start to enable a directory storage pool, thereby allowing allowing volumes of the pool to be used as guest disk images.

   # virsh pool-start guest_images
   Pool guest_images started
   # virsh pool-list --all
   Name                 State      Autostart
   -----------------------------------------
   default             active     yes
   guest_images    active     no
   

5. Turn on autostart

   Turn on autostart for the storage pool. Autostart configures the libvirtd service to start the storage pool when the service starts.

   # virsh pool-autostart guest_images
   Pool guest_images marked as autostarted
   # virsh pool-list --all
   Name                 State      Autostart
   -----------------------------------------
   default              active     yes
   guest_images         active     yes
   

6. Verify the storage pool configuration

   Verify the storage pool was created correctly, the size is reported correctly, and the state is reported as running. If you want the pool to be accessible even if the guest virtual machine is not running, make sure that Persistent is reported as yes. If you want the pool to start automatically when the service starts, make sure that Autostart is reported as yes.

   # virsh pool-info guest_images
   Name:           guest_images
   UUID:           779081bf-7a82-107b-2874-a19a9c51d24c
   State:          running
   Persistent:     yes
   Autostart:      yes
   Capacity:       49.22 GB
   Allocation:     12.80 GB
   Available:      36.41 GB
   
   # ls -la /guest_images
   total 8
   drwx------.  2 root root 4096 May 30 02:44 .
   dr-xr-xr-x. 26 root root 4096 May 30 02:44 ..
   #
   

1. To avoid any issues with other guest virtual machines using the same pool, it is best to stop the storage pool and release any resources in use by it.

   # virsh pool-destroy guest_images_disk

2. Optionally, if you want to remove the directory where the storage pool resides use the following command:

   # virsh pool-delete guest_images_disk

3. Remove the storage pool's definition

   # virsh pool-undefine guest_images_disk

1. Optional: Create new partition for LVM volumes

   These steps describe how to create a new partition and LVM volume group on a new hard disk drive.

   Warning

   This procedure will remove all data from the selected storage device.

   1. Create a new partition

      Use the fdisk command to create a new disk partition from the command line. The following example creates a new partition that uses the entire disk on the storage device /dev/sdb.

      # fdisk /dev/sdb
      Command (m for help):
      

      Press n for a new partition.
   2. Press p for a primary partition.

      Command action
         e   extended
         p   primary partition (1-4)
      

   3. Choose an available partition number. In this example the first partition is chosen by entering 1.

      Partition number (1-4): 1

   4. Enter the default first cylinder by pressing Enter.

      First cylinder (1-400, default 1):
      

   5. Select the size of the partition. In this example the entire disk is allocated by pressing Enter.

      Last cylinder or +size or +sizeM or +sizeK (2-400, default 400):
      

   6. Set the type of partition by pressing t.

      Command (m for help): t

   7. Choose the partition you created in the previous steps. In this example, the partition number is 1.

      Partition number (1-4): 1

   8. Enter 8e for a Linux LVM partition.

      Hex code (type L to list codes): 8e

   9. write changes to disk and quit.

      Command (m for help): w
      Command (m for help): q

   10. Create a new LVM volume group

       Create a new LVM volume group with the vgcreate command. This example creates a volume group named guest_images_lvm.

       # vgcreate guest_images_lvm /dev/sdb1
         Physical volume "/dev/vdb1" successfully created
         Volume group "guest_images_lvm" successfully created
       

   The new LVM volume group, guest_images_lvm, can now be used for an LVM-based storage pool.

2. Open the storage pool settings

   1. In the virt-manager graphical interface, select the host from the main window.
      Open the Edit menu and select Connection Details
      

      Figure 12.12. Connection details

   2. Click on the Storage tab.
      

      Figure 12.13. Storage tab

3. Create the new storage pool

   1. Start the Wizard

      Press the + button (the add pool button). The Add a New Storage Pool wizard appears.
      Choose a Name for the storage pool. We use guest_images_lvm for this example. Then change the Type to logical: LVM Volume Group, and
      

      Figure 12.14. Add LVM storage pool

      Press the Forward button to continue.

   2. Add a new pool (part 2)

      Change the Target Path field. This example uses /guest_images.
      Now fill in the Target Path and Source Path fields, then tick the Build Pool check box.
      • Use the Target Path field to either select an existing LVM volume group or as the name for a new volume group. The default format is /dev/storage_pool_name.
        This example uses a new volume group named /dev/guest_images_lvm.
      • The Source Path field is optional if an existing LVM volume group is used in the Target Path.
        For new LVM volume groups, input the location of a storage device in the Source Path field. This example uses a blank partition /dev/sdc.
      • The Build Pool check box instructs virt-manager to create a new LVM volume group. If you are using an existing volume group you should not select the Build Pool check box.
        This example is using a blank partition to create a new volume group so the Build Pool check box must be selected.
      

      Figure 12.15. Add target and source

      Verify the details and press the Finish button format the LVM volume group and create the storage pool.

   3. Confirm the device to be formatted

      A warning message appears.
      

      Figure 12.16. Warning message

      Press the Yes button to proceed to erase all data on the storage device and create the storage pool.

4. Verify the new storage pool

   The new storage pool will appear in the list on the left after a few seconds. Verify the details are what you expect, 465.76 GB Free in our example. Also verify the State field reports the new storage pool as Active.
   It is generally a good idea to have the Autostart check box enabled, to ensure the storage pool starts automatically with libvirtd.
   

   Figure 12.17. Confirm LVM storage pool details

   Close the Host Details dialog, as the task is now complete.

1. To avoid any issues with other guest virtual machines using the same pool, it is best to stop the storage pool and release any resources in use by it. To do this, select the storage pool you want to stop and click the red X icon at the bottom of the Storage window.
   

   Figure 12.18. Stop Icon

2. Delete the storage pool by clicking the Trash can icon. This icon is only enabled if you stop the storage pool first.

Procedure 12.3. Creating an LVM-based storage pool with virsh

1. Define the pool name guest_images_lvm.

   # virsh pool-define-as guest_images_lvm logical - - /dev/sdc libvirt_lvm \ /dev/libvirt_lvm
   Pool guest_images_lvm defined
   

2. Build the pool according to the specified name. If you are using an already existing volume group, skip this step.

   # virsh pool-build guest_images_lvm
   
   Pool guest_images_lvm built
   

3. Initialize the new pool.

   # virsh pool-start guest_images_lvm
   
   Pool guest_images_lvm started
   

4. Show the volume group information with the vgs command.

   # vgs
   VG          #PV #LV #SN Attr   VSize   VFree
   libvirt_lvm   1   0   0 wz--n- 465.76g 465.76g
   

5. Set the pool to start automatically.

   # virsh pool-autostart guest_images_lvm
   Pool guest_images_lvm marked as autostarted
   

6. List the available pools with the virsh command.

   # virsh pool-list --all
   Name                 State      Autostart
   -----------------------------------------
   default              active     yes
   guest_images_lvm     active     yes
   

7. The following commands demonstrate the creation of three volumes (volume1, volume2 and volume3) within this pool.

   # virsh vol-create-as guest_images_lvm volume1 8G
   Vol volume1 created
   
   # virsh vol-create-as guest_images_lvm volume2 8G
   Vol volume2 created
   
   # virsh vol-create-as guest_images_lvm volume3 8G
   Vol volume3 created
   

8. List the available volumes in this pool with the virsh command.

   # virsh vol-list guest_images_lvm
   Name                 Path
   -----------------------------------------
   volume1              /dev/libvirt_lvm/volume1
   volume2              /dev/libvirt_lvm/volume2
   volume3              /dev/libvirt_lvm/volume3
   

9. The following two commands (lvscan and lvs) display further information about the newly created volumes.

   # lvscan
   ACTIVE            '/dev/libvirt_lvm/volume1' [8.00 GiB] inherit
   ACTIVE            '/dev/libvirt_lvm/volume2' [8.00 GiB] inherit
   ACTIVE            '/dev/libvirt_lvm/volume3' [8.00 GiB] inherit
   
   # lvs
   LV       VG            Attr     LSize   Pool Origin Data%  Move Log Copy%  Convert
   volume1  libvirt_lvm   -wi-a-   8.00g
   volume2  libvirt_lvm   -wi-a-   8.00g
   volume3  libvirt_lvm   -wi-a-   8.00g
   

1. To avoid any issues with other guests using the same pool, it is best to stop the storage pool and release any resources in use by it.

   # virsh pool-destroy guest_images_disk

2. Optionally, if you want to remove the directory where the storage pool resides use the following command:

   # virsh pool-delete guest_images_disk

3. Remove the storage pool's definition

   # virsh pool-undefine guest_images_disk

Procedure 12.4. Creating an iSCSI target

1. Install the required packages

   Install the scsi-target-utils package and all dependencies

   # yum install scsi-target-utils

2. Start the tgtd service

   The tgtd service host physical machines SCSI targets and uses the iSCSI protocol to host physical machine targets. Start the tgtd service and make the service persistent after restarting with the chkconfig command.

   # service tgtd start
   # chkconfig tgtd on

3. Optional: Create LVM volumes

   LVM volumes are useful for iSCSI backing images. LVM snapshots and resizing can be beneficial for guest virtual machines. This example creates an LVM image named virtimage1 on a new volume group named virtstore on a RAID5 array for hosting guest virtual machines with iSCSI.

   1. Create the RAID array

      Creating software RAID5 arrays is covered by the Red Hat Enterprise Linux Deployment Guide.

   2. Create the LVM volume group

      Create a volume group named virtstore with the vgcreate command.

      # vgcreate virtstore /dev/md1

   3. Create a LVM logical volume

      Create a logical volume group named virtimage1 on the virtstore volume group with a size of 20GB using the lvcreate command.

      # lvcreate --size 20G -n virtimage1 virtstore

      The new logical volume, virtimage1, is ready to use for iSCSI.

4. Optional: Create file-based images

   File-based storage is sufficient for testing but is not recommended for production environments or any significant I/O activity. This optional procedure creates a file based imaged named virtimage2.img for an iSCSI target.

   1. Create a new directory for the image

      Create a new directory to store the image. The directory must have the correct SELinux contexts.

      # mkdir -p /var/lib/tgtd/virtualization

   2. Create the image file

      Create an image named virtimage2.img with a size of 10GB.

      # dd if=/dev/zero of=/var/lib/tgtd/virtualization/virtimage2.img bs=1M seek=10000 count=0

   3. Configure SELinux file contexts

      Configure the correct SELinux context for the new image and directory.

      # restorecon -R /var/lib/tgtd

      The new file-based image, virtimage2.img, is ready to use for iSCSI.

5. Create targets

   Targets can be created by adding a XML entry to the /etc/tgt/targets.conf file. The target attribute requires an iSCSI Qualified Name (IQN). The IQN is in the format:

   iqn.yyyy-mm.reversed domain name:optional identifier text

   Where:
   • yyyy-mm represents the year and month the device was started (for example: 2010-05);
   • reversed domain name is the host physical machines domain name in reverse (for example server1.example.com in an IQN would be com.example.server1); and
   • optional identifier text is any text string, without spaces, that assists the administrator in identifying devices or hardware.
   This example creates iSCSI targets for the two types of images created in the optional steps on server1.example.com with an optional identifier trial. Add the following to the /etc/tgt/targets.conf file.

   <target iqn.2010-05.com.example.server1:iscsirhel6guest>
      backing-store /dev/virtstore/virtimage1  #LUN 1
      backing-store /var/lib/tgtd/virtualization/virtimage2.img  #LUN 2
      write-cache off
   </target>
   

   Ensure that the /etc/tgt/targets.conf file contains the default-driver iscsi line to set the driver type as iSCSI. The driver uses iSCSI by default.

   Important

   This example creates a globally accessible target without access control. Refer to the scsi-target-utils for information on implementing secure access.

6. Restart the tgtd service

   Restart the tgtd service to reload the configuration changes.

   # service tgtd restart

7. iptables configuration

   Open port 3260 for iSCSI access with iptables.

   # iptables -I INPUT -p tcp -m tcp --dport 3260 -j ACCEPT
   # service iptables save
   # service iptables restart

8. Verify the new targets

   View the new targets to ensure the setup was successful with the tgt-admin --show command.

   # tgt-admin --show
   Target 1: iqn.2010-05.com.example.server1:iscsirhel6guest
   System information:
   Driver: iscsi
   State: ready
   I_T nexus information:
   LUN information:
   LUN: 0
       Type: controller
       SCSI ID: IET     00010000
       SCSI SN: beaf10
       Size: 0 MB
       Online: Yes
       Removable media: No
       Backing store type: rdwr
       Backing store path: None
   LUN: 1
       Type: disk
       SCSI ID: IET     00010001
       SCSI SN: beaf11
       Size: 20000 MB
       Online: Yes
       Removable media: No
       Backing store type: rdwr
       Backing store path: /dev/virtstore/virtimage1
   LUN: 2
       Type: disk
       SCSI ID: IET     00010002
       SCSI SN: beaf12
       Size: 10000 MB
       Online: Yes
       Removable media: No
       Backing store type: rdwr
       Backing store path: /var/lib/tgtd/virtualization/virtimage2.img
   Account information:
   ACL information:
   ALL
   

   Warning

   The ACL list is set to all. This allows all systems on the local network to access this device. It is recommended to set host physical machine access ACLs for production environments.

9. Optional: Test discovery

   Test whether the new iSCSI device is discoverable.

   # iscsiadm --mode discovery --type sendtargets --portal server1.example.com
   127.0.0.1:3260,1 iqn.2010-05.com.example.server1:iscsirhel6guest

10. Optional: Test attaching the device

    Attach the new device (iqn.2010-05.com.example.server1:iscsirhel6guest) to determine whether the device can be attached.

    # iscsiadm -d2 -m node --login
    scsiadm: Max file limits 1024 1024
    
    Logging in to [iface: default, target: iqn.2010-05.com.example.server1:iscsirhel6guest, portal: 10.0.0.1,3260]
    Login to [iface: default, target: iqn.2010-05.com.example.server1:iscsirhel6guest, portal: 10.0.0.1,3260] successful.

    Detach the device.

    # iscsiadm -d2 -m node --logout
    scsiadm: Max file limits 1024 1024
    
    Logging out of session [sid: 2, target: iqn.2010-05.com.example.server1:iscsirhel6guest, portal: 10.0.0.1,3260
    Logout of [sid: 2, target: iqn.2010-05.com.example.server1:iscsirhel6guest, portal: 10.0.0.1,3260] successful.

Procedure 12.5. Adding an iSCSI device to virt-manager

1. Open the host physical machine's storage tab

   Open the Storage tab in the Connection Details window.
   1. Open virt-manager.
   2. Select a host physical machine from the main virt-manager window. Click Edit menu and select Connection Details.
      

      Figure 12.19. Connection details

   3. Click on the Storage tab.
      

      Figure 12.20. Storage menu

2. Add a new pool (part 1)

   Press the + button (the add pool button). The Add a New Storage Pool wizard appears.
   

   Figure 12.21. Add an iscsi storage pool name and type

   Choose a name for the storage pool, change the Type to iscsi, and press Forward to continue.

3. Add a new pool (part 2)

   You will need the information you used in Section 12.5, “iSCSI-based Storage Pools” and Procedure 12.4, “Creating an iSCSI target” to complete the fields in this menu.
   1. Enter the iSCSI source and target. The Format option is not available as formatting is handled by the guest virtual machines. It is not advised to edit the Target Path. The default target path value, /dev/disk/by-path/, adds the drive path to that directory. The target path should be the same on all host physical machines for migration.
   2. Enter the host name or IP address of the iSCSI target. This example uses host1.example.com.
   3. In the Source Pathfield, enter the iSCSI target IQN. If you look at Procedure 12.4, “Creating an iSCSI target” in Section 12.5, “iSCSI-based Storage Pools”, this is the information you added in the /etc/tgt/targets.conf file. This example uses iqn.2010-05.com.example.server1:iscsirhel6guest.
   4. Check the IQN check box to enter the IQN for the initiator. This example uses iqn.2010-05.com.example.host1:iscsirhel6.
   5. Click Finish to create the new storage pool.
   

   Figure 12.22. Create an iscsi storage pool

1. To avoid any issues with other guest virtual machines using the same pool, it is best to stop the storage pool and release any resources in use by it. To do this, select the storage pool you want to stop and click the red X icon at the bottom of the Storage window.
   

   Figure 12.23. Stop Icon

2. Delete the storage pool by clicking the Trash can icon. This icon is only enabled if you stop the storage pool first.

1. Use pool-define-as to define the pool from the command line

   Storage pool definitions can be created with the virsh command line tool. Creating storage pools with virsh is useful for systems administrators using scripts to create multiple storage pools.
   The virsh pool-define-as command has several parameters which are accepted in the following format:

   virsh pool-define-as name type source-host source-path source-dev source-name target

   The parameters are explained as follows:

   type

   defines this pool as a particular type, iscsi for example

   name

   must be unique and sets the name for the storage pool

   source-host and source-path

   the host name and iSCSI IQN respectively

   source-dev and source-name

   these parameters are not required for iSCSI-based pools, use a - character to leave the field blank.

   target

   defines the location for mounting the iSCSI device on the host physical machine

   The example below creates the same iSCSI-based storage pool as the previous step.

   #   virsh pool-define-as --name scsirhel6guest --type iscsi \
        --source-host server1.example.com \
        --source-dev iqn.2010-05.com.example.server1:iscsirhel6guest
        --target /dev/disk/by-path
   Pool iscsirhel6guest defined

2. Verify the storage pool is listed

   Verify the storage pool object is created correctly and the state reports as inactive.

   # virsh pool-list --all
   Name                 State      Autostart
   -----------------------------------------
   default              active     yes
   iscsirhel6guest      inactive   no

3. Start the storage pool

   Use the virsh command pool-start for this. pool-start enables a directory storage pool, allowing it to be used for volumes and guest virtual machines.

   # virsh pool-start guest_images_disk
   Pool guest_images_disk started
   # virsh pool-list --all
   Name                 State      Autostart
   -----------------------------------------
   default              active     yes
   iscsirhel6guest      active     no
   

4. Turn on autostart

   Turn on autostart for the storage pool. Autostart configures the libvirtd service to start the storage pool when the service starts.

   # virsh pool-autostart iscsirhel6guest
   Pool iscsirhel6guest marked as autostarted

   Verify that the iscsirhel6guest pool has autostart set:

   # virsh pool-list --all
   Name                 State      Autostart
   -----------------------------------------
   default              active     yes
   iscsirhel6guest      active     yes
   

5. Verify the storage pool configuration

   Verify the storage pool was created correctly, the sizes reported correctly, and the state reports as running.

   # virsh pool-info iscsirhel6guest
   Name:           iscsirhel6guest
   UUID:           afcc5367-6770-e151-bcb3-847bc36c5e28
   State:          running
   Persistent:     unknown
   Autostart:      yes
   Capacity:       100.31 GB
   Allocation:     0.00
   Available:      100.31 GB
   

1. To avoid any issues with other guest virtual machines using the same pool, it is best to stop the storage pool and release any resources in use by it.

   # virsh pool-destroy guest_images_disk

2. Remove the storage pool's definition

   # virsh pool-undefine guest_images_disk

1. Open the host physical machine's storage tab

   Open the Storage tab in the Host Details window.
   1. Open virt-manager.
   2. Select a host physical machine from the main virt-manager window. Click Edit menu and select Connection Details.
      

      Figure 12.24. Connection details

   3. Click on the Storage tab.
      

      Figure 12.25. Storage tab

2. Create a new pool (part 1)

   Press the + button (the add pool button). The Add a New Storage Pool wizard appears.
   

   Figure 12.26. Add an NFS name and type

   Choose a name for the storage pool and press Forward to continue.

3. Create a new pool (part 2)

   Enter the target path for the device, the host name and the NFS share path. Set the Format option to NFS or auto (to detect the type). The target path must be identical on all host physical machines for migration.
   Enter the host name or IP address of the NFS server. This example uses server1.example.com.
   Enter the NFS path. This example uses /nfstrial.
   

   Figure 12.27. Create an NFS storage pool

   Press Finish to create the new storage pool.

1. To avoid any issues with other guests using the same pool, it is best to stop the storage pool and release any resources in use by it. To do this, select the storage pool you want to stop and click the red X icon at the bottom of the Storage window.
   

   Figure 12.28. Stop Icon

2. Delete the storage pool by clicking the Trash can icon. This icon is only enabled if you stop the storage pool first.

Procedure 12.6. Creating a vHBA

1. Locate HBAs on the host system

   To locate the HBAs on your host system, examine the SCSI devices on the host system to locate a scsi_host with vport capability.
   Run the following command to retrieve a scsi_host list:

   # virsh nodedev-list --cap scsi_host
   scsi_host0
   scsi_host1
   scsi_host2
   scsi_host3
   scsi_host4
   

   For each scsi_host, run the following command to examine the device XML for the line <capability type='vport_ops'>, which indicates a scsi_host with vport capability.

   # virsh nodedev-dumpxml scsi_hostN

2. Check the HBA's details

   Use the virsh nodedev-dumpxml HBA_device command to see the HBA's details.
   The XML output from the virsh nodedev-dumpxml command will list the fields <name>, <wwnn>, and <wwpn>, which are used to create a vHBA. The <max_vports> value shows the maximum number of supported vHBAs.

    # virsh nodedev-dumpxml scsi_host3
   <device>
     <name>scsi_host3</name>
     <path>/sys/devices/pci0000:00/0000:00:04.0/0000:10:00.0/host3</path>
     <parent>pci_0000_10_00_0</parent>
     <capability type='scsi_host'>
       <host>3</host>
       <capability type='fc_host'>
         <wwnn>20000000c9848140</wwnn>
         <wwpn>10000000c9848140</wwpn>
         <fabric_wwn>2002000573de9a81</fabric_wwn>
       </capability>
       <capability type='vport_ops'>
         <max_vports>127</max_vports>
         <vports>0</vports>
       </capability>
     </capability>
   </device>   

   In this example, the <max_vports> value shows there are a total 127 virtual ports available for use in the HBA configuration. The <vports> value shows the number of virtual ports currently being used. These values update after creating a vHBA.

3. Create a vHBA host device

   Create an XML file similar to the following (in this example, named vhba_host3.xml) for the vHBA host.

   # cat vhba_host3.xml
      <device>
        <parent>scsi_host3</parent>
        <capability type='scsi_host'>
          <capability type='fc_host'>
          </capability>
        </capability>
      </device>   

   The <parent> field specifies the HBA device to associate with this vHBA device. The details in the <device> tag are used in the next step to create a new vHBA device for the host. See http://libvirt.org/formatnode.html for more information on the nodedev XML format.

4. Create a new vHBA on the vHBA host device

   To create a vHBA on vhba_host3, use the virsh nodedev-create command:

   # virsh nodedev-create vhba_host3.xml
   Node device scsi_host5 created from vhba_host3.xml

5. Verify the vHBA

   Verify the new vHBA's details (scsi_host5) with the virsh nodedev-dumpxml command:

   # virsh nodedev-dumpxml scsi_host5
   <device>
     <name>scsi_host5</name>
     <path>/sys/devices/pci0000:00/0000:00:04.0/0000:10:00.0/host3/vport-3:0-0/host5</path>
     <parent>scsi_host3</parent>
     <capability type='scsi_host'>
       <host>5</host>
       <capability type='fc_host'>
         <wwnn>5001a4a93526d0a1</wwnn>
         <wwpn>5001a4ace3ee047d</wwpn>
         <fabric_wwn>2002000573de9a81</fabric_wwn>
       </capability>
     </capability>
   </device>  

1. Create a SCSI storage pool

   To create a vHBA configuration, first create a libvirt 'scsi' storage pool XML file based on the vHBA using the format below.

   Note

   Ensure you use the vHBA created in Procedure 12.6, “Creating a vHBA” as the host name, modifying the vHBA name scsi_hostN to hostN for the storage pool configuration. In this example, the vHBA is named scsi_host5, which is specified as <adapter name='host5'/> in a Red Hat Enterprise Linux 6 libvirt storage pool.
   It is recommended to use a stable location for the <path> value, such as one of the /dev/disk/by-{path|id|uuid|label} locations on your system. More information on <path> and the elements within <target> can be found at http://libvirt.org/formatstorage.html.
   In this example, the 'scsi' storage pool is named vhbapool_host3.xml:

     <pool type='scsi'>
        <name>vhbapool_host3</name>
        <uuid>e9392370-2917-565e-692b-d057f46512d6</uuid>
        <capacity unit='bytes'>0</capacity>
        <allocation unit='bytes'>0</allocation>
        <available unit='bytes'>0</available>
        <source>
          <adapter name='host5'/>
        </source>
         <target>
           <path>/dev/disk/by-path</path>
           <permissions>
             <mode>0700</mode>
             <owner>0</owner>
             <group>0</group>
           </permissions>
         </target>
       </pool> 

2. Define the pool

   To define the storage pool (named vhbapool_host3 in this example), use the virsh pool-define command:

            # virsh pool-define vhbapool_host3.xml
            Pool vhbapool_host3 defined from vhbapool_host3.xml
   

3. Start the pool

   Start the storage pool with the following command:

   # virsh pool-start vhbapool_host3
   Pool vhbapool_host3 started
   

4. Enable autostart

   Finally, to ensure that subsequent host reboots will automatically define vHBAs for use in virtual machines, set the storage pool autostart feature (in this example, for a pool named vhbapool_host3):

   # virsh pool-autostart vhbapool_host3

1. Find available LUNs

   First, use the virsh vol-list command in order to generate a list of available LUNs on the vHBA. For example:

   # virsh vol-list vhbapool_host3
    Name                 Path
   ------------------------------------------------------------------------------
    unit:0:4:0           /dev/disk/by-path/pci-0000:10:00.0-fc-0x5006016844602198-lun-0
    unit:0:5:0           /dev/disk/by-path/pci-0000:10:00.0-fc-0x5006016044602198-lun-0

   The list of LUN names displayed will be available for use as disk volumes in virtual machine configurations.

2. Add the vHBA LUN to the virtual machine

   Add the vHBA LUN to the virtual machine by specifying in the virtual machine's XML:
   • the device type as lun or disk in the <disk> parameter, and
   • the source device in the <source> parameter. Note this can be entered as /dev/sdaN, or as a symbolic link generated by udev in /dev/disk/by-path|by-id|by-uuid|by-label, which can be found by running the virsh vol-list pool command.
   For example:

      <disk type='block' device='lun'>
        <driver name='qemu' type='raw'/>
        <source dev='/dev/disk/by-path/pci-0000\:04\:00.1-fc-0x203400a0b85ad1d7-lun-0'/>
        <target dev='sda' bus='scsi'/>
      </disk>
         

Procedure 13.1. Adding file-based storage

1. Create a storage file or use an existing file (such as an IMG file). Note that both of the following commands create a 4GB file which can be used as additional storage for a guest:
   • Pre-allocated files are recommended for file-based storage images. Create a pre-allocated file using the following dd command as shown:

     # dd if=/dev/zero of=/var/lib/libvirt/images/FileName.img bs=1M count=4096

   • Alternatively, create a sparse file instead of a pre-allocated file. Sparse files are created much faster and can be used for testing, but are not recommended for production environments due to data integrity and performance issues.

     # dd if=/dev/zero of=/var/lib/libvirt/images/FileName.img bs=1M seek=4096 count=0

2. Create the additional storage by writing a <disk> element in a new file. In this example, this file will be known as NewStorage.xml.
   A <disk> element describes the source of the disk, and a device name for the virtual block device. The device name should be unique across all devices in the guest, and identifies the bus on which the guest will find the virtual block device. The following example defines a virtio block device whose source is a file-based storage container named FileName.img:

   <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='/var/lib/libvirt/images/FileName.img'/>
      <target dev='vdb'/>
   </disk>
   

   Device names can also start with "hd" or "sd", identifying respectively an IDE and a SCSI disk. The configuration file can also contain an <address> sub-element that specifies the position on the bus for the new device. In the case of virtio block devices, this should be a PCI address. Omitting the <address> sub-element lets libvirt locate and assign the next available PCI slot.
3. Attach the CD-ROM as follows:

   <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='/var/lib/libvirt/images/FileName.img'/>
      <readonly/>
      <target dev='hdc'/>
   </disk >
   

4. Add the device defined in NewStorage.xml with your guest (Guest1):

   # virsh attach-device --config Guest1 ~/NewStorage.xml

   Note

   This change will only apply after the guest has been destroyed and restarted. In addition, persistent devices can only be added to a persistent domain, that is a domain whose configuration has been saved with virsh define command.
   If the guest is running, and you want the new device to be added temporarily until the guest is destroyed, omit the --config option:

   # virsh attach-device Guest1 ~/NewStorage.xml

   Note

   The virsh command allows for an attach-disk command that can set a limited number of parameters with a simpler syntax and without the need to create an XML file. The attach-disk command is used in a similar manner to the attach-device command mentioned previously, as shown:

   # virsh attach-disk Guest1 /var/lib/libvirt/images/FileName.img vdb --cache none --driver qemu --subdriver raw
   

   Note that the virsh attach-disk command also accepts the --config option.
5. Start the guest machine (if it is currently not running):

   # virsh start Guest1

   Note

   The following steps are Linux guest specific. Other operating systems handle new storage devices in different ways. For other systems, refer to that operating system's documentation.

6. Partitioning the disk drive

   The guest now has a hard disk device called /dev/vdb. If required, partition this disk drive and format the partitions. If you do not see the device that you added, then it indicates that there is an issue with the disk hotplug in your guest's operating system.
   1. Start fdisk for the new device:

      # fdisk /dev/vdb
      Command (m for help):
      

   2. Type n for a new partition.
   3. The following appears:

      Command action
      e   extended
      p   primary partition (1-4)
      

      Type p for a primary partition.
   4. Choose an available partition number. In this example, the first partition is chosen by entering 1.

      Partition number (1-4): 1

   5. Enter the default first cylinder by pressing Enter.

      First cylinder (1-400, default 1):

   6. Select the size of the partition. In this example the entire disk is allocated by pressing Enter.

      Last cylinder or +size or +sizeM or +sizeK (2-400, default 400):

   7. Enter t to configure the partition type.

      Command (m for help): t

   8. Select the partition you created in the previous steps. In this example, the partition number is 1 as there was only one partition created and fdisk automatically selected partition 1.

      Partition number (1-4): 1

   9. Enter 83 for a Linux partition.

      Hex code (type L to list codes): 83

   10. Enter w to write changes and quit.

       Command (m for help): w
       

   11. Format the new partition with the ext3 file system.

       # mke2fs -j /dev/vdb1

7. Create a mount directory, and mount the disk on the guest. In this example, the directory is located in myfiles.

   # mkdir /myfiles
   # mount /dev/vdb1 /myfiles
   

   The guest now has an additional virtualized file-based storage device. Note however, that this storage will not mount persistently across reboot unless defined in the guest's /etc/fstab file:

   /dev/vdb1    /myfiles    ext3     defaults    0 0

Procedure 13.2. Adding physical block devices to guests

1. This procedure describes how to add a hard drive on the host physical machine to a guest. It applies to all physical block devices, including CD-ROM, DVD and floppy devices.
   Physically attach the hard disk device to the host physical machine. Configure the host physical machine if the drive is not accessible by default.
2. Do one of the following:
   1. Create the additional storage by writing a disk element in a new file. In this example, this file will be known as NewStorage.xml. The following example is a configuration file section which contains an additional device-based storage container for the host physical machine partition /dev/sr0:

      <disk type='block' device='disk'>
            <driver name='qemu' type='raw' cache='none'/>
            <source dev='/dev/sr0'/>
            <target dev='vdc' bus='virtio'/>
      </disk>
      

   2. Follow the instruction in the previous section to attach the device to the guest virtual machine. Alternatively, you can use the virsh attach-disk command, as shown:

      # virsh attach-disk Guest1 /dev/sr0 vdc
      

      Note that the following options are available:
      • The virsh attach-disk command also accepts the --config, --type, and --mode options, as shown:

        # virsh attach-disk Guest1 /dev/sr0 vdc --config --type cdrom --mode readonly

      • Additionally, --type also accepts --type disk in cases where the device is a hard drive.
3. The guest virtual machine now has a new hard disk device called /dev/vdc on Linux (or something similar, depending on what the guest virtual machine OS chooses) or D: drive (for example) on Windows. You can now initialize the disk from the guest virtual machine, following the standard procedures for the guest virtual machine's operating system. Refer to Procedure 13.1, “Adding file-based storage” for an example.

   Warning

   When adding block devices to a guest, make sure to follow security considerations. This information is discussed in more detail in the Red Hat Enterprise Linux Virtualization Security Guide which can be found at https://access.redhat.com/site/documentation/.

   Important

   Guest virtual machines should not be given write access to whole disks or block devices (for example, /dev/sdb). Guest virtual machines with access to whole block devices may be able to modify volume labels, which can be used to compromise the host physical machine system. Use partitions (for example, /dev/sdb1) or LVM volumes to prevent this issue.

Procedure 14.2. virsh blockcommit

• Run the following command:

  # virsh blockcommit $dom $disk -base snap1 -top snap2 -wait -verbose

  The contents of snap2 are moved into snap1, resulting in:
  base ← snap1 ← active. Snap2 is no longer valid and can be deleted

  Warning

  blockcommit will corrupt any file that depends on the -base option (other than files that depend on the -top option, as those files now point to the base). To prevent this, do not commit changes into files shared by more than one guest. The -verbose option allows the progress to be printed on the screen.

Procedure 14.3. Using blockpull to Shorten a Backing Chain

1. It may be helpful to run this command prior to running blockpull:

   # virsh snapshot-create-as $dom $name - disk-only

2. If the chain looks like this: base ← snap1 ← snap2 ← active run the following:

   # virsh blockpull $dom $disk snap1

   This command makes 'snap1' the backing file of active, by pulling data from snap2 into active resulting in: base ← snap1 ← active.
3. Once the blockpull is complete, the libvirt tracking of the snapshot that created the extra image in the chain is no longer useful. Delete the tracking on the outdated snapshot with this command:

   # virsh snapshot-delete $dom $name - metadata

• Run the following command: blockresize [domain] [path size] where:
  • Domain is the unique target name or source file of the domain whose size you want to change
  • Path size is a scaled integer which defaults to KiB (blocks of 1024 bytes) if there is no suffix. You must use a suffix of "B" to for bytes.

1. Run the virsh edit [guestname] command to edit the XML configuration file for the guest virtual machine.
2. In the address type tag, add a multifunction='on' entry for function='0x0'.
   This enables the guest virtual machine to use the multifunction PCI devices.

   <disk type='file' device='disk'>
   <driver name='qemu' type='raw' cache='none'/>
   <source file='/var/lib/libvirt/images/rhel62-1.img'/>
   <target dev='vda' bus='virtio'/>
   <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/
   </disk>
   

   For a PCI device with two functions, amend the XML configuration file to include a second device with the same slot number as the first device and a different function number, such as function='0x1'.
   For Example:

   <disk type='file' device='disk'>
   <driver name='qemu' type='raw' cache='none'/>
   <source file='/var/lib/libvirt/images/rhel62-1.img'/>
   <target dev='vda' bus='virtio'/>
   <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
   </disk>
   <disk type='file' device='disk'>
   <driver name='qemu' type='raw' cache='none'/>
   <source file='/var/lib/libvirt/images/rhel62-2.img'/>
   <target dev='vdb' bus='virtio'/>
   <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
   </disk>
   

3. lspci output from the KVM guest virtual machine shows:

   $ lspci
   
   00:05.0 SCSI storage controller: Red Hat, Inc Virtio block device
   00:05.1 SCSI storage controller: Red Hat, Inc Virtio block device
   

Procedure 14.4. Workaround for Red Hat Enterprise Linux 6 guests

1. Install the acpid package

   The acpid service listen and processes ACPI requests.
   Log into the guest virtual machine and install the acpid package on the guest virtual machine:

   # yum install acpid

2. Enable the acpid service

   Set the acpid service to start during the guest virtual machine boot sequence and start the service:

   # chkconfig acpid on
   # service acpid start

3. Prepare guest domain xml

   Edit the domain XML file to include the following element. Replace the virtio serial port with org.qemu.guest_agent.0 and use your guest's name instead of $guestname

   
   <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/{$guestname}.agent'/>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
   </channel>
       
   
   

   Figure 14.2. Guest XML replacement

4. Install the QEMU guest agent

   Install the QEMU guest agent (QEMU-GA) and start the service as directed in Chapter 10, QEMU-img and QEMU Guest Agent. If you are running a Windows guest there are instructions in this chapter for that as well.

5. Shutdown the guest

   1. Run the following commands

      # virsh list --all  - this command lists all of the known domains
         Id Name              State
      ----------------------------------
         rhel6                running
      

   2. Shut down the guest virtual machine

      # virsh shutdown rhel6
      
      Domain rhel6 is being shutdown
      

   3. Wait a few seconds for the guest virtual machine to shut down.

      # virsh list --all
       Id Name                 State
      ----------------------------------
        . rhel6                shut off
      

   4. Start the domain named rhel6, with the XML file you edited.

      # virsh start rhel6

   5. Shut down the acpi in the rhel6 guest virtual machine.

      # virsh shutdown --mode acpi rhel6 

   6. List all the domains again, rhel6 should still be on the list, and it should indicate it is shut off.

      # virsh list --all
         Id Name                 State
      ----------------------------------
         rhel6                shut off
      

   7. Start the domain named rhel6, with the XML file you edited.

      # virsh start rhel6

   8. Shut down the rhel6 guest virtual machine guest agent.

      # virsh shutdown --mode agent rhel6

   9. List the domains. rhel6 should still be on the list, and it should indicate it is shut off

      # virsh list --all
         Id Name                 State
      ----------------------------------
         rhel6                shut off
      

Procedure 14.5. Changing the libvirt-guests service parameters to allow for the graceful shutdown of guests

1. Open the configuration file

   The configuration file is located in /etc/sysconfig/libvirt-guests. Edit the file, remove the comment mark (#) and change the ON_SHUTDOWN=suspend to ON_SHUTDOWN=shutdown. Remember to save the change.

   $ vi /etc/sysconfig/libvirt-guests
   
   # URIs to check for running guests
   # example: URIS='default xen:/// vbox+tcp://host/system lxc:///'
   #URIS=default
   
   # action taken on host boot
   # - start   all guests which were running on shutdown are started on boot
   #           regardless on their autostart settings                                 
   # - ignore  libvirt-guests init script won't start any guest on boot, however,     
   #           guests marked as autostart will still be automatically started by      
   #           libvirtd                                                               
   #ON_BOOT=start                                                                     
   
   # Number of seconds to wait between each guest start. Set to 0 to allow            
   # parallel startup.
   #START_DELAY=0
   
   # action taken on host shutdown
   # - suspend   all running guests are suspended using virsh managedsave
   # - shutdown  all running guests are asked to shutdown. Please be careful with
   #             this settings since there is no way to distinguish between a
   #             guest which is stuck or ignores shutdown requests and a guest
   #             which just needs a long time to shutdown. When setting
   #             ON_SHUTDOWN=shutdown, you must also set SHUTDOWN_TIMEOUT to a
   #             value suitable for your guests.
   ON_SHUTDOWN=shutdown
   
   # If set to non-zero, shutdown will suspend guests concurrently. Number of
   # guests on shutdown at any time will not exceed number set in this variable.
   #PARALLEL_SHUTDOWN=0
   
   # Number of seconds we're willing to wait for a guest to shut down. If parallel
   # shutdown is enabled, this timeout applies as a timeout for shutting down all
   # guests on a single URI defined in the variable URIS. If this is 0, then there
   # is no time out (use with caution, as guests might not respond to a shutdown
   # request). The default value is 300 seconds (5 minutes).
   #SHUTDOWN_TIMEOUT=300
   
   # If non-zero, try to bypass the file system cache when saving and
   # restoring guests, even though this may give slower operation for
   # some file systems.
   #BYPASS_CACHE=0

   	URIS - checks the specified connections for a running guest. The Default setting functions in the same manner as virsh does when no explicit URI is set In addition, one can explicitly set the URI from /etc/libvirt/libvirt.conf. It should be noted that when using the libvirt configuration file default setting, no probing will be used.
   	ON_BOOT - specifies the action to be done to / on the guests when the host boots. The start option starts all guests that were running prior to shutdown regardless on their autostart settings. The ignore option will not start the formally running guest on boot, however, any guest marked as autostart will still be automatically started by libvirtd.
   	The START_DELAY - sets a delay interval in between starting up the guests. This time period is set in seconds. Use the 0 time setting to make sure there is no delay and that all guests are started simultaneously.
   	ON_SHUTDOWN - specifies the action taken when a host shuts down. Options that can be set include: suspend which suspends all running guests using virsh managedsave and shutdown which shuts down all running guests. It is best to be careful with using the shutdown option as there is no way to distinguish between a guest which is stuck or ignores shutdown requests and a guest that just needs a longer time to shutdown. When setting the ON_SHUTDOWN=shutdown, you must also set SHUTDOWN_TIMEOUT to a value suitable for the guests.
   	PARALLEL_SHUTDOWN Dictates that the number of guests on shutdown at any time will not exceed number set in this variable and the guests will be suspended concurrently. If set to 0, then guests are not shutdown concurrently.
   	Number of seconds to wait for a guest to shut down. If SHUTDOWN_TIMEOUT is enabled, this timeout applies as a timeout for shutting down all guests on a single URI defined in the variable URIS. If SHUTDOWN_TIMEOUT is set to 0, then there is no time out (use with caution, as guests might not respond to a shutdown request). The default value is 300 seconds (5 minutes).
   	BYPASS_CACHE can have 2 values, 0 to disable and 1 to enable. If enabled it will by-pass the file system cache when guests are restored. Note that setting this may effect performance and may cause slower operation for some file systems.

2. Start libvirt-guests service

   If you have not started the service, start the libvirt-guests service. Do not restart the service as this will cause all running domains to shutdown.