Chapter 21. Xen live migration
The Xen hypervisor supports Virtualization Migration for para-virtualized guests and fully virtualized guests. Migration is only supported on Red Hat Enterprise Linux 5.1 and newer systems. Migration can be performed offline or live.
- Offline migration suspends the guest on the original host, transfers it to the destination host and then resumes it once the guest is fully transferred. Offline migration uses the
virsh migratecommand.# virsh migrate GuestName libvirtURI
- A live migration keeps the guest running on the source host and begins moving the memory without stopping the guest. All modified memory pages are monitored for changes and sent to the destination while the image is sent. The memory is updated with the changed pages. The process continues until the amount of pause time allowed for the guest equals the predicted time for the final few pages to be transfer. The Xen hypervisor estimates the time remaining and attempts to transfer the maximum amount of page files from the source to the destination until Xen predicts the amount of remaining pages can be transferred during a very brief time while the guest is paused. The registers are loaded on the new host and the guest is then resumed on the destination host. If the guest cannot be merged (which happens when guests are under extreme loads) the guest is paused and then an offline migration is started instead.Live migration uses the
--liveoption for thevirsh migratecommand.# virsh migrate--live GuestName libvirtURI
Important
Migration is presently unsupported on the Itanium® architecture.
To enable migration with Xen a few changes must be made to the
/etc/xen/xend-config.sxp configuration file. By default, migration is disabled as migration can be a potential security hazard if incorrectly configured. Opening the migration port can allow an unauthorized host to initiate a migration or connect to the migration ports. Authentication and authorization are not configured for migration requests and the only control mechanism is based on hostnames and IP addresses. Special care should be taken to ensure the migration port is not accessible to unauthorized hosts.
Important
IP address and hostname filters only offer minimal security. Both of these attributes can be forged if the attacker knows the address or hostname of the migration client. The best method for securing migration is to isolate the network from external and unauthorized internal connections.
Enabling migration
Modify the following entries in /etc/xen/xend-config.sxp to enable migration. Modify the values, when necessary, and remove the comments (the # symbol) preceding the following parameters:
(xend-relocation-server yes)- The default value, which disables migration, is
no. Change the value ofxend-relocation-servertoyesto enable migration. (xend-relocation-port 8002)- The parameter,
(xend-relocation-port), specifies the portxendshould use for the relocation interface, ifxend-relocation-serveris set toyesThe default value of this variable should work for most installations. If you change the value make sure you are using an unused port on the relocation server.The port set by thexend-relocation-portparameter must be open on both systems. (xend-relocation-address '')(xend-relocation-address)is the address thexendlistens for migration commands on therelocation-socketconnection ifxend-relocation-serveris set.The default is to listen on all active interfaces. The(xend-relocation-address)parameter restricts the migration server to only listen to a specific interface. The default value in/etc/xen/xend-config.sxpis an empty string(''). This value should be replaced with a single, valid IP address. For example:(xend-relocation-address '10.0.0.1')
(xend-relocation-hosts-allow '')- The
(xend-relocation-hosts-allow 'hosts')parameter controls which hostnames can communicate on the relocation port.Unless you are using SSH or TLS, the guest's virtual memory is transferred in raw form without encryption of the communication. Modify thexend-relocation-hosts-allowoption to restrict access to the migration server.If the value is empty, as denoted in the example above by an empty string surrounded by single quotes, then all connections are allowed. This assumes the connection arrives on a port and interface which the relocation server listens on, see alsoxend-relocation-portandxend-relocation-address.Otherwise, the(xend-relocation-hosts-allow)parameter should be a sequence of regular expressions separated by spaces. Any host with a fully-qualified domain name or an IP address which matches one of these regular expressions will be accepted.An example of a(xend-relocation-hosts-allow)attribute:(xend-relocation-hosts-allow '^localhost$ ^localhost\\.localdomain$')
After you have configured the parameters in your configuration file, restart the Xen service.
# service xend restart
21.1. A live migration example
Below is an example of how to setup a simple environment for live migration. This configuration is using NFS for the shared storage. NFS is suitable for demonstration environments but for a production environment a more robust shared storage configuration using Fibre Channel or iSCSI and GFS is recommended.
The configuration below consists of two servers (
et-virt07 and et-virt08), both of them are using eth1 as their default network interface hence they are using xenbr1 as their Xen networking bridge. We are using a locally attached SCSI disk (/dev/sdb) on et-virt07 for shared storage using NFS.
Setup for live migration
Create and mount the directory used for the migration:
# mkdir /var/lib/libvirt/images # mount /dev/sdb /var/lib/libvirt/images
Warning
Ensure the directory is exported with the correct options. If you are exporting the default directory
/var/lib/libvirt/images/ make sure you only export /var/lib/libvirt/images/ and not/var/lib/xen/ as this directory is used by the xend daemon and other tools. Sharing /var/lib/xen/ will cause unpredictable behavior.
# cat /etc/exports /var/lib/libvirt/images *(rw,async,no_root_squash)
Verify it is exported via NFS:
# showmount -e et-virt07 Export list for et-virt07: /var/lib/libvirt/images *
Install the guest
The install command in the example used for installing the guest:
# virt-install -p -f /var/lib/libvirt/images/testvm1.dsk -s 5 -n\ testvm1 --vnc -r 1024 -l http://example.com/RHEL5-tree\ Server/x86-64/os/ -b xenbr1
For step by step installation instructions, see Chapter 8, Guest operating system installation procedures.
Verify environment for migration
Make sure the virtualized network bridges are configured correctly and have the same name on both hosts:
[et-virt08 ~]# brctl show bridge name bridge id STP enabled interfaces xenbr1 8000.feffffffffff no peth1 vif0.1
[et-virt07 ~]# brctl show bridge name bridge id STP enabled interfaces xenbr1 8000.feffffffffff no peth1 vif0.1
Verify the relocation parameters are configured on both hosts:
[et-virt07 ~]# grep xend-relocation /etc/xen/xend-config.sxp |grep -v '#' (xend-relocation-server yes) (xend-relocation-port 8002) (xend-relocation-address '') (xend-relocation-hosts-allow '')
[et-virt08 ~]# grep xend-relocation /etc/xen/xend-config.sxp |grep -v '#' (xend-relocation-server yes) (xend-relocation-port 8002) (xend-relocation-address '') (xend-relocation-hosts-allow '')
Make sure the relocation server has started and is listening on the dedicated port for Xen migrations (8002):
[et-virt07 ~]# lsof -i :8002 COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME python 3445 root 14u IPv4 10223 TCP *:teradataordbms (LISTEN)
[et-virt08 ~]# lsof -i :8002 COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME python 3252 root 14u IPv4 10901 TCP *:teradataordbms (LISTEN)
That the default
/var/lib/libvirt/images directory is available and mounted with networked storage on both hosts. Shared, networked storage is required for migrations.
[et-virt08 ~]# df /var/lib/libvirt/images Filesystem 1K-blocks Used Available Use% Mounted on et-virt07:/var/lib/libvirt/images 70562400 2379712 64598336 4% /var/lib/libvirt/images
[et-virt08 ~]# file /var/lib/libvirt/images/testvm1.dsk /var/lib/libvirt/images/testvm1.dsk: x86 boot sector; partition 1: ID=0x83, active, starthead 1, startsector 63, 208782 sectors; partition 2: ID=0x8e, starthead 0, startsector 208845, 10265535 sectors, code offset 0x48
[et-virt08 ~]# touch /var/lib/libvirt/images/foo [et-virt08 ~]# rm -f /var/lib/libvirt/images/foo
Verify saving and restoring the guest
Start the virtual machine (if the virtual machine is not on):
[et-virt07 ~]# virsh list Id Name State ---------------------------------- Domain-0 running
[et-virt07 ~]# virsh start testvm1 Domain testvm1 started
Verify the virtual machine is running:
[et-virt07 ~]# virsh list Id Name State ---------------------------------- Domain-0 running testvm1 blocked
Save the virtual machine on the local host:
[et-virt07 images]# time virsh save testvm1 testvm1.sav real 0m15.744s user 0m0.188s sys 0m0.044s
[et-virt07 images]# ls -lrt testvm1.sav -rwxr-xr-x 1 root root 1075657716 Jan 12 06:46 testvm1.sav
[et-virt07 images]# virsh list Id Name State ---------------------------------- Domain-0 running
Restore the virtual machine on the local host:
[et-virt07 images]# virsh restore testvm1.sav
[et-virt07 images]# virsh list Id Name State ---------------------------------- Domain-0 running testvm1 blocked
Start the live migration of
domain-id from et-virt08 to et-virt07. The hostname you are migrating to and <domain-id> must be replaced with valid values. This example uses the et-virt08 host which must have SSH access to et-virt07
[et-virt08 ~]# xm migrate --live testvm1 et-virt07
Verify the virtual machine is no longer present on
et-virt08
[et-virt08 ~]# virsh list Id Name State ---------------------------------- Domain-0 running
Verify the virtual machine has been migrated to
et-virt07:
[et-virt07 ~]# virsh list Id Name State ---------------------------------- Domain-0 running testvm1 running
Testing the progress and initiating the live migration
Create the following script inside the virtual machine to log date and hostname during the migration. This script performs I/O tasks on the guest's file system.
#!/bin/bash while true do touch /var/tmp/$$.log echo `hostname` >> /var/tmp/$$.log echo `date` >> /var/tmp/$$.log cat /var/tmp/$$.log df /var/tmp ls -l /var/tmp/$$.log sleep 3 done
Remember, that script is only for testing purposes and unnecessary for a live migration in a production environment.
Verify the virtual machine is running on et-virt08 before we try to migrate it to
et-virt07:
[et-virt08 ~]# virsh list Id Name State ---------------------------------- Domain-0 running testvm1 blocked
Initiate a live migration to
et-virt07. You can add the time command to see how long the migration takes:
[et-virt08 ~]# xm migrate --live testvm1 et-virt07
run the script inside the guest:
# ./doit dhcp78-218.lab.boston.redhat.com Fri Jan 12 02:26:27 EST 2007 Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/VolGroup00-LogVol00 2983664 2043120 786536 73% / -rw-r--r-- 1 root root 62 Jan 12 02:26 /var/tmp/2279.log dhcp78-218.lab.boston.redhat.com Fri Jan 12 02:26:27 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 02:26:30 EST 2007 Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/VolGroup00-LogVol00 2983664 2043120 786536 73% / -rw-r--r-- 1 root root 124 Jan 12 02:26 /var/tmp/2279.log dhcp78-218.lab.boston.redhat.com Fri Jan 12 02:26:27 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 02:26:30 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 02:26:33 EST 2007 Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/VolGroup00-LogVol00 2983664 2043120 786536 73% / -rw-r--r-- 1 root root 186 Jan 12 02:26 /var/tmp/2279.log Fri Jan 12 02:26:45 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 02:26:48 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 02:26:51 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 06:54:57 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 06:55:00 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 06:55:03 EST 2007 Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/VolGroup00-LogVol00 2983664 2043120 786536 73% / -rw-r--r-- 1 root root 744 Jan 12 06:55 /var/tmp/2279.log dhcp78-218.lab.boston.redhat.com Fri Jan 12 02:26:27 EST 2007
Verify the virtual machine has been shut down on
et-virt08:
[et-virt08 ~]# virsh list Id Name State ---------------------------------- Domain-0 running
Verify the virtual machine has started up on
et-virt07:
[et-virt07 images]# virsh list Id Name State ---------------------------------- Domain-0 running testvm1 blocked
Run through another cycle migrating from
et-virt07 to et-virt08. Initiate a migration from et-virt07 to et-virt08:
[et-virt07 images]# xm migrate --live testvm1 et-virt08
Verify the virtual machine has been shut down:
[et-virt07 images]# virsh list Id Name State ---------------------------------- Domain-0 running
Before initiating the migration start the simple script in the guest and note the change in time when migrating the guest:
# ./doit dhcp78-218.lab.boston.redhat.com Fri Jan 12 06:57:53 EST 2007 Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/VolGroup00-LogVol00 2983664 2043120 786536 73% / -rw-r--r-- 1 root root 62 Jan 12 06:57 /var/tmp/2418.log dhcp78-218.lab.boston.redhat.com Fri Jan 12 06:57:53 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 06:57:56 EST 2007 Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/VolGroup00-LogVol00 2983664 2043120 786536 73% / -rw-r--r-- 1 root root 124 Jan 12 06:57 /var/tmp/2418.log dhcp78-218.lab.boston.redhat.com Fri Jan 12 06:57:53 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 06:57:56 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 06:58:00 EST 2007 Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/VolGroup00-LogVol00 2983664 2043120 786536 73% / -rw-r--r-- 1 root root 186 Jan 12 06:57 /var/tmp/2418.log dhcp78-218.lab.boston.redhat.com Fri Jan 12 06:57:53 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 06:57:56 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 06:58:00 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 02:30:00 EST 2007 Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/VolGroup00-LogVol00 2983664 2043120 786536 73% / -rw-r--r-- 1 root root 248 Jan 12 02:30 /var/tmp/2418.log dhcp78-218.lab.boston.redhat.com Fri Jan 12 06:57:53 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 06:57:56 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 06:58:00 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 02:30:00 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 02:30:03 EST 2007 Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/VolGroup00-LogVol00 2983664 2043120 786536 73% / -rw-r--r-- 1 root root 310 Jan 12 02:30 /var/tmp/2418.log dhcp78-218.lab.boston.redhat.com Fri Jan 12 06:57:53 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 06:57:56 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 06:58:00 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 02:30:00 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 02:30:03 EST 2007 dhcp78-218.lab.boston.redhat.com Fri Jan 12 02:30:06 EST 2007 Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/VolGroup00-LogVol00 2983664 2043120 786536 73% / -rw-r--r-- 1 root root 372 Jan 12 02:30 /var/tmp/2418.log
After the migration command completes on
et-virt07 verify on et-virt08 that the virtual machine has started:
[et-virt08 ~]# virsh list Id Name State ---------------------------------- Domain-0 running testvm1 blocked
and run another cycle:
[et-virt08 ~]# time virsh migrate --live testvm1 et-virt07 real 0m10.378s user 0m0.068s sys 0m0.052s
At this point you have successfully performed an offline and a live migration test.
