Chapter 21. Xen live migration
- Offline migration suspends the guest on the original host, transfers it to the destination host and then resumes it once the guest is fully transferred. Offline migration uses the
virsh migrate
command.Copy to Clipboard Copied! Toggle word wrap Toggle overflow virsh migrate GuestName libvirtURI
# virsh migrate GuestName libvirtURI
- A live migration keeps the guest running on the source host and begins moving the memory without stopping the guest. All modified memory pages are monitored for changes and sent to the destination while the image is sent. The memory is updated with the changed pages. The process continues until the amount of pause time allowed for the guest equals the predicted time for the final few pages to be transfer. The Xen hypervisor estimates the time remaining and attempts to transfer the maximum amount of page files from the source to the destination until Xen predicts the amount of remaining pages can be transferred during a very brief time while the guest is paused. The registers are loaded on the new host and the guest is then resumed on the destination host. If the guest cannot be merged (which happens when guests are under extreme loads) the guest is paused and then an offline migration is started instead.Live migration uses the
--live
option for thevirsh migrate
command.Copy to Clipboard Copied! Toggle word wrap Toggle overflow virsh migrate--live GuestName libvirtURI
# virsh migrate--live GuestName libvirtURI
Important
/etc/xen/xend-config.sxp
configuration file. By default, migration is disabled as migration can be a potential security hazard if incorrectly configured. Opening the migration port can allow an unauthorized host to initiate a migration or connect to the migration ports. Authentication and authorization are not configured for migration requests and the only control mechanism is based on hostnames and IP addresses. Special care should be taken to ensure the migration port is not accessible to unauthorized hosts.
Important
Modify the following entries in /etc/xen/xend-config.sxp
to enable migration. Modify the values, when necessary, and remove the comments (the #
symbol) preceding the following parameters:
(xend-relocation-server yes)
- The default value, which disables migration, is
no
. Change the value ofxend-relocation-server
toyes
to enable migration. (xend-relocation-port 8002)
- The parameter,
(xend-relocation-port)
, specifies the portxend
should use for the relocation interface, ifxend-relocation-server
is set toyes
The default value of this variable should work for most installations. If you change the value make sure you are using an unused port on the relocation server.The port set by thexend-relocation-port
parameter must be open on both systems. (xend-relocation-address '')
(xend-relocation-address)
is the address thexend
listens for migration commands on therelocation-socket
connection ifxend-relocation-server
is set.The default is to listen on all active interfaces. The(xend-relocation-address)
parameter restricts the migration server to only listen to a specific interface. The default value in/etc/xen/xend-config.sxp
is an empty string(''
). This value should be replaced with a single, valid IP address. For example:Copy to Clipboard Copied! Toggle word wrap Toggle overflow (xend-relocation-address '10.0.0.1')
(xend-relocation-address '10.0.0.1')
(xend-relocation-hosts-allow '')
- The
(xend-relocation-hosts-allow 'hosts')
parameter controls which hostnames can communicate on the relocation port.Unless you are using SSH or TLS, the guest's virtual memory is transferred in raw form without encryption of the communication. Modify thexend-relocation-hosts-allow
option to restrict access to the migration server.If the value is empty, as denoted in the example above by an empty string surrounded by single quotes, then all connections are allowed. This assumes the connection arrives on a port and interface which the relocation server listens on, see alsoxend-relocation-port
andxend-relocation-address
.Otherwise, the(xend-relocation-hosts-allow)
parameter should be a sequence of regular expressions separated by spaces. Any host with a fully-qualified domain name or an IP address which matches one of these regular expressions will be accepted.An example of a(xend-relocation-hosts-allow)
attribute:Copy to Clipboard Copied! Toggle word wrap Toggle overflow (xend-relocation-hosts-allow '^localhost$ ^localhost\\.localdomain$')
(xend-relocation-hosts-allow '^localhost$ ^localhost\\.localdomain$')
service xend restart
# service xend restart
21.1. A live migration example
et-virt07
and et-virt08
), both of them are using eth1
as their default network interface hence they are using xenbr1
as their Xen networking bridge. We are using a locally attached SCSI disk (/dev/sdb
) on et-virt07
for shared storage using NFS.
Create and mount the directory used for the migration:
mkdir /var/lib/libvirt/images mount /dev/sdb /var/lib/libvirt/images
# mkdir /var/lib/libvirt/images
# mount /dev/sdb /var/lib/libvirt/images
Warning
/var/lib/libvirt/images/
make sure you only export /var/lib/libvirt/images/
and not/var/lib/xen/
as this directory is used by the xend
daemon and other tools. Sharing /var/lib/xen/
will cause unpredictable behavior.
cat /etc/exports /var/lib/libvirt/images *(rw,async,no_root_squash)
# cat /etc/exports
/var/lib/libvirt/images *(rw,async,no_root_squash)
showmount -e et-virt07
# showmount -e et-virt07
Export list for et-virt07:
/var/lib/libvirt/images *
The install command in the example used for installing the guest:
virt-install -p -f /var/lib/libvirt/images/testvm1.dsk -s 5 -n\ testvm1 --vnc -r 1024 -l http://example.com/RHEL5-tree\ Server/x86-64/os/ -b xenbr1
# virt-install -p -f /var/lib/libvirt/images/testvm1.dsk -s 5 -n\
testvm1 --vnc -r 1024 -l http://example.com/RHEL5-tree\
Server/x86-64/os/ -b xenbr1
Make sure the virtualized network bridges are configured correctly and have the same name on both hosts:
brctl show
[et-virt08 ~]# brctl show
bridge name bridge id STP enabled interfaces
xenbr1 8000.feffffffffff no peth1
vif0.1
brctl show
[et-virt07 ~]# brctl show
bridge name bridge id STP enabled interfaces
xenbr1 8000.feffffffffff no peth1
vif0.1
grep xend-relocation /etc/xen/xend-config.sxp |grep -v '#'
[et-virt07 ~]# grep xend-relocation /etc/xen/xend-config.sxp |grep -v '#'
(xend-relocation-server yes)
(xend-relocation-port 8002)
(xend-relocation-address '')
(xend-relocation-hosts-allow '')
grep xend-relocation /etc/xen/xend-config.sxp |grep -v '#'
[et-virt08 ~]# grep xend-relocation /etc/xen/xend-config.sxp |grep -v '#'
(xend-relocation-server yes)
(xend-relocation-port 8002)
(xend-relocation-address '')
(xend-relocation-hosts-allow '')
lsof -i :8002
[et-virt07 ~]# lsof -i :8002
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
python 3445 root 14u IPv4 10223 TCP *:teradataordbms (LISTEN)
lsof -i :8002
[et-virt08 ~]# lsof -i :8002
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
python 3252 root 14u IPv4 10901 TCP *:teradataordbms (LISTEN)
/var/lib/libvirt/images
directory is available and mounted with networked storage on both hosts. Shared, networked storage is required for migrations.
df /var/lib/libvirt/images
[et-virt08 ~]# df /var/lib/libvirt/images
Filesystem 1K-blocks Used Available Use% Mounted on
et-virt07:/var/lib/libvirt/images 70562400 2379712 64598336 4% /var/lib/libvirt/images
file /var/lib/libvirt/images/testvm1.dsk
[et-virt08 ~]# file /var/lib/libvirt/images/testvm1.dsk
/var/lib/libvirt/images/testvm1.dsk: x86 boot sector; partition 1: ID=0x83,
active, starthead 1, startsector 63, 208782 sectors; partition 2: ID=0x8e,
starthead 0, startsector 208845, 10265535 sectors, code offset 0x48
touch /var/lib/libvirt/images/foo rm -f /var/lib/libvirt/images/foo
[et-virt08 ~]# touch /var/lib/libvirt/images/foo
[et-virt08 ~]# rm -f /var/lib/libvirt/images/foo
Start the virtual machine (if the virtual machine is not on):
virsh list
[et-virt07 ~]# virsh list
Id Name State
----------------------------------
Domain-0 running
virsh start testvm1
[et-virt07 ~]# virsh start testvm1
Domain testvm1 started
virsh list
[et-virt07 ~]# virsh list
Id Name State
----------------------------------
Domain-0 running
testvm1 blocked
time virsh save testvm1 testvm1.sav
[et-virt07 images]# time virsh save testvm1 testvm1.sav
real 0m15.744s
user 0m0.188s
sys 0m0.044s
ls -lrt testvm1.sav
[et-virt07 images]# ls -lrt testvm1.sav
-rwxr-xr-x 1 root root 1075657716 Jan 12 06:46 testvm1.sav
virsh list
[et-virt07 images]# virsh list
Id Name State
----------------------------------
Domain-0 running
virsh restore testvm1.sav
[et-virt07 images]# virsh restore testvm1.sav
virsh list
[et-virt07 images]# virsh list
Id Name State
----------------------------------
Domain-0 running
testvm1 blocked
domain-id
from et-virt08
to et-virt07
. The hostname you are migrating to and <domain-id> must be replaced with valid values. This example uses the et-virt08
host which must have SSH access to et-virt07
xm migrate --live testvm1 et-virt07
[et-virt08 ~]# xm migrate --live testvm1 et-virt07
et-virt08
virsh list
[et-virt08 ~]# virsh list
Id Name State
----------------------------------
Domain-0 running
et-virt07
:
virsh list
[et-virt07 ~]# virsh list
Id Name State
----------------------------------
Domain-0 running
testvm1 running
Create the following script inside the virtual machine to log date and hostname during the migration. This script performs I/O tasks on the guest's file system.
#!/bin/bash while true do touch /var/tmp/$$.log echo `hostname` >> /var/tmp/$$.log echo `date` >> /var/tmp/$$.log cat /var/tmp/$$.log df /var/tmp ls -l /var/tmp/$$.log sleep 3 done
#!/bin/bash
while true
do
touch /var/tmp/$$.log
echo `hostname` >> /var/tmp/$$.log
echo `date` >> /var/tmp/$$.log
cat /var/tmp/$$.log
df /var/tmp
ls -l /var/tmp/$$.log
sleep 3
done
et-virt07
:
virsh list
[et-virt08 ~]# virsh list
Id Name State
----------------------------------
Domain-0 running
testvm1 blocked
et-virt07
. You can add the time
command to see how long the migration takes:
xm migrate --live testvm1 et-virt07
[et-virt08 ~]# xm migrate --live testvm1 et-virt07
./doit
# ./doit
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:26:27 EST 2007
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
2983664 2043120 786536 73% /
-rw-r--r-- 1 root root 62 Jan 12 02:26 /var/tmp/2279.log
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:26:27 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:26:30 EST 2007
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
2983664 2043120 786536 73% /
-rw-r--r-- 1 root root 124 Jan 12 02:26 /var/tmp/2279.log
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:26:27 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:26:30 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:26:33 EST 2007
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
2983664 2043120 786536 73% /
-rw-r--r-- 1 root root 186 Jan 12 02:26 /var/tmp/2279.log
Fri Jan 12 02:26:45 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:26:48 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:26:51 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:54:57 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:55:00 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:55:03 EST 2007
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
2983664 2043120 786536 73% /
-rw-r--r-- 1 root root 744 Jan 12 06:55 /var/tmp/2279.log
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:26:27 EST 2007
et-virt08
:
virsh list
[et-virt08 ~]# virsh list
Id Name State
----------------------------------
Domain-0 running
et-virt07
:
virsh list
[et-virt07 images]# virsh list
Id Name State
----------------------------------
Domain-0 running
testvm1 blocked
et-virt07
to et-virt08
. Initiate a migration from et-virt07
to et-virt08
:
xm migrate --live testvm1 et-virt08
[et-virt07 images]# xm migrate --live testvm1 et-virt08
virsh list
[et-virt07 images]# virsh list
Id Name State
----------------------------------
Domain-0 running
./doit
# ./doit
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:53 EST 2007
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
2983664 2043120 786536 73% /
-rw-r--r-- 1 root root 62 Jan 12 06:57 /var/tmp/2418.log
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:53 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:56 EST 2007
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
2983664 2043120 786536 73% /
-rw-r--r-- 1 root root 124 Jan 12 06:57 /var/tmp/2418.log
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:53 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:56 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:58:00 EST 2007
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
2983664 2043120 786536 73% /
-rw-r--r-- 1 root root 186 Jan 12 06:57 /var/tmp/2418.log
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:53 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:56 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:58:00 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:30:00 EST 2007
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
2983664 2043120 786536 73% /
-rw-r--r-- 1 root root 248 Jan 12 02:30 /var/tmp/2418.log
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:53 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:56 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:58:00 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:30:00 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:30:03 EST 2007
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
2983664 2043120 786536 73% /
-rw-r--r-- 1 root root 310 Jan 12 02:30 /var/tmp/2418.log
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:53 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:56 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:58:00 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:30:00 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:30:03 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:30:06 EST 2007
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
2983664 2043120 786536 73% /
-rw-r--r-- 1 root root 372 Jan 12 02:30 /var/tmp/2418.log
et-virt07
verify on et-virt08
that the virtual machine has started:
virsh list
[et-virt08 ~]# virsh list
Id Name State
----------------------------------
Domain-0 running
testvm1 blocked
time virsh migrate --live testvm1 et-virt07
[et-virt08 ~]# time virsh migrate --live testvm1 et-virt07
real 0m10.378s
user 0m0.068s
sys 0m0.052s