Virtualization Security Guide
Securing hosts, guests, and shared infrastructure in virtualized environments on RHEL
Abstract
Chapter 1. Introduction
1.1. Virtualized and Non-Virtualized Environments
In a non-virtualized environment, hosts are separated from each other physically and each host has a self-contained environment, which consists of services such as a web server, or a DNS server. These services communicate directly to their own user space, host kernel and physical host, offering their services directly to the network.
Figure 1.1. Non-Virtualized Environment
In a virtualized environment, several operating systems can be housed (as guest virtual machines) within a single host kernel and physical host.
Figure 1.2. Virtualized Environment
1.2. Why Virtualization Security Matters
- The host and hypervisor become prime targets; they are often a single point of failure for guests and data.
- Virtual machines can interfere with each other in undesirable ways. If no access controls are in place to help prevent this, one malicious guest can bypass a vulnerable hypervisor and directly access other resources on the host system, such as the storage of other guests.
- Resources and services can become difficult to track and maintain; with rapid deployment of virtualized systems comes an increased need for management of resources, including sufficient patching, monitoring and maintenance.
- Resources such as storage can be spread across, and dependent upon, several machines. This can lead to overly complex environments and poorly managed and maintained systems.
- Virtualization does not remove any of the traditional security risks present in your environment; the entire solution stack, not just the virtualization layer, must be secured.
Chapter 2. Host Security
2.1. Securing the Host Physical Machine
- Ensure that SELinux is configured properly for your installation and is operating in enforcing mode:
#
setenforce 1
In addition to being a good security practice, the advanced virtualization security functionality provided by sVirt relies on SELinux. See Chapter 4, sVirt for more information on SELinux and sVirt. - Remove or disable any unnecessary services such as
AutoFS
,NFS
,FTP
,HTTP
,NIS
,telnetd
, orsendmail
. - Only add the minimum number of user accounts needed for platform management on the server and remove unnecessary user accounts. Limit direct access to the system to only those users who have a need to manage the system. Consider disallowing shared root access and instead use tools such as
sudo
to grant privileged access to administrators based on their administrative roles. - Avoid running any unessential applications on your host. Running applications on the host may impact virtual machine performance and can affect server stability. Any application that may crash the server will also cause all virtual machines on the server to fail. In addition, vulnerable applications can become vectors for an attack on the host.
- Use a central location for virtual machine installations and images. Virtual machine images should be stored under
/var/lib/libvirt/images/
. If you are using a different directory for your virtual machine images make sure you add the directory to your SELinux policy and relabel it before starting the installation. Use of shareable, network storage in a central location is highly recommended. - Run only the services necessary to support the use and management of your guest systems. If you need to provide additional services, such as file or print services, consider running those services on a Red Hat Enterprise Linux guest.
- Ensure that auditing is enabled on the host system and that libvirt is configured to generate audit records. When auditing is enabled, libvirt generates audit records for changes to guest configuration and start/stop events, which can help you track the guest's state. In addition, the libvirt audit events can also be viewed using the specialized
auvirt
utility. For more information, use theman auvirt
command. - Ensure that any remote management of the system takes place only over secured network channels. Utilities such as SSH and network protocols such as TLS or SSL provide both authentication and data encryption to help ensure that only approved administrators can manage the system remotely.
- Ensure that the firewall is configured properly for your installation and is activated at boot. Only network ports needed for the use and management of the system should be allowed.
- Do not grant guests with direct access to entire disks or block devices (for example,
/dev/sdb
); instead, use partitions (for example,/dev/sdb1
) or LVM volumes for guest storage. - Attaching a USB device, Physical Function or physical device when SR-IOV is not available to a virtual machine could provide access to the device which is sufficient enough to overwrite that device's firmware. This presents a potential security issue by which an attacker could overwrite the device's firmware with malicious code and cause problems when moving the device between virtual machines or at host boot time.It is advised to use SR-IOV Virtual Function device assignment where applicable.
Note
2.2. Client Access Control
libvirtd
daemon has three levels of access control:
- All connections start off in an unauthenticated state, where the only API operations allowed are those required to complete authentication.
- After successful authentication, a connection either has full, unrestricted access to all libvirt API calls, or is locked down to only "read only" operations, according to what socket the client connection originated on.
- The access control framework allows authenticated connections to have fine-grained permission rules to be defined by the administrator.
2.2.1. Access Control Drivers
none
driver is used, which performs no access control checks at all. Currently, libvirt provides support for using polkit as a real access control driver. To learn how to use the polkit access driver see the configuration documentation.
/etc/libvirt/libvirtd.conf
configuration file, using the access_drivers
parameter. This parameter accepts an array of access control driver names. If more than one access driver is requested, then all must succeed in order for access to be granted. To enable 'polkit' as the driver, use the augtool
command:
# augtool -s set '/files/etc/libvirt/libvirtd.conf/access_drivers[1]' polkit
# augtool -s rm /files/etc/libvirt/libvirtd.conf/access_drivers
libvirtd.conf
to take effect, restart the libvirtd
service.
# systemctl restart libvirtd.service
2.2.2. Objects and Permissions
2.2.3. Security Concerns when Adding Block Devices to a Guest
- The host physical machine should not use file system labels to identify file systems in the
fstab
file, theinitrd
file or on the kernel command line. Doing so presents a security risk if guest virtual machines have write access to whole partitions or LVM volumes, because a guest virtual machine could potentially write a file-system label belonging to the host physical machine, to its own block device storage. Upon reboot of the host physical machine, the host physical machine could then mistakenly use the guest virtual machine's disk as a system disk, which would compromise the host physical machine system.It is preferable to use the UUID of a device to identify it in the/etc/fstab
file, the/dev/initrd
file, or on the kernel command line. - Guest virtual machines should not be given write access to entire disks or block devices (for example,
/dev/sdb
). Guest virtual machines with access to entire block devices may be able to modify volume labels, which can be used to compromise the host physical machine system. Use partitions (for example,/dev/sdb1
) or LVM volumes to prevent this problem. See LVM Administration with CLI Commands or LVM Configuration Examples for information on LVM administration and configuration examples.If you are using raw access to partitions, for example/dev/sdb1
or raw disks such as/dev/sdb,
you should configure LVM to only scan disks that are safe, using theglobal_filter
setting. See the Logical Volume Manager Administration Guide for an example of an LVM configuration script using theglobal_filter
command.
2.3. Special Considerations for Public Cloud Operators
- Disallow any direct hardware access from the guest. PCI, USB, FireWire, Thunderbolt, eSATA, and other device passthrough mechanisms make management difficult and often rely on the underlying hardware to enforce separation between the guests.
- Isolate the cloud operator's private management network from the customer guest network, and customer networks from one another, so that:
- The guests cannot access the host systems over the network.
- One customer cannot access another customer's guest systems directly through the cloud provider's internal network.
Chapter 3. Guest Security
3.1. Why Guest Security Matters
3.2. Guest Security Recommended Practices
- With all management of the guest likely taking place remotely, ensure that the management of the system takes place only over secured network channels. Tools such as SSH and network protocols such as TLS or SSL provide both authentication and data encryption to ensure that only approved administrators can manage the system remotely.
- Some virtualization technologies use special guest agents or drivers to enable some virtualization specific features. Ensure that these agents and applications are secured using the standard Red Hat Enterprise Linux security features, such as SELinux.
- In virtualized environments, a greater risk exists of sensitive data being accessed outside the protection boundaries of the guest system. Protect stored sensitive data using encryption tools such as dm-crypt and GnuPG; although special care needs to be taken to ensure the confidentiality of the encryption keys.
Note
3.3. Kernel Address Space Randomization
nokaslr
string to the guest's kernel command line. To edit the guest boot options, use the following command, where guestname is the name of your guest:
# virt-edit -d guestname /etc/default/grub
GRUB_CMDLINE_LINUX
line, for example:
GRUB_CMDLINE_LINUX="rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet nokaslr"
Important
crash
utility. To fix this, add the <vmcoreinfo/>
element to the <features>
section of the XML configuration files of your guests.
<vmcoreinfo/>
fails if the destination host is using an OS that does not support <vmcoreinfo/>
. These include Red Hat Enterprise Linux 7.4 and earlier, as well as Red Hat Enterprise Linux 6.9 and earlier.
3.4. Creating a SecureBoot Red Hat Enterprise Linux 7 Guest with virt-manager
Procedure 3.1. Creating a SecureBoot Red Hat Enterprise Linux 7 guest virtual machine with virt-manager using local installation media
- Perform steps 1 to 6 of Creating a Red Hat Enterprise Linux 7 Guest with virt-manager.
Name and final configuration
Name the virtual machine. Virtual machine names can contain letters, numbers and the following characters: underscores (_
), periods (.
), and hyphens (-
). Virtual machine names must be unique for migration and cannot consist only of numbers.By default, the virtual machine will be created with network address translation (NAT) for a network called 'default' . To change the network selection, clickNetwork selection
and select a host device and source mode.Figure 3.1. Verifying the configuration
To further configure the virtual machine's hardware, check thecheck box to change the guest's storage or network devices, to use the paravirtualized (virtio) drivers or to add additional devices. Verify the settings of the virtual machine and click when you are satisfied. This will open a new wizard for futher configuring your virtual machine.Customize virtual machine hardware
In the overview section of the wizard, select Q35 in the Chipset drop-down menu.In the Firmware drop-down menu, select UEFI x86_64.Figure 3.2. The configure hardware window
Verify the settings of the virtual machine and clickwhen you are satisfied.Clickto create a virtual machine with the specified networking settings, virtualization type, and architecture.
Chapter 4. sVirt
4.1. Introduction
Figure 4.1. Attack path isolated by SELinux
Note
4.2. SELinux and Mandatory Access Control (MAC)
4.3. sVirt Configuration
setsebool boolean_name {on|off}
for a temporary change, or setsebool -P boolean_name {on|off}
to make the change persistent across reboots.
getsebool -a|grep virt
.
SELinux Boolean | Description |
---|---|
staff_use_svirt | Enables staff users to create and transition to sVirt domains. |
unprivuser_use_svirt | Enables unprivileged users to create and transition to sVirt domains. |
virt_sandbox_use_audit | Enables sandbox containers to send audit messages. |
virt_sandbox_use_netlink | Enables sandbox containers to use netlink system calls. |
virt_sandbox_use_sys_admin | Enables sandbox containers to use sys_admin system calls, such as mount. |
virt_transition_userdomain | Enables virtual processes to run as user domains. |
virt_use_comm | Enables virt to use serial/parallel communication ports. |
virt_use_execmem | Enables confined virtual guests to use executable memory and executable stack. |
virt_use_fusefs | Enables virt to read FUSE mounted files. |
virt_use_nfs | Enables virt to manage NFS mounted files. |
virt_use_rawip | Enables virt to interact with rawip sockets. |
virt_use_samba | Enables virt to manage CIFS mounted files. |
virt_use_sanlock | Enables confined virtual guests to interact with the sanlock. |
virt_use_usb | Enables virt to use USB devices. |
virt_use_xserver | Enables virtual machine to interact with the X Window System. |
Note
4.4. sVirt Labeling
virsh edit guest_name
command and add or edit <seclabel> elements as described in the sections below. <seclabel> can be used as a root element for the entire guest, or it can be specified as a sub-element of the <source> element for selecting a specific sVirt label of the given device.
4.4.1. Types of sVirt Labels
Type | SELinux Context | Description/Effect |
---|---|---|
Virtual Machine Processes | system_u:system_r:svirt_t:MCS1 | MCS1 is a randomly selected field. Currently approximately 500,000 labels are supported. |
Virtual Machine Image | system_u:object_r:svirt_image_t:MCS1 | Only svirt_t processes with the same MCS1 fields are able to read/write these image files and devices. |
Virtual Machine Shared Read/Write Content | system_u:object_r:svirt_image_t:s0 | All svirt_t processes are allowed to write to the svirt_image_t:s0 files and devices. |
Virtual Machine Shared Shared Read Only content | system_u:object_r:svirt_content_t:s0 | All svirt_t processes are able to read files/devices with this label. |
Virtual Machine Image | system_u:object_r:virt_content_t:s0 | System default label used when an image exits. No svirt_t virtual processes are allowed to read files/devices with this label. |
4.4.2. Dynamic Configuration
# ps -eZ | grep qemu-kvm
system_u:system_r:svirt_t:s0:c87,c520 27950 ? 00:00:17 qemu-kvm
qemu-kvm
process has a base label of system_u:system_r:svirt_t:s0
. The libvirt system has generated a unique MCS label of c87,c520
for this process. The base label and the MCS label are combined to form the complete security label for the process. Likewise, libvirt takes the same MCS label and base label to form the image label. This image label is then automatically applied to all host files that the VM is required to access, such as disk images, disk devices, PCI devices, USB devices, and kernel/initrd files. Each process is isolated from other virtual machines with different labels.
c87,c520
in this case) as applied to the guest disk image file in /var/lib/libvirt/images
:
# ls -lZ /var/lib/libvirt/images/*
system_u:object_r:svirt_image_t:s0:c87,c520 image1
<seclabel type='dynamic' model='selinux' relabel='yes'> <label>system_u:system_r:svirt_t:s0:c87,c520</label> <imagelabel>system_u:object_r:svirt_image_t:s0:c87,c520</imagelabel> </seclabel>
4.4.3. Dynamic Configuration with Base Labeling
<baselabel>
option can be configured manually in the XML guest configuration, as shown in this example:
<seclabel type='dynamic' model='selinux' relabel='yes'> <baselabel>system_u:system_r:svirt_custom_t:s0</baselabel> <label>system_u:system_r:svirt_custom_t:s0:c87,c520</label> <imagelabel>system_u:object_r:svirt_image_t:s0:c87,c520</imagelabel> </seclabel>
4.4.4. Static Configuration with Dynamic Resource Labeling
<seclabel type='static' model='selinux' relabel='yes'> <label>system_u:system_r:svirt_custom_t:s0:c87,c520</label> </seclabel>
4.4.5. Static Configuration without Resource Labeling
<seclabel type='static' model='selinux' relabel='no'> <label>system_u:system_r:svirt_custom_t:s0:c87,c520</label> </seclabel>
4.4.6. sVirt Labeling and NFS
virt_var_lib_t
for the root of the NFS directory that you are exporting for guest sharing. For example, if you are exporting the /exports/nfs/
directory, use the following commands:
#semanage fcontext -a -t virt_var_lib_t '/exports/nfs/'
#restorecon -Rv /exports/nfs/
libvirt
dynamically generates an sVirt label for a guest virtual machines on a NFS volume, it only guarantees label uniqueness within a single host. This means that if a high number of guests across multiple hosts share a NFS volume, it is possible for duplicate labels to occur, which creates a potential vulnerability.
- Use a different NFS volume for each virtualization host. In addition, when performing guest migration, copy the guest storage by using the
--migrate-disks
and--copy-storage-all
options. - When creating a new guest with the
virt-install
command, set a static label for the guest by:- Using the
--security
option. For example:#
virt-install --name guest1-rhel7 --memory 2048 --vcpus 2 --disk size=8 --cdrom /home/username/Downloads/rhel-workstation-7.4-x86_64-dvd.iso --os-variant rhel7 --security model=selinux,label='system_u:object_r:svirt_image_t:s0:c100,c200'
This sets the security label for all disks on the guest. - Using the
--disk
option with theseclabel
parameter. For example:#
virt-install --name guest1-rhel7 --memory 2048 --vcpus 2 --disk /path/to/disk.img,seclabel.model=selinux,seclabel.label='system_u:object_r:svirt_image_t:s0:c100,c200' --cdrom /home/username/Downloads/rhel-workstation-7.4-x86_64-dvd.iso --os-variant rhel7
This sets the security label only on the specified disks.
Chapter 5. Network Security in a Virtualized Environment
5.1. Network Security Overview
5.2. Network Security Recommended Practices
- Ensure that remote management of the system takes place only over secured network channels. Tools such as SSH and network protocols such as TLS or SSL provide both authentication and data encryption to assist with secure and controlled access to systems.
- Ensure that guest applications transferring sensitive data do so over secured network channels. If protocols such as TLS or SSL are not available, consider using one like IPsec.
- Configure firewalls and ensure they are activated at boot. Only network ports needed for the use and management of the system should be allowed. Test and review firewall rules regularly.
5.2.1. Securing Connectivity to SPICE
5.2.2. Securing Connectivity to Storage
Note
Appendix A. Further Information
A.1. SELinux and sVirt
- Main sVirt website: http://selinuxproject.org/page/SVirt.
- Dan Walsh's blog: http://danwalsh.livejournal.com/.
A.2. Virtualization Security
- NIST (National Institute of Standards and Technology) full virtualization security guidelines: http://www.nist.gov/itl/csd/virtual-020111.cfm.
Appendix B. Revision History
Revision History | |||
---|---|---|---|
Revision 1.0-22 | Thu May 23 2019 | ||
| |||
Revision 1.0-21 | Thu Oct 25 2018 | ||
| |||
Revision 1.0-21 | Thu Aug 14 2018 | ||
| |||
Revision 1.0-20 | Thu Apr 5 2018 | ||
| |||
Revision 1.0-18 | Thu Jul 27 2017 | ||
| |||
Revision 1.0-15 | Mon Oct 17 2016 | ||
| |||
Revision 1.0-9 | Thu Oct 08 2015 | ||
| |||
Revision 1.0-8 | Wed Feb 18 2015 | ||
|