Instance security

You can use different security controls when you run instances in a virtualized environment compared to when you deploy instances on bare metal. The Compute service (nova) architecture is split across two domains:

  • The control plane services, nova-api, nova-scheduler, nova-conductor, and nova-novncproxy, run as pods on the Red Hat OpenShift Container Platform (RHOCP) cluster.
  • The nova-compute service and the KVM/libvirt run on data plane Red Hat Enterprise Linux 9.4 Compute nodes that are managed by OpenStackDataPlaneNodeSet custom resources (CR)s.

Entropy for instances

The quality and source of random data available to an instance is known as entropy. Cryptographic operations depend on high-quality randomness, and entropy starvation can cause slow startup times, for example, during SSH key generation.

To provide a high-quality entropy source to instances, you need sufficient hardware random number generators (HRNGs) in the cloud. A modern HRNG can support 50 to 100 Compute nodes for normal workloads.

VirtIO RNG (default)
VirtIO RNG devices are enabled by default for all instances in Red Hat OpenStack Services on OpenShift (RHOSO) 18.0. The default entropy source is /dev/urandom on the Compute node, which provides sufficient entropy to instances at start. To disable VirtIO RNG for a specific flavor, set the hw_rng:allowed parameter to False.
Using a hardware RNG

To configure VirtIO RNG to use a hardware RNG, create a ConfigMap with the custom nova configuration and reference it in the OpenStackDataPlaneNodeSet custom resource (CR) for the Compute nodes:

apiVersion: v1
kind: ConfigMap
metadata:
  name: nova-extra-config
  namespace: openstack
data:
  19-hw-rng.conf: |
    [libvirt]
    rng_dev_path = /dev/hwrng
Note
Ensure that you use the same ConfigMap name as the configMapRef that is used by the OpenStackDataPlaneNodeSet CR. All Compute node sets referencing this ConfigMap are affected.

Instance scheduling and compute node placement

Before you create an instance, the nova-scheduler on the Red Hat OpenShift Container Platform (RHOCP) control plane selects the Compute node. Use FilterScheduler to enforce security and compliance requirements such as data isolation, availability zone separation, and affinity rules.

The following are some examples of filter schedulers:

  • Resource-based filters determine placement based on hypervisor resource usage, such as RAM, CPU, I/O utilization.
  • Image-based filters delegate placement based on image metadata, such as operating system, image type.
  • Environment-based filters place instances based on external criteria, such as availability zones, IP ranges, server group membership.
  • Custom criteria place instances based on administrator-defined metadata or trusts.

You can apply multiple filters simultaneously. For example, ServerGroupAffinityFilter ensures that an instance is created on a member of a specific host set, while ServerGroupAntiAffinityFilter ensures the instance is not created on another specific host set.

Configuring scheduler filters

Previously, you used the NovaSchedulerEnabledFilters heat parameter to configure scheduler filters. In Red Hat OpenStack Services on OpenShift (RHOSO), you configure scheduler filters with the customServiceConfig field in the nova template of the OpenStackControlPlane custom resource (CR):

spec:
  nova:
    template:
      customServiceConfig: |
        [filter_scheduler]
        enabled_filters = AggregateInstanceExtraSpecsFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,SameHostFilter,DifferentHostFilter,PciPassthroughFilter,NUMATopologyFilter
Note
Consider disabling filters that parse user-provided objects, such as arbitrary metadata, unless you have a specific requirement. Filters that process untrusted input are less secure.

Trusted images

On virtualization and containerization platforms, image integrity verification is a critical security requirement. A chain of trust must extend from the image source to the Compute node that hosts it.

You can obtain trusted images in many ways, such as the following:

  • Download from Red Hat from the official Red Hat source and validate with the SHA256 checksum.
  • Build a custom image from a RHEL installation process using the RHOSO Image service documentation.
  • Use Red Hat Enterprise Linux image builder (composer) to produce minimal, purpose-built images with only the components required for their workload.

Apply your organization’s hardening guidelines to all images before uploading them to the Image service (glance). For more information about creating operating system images for instances, see Virtual machine image management.

Image signature verification

You can verify the signature of an Image service (glance) image to ensure that the image has not been modified before the Compute service (nova) starts an instance from the image.

Images are signed using a certificate stored in the Key Manager service (barbican). The signature and certificate metadata for the image are stored as glance image properties, namely img_signature, img_signature_hash_method, img_signature_key_type, img_signature_certificate_uuid. When nova downloads the image to start an instance, it validates the signature against the stored certificate before proceeding.

Previously, you used the VerifyGlanceSignatures heat parameter and an openstack overcloud deploy run heat parameter to verify image signatures. In Red Hat OpenStack Services on OpenShift (RHOSO), you configure image signature verification with the customServiceConfig field in the nova template of the OpenStackControlPlane custom resource (CR):

spec:
  nova:
    template:
      customServiceConfig: |
        [glance]
        verify_glance_signatures = True
        enable_certificate_validation = True

After you modify the OpenStackControlPlane CR, the OpenStack Operator applies the change and restarts the affected nova pods. The configuration is also propagated to the nova-compute service on data plane Compute nodes from the nova-extra-configConfigMap mechanism.

If you enable image signature verification, any image that is unsigned or improperly signed fails verification and the instance does not start. Ensure that all production images are properly signed before you enable this feature in a production environment.

The Key Manager service (barbican) is enabled by default in RHOSO. You can use certificate storage for image signing without additional deployment configuration.

Secure instance migration

You can migrate instances from one Compute node to another Compute node. When live migrate an instance, you transfer the contents of the disk and memory of a running instance in plain text on the control plane network. The following are some of the risks involved with live migration:

  • denial of service (DoS).
  • Data exposure and manipulation.
  • Code injection.
Note
You can enable TLS-Everywhere (TLS-e) to encrypt the live migration.

Updates and patches

When you update the hypervisor or the Compute node operating system, the updates do not propagate to guest instances.

Previously, you used the openstack overcloud update command to update. In Red Hat OpenStack Services on OpenShift (RHOSO), you use OpenStackDataPlaneNodeSet custom resource (CR)s and the data plane update services to manage updates.

Split update workflow (RHOSO 18.0.18 and newer)

The update process is separated into two phases:

  • OpenStack service update that updates containers and essential packages.
  • System-level update that updates all the packages and the kernel.
Rolling updates
You can update Compute nodes incrementally in stages while instances run on nodes that are not being updated. You can use live migration to evacuate instances from a node before you update. This approach is suitable for environments where all workloads support live migration.
In-place updates
You can update Compute nodes in place without migrating instances. To avoid unplanned instance disruption, you can defer kernel upgrades. In-place updates are the only option for achieving zero workload downtime on nodes that host instances that you cannot live migrate, such as those using PCI passthrough. Do not perform in-place updates on nodes running OVS-DPDK, because the update process disrupts data plane networking for active instances.

Assign clear ownership of guest VM images. Instance owners are responsible for hardening, patching, and maintaining the security of their guest operating systems. Cloud operators should provide patch management guidance and enforce update Service Level Agreement (SLA)s where applicable.

Security groups

You can use security groups to provide network-level access control for instances. You implement security groups with OVN ACLs, instead of with iptables.

Security group rules are applied to incoming traffic by default. For outgoing traffic restrictions, you can use additional OVN-based policies or host-based firewalls within the instance.

  • Keep security groups and port security enabled unless you have a specific, documented requirement. For a defense-in-depth approach, you can use security groups at the network layer, and host-based firewalls at the instance layer.
  • Apply granular rules instead of permissive security group rules, for example, allowing all ingress from 0.0.0.0/0. Create specific rules for each required protocol, port, and source address range.
  • Audit security group changes. In RHOSO, every security group API call passes through nova/neutron pods and is logged. You can forward these logs to a Security Information and Event Management (SIEM) and create alerts for unexpected rule modifications,such as rules that open broad ingress access.
Note
Security group rules and host-based firewall rules can conflict and deny legitimate traffic. When troubleshooting connectivity issues, check both layers.

Instance console security

Access the console of your instance with the browser-based VNC client, noVNC. nova-novncproxy runs as a pod on the Red Hat OpenShift Container Platform (RHOCP) cluster. The proxy endpoint is exposed with a RHOCP Service, either MetalLB or Route.

You can improve the security of the console with the following best practices:

Ensure TLS encryption on the VNC proxy endpoint
In RHOSO, Transport Layer Security (TLS) is managed by cert-manager. Verify the nova-novncproxy` endpoint is served over TLS by checking the corresponding Certificate custom resource (CR) in the openstack namespace.
Enable VeNCrypt authentication
You can use VeNCrypt to provide TLS-encrypted and x509-authenticated connections between the noVNC proxy pod and the VNC server on Compute nodes. Configure the vnc.auth_schemes option in the nova-compute configuration ConfigMap to require vencrypt, and remove none from the accepted schemes.
Restrict access to the proxy endpoint
You can use MetalLBloadBalancerSourceRanges or RHOCP Route IP whitelisting to limit the networks that can reach the noVNC proxy. Do not expose the noVNC proxy to the public internet unless explicitly required.

Certificate and key injection

When you create an instance with the command openstack server create --key-name, the specified SSH public key is made available to cloud-init, which installs it into the appropriate user’s ~/.ssh/authorized_keys file.

The following are some best practices for certificate and key injection:

Use SSH key pairs instead of passwords
Key-based authentication is more resistant to brute-force attacks than password authentication.
Rotate SSH key pairs
Create new key pairs periodically and rotate the keys through Ansible roles.
Use a configuration drive for key injection in isolated networks
If an instance does not have access to the metadata service, for example, on a network without a DHCP agent or router, you can use a configuration drive to deliver the SSH key. In RHOSO, the configuration drive is enabled by default using ISO format and live migration works, when you use nova microversion 2.25 or newer.

RHOCP security controls for compute services

The Compute service (nova) control plane services run as pods on Red Hat OpenShift Container Platform (RHOCP), therefore you can implement platform-level security controls, such as the following:

Monitor nova pod lifecycle
Unexpected restarts or CrashLoopBackOff of nova-scheduler, nova-conductor, or nova-api pods should trigger alerts that might indicate a misconfiguration, resource exhaustion, or an active attack.
Protect nova Secrets
The openstack namespace contains Kubernetes Secrets holding nova database credentials, RabbitMQ credentials, and keystone service account passwords. You can use RHOCP RBAC and etcd encryption to protect these Kubernetes Secrets.
Verify container image provenance
Ensure that all nova container images are pulled from the authenticated Red Hat registry and are signed. Do not allow images from untrusted registries.