Chapter 13. Managing memory devices

A Compute Express Link (CXL) memory device transitions through several states in response to specific system events. The terms "Level 0" through "Level 4" describe the state transitions of a CXL memory device during hot-plug operations.

In RHEL, CXL memory is managed in units called memory regions. A memory region can correspond to a single CXL device or span multiple devices by specifying interleave during creation. Memory incorporated into memory regions must be brought "online" before it is available to the operating system.

You must modify or configure memory regions manually in several scenarios. For example, during a hot-add event, the driver detects the hardware connection but does not automatically configure memory regions or bring the memory online. Similarly, you must manually create memory regions when you want to interleave memory across multiple CXL devices to improve performance, as the driver cannot automatically determine your desired interleave topology.

In Red Hat Enterprise Linux (RHEL), memory used by the kernel or drivers cannot be set to an offline state or hot-removed. RHEL cannot migrate data from these areas, which can block hot-remove operations.

By default, the kernel and drivers can use all memory areas. This behavior prevents other memory devices, such as Compute Express Link (CXL), from being hot-plugged or hot-removed. To avoid this limitation, RHEL allows you to create a ZONE_MOVABLE. This memory area is not used by the kernel or drivers, ensuring that it can be safely hot-removed.

For bare metal systems, RHEL automatically configures ZONE_MOVABLE through the default udev rules for hot-plugged memory. This configuration ensures safer hot-removal of devices like CXL without manual configuration.

The kernel requires a certain amount of memory capacity to manage hot-added memory, known as the memmap area. This area uses approximately 1.6% of the memory size, for example, 16GB for 1TB of memory. By default, this is allocated from ZONE_NORMAL. If ZONE_NORMAL has insufficient amount of free memory, the allocation could trigger the Out of Memory (OOM) Killer.

To prevent this, you can allocate the memmap area from the hot-added memory itself. Specific devices, such as Compute Express Link (CXL) devices and standard Dual Inline Memory Modules (DIMMs), support this feature.

To dynamically increase system memory, you can hot-plug memory devices, such as Compute Express Link (CXL) modules or Dual Inline Memory Modules (DIMMs), in Red Hat Enterprise Linux.

To safely hot-remove a Compute Express Link (CXL) memory device, you must ensure that the memory is no longer in use by the kernel. You can safely hot-remove a CXL memory device by taking the memory offline and disabling the associated memory regions.

You can verify the status and configuration of Compute Express Link (CXL) memory devices by using various command-line tools.

When using Compute Express Link (CXL) memory as shared memory between servers, ensure the device is set to Level 3 (Memory region enabled but not online for the OS).

The standard OS memory management system operates at level 4. However, using shared memory between servers requires a dedicated interface and cannot be handled by traditional RHEL memory management.

To use CXL memory as shared memory between servers, keep it offline. To keep the memory offline, you must create custom udev rules to prevent the automatic online operation that RHEL performs by default.

When planning to hot-remove memory devices, consider the following potential issues and limitations.

[ 6381.169189] page dumped because: migration failure
[ 6381.181353] migrating pfn 8842600 failed ret:2
...

This indicates a failure to move data from the memory targeted for removal. The kernel retries memory migration, and it usually completes successfully. However, hot removal fails depending on system conditions:

Stop these processes before performing the hot removal.