Chapter 15. Backup and restore


15.1. Backup and restore by using VM snapshots

You can back up and restore virtual machines (VMs) by using snapshots.

Snapshots are supported by the following storage providers:

  • Red Hat OpenShift Data Foundation
  • Any other cloud storage provider with the Container Storage Interface (CSI) driver that supports the Kubernetes Volume Snapshot API

To create snapshots of a VM in the Running state with the highest integrity, install the QEMU guest agent if it is not included with your operating system. The QEMU guest agent is included with the default Red Hat templates.

Important

Online snapshots are supported for virtual machines that have hot plugged virtual disks. However, hot plugged disks that are not in the virtual machine specification are not included in the snapshot.

Ensure that the QEMU guest agent is installed and running on the virtual machine before you take an online snapshot.

The QEMU guest agent stops responding to file system operations to ensure that the snapshot captures a consistent state.

The QEMU guest agent takes a consistent snapshot by attempting to quiesce the VM file system. This ensures that in-flight I/O is written to the disk before the snapshot is taken. If the guest agent is not present, quiescing is not possible and a best-effort snapshot is taken.

The conditions under which a snapshot is taken are reflected in the snapshot indications that are displayed in the web console or CLI. If these conditions do not meet your requirements, try creating the snapshot again or use an offline snapshot

15.1.1. About snapshots

A snapshot represents the state and data of a virtual machine (VM) at a specific point in time. You can use a snapshot to restore an existing VM to a previous state (represented by the snapshot) for backup and disaster recovery or to rapidly roll back to a previous development version.

A VM snapshot is created from a VM that is powered off (Stopped state) or powered on (Running state).

When taking a snapshot of a running VM, the controller checks that the QEMU guest agent is installed and running. If so, it freezes the VM file system before taking the snapshot, and thaws the file system after the snapshot is taken.

The snapshot stores a copy of each Container Storage Interface (CSI) volume attached to the VM and a copy of the VM specification and metadata. Snapshots cannot be changed after creation.

You can perform the following snapshot actions:

  • Create a new snapshot
  • Create a clone of a virtual machine from a snapshot

    Important

    Cloning a VM with a vTPM device attached to it or creating a new VM from its snapshot is not supported.

  • List all snapshots attached to a specific VM
  • Restore a VM from a snapshot
  • Delete an existing VM snapshot

The VM snapshot feature introduces three new API objects defined as custom resource definitions (CRDs) for managing snapshots:

  • VirtualMachineSnapshot: Represents a user request to create a snapshot. It contains information about the current state of the VM.
  • VirtualMachineSnapshotContent: Represents a provisioned resource on the cluster (a snapshot). It is created by the VM snapshot controller and contains references to all resources required to restore the VM.
  • VirtualMachineRestore: Represents a user request to restore a VM from a snapshot.

The VM snapshot controller binds a VirtualMachineSnapshotContent object with the VirtualMachineSnapshot object for which it was created, with a one-to-one mapping.

You can configure application-consistent snapshots and backups for Linux or Windows virtual machines (VMs) through a cycle of freezing and thawing. For any application, you can configure a script on a Linux VM or register on a Windows VM to be notified when a snapshot or backup is due to begin.

On a Linux VM, freeze and thaw processes trigger automatically when a snapshot is taken or a backup is started by using, for example, a plugin from Velero or another backup vendor. The freeze process, performed by QEMU Guest Agent (QEMU GA) freeze hooks, ensures that before the snapshot or backup of a VM occurs, all of the VM’s filesystems are frozen and each appropriately configured application is informed that a snapshot or backup is about to start. This notification affords each application the opportunity to quiesce its state. Depending on the application, quiescing might involve temporarily refusing new requests, finishing in-progress operations, and flushing data to disk. The operating system is then directed to quiesce the filesystems by flushing outstanding writes to disk and freezing new write activity. All new connection requests are refused. When all applications have become inactive, the QEMU GA freezes the filesystems, and a snapshot is taken or a backup initiated. After the taking of the snapshot or start of the backup, the thawing process begins. Filesystems writing is reactivated and applications receive notification to resume normal operations.

The same cycle of freezing and thawing is available on a Windows VM. Applications register with the Volume Shadow Copy Service (VSS) to receive notifications that they should flush out their data because a backup or snapshot is imminent. Thawing of the applications after the backup or snapshot is complete returns them to an active state. For more details, see the Windows Server documentation about the Volume Shadow Copy Service.

You can create a snapshot of a virtual machine (VM) by using the OpenShift Container Platform web console.

Prerequisites

  • The snapshot feature gate is enabled in the YAML configuration of the kubevirt CR.
  • The VM snapshot includes disks that meet the following requirements:

    • The disks are data volumes or persistent volume claims.
    • The disks belong to a storage class that supports Container Storage Interface (CSI) volume snapshots.
    • The disks are bound to a persistent volume (PV) and populated with a datasource.

Procedure

  1. Navigate to Virtualization VirtualMachines in the web console.
  2. Select a VM to open the VirtualMachine details page.
  3. Click the Snapshots tab and then click Take Snapshot.

    Alternatively, right-click the VM and select Create snapshot from the menu.

  4. Enter the snapshot name.
  5. Expand Disks included in this Snapshot to see the storage volumes to be included in the snapshot.
  6. If your VM has disks that cannot be included in the snapshot and you wish to proceed, select I am aware of this warning and wish to proceed.
  7. Click Save.

15.1.4. Creating a snapshot by using the CLI

You can create a virtual machine (VM) snapshot for an offline or online VM by creating a VirtualMachineSnapshot object.

Prerequisites

  • Ensure the Snapshot feature gate is enabled for the kubevirt CR by using the following command:

    $ oc get kubevirt kubevirt-hyperconverged -n openshift-cnv -o yaml

    Truncated output:

    spec:
      developerConfiguration:
        featureGates:
          - Snapshot
  • Ensure that the VM snapshot includes disks that meet the following requirements:

    • The disks are data volumes or persistent volume claims.
    • The disks belong to a storage class that supports Container Storage Interface (CSI) volume snapshots.
    • The disks are bound to a persistent volume (PV) and populated with a datasource.
  • Install the OpenShift CLI (oc).
  • Optional: Power down the VM for which you want to create a snapshot.

Procedure

  1. Create a YAML file to define a VirtualMachineSnapshot object that specifies the name of the new VirtualMachineSnapshot and the name of the source VM as in the following example:

    apiVersion: snapshot.kubevirt.io/v1beta1
    kind: VirtualMachineSnapshot
    metadata:
      name: <snapshot_name>
    spec:
      source:
        apiGroup: kubevirt.io
        kind: VirtualMachine
        name: <vm_name>
  2. Create the VirtualMachineSnapshot object:

    $ oc create -f <snapshot_name>.yaml

    The snapshot controller creates a VirtualMachineSnapshotContent object, binds it to the VirtualMachineSnapshot, and updates the status and readyToUse fields of the VirtualMachineSnapshot object.

Verification

  1. Optional: During the snapshot creation process, you can use the wait command to monitor the status of the snapshot and wait until it is ready for use:

    1. Enter the following command:

      $ oc wait <vm_name> <snapshot_name> --for condition=Ready
    2. Verify the status of the snapshot:

      • InProgress - The snapshot operation is still in progress.
      • Succeeded - The snapshot operation completed successfully.
      • Failed - The snapshot operaton failed.

        Note

        Online snapshots have a default time deadline of five minutes (5m). If the snapshot does not complete successfully in five minutes, the status is set to failed. Afterwards, the file system will be thawed and the VM unfrozen but the status remains failed until you delete the failed snapshot image.

        To change the default time deadline, add the FailureDeadline attribute to the VM snapshot spec with the time designated in minutes (m) or in seconds (s) that you want to specify before the snapshot operation times out.

        To set no deadline, you can specify 0, though this is generally not recommended, as it can result in an unresponsive VM.

        If you do not specify a unit of time such as m or s, the default is seconds (s).

  2. Verify that the VirtualMachineSnapshot object is created and bound with VirtualMachineSnapshotContent and that the readyToUse flag is set to true:

    $ oc describe vmsnapshot <snapshot_name>

    Example output:

    apiVersion: snapshot.kubevirt.io/v1beta1
    kind: VirtualMachineSnapshot
    metadata:
      creationTimestamp: "2020-09-30T14:41:51Z"
      finalizers:
      - snapshot.kubevirt.io/vmsnapshot-protection
      generation: 5
      name: mysnap
      namespace: default
      resourceVersion: "3897"
      selfLink: /apis/snapshot.kubevirt.io/v1beta1/namespaces/default/virtualmachinesnapshots/my-vmsnapshot
      uid: 28eedf08-5d6a-42c1-969c-2eda58e2a78d
    spec:
      source:
        apiGroup: kubevirt.io
        kind: VirtualMachine
        name: my-vm
    status:
      conditions:
      - lastProbeTime: null
        lastTransitionTime: "2020-09-30T14:42:03Z"
        reason: Operation complete
        status: "False"
        type: Progressing
      - lastProbeTime: null
        lastTransitionTime: "2020-09-30T14:42:03Z"
        reason: Operation complete
        status: "True"
        type: Ready
      creationTime: "2020-09-30T14:42:03Z"
      readyToUse: true
      sourceUID: 355897f3-73a0-4ec4-83d3-3c2df9486f4f
      virtualMachineSnapshotContentName: vmsnapshot-content-28eedf08-5d6a-42c1-969c-2eda58e2a78d
      indications:
        - Online
      includedVolumes:
        - name: rootdisk
          kind: PersistentVolumeClaim
          namespace: default
        - name: datadisk1
          kind: DataVolume
          namespace: default

    where:

    status

    The status field of the Progressing condition specifies if the snapshot is still being created.

    The status field of the Ready condition specifies if the snapshot creation process is complete.

    readyToUse
    Specifies if the snapshot is ready to be used.
    virtualMachineSnapshotContentName
    Specifies that the snapshot is bound to a VirtualMachineSnapshotContent object created by the snapshot controller.
    indications
    Specifies additional information about the snapshot, such as whether it is an online snapshot, or whether it was created with QEMU guest agent running.
    includedVolumes
    Lists the storage volumes that are part of the snapshot, as well as their parameters.
  3. Check the includedVolumes section in the snapshot description to verify that the expected PVCs are included in the snapshot.

Snapshot indications are contextual information about online virtual machine (VM) snapshot operations. Indications are not available for offline virtual machine (VM) snapshot operations. Indications are helpful in describing details about the online snapshot creation.

Prerequisites

  • You must have attempted to create an online VM snapshot.

Procedure

  1. Display the output from the snapshot indications by performing one of the following actions:

    • Use the command line to view indicator output in the status stanza of the VirtualMachineSnapshot object YAML.
    • In the web console, click VirtualMachineSnapshot Status in the Snapshot details screen.
  2. Verify the status of your online VM snapshot by viewing the values of the status.indications parameter:

    • Online indicates that the VM was running during online snapshot creation.
    • GuestAgent indicates that the QEMU guest agent was active and successfully quiesced the guest file system for the online snapshot. This results in an application-consistent snapshot, preserving data integrity as if the applications had been gracefully shut down.
    • NoGuestAgent indicates that the QEMU guest agent was not installed, or not ready to quiesce the file system during the online snapshot. This results in a crash-consistent snapshot, which captures the VM’s state like an abrupt power-off. As a result, application consistency is not guaranteed, which causes a risk of data issues for critical applications. For higher reliability, install and run the guest agent, or retry the snapshot.
    • QuiesceFailed indicates that an attempt to quiesce the file system failed during the online snapshot process. This means that the snapshot was created, but it is not necessarily application-consistent. To achieve proper consistency, retry the snapshot.

You can restore a virtual machine (VM) to a previous configuration represented by a snapshot in the OpenShift Container Platform web console.

Procedure

  1. Navigate to Virtualization VirtualMachines in the web console.
  2. Select a VM to open the VirtualMachine details page.
  3. If the VM is running, click the Options menu kebab and select Stop to power it down.
  4. Click the Snapshots tab to view a list of snapshots associated with the VM.
  5. Select a snapshot to open the Snapshot Details screen.
  6. Click the Options menu kebab and select Restore VirtualMachine from snapshot.
  7. Optional: In the Volume restore policy section, select how restored persistent volume claims (PVCs) are named:

    • Prefix target name - The restored PVC names use the target VM name as a prefix. This is the default setting.
    • In place - The restored PVCs overwrite the original PVCs with the same names.
    • Randomize names - The restored PVC names are randomly generated.
  8. Click Restore.
  9. Optional: You can also create a new VM based on the snapshot. To do so:

    1. In the Options menu kebab of the snapshot, select Create VirtualMachine from Snapshot.
    2. Provide a name for the new VM.
    3. Click Create

You can restore an existing virtual machine (VM) to a previous configuration by using the command line. You can only restore from an offline VM snapshot.

Prerequisites

  • Install the OpenShift CLI (oc).
  • Power down the VM you want to restore.
  • Optional: Adjust what happens if the target VM is not fully stopped (ready). To do so, set the targetReadinessPolicy parameter in the vmrestore YAML configuration to one of the following values:

    • FailImmediate - The restore process fails immediately if the VM is not ready.
    • StopTarget - If the VM is not ready, it gets stopped, and the restore process starts.
    • WaitGracePeriod 5 - The restore process waits for a set amount of time, in minutes, for the VM to be ready. This is the default setting, with the default value set to 5 minutes.
    • WaitEventually - The restore process waits indefinitely for the VM to be ready.
  • Optional: To control the naming of restored persistent volume claims (PVCs), you can set the volumeRestorePolicy parameter to one of the following values:

    • PrefixTargetName - The restored PVC names use the target VM name as a prefix: <vm_name>-<volume_name>.
    • RandomizeNames - The system generates the restored PVC names randomly: restore-<uid>-<volume_name>.
    • InPlace - The restored PVCs overwrite the original PVCs. The system deletes the original PVCs if they exist and creates new PVCs with the same names. This is the default setting.
  • Optional: To control how restored persistent volume claims (PVCs) are named, you can set the volumeRestorePolicy parameter to one of the following values:

    • PrefixTargetName - The restored PVC names use the target VM name as a prefix: <vm_name>-<volume_name>. This is the default setting.
    • RandomizeNames - The restored PVC names are randomly generated: restore-<uid>-<volume_name>.
    • InPlace - The restored PVCs overwrite the original PVCs. The original PVCs are deleted if they exist, and new PVCs are created with the same names.

Procedure

  1. Create a YAML file to define a VirtualMachineRestore object that specifies the name of the VM you want to restore and the name of the snapshot to be used as the source as in the following example:

    apiVersion: snapshot.kubevirt.io/v1beta1
    kind: VirtualMachineRestore
    metadata:
      name: <vm_restore>
    spec:
      target:
        apiGroup: kubevirt.io
        kind: VirtualMachine
        name: <vm_name>
      virtualMachineSnapshotName: <snapshot_name>
      volumeRestorePolicy: PrefixTargetName

    Where:

    • volumeRestorePolicy: Optional. The volume restore policy determines how restored PVCs are named. Valid values are PrefixTargetName (default), RandomizeNames, or InPlace.
  2. Optional: To customize the names, labels, and annotations of individual restored persistent volume claims (PVCs), add the volumeRestoreOverrides parameter to the YAML file:

    apiVersion: snapshot.kubevirt.io/v1beta1
    kind: VirtualMachineRestore
    metadata:
      name: <vm_restore>
    spec:
      target:
        apiGroup: kubevirt.io
        kind: VirtualMachine
        name: <vm_name>
      virtualMachineSnapshotName: <snapshot_name>
      volumeRestoreOverrides:
      - volumeName: <volume_name>
        restoreName: <custom_pvc_name>
        labels:
          custom-label: <label_value>
        annotations:
          custom-annotation: <annotation_value>

    Where:

    • volumeName: Required. The name of the volume from the snapshot to customize.
    • restoreName: Optional. The custom name for the restored PVC. If not specified, the PVC name is determined by the volumeRestorePolicy setting.
    • labels: Optional. Custom labels to add to the restored PVC. These labels are merged with any existing labels from the source PVC.
    • annotations: Optional. Custom annotations to add to the restored PVC. These annotations are merged with any existing annotations from the source PVC.
  3. Create the VirtualMachineRestore object:

    $ oc create -f <vm_restore>.yaml

    The snapshot controller updates the status fields of the VirtualMachineRestore object and replaces the existing VM configuration with the snapshot content.

Verification

  • Verify that the VM is restored to the previous state represented by the snapshot and that the status.complete flag is set to true:

    $ oc get vmrestore <vm_restore>

    Example output:

    apiVersion: snapshot.kubevirt.io/v1beta1
    kind: VirtualMachineRestore
    metadata:
      creationTimestamp: "2020-09-30T14:46:27Z"
      generation: 5
      name: my-vmrestore
      namespace: default
      ownerReferences:
      - apiVersion: kubevirt.io/v1
        blockOwnerDeletion: true
        controller: true
        kind: VirtualMachine
        name: my-vm
        uid: 355897f3-73a0-4ec4-83d3-3c2df9486f4f
      resourceVersion: "5512"
      uid: 71c679a8-136e-46b0-b9b5-f57175a6a041
    spec:
      target:
        apiGroup: kubevirt.io
        kind: VirtualMachine
        name: my-vm
      virtualMachineSnapshotName: my-vmsnapshot
    status:
      complete: true
      conditions:
      - lastProbeTime: null
        lastTransitionTime: "2020-09-30T14:46:28Z"
        reason: Operation complete
        status: "False"
        type: Progressing
      - lastProbeTime: null
        lastTransitionTime: "2020-09-30T14:46:28Z"
        reason: Operation complete
        status: "True"
        type: Ready
      deletedDataVolumes:
      - test-dv1
      restoreTime: "2020-09-30T14:46:28Z"
      restores:
      - dataVolumeName: restore-71c679a8-136e-46b0-b9b5-f57175a6a041-datavolumedisk1
        persistentVolumeClaim: restore-71c679a8-136e-46b0-b9b5-f57175a6a041-datavolumedisk1
        volumeName: datavolumedisk1
        volumeSnapshotName: vmsnapshot-28eedf08-5d6a-42c1-969c-2eda58e2a78d-volume-datavolumedisk1
    Note

    If the Progressing condition has status: "True", the VM is still being restored.

You can delete an existing virtual machine (VM) snapshot by using the web console.

Procedure

  1. Navigate to Virtualization VirtualMachines in the web console.
  2. Select a VM to open the VirtualMachine details page.
  3. Click the Snapshots tab to view a list of snapshots associated with the VM.
  4. Click the Options menu kebab beside a snapshot and select Delete snapshot.
  5. Click Delete.

You can delete an existing virtual machine (VM) snapshot by deleting the appropriate VirtualMachineSnapshot object.

Prerequisites

  • Install the OpenShift CLI (oc).

Procedure

  • Delete the VirtualMachineSnapshot object:

    $ oc delete vmsnapshot <snapshot_name>

    The snapshot controller deletes the VirtualMachineSnapshot along with the associated VirtualMachineSnapshotContent object.

Verification

  • Verify that the snapshot is deleted and no longer attached to this VM:

    $ oc get vmsnapshot

15.2. Backing up and restoring virtual machines

Back up and restore virtual machines by using the OpenShift API for Data Protection.

Important

Red Hat supports using OpenShift Virtualization 4.14 or later with OADP 1.3.x or later.

OADP versions earlier than 1.3.0 are not supported for back up and restore of OpenShift Virtualization.

As a cluster administrator, you can install the OpenShift API for Data Protection (OADP) with OpenShift Virtualization by installing the OADP Operator and configuring a backup location. You can then install the Data Protection Application.

To install the OADP Operator in a restricted network environment, you must first disable the default software catalog sources and mirror the Operator catalog.

Note

OpenShift API for Data Protection with OpenShift Virtualization supports the following backup and restore storage options:

  • Container Storage Interface (CSI) backups
  • Container Storage Interface (CSI) backups with DataMover

The following storage options are excluded:

  • File system backup and restore
  • Volume snapshot backup and restore

The latest version of the OADP Operator installs Velero 1.16.

Warning

Red Hat support is limited to only the following options:

  • CSI backups
  • CSI backups with DataMover.

Prerequisites

  • Access to the cluster as a user with the cluster-admin role.

Procedure

  1. Install the OADP Operator according to the instructions for your storage provider.
  2. Install the Data Protection Application (DPA) with the kubevirt and openshift OADP plug-ins.
  3. Back up virtual machines by creating a Backup custom resource (CR).

    You restore the Backup CR by creating a Restore CR.

15.2.2. Installing the Data Protection Application

You install the Data Protection Application (DPA) by creating an instance of the DataProtectionApplication API.

Prerequisites

  • You must install the OADP Operator.
  • You must configure object storage as a backup location.
  • If you use snapshots to back up PVs, your cloud provider must support either a native snapshot API or Container Storage Interface (CSI) snapshots.
  • If the backup and snapshot locations use the same credentials, you must create a Secret with the default name, cloud-credentials.

    Note

    If you do not want to specify backup or snapshot locations during the installation, you can create a default Secret with an empty credentials-velero file. If there is no default Secret, the installation will fail.

Procedure

  1. Click Ecosystem Installed Operators and select the OADP Operator.
  2. Under Provided APIs, click Create instance in the DataProtectionApplication box.
  3. Click YAML View and update the parameters of the DataProtectionApplication manifest:

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    metadata:
      name: <dpa_sample>
      namespace: openshift-adp
    spec:
      configuration:
        velero:
          defaultPlugins:
            - kubevirt
            - gcp
            - csi
            - openshift
          resourceTimeout: 10m
        nodeAgent:
          enable: true
          uploaderType: kopia
          podConfig:
            nodeSelector: <node_selector>
      backupLocations:
        - velero:
            provider: gcp
            default: true
            credential:
              key: cloud
              name: <default_secret>
            objectStorage:
              bucket: <bucket_name>
              prefix: <prefix>

    where:

    namespace
    Specifies the default namespace for OADP which is openshift-adp. The namespace is a variable and is configurable.
    kubevirt
    Specifies that the kubevirt plugin is mandatory for OpenShift Virtualization.
    gcp
    Specifies the plugin for the backup provider, for example, gcp, if it exists.
    csi
    Specifies that the csi plugin is mandatory for backing up PVs with CSI snapshots. The csi plugin uses the Velero CSI beta snapshot APIs. You do not need to configure a snapshot location.
    openshift
    Specifies that the openshift plugin is mandatory.
    resourceTimeout
    Specifies how many minutes to wait for several Velero resources such as Velero CRD availability, volumeSnapshot deletion, and backup repository availability, before timeout occurs. The default is 10m.
    nodeAgent
    Specifies the administrative agent that routes the administrative requests to servers.
    enable
    Set this value to true if you want to enable nodeAgent and perform File System Backup.
    uploaderType
    Specifies the uploader type. Enter kopia as your uploader to use the Built-in DataMover. The nodeAgent deploys a daemon set, which means that the nodeAgent pods run on each working node. You can configure File System Backup by adding spec.defaultVolumesToFsBackup: true to the Backup CR.
    nodeSelector
    Specifies the nodes on which Kopia are available. By default, Kopia runs on all nodes.
    provider
    Specifies the backup provider.
    name
    Specifies the correct default name for the Secret, for example, cloud-credentials-gcp, if you use a default plugin for the backup provider. If specifying a custom name, then the custom name is used for the backup location. If you do not specify a Secret name, the default name is used.
    bucket
    Specifies a bucket as the backup storage location. If the bucket is not a dedicated bucket for Velero backups, you must specify a prefix.
    prefix
    Specifies a prefix for Velero backups, for example, velero, if the bucket is used for multiple purposes.
  4. Click Create.

Verification

  1. Verify the installation by viewing the OpenShift API for Data Protection (OADP) resources by running the following command:

    $ oc get all -n openshift-adp
    NAME                                                     READY   STATUS    RESTARTS   AGE
    pod/oadp-operator-controller-manager-67d9494d47-6l8z8    2/2     Running   0          2m8s
    pod/node-agent-9cq4q                                     1/1     Running   0          94s
    pod/node-agent-m4lts                                     1/1     Running   0          94s
    pod/node-agent-pv4kr                                     1/1     Running   0          95s
    pod/velero-588db7f655-n842v                              1/1     Running   0          95s
    
    NAME                                                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
    service/oadp-operator-controller-manager-metrics-service   ClusterIP   172.30.70.140    <none>        8443/TCP   2m8s
    service/openshift-adp-velero-metrics-svc                   ClusterIP   172.30.10.0      <none>        8085/TCP   8h
    
    NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
    daemonset.apps/node-agent    3         3         3       3            3           <none>          96s
    
    NAME                                                READY   UP-TO-DATE   AVAILABLE   AGE
    deployment.apps/oadp-operator-controller-manager    1/1     1            1           2m9s
    deployment.apps/velero                              1/1     1            1           96s
    
    NAME                                                           DESIRED   CURRENT   READY   AGE
    replicaset.apps/oadp-operator-controller-manager-67d9494d47    1         1         1       2m9s
    replicaset.apps/velero-588db7f655                              1         1         1       96s
  2. Verify that the DataProtectionApplication (DPA) is reconciled by running the following command:

    $ oc get dpa dpa-sample -n openshift-adp -o jsonpath='{.status}'
    {"conditions":[{"lastTransitionTime":"2023-10-27T01:23:57Z","message":"Reconcile complete","reason":"Complete","status":"True","type":"Reconciled"}]}
  3. Verify the type is set to Reconciled.
  4. Verify the backup storage location and confirm that the PHASE is Available by running the following command:

    $ oc get backupstoragelocations.velero.io -n openshift-adp
    NAME           PHASE       LAST VALIDATED   AGE     DEFAULT
    dpa-sample-1   Available   1s               3d16h   true

Recover individual files from virtual machine backups without restoring the entire VM. Browse, compare, and download files from multiple backups simultaneously through a web browser or SSH-based tools.

15.3.1. What problem is OADP VMFR solving

Recover individual files from virtual machine (VM) backups without restoring the entire VM. Browse multiple backups simultaneously and retrieve only the files you need through standard tools such as a web browser or rsync.

Current VM backup recovery workflows require you to restore an entire virtual machine to access a single file. This uses substantial cluster resources and time. The virtual machine file restore (VMFR) feature addresses this problem by providing a Kubernetes-native mechanism for file-level recovery from VM backups created by OADP.

The VMFR feature uses a two-phase approach:

  1. Backup discovery: Identify which Velero backups contain a specified virtual machine by creating a VirtualMachineBackupsDiscovery (VMBD) custom resource (CR).
  2. File restore: Make the discovered backup files accessible for browsing and downloading by creating a VirtualMachineFileRestore (VMFR) CR.

OADP VMFR offers the following benefits:

  • You can recover individual files from VM backups without restoring the entire VM.
  • You can browse and compare files across multiple backup versions simultaneously.
  • You can access restored files through a web browser or SSH-based tools such as rsync, scp, and sftp.
  • All operations are managed within the cluster by using standard Kubernetes resources.
  • Temporary namespaces isolate file-serving resources and are cleaned up automatically.

15.3.2. When to use OADP VMFR

Review the following scenarios where OADP virtual machine file restore (VMFR) addresses common file recovery challenges. This helps you determine whether VMFR is the appropriate solution for your use case.

Configuration issue investigation
You have a production virtual machine (VM) that has a configuration issue. You need to compare configuration files from before and after the incident. Without VMFR, you have to restore multiple full VMs and perform manual file comparisons, which uses significant time and resources. With VMFR, you can browse files from multiple backup versions simultaneously and compare configurations without restoring any VMs.
Selective file recovery
You accidentally delete critical documents from a VM. The files exist in a recent backup, but restoring the entire VM would overwrite changes made since that backup. With VMFR, you can recover the specific files you need without losing the current VM state or overwriting recent modifications.
Multi-VM backup discovery
You have a namespace that runs multiple VMs with daily backups over several weeks. You need to recover a file from a specific VM but do not know which backup contains it. Without VMFR, you must inspect each backup individually or attempt multiple restores. With VMFR, you can create a VirtualMachineBackupsDiscovery CR to identify which backups contain the target VM and then restore files from the target backup.

15.3.3. OADP VMFR custom resources

Use OADP virtual machine file restore (VMFR) custom resources to discover VM backups and restore individual files from those backups.

The OADP VMFR feature uses the following custom resources (CRs) to perform file-level restore operations:

Expand
Table 15.1. Custom resources
CRDescription

VirtualMachineBackupsDiscovery (VMBD)

Identifies which Velero backups contain a specified virtual machine. Returns categorized results of valid and invalid backups.

VirtualMachineFileRestore (VMFR)

Orchestrates the workflow for making discovered backup files accessible through web or SSH access methods.

15.3.4. How OADP VMFR works

Review how OADP virtual machine file restore (VMFR) processes file-level restore requests through a two-phase workflow that discovers virtual machine (VM) backups and makes their contents accessible for browsing and downloading.

15.3.4.1. Backup discovery phase

The backup discovery phase identifies which Velero backups contain a specified VM. When you create a VirtualMachineBackupsDiscovery (VMBD) CR, the discovery controller performs the following steps:

  1. Compiles a list of candidate backups from explicitly named backups or all cluster backups.
  2. Filters candidates by time range if you specify startTime and endTime boundaries.
  3. Validates that each candidate backup is in the Completed phase.
  4. Verifies that the specified VM is present in each candidate backup by querying Velero metadata.
  5. Updates the VMBD status with categorized results of valid and invalid backups.

15.3.4.2. File restore phase

The file restore phase makes the discovered backup files accessible. When you create a VirtualMachineFileRestore (VMFR) CR, the restore controller performs the following steps:

  1. Validates that the referenced VMBD CR exists and is in the Completed phase.
  2. Verifies that the selected backups exist in the valid discovery results.
  3. Extracts persistent volume claim (PVC) metadata from the selected backup manifests.
  4. Creates a temporary namespace to isolate file-serving resources.
  5. Triggers Velero restore objects for the PVCs for each backup. The restore objects use includedResources to restore only PersistentVolumeClaims (PVCs) and VolumeSnapshots, and use orLabelSelectors to target specific PVCs by their velero.kubevirt.io/pvc-uid label. A namespaceMapping redirects the restored PVCs to the temporary namespace.
  6. Creates a file-serving pod with an initialization container and access sidecars.

15.3.4.3. File-serving pod architecture

The file-serving pod uses libguestfs and FUSE to mount VM disk images without requiring privileged access. The initialization container performs the following steps:

  1. Scans PVCs to locate VM disk image files.
  2. Detects disk image formats, including qcow2 and raw.
  3. Mounts disk images as read-only under the /backups/ directory.
  4. Organizes files by backup name and PVC name.

After initialization, the sidecars provide file access through web or SSH methods.

The mounted directory structure provides an intuitive organization as shown in the following example:

/backups/
  <backup_name_1>/
    <vm_disk_root>/
      etc/
      var/
    <vm_disk_data>/
      application-data/
  <backup_name_2>/
    <vm_disk_root>/
      etc/
      var/

15.3.5. OADP VMFR prerequisites

Configure your cluster environment to enable OADP virtual machine file restore (VMFR) operations by meeting the following prerequisites. This helps you perform file-level restore from virtual machine backups.

  • You have installed the OADP Operator.
  • You have configured the DataProtectionApplication (DPA) CR with the vmFileRestore.enable field set to true.
  • The DPA CR includes the kubevirt Velero plugin in the defaultPlugins list.
  • OpenShift Virtualization is installed and running on the cluster.
  • You have a default storage class configured on the cluster.
  • You have existing Velero backups that contain virtual machine data.

15.3.6. OADP VMFR file access methods

Access restored files from virtual machine (VM) backups through a web browser or SSH-based tools. OADP virtual machine file restore (VMFR) provides access methods that you can configure individually or together.

15.3.6.1. Web browser access

The web access method uses a FileBrowser container to serve an HTTPS interface for browsing and downloading files. Web browser access provides the following capabilities:

  • Directory navigation across backup versions
  • File preview for common text and image formats
  • Individual file or directory archive downloads
  • Credential management through Kubernetes secrets
  • ClusterIP service exposure for internal cluster access
  • Optional external route exposure for access outside the cluster by setting exposeExternally: true in the VMFR CR

15.3.6.2. SSH-based access

The SSH access method provides command-line file transfer capabilities through key-based authentication. When you configure fileAccess.ssh: {} in the VMFR CR, the controller autogenerates an SSH key pair and stores it in a Kubernetes secret. SSH-based access supports the following tools:

  • scp for individual file transfer
  • sftp for interactive file browsing sessions
  • rsync over SSH for synchronizing files and directories

SSH access uses the following defaults:

  • Default username: oadp
  • Default port: 2222
  • Remote path format: /restores/<date>/<backup_name>/<vm_name>/<path_to_file>
  • SSH access uses key-based authentication only. Password-based logins are not supported.

15.3.7. OADP VMFR limitations

Review the limitations of OADP virtual machine file restore (VMFR) to understand which operations are restricted. This helps you plan appropriate file-level restore strategies within the supported functionality.

The following limitations apply to OADP VMFR:

  • VMFR supports only VM backups created by OADP with the kubevirt Velero plugin. Backups created without this plugin are not supported.
  • Restored files are mounted as read-only. You cannot modify files directly in the backup.
  • VMFR supports the following disk image formats: qcow2 and raw. Other disk image formats are not supported.
  • VMFR supports the following file systems: ext4, xfs, ntfs, and fat. Other file systems are not supported.
  • File-serving pods require sufficient disk capacity to mount and serve backup data. Each restore operation mounts the PVC at the size it was at the time of backup. Ensure that your cluster has adequate disk resources available.
  • The VMFR CR must reference a VirtualMachineBackupsDiscovery CR that is in the Completed phase. You cannot create a VMFR CR without a completed discovery.
  • VMFR does not replace full VM restore capabilities. File-level restore and full VM restore are separate workflows that coexist.
  • VMFR does not support hot-mount capabilities into running VMs.

15.3.8. OADP VMFR security considerations

Review the security considerations for OADP virtual machine file restore (VMFR) to understand how access control, data protection, and multi-tenancy are managed. This helps you to plan secure file-level restore operations.

Access control

VMFR enforces the following access control policies:

  • File-serving resources are created in a temporary namespace that is separate from the OADP namespace. This namespace is owned by the VirtualMachineFileRestore CR and is cleaned up automatically when you delete the CR.
  • The discovery controller requires read-only access to Velero Backup resources.
  • The file restore controller requires create and delete access for namespaces, pods, services, and PVCs.
  • Cluster administrators require create, read, update, and delete access to VirtualMachineBackupsDiscovery and VirtualMachineFileRestore resources in the OADP namespace.
  • Controllers authenticate to object storage by using the Velero backup storage location credentials. Controllers have read-only access to backup data and never modify backups.
Data protection

VMFR protects data in transit by using the following mechanisms:

  • Web browser access uses HTTPS endpoints with TLS certificates.
  • SSH access uses auto generated key pairs.
  • Credentials are stored in Kubernetes secrets.
Multi-tenancy

VMFR supports multi-tenancy through namespace isolation:

  • Multiple VirtualMachineFileRestore resources can exist simultaneously. Each resource creates an isolated temporary namespace.
  • Resources from different restores cannot interfere with each other.
  • You can restore the same VM from different backups in parallel.
  • Temporary namespace names are derived from the VirtualMachineFileRestore resource name, and PVC names in the temporary namespace use the backup name as a suffix to prevent naming conflicts.

15.3.9. OADP VMFR discovery and restore phases

Review the status phases of VirtualMachineBackupsDiscovery (VMBD) and VirtualMachineFileRestore (VMFR) custom resources (CRs) to track the progress of file-level restore operations. This helps you monitor and troubleshoot VMFR requests.

Expand
Table 15.2. VirtualMachineBackupsDiscovery phases
ValueDescription

New

The VMBD CR creation request is accepted but discovery has not yet started.

InProgress

The controller is actively validating candidate backups and verifying VM presence.

Completed

All candidate backups are processed and results are available in the VMBD status.

PartiallyFailed

Some candidate backups were validated but others failed verification.

Failed

The discovery process failed. Check the VMBD status conditions for error details.

Expand
Table 15.3. VirtualMachineFileRestore phases
ValueDescription

New

The VMFR CR creation request is accepted but the restore process has not yet started.

InProgress

The controller is restoring PVCs and creating file-serving resources.

Completed

All PVCs are restored and file-serving endpoints are available.

PartiallyFailed

Some PVCs were restored but others failed. Partial file access might be available.

Failed

The restore process failed. Check the VMFR status conditions for error details.

Deleting

The VMFR CR is marked for deletion. The controller is cleaning up file-serving resources, restored PVCs, and the temporary namespace.

Expand
Table 15.4. PVC restore state values
ValueDescription

available

The PVC is restored and the file system is mounted and accessible.

processing

The PVC restore is in progress.

failed

The PVC restore failed.

backup-deleted

The source backup for this PVC has been deleted.

backup-missing

The source backup for this PVC cannot be found.

unsupported-plugin

The backup was created with an unsupported plugin format.

extraction-failed

The file system extraction from the disk image failed.

15.4. Use virtual machine file restore

Discover VM backups, restore files, and access restored files through a web browser or SSH-based tools.

15.4.1. Enable OADP VMFR

Enable OADP virtual machine file restore (VMFR) by configuring the DataProtectionApplication (DPA) custom resource (CR) with the vmFileRestore section. You can allow file-level restore operations for VM backups.

Prerequisites

  • You are logged in to the cluster with the cluster-admin role.
  • The OADP Operator is installed.
  • The DataProtectionApplication (DPA) CR is configured.
  • OpenShift Virtualization is installed and running on the cluster.
  • You have a default storage class configured on the cluster.

Procedure

  1. Edit the DataProtectionApplication CR to enable the VMFR feature:

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    metadata:
      name: oadp-backup
      namespace: openshift-adp
    spec:
      configuration:
        nodeAgent:
          enable: true
          uploaderType: kopia
        velero:
          defaultPlugins:
            - kubevirt
            - csi
            - openshift
            - aws
          disableFsBackup: false
      vmFileRestore:
        enable: true
      backupLocations:
        - velero:
            config:
              profile: "default"
              region: <region>
            provider: aws
            default: true
            credential:
              key: cloud
              name: <cloud_credentials>
            objectStorage:
              bucket: <bucket_name>
              prefix: velero

    where:

    kubevirt
    Specifies the kubevirt Velero plugin in the defaultPlugins list. This plugin is required for VM backup and file-level restore operations.
    vmFileRestore
    Specifies the section in the DPA spec to enable the VMFR feature.
    enable
    Specifies whether to enable the VMFR feature. Set to true to enable the feature.
  2. Apply the DPA configuration by running the following command:

    $ oc apply -f <dpa_cr_filename>

    Replace <dpa_cr_filename> with the file name containing the DPA CR configuration.

Verification

  1. To verify that the DPA is reconciled with the VMFR feature enabled, run the following command:

    $ oc get dpa -n openshift-adp -o yaml

    In the output, verify that the status.conditions section includes a condition with type: VMFileRestoreReady and status: "True".

  2. To verify that the oadp-vm-file-restore-controller-manager pod is running, run the following command:

    $ oc get pod -n openshift-adp

    The output should include a running oadp-vm-file-restore-controller-manager pod.

15.4.2. Create a VirtualMachineBackupsDiscovery CR

Create a VirtualMachineBackupsDiscovery (VMBD) custom resource (CR) to identify which Velero backups contain a specified virtual machine (VM). You can locate available backups before performing a file-level restore.

After you create a VMBD CR, the CR undergoes the following phases:

  • The initial phase for the CR is New.
  • The controller compiles candidate backups and verifies VM presence in each backup.
  • Upon successful discovery, the status.phase field of the VMBD CR is updated to Completed.
Important

Create all VMFR custom resources in the protected namespace, which is openshift-adp by default.

Prerequisites

  • You are logged in to the cluster with the cluster-admin role.
  • You have installed the OADP Operator.
  • You have configured the DataProtectionApplication (DPA) CR with the VMFR feature enabled.
  • You have existing Velero backups that contain virtual machine data.

Procedure

  1. Create a VirtualMachineBackupsDiscovery CR YAML manifest file with the following configuration:

    apiVersion: oadp.openshift.io/v1alpha1
    kind: VirtualMachineBackupsDiscovery
    metadata:
      name: find-my-vm-backups
      namespace: openshift-adp
    spec:
      virtualMachineName: "production-web-server"
      virtualMachineNamespace: "production"

    where:

    name
    Specifies a name for the VMBD CR. For example, find-my-vm-backups.
    namespace
    Specifies the namespace where the VMBD CR is created. This must be the OADP protected namespace, typically openshift-adp.
    virtualMachineName
    Specifies the name of the virtual machine to search for in backups. For example, production-web-server.
    virtualMachineNamespace
    Specifies the namespace of the target virtual machine. For example, production.
  2. Optional: To filter backups by a time range, add startTime and endTime fields to the spec section:

    apiVersion: oadp.openshift.io/v1alpha1
    kind: VirtualMachineBackupsDiscovery
    metadata:
      name: find-my-vm-backups
      namespace: openshift-adp
    spec:
      virtualMachineName: "production-web-server"
      virtualMachineNamespace: "production"
      startTime: "2025-08-01"
      endTime: "2025-09-01"

    where:

    startTime
    Specifies the start of the time range to filter backups. Backups created before this date are excluded.
    endTime
    Specifies the end of the time range to filter backups. Backups created after this date are excluded.
  3. Optional: To discover specific backups by name, add the requestedBackups field to the spec section:

    apiVersion: oadp.openshift.io/v1alpha1
    kind: VirtualMachineBackupsDiscovery
    metadata:
      name: find-my-vm-backups
      namespace: openshift-adp
    spec:
      virtualMachineName: "production-web-server"
      virtualMachineNamespace: "production"
      requestedBackups:
        - "initial-backup-from-2024-01-01"
        - "last-working-from-2025-07-28"

    where:

    requestedBackups
    Specifies a list of backup names to include in the discovery. These backups are included regardless of any time range filter.
  4. To apply the VMBD CR configuration, run the following command:

    $ oc apply -f <vmbd_cr_filename>

    Replace <vmbd_cr_filename> with the file name containing the VMBD CR configuration.

Verification

  • To verify that the VMBD CR is successfully created and discovery is complete, run the following command:

    $ oc get vmbd find-my-vm-backups -n openshift-adp -o yaml
    apiVersion: oadp.openshift.io/v1alpha1
    kind: VirtualMachineBackupsDiscovery
    metadata:
      name: find-my-vm-backups
      namespace: openshift-adp
    spec:
      virtualMachineName: "production-web-server"
      virtualMachineNamespace: "production"
    status:
      phase: Completed
      validBackups:
        - name: "backup-2025-09-20"
          namespace: openshift-adp
          createdAt: "2025-09-20T02:00:00Z"
        - name: "backup-2025-09-15"
          namespace: openshift-adp
          createdAt: "2025-09-15T02:00:00Z"
      backupDiscoveryProgress:
        - name: "backup-2025-09-20"
          namespace: openshift-adp
          status: Completed
          message: "VM found in backup"
          createdAt: "2025-09-20T02:00:00Z"
        - name: "backup-2025-09-15"
          namespace: openshift-adp
          status: Completed
          message: "VM found in backup"
          createdAt: "2025-09-15T02:00:00Z"
      conditions:
        - type: Ready
          status: "True"
          message: "Successfully discovered 2 valid backups"
          reason: DiscoverySuccessful
      discoveryStats:
        totalCandidates: 2
        completed: 2
        failed: 0
        inProgress: 0
        pending: 0
        skipped: 0
      observedGeneration: 1

    where:

    phase: Completed
    Specifies that the discovery process is complete.
    validBackups
    Specifies the list of backups that contain the specified virtual machine. Each entry includes the name, namespace, and createdAt timestamp.
    backupDiscoveryProgress
    Specifies the discovery progress for each candidate backup, including the status and message.
    discoveryStats
    Specifies the total number of candidate backups processed and the count of completed, failed, in-progress, pending, and skipped results.
    observedGeneration
    Specifies the last generation value processed by the controller.

15.4.3. Create a VirtualMachineFileRestore CR

Create a VirtualMachineFileRestore (VMFR) custom resource (CR) to make files from discovered virtual machine (VM) backups accessible for browsing and downloading. You can recover individual files without restoring the entire VM.

After you create a VMFR CR, the CR undergoes the following phases:

  • The initial phase for the CR is New.
  • The controller validates the referenced discovery, restores PVCs, and creates file-serving resources.
  • Upon successful setup, the status.phase field of the VMFR CR is updated to Completed.
Note

All VMFR custom resources must be created in the protected namespace, which is openshift-adp by default.

Prerequisites

  • You are logged in to the cluster with the cluster-admin role.
  • You have installed the OADP Operator.
  • You have configured the DataProtectionApplication (DPA) CR with the VMFR feature enabled.
  • You have created a VirtualMachineBackupsDiscovery (VMBD) CR and its status.phase is Completed.

Procedure

  1. [Optional] Create a secret containing the credentials for accessing the file browser:

    If you do not create a secret, the VMFR controller creates a secret for you and references it in the VMFR CR.

    apiVersion: v1
    kind: Secret
    metadata:
      name: vmfr-credentials
      namespace: openshift-adp
    type: Opaque
    data:
      password: <base64_encoded_password>
      username: <base64_encoded_username>

    where:

    password
    Specifies the base64-encoded password for the file browser. The password must be at least 12 characters long before encoding.
    username
    Specifies the base64-encoded username for the file browser.
  2. Create a VirtualMachineFileRestore CR YAML manifest file with the following configuration.

    To configure file browser access:

    apiVersion: oadp.openshift.io/v1alpha1
    kind: VirtualMachineFileRestore
    metadata:
      name: restore-config-files
      namespace: openshift-adp
    spec:
      backupsDiscoveryRef: find-my-vm-backups
      selectedBackups:
        - backup-2025-09-20
        - backup-2025-09-15
      fileAccess:
        fileBrowser:
          credentialsSecretRef:
            name: vmfr-credentials
          exposeExternally: true

    To configure SSH access:

    apiVersion: oadp.openshift.io/v1alpha1
    kind: VirtualMachineFileRestore
    metadata:
      name: restore-config-files
      namespace: openshift-adp
    spec:
      backupsDiscoveryRef: find-my-vm-backups
      selectedBackups:
        - backup-2025-09-20
        - backup-2025-09-15
      fileAccess:
        ssh: {}

    where:

    name
    Specifies a name for the VMFR CR. For example, restore-config-files.
    namespace
    Specifies the namespace where the VMFR CR is created. This must be the OADP protected namespace, typically openshift-adp.
    backupsDiscoveryRef
    Specifies the name of the VMBD CR that contains the discovery results. This VMBD CR must be in the Completed phase.
    selectedBackups
    Specifies a list of backup names from the VMBD valid results to restore. A Velero restore operation is created for each backup in this list.
    fileAccess
    Specifies the configuration for accessing the restored files.
    fileBrowser
    Specifies the file browser configuration for web-based access.
    credentialsSecretRef
    Specifies the name of the Kubernetes secret that contains the credentials for file browser access.
    exposeExternally
    Specifies whether to create an external route for accessing the file browser. Set to true to create a publicly accessible route.
    ssh

    Specifies the SSH access configuration. Set to {} to enable SSH access with autogenerated credentials. The controller generates an SSH key pair and stores it in a Kubernetes secret.

    Optionally, you can also specify a username, and publickey for enabling the SSH access as shown in the following example:

    ...
       ssh:
         username: fedora
         publicKey: ""
  3. To apply the VMFR CR configuration, run the following command:

    $ oc apply -f <vmfr_cr_filename>

    Replace <vmfr_cr_filename> with the file name containing the VMFR CR configuration.

Verification

  • To verify that the VMFR CR is successfully created and file access is available, run the following command:

    $ oc get vmfr restore-config-files -n openshift-adp -o yaml
    apiVersion: oadp.openshift.io/v1alpha1
    kind: VirtualMachineFileRestore
    metadata:
      name: restore-config-files
      namespace: openshift-adp
    spec:
      backupsDiscoveryRef: find-my-vm-backups
      selectedBackups:
        - backup-2025-09-20
        - backup-2025-09-15
      fileAccess:
        fileBrowser:
          credentialsSecretRef:
            name: vmfr-credentials
          exposeExternally: true
    status:
      phase: Completed
      fileServingInfo:
        fileBrowser:
          clusterAccess: https://restore-config-files-fileserver-svc.production-production-web-server-a1b2c3.svc.cluster.local:8443
          publicAccess: https://restore-config-files.vmfr.apps.example.com
          credentialsSecretRef:
            name: restore-config-files-filebrowser-xk4wm
            namespace: production-production-web-server-a1b2c3
      conditions:
        - type: Ready
          status: "True"
          message: "File restore completed, files accessible via file server and external route"
          reason: Completed
        - type: Available
          status: "True"
          message: "File server is accessible and serving files"
          reason: FileServerAvailable
      pvcRestores:
        - pvcName: production-web-server-dv
          pvcNamespace: production
          pvcUID: 05ac1521-2a16-4a71-b81f-ccad592b89cd
          restores:
            - veleroBackupName: backup-2025-09-20
              veleroBackupNamespace: openshift-adp
              veleroRestoreName: vmfr-restore-config-files-backup-2025-09-20-89lfl
              veleroRestoreNamespace: openshift-adp
              phase: Completed
              state: available
              timestamp: "2025-09-20T02:00:00Z"
            - veleroBackupName: backup-2025-09-15
              veleroBackupNamespace: openshift-adp
              veleroRestoreName: vmfr-restore-config-files-backup-2025-09-15-bfqx4
              veleroRestoreNamespace: openshift-adp
              phase: Completed
              state: available
              timestamp: "2025-09-15T02:00:00Z"
          size: 150Mi
      createdNamespace: production-production-web-server-a1b2c3
      observedGeneration: 1

    where:

    phase: Completed
    Specifies that the file restore process is complete and file-serving endpoints are available.
    fileServingInfo.fileBrowser
    Specifies the endpoint details and credentials for accessing the restored files through the file browser.
    clusterAccess
    Specifies the cluster-internal URL for accessing the file browser.
    publicAccess
    Specifies the externally accessible URL for the file browser. This field is present only when exposeExternally is set to true.
    credentialsSecretRef
    Specifies the Kubernetes secret that contains the generated credentials for accessing the file browser interface, including the name and namespace of the secret.
    pvcRestores
    Specifies the restore status grouped by PVC. Each PVC entry includes the pvcName, pvcNamespace, pvcUID, and a restores list with details for each backup restore.
    state
    Specifies the state of the restored PVC. Possible values include available, processing, failed, backup-deleted, backup-missing, unsupported-plugin, and extraction-failed.
    createdNamespace
    Specifies the temporary namespace created for hosting the file-serving resources. This namespace is cleaned up when you delete the VMFR CR.

Access restored virtual machine (VM) files through a web browser by using the file browser interface provided by the VirtualMachineFileRestore (VMFR) custom resource (CR). You can browse, preview, and download files from VM backups.

Prerequisites

  • You are logged in to the cluster with the cluster-admin role.
  • A VirtualMachineFileRestore (VMFR) CR with the fileAccess.fileBrowser section configured exists.
  • The VMFR CR status.phase is Completed.

Procedure

  1. To retrieve the file browser access URLs, run the following command:

    $ oc get vmfr <vmfr_cr_name> -n openshift-adp -o jsonpath='{.status.fileServingInfo.fileBrowser}'

    Replace <vmfr_cr_name> with the name of the VMFR CR. The output includes the clusterAccess URL for cluster-internal access and the publicAccess URL if exposeExternally is set to true.

  2. If the VMFR CR has exposeExternally set to true, open a web browser and navigate to the publicAccess URL from the status output.

    If exposeExternally is not enabled, set up port forwarding to the file-serving service by running the following command:

    $ oc port-forward svc/vmfr-<vmfr_cr_name>-fileserver-svc -n <restore_namespace> 8443:8443

    Replace <vmfr_cr_name> with the name of the VMFR CR and <restore_namespace> with the namespace from the status.createdNamespace field. Then navigate to https://localhost:8443 in your web browser.

  3. Log in by using the credentials from the secret you created for file browser access.

    Figure 15.1. OADP VM File Restore Browser login page

    OADP VM File Restore Browser login page
  4. Browse the files organized by date, backup name, and PVC name.

    Figure 15.2. File browser listing showing backup contents

    File browser listing showing backup contents
  5. View the file content by selecting the file. To download a file, select the file and click Download. To download a directory as an archive, select the directory and click Download.

    Figure 15.3. File preview in the file browser

    File preview in the file browser

15.4.5. Access restored files through SSH

Access restored virtual machine (VM) files through SSH by using rsync, scp, or sftp with the VirtualMachineFileRestore (VMFR) custom resource (CR). You can transfer files from VM backups efficiently.

When you configure SSH access, the VMFR controller autogenerates an SSH key pair and stores it in a Kubernetes secret. The default SSH username is oadp. The SSH file server listens on port 2222.

The remote path for restored files follows the format /restores/<date>/<backup_name>/<vm_name>/<path_to_file>.

Prerequisites

  • You are logged in to the cluster with the cluster-admin role.
  • You have created a VirtualMachineFileRestore (VMFR) CR with the fileAccess.ssh section configured.
  • The VMFR CR status.phase is Completed.

Procedure

  1. To retrieve the SSH access information, run the following command:

    $ oc get vmfr <vmfr_cr_name> -o jsonpath='{.status.fileServingInfo.ssh}' | jq

    Replace <vmfr_cr_name> with the name of the VMFR CR. The output includes the clusterAccess URL and credentialsSecretRef containing the name and namespace of the generated SSH key secret.

  2. Retrieve the private key from the generated secret and save it to a file:

    $ oc get secret <secret_name> -n <secret_namespace> -o jsonpath='{.data.privateKey}' | base64 -d > id-rsa

    Replace <secret_name> and <secret_namespace> with the values from the status.fileServingInfo.ssh.credentialsSecretRef field.

  3. Set the correct permissions on the private key file:

    $ chmod 600 id-rsa
  4. Get the name of the file server service created in the VMFR namespace:

    $ oc get svc -n <created_namespace> | grep fileserver

    Replace <created_namespace> with the value from the status.createdNamespace field of the VMFR CR.

  5. To copy a file from the backup by using scp, run the following command:

    $ scp -P 2222 -i id-rsa \
      -o StrictHostKeyChecking=no \
      -o UserKnownHostsFile=/dev/null \
      oadp@<fileserver_svc>.<created_namespace>.svc.cluster.local:<remote_path> \
      <local_destination>

    where:

    <fileserver_svc>
    Specifies the name of the file server service.
    <created_namespace>
    Specifies the namespace from the status.createdNamespace field.
    <remote_path>
    Specifies the path to the file in the format /restores/<date>/<backup_name>/<vm_name>/<path_to_file>.
    <local_destination>
    Specifies the local file path to save the restored file.
  6. To start an interactive SFTP session, run the following command:

    $ sftp -P 2222 -i id-rsa \
      -o StrictHostKeyChecking=no \
      oadp@<fileserver_svc>.<created_namespace>.svc.cluster.local

15.4.6. Delete a VirtualMachineFileRestore CR

Delete a VirtualMachineFileRestore (VMFR) custom resource (CR) to clean up file-serving resources after you have recovered the files you need. This helps you free cluster resources used by the file-serving pod and temporary namespace.

When you delete a VMFR CR, the controller performs the following cleanup operations:

  • Stops the file-serving pod and associated services.
  • Deletes the restored PVCs.
  • Deletes the temporary namespace if one was automatically created.

Prerequisites

  • You are logged in to the cluster with the cluster-admin role.
  • You have a VirtualMachineFileRestore CR that you want to delete.

Procedure

  • To delete a VirtualMachineFileRestore CR, run the following command:

    $ oc delete vmfr <vmfr_cr_name> -n openshift-adp

    Replace <vmfr_cr_name> with the name of the VMFR CR.

Verification

  • To verify that the VMFR CR is deleted and resources are cleaned up, run the following command:

    $ oc get vmfr -n openshift-adp

    The deleted VMFR CR should not appear in the output.

15.4.7. Test the VM file restore workflow

Complete an end-to-end workflow that creates a VM, backs it up, and restores individual files through SSH to test the VM file restore feature or verify your configuration.

Prerequisites

  • You are logged in to the cluster with the cluster-admin role.
  • You have installed the OADP Operator.
  • OpenShift Virtualization is installed and running on the cluster.
  • You have installed the virtctl CLI tool to access the VM.

Procedure

  1. Create a secret for the cloud storage credentials by running the following command:

    $ oc create secret generic <secret_name> -n openshift-adp --from-file cloud=<credentials_file_path>

    where:

    <secret_name>
    Specifies the name of the cloud credentials secret.
    <credentials_file_path>
    Specifies the path to the file that contains the cloud storage credentials.
  2. Create a DataProtectionApplication (DPA) CR with the VMFR feature enabled and the kubevirt plugin:

    apiVersion: oadp.openshift.io/v1alpha1
    kind: DataProtectionApplication
    metadata:
      name: vmfr-dpa
      namespace: openshift-adp
    spec:
      backupLocations:
        - velero:
            credential:
              key: cloud
              name: <secret_name>
            default: true
            objectStorage:
              bucket: <bucket_name>
              prefix: velero
            provider: <provider>
      configuration:
        velero:
          defaultPlugins:
            - csi
            - openshift
            - kubevirt
            - <provider>
          disableFsBackup: false
        nodeAgent:
          enable: true
          uploaderType: kopia
      vmFileRestore:
        enable: true

    where:

    <secret_name>
    Specifies the name of the cloud credentials secret you created.
    <bucket_name>
    Specifies the name of the object storage bucket.
    <provider>
    Specifies the cloud provider plugin, such as aws, gcp, or azure.
    vmFileRestore
    Enables the VMFR feature by setting the enable field to true.
  3. Apply the DPA configuration by running the following command:

    $ oc apply -f <dpa_cr_filename>
  4. Verify that the DPA is reconciled and the VMFR feature is enabled by running the following command:

    $ oc get dpa -n openshift-adp -o yaml

    In the output, verify that the status.conditions section includes a condition with type: VMFileRestoreReady and status: "True".

  5. Verify that the velero, nodeAgent, and oadp-vm-file-restore-controller-manager pods are running by running the following command:

    $ oc get pod -n openshift-adp
  6. Generate an SSH key pair for accessing the VM:

    $ ssh-keygen -t ed25519 -f ~/.ssh/<vm_key_name> -C "<vm_key_name>"

    Replace <vm_key_name> with a name for the SSH folder and the key file.

  7. Create a namespace for the VM by running the following command:

    $ oc create ns <vm_namespace>
  8. Create a Kubernetes secret with the SSH public key in the VM namespace:

    $ oc create secret generic <ssh_secret_name> \
      --from-file=key=<path_to_public_key> \
      -n <vm_namespace>

    where:

    <ssh_secret_name>
    Specifies the name of the secret that contains the SSH public key.
    <path_to_public_key>
    Specifies the path to the SSH public key file you created in an earlier step. For example, $HOME/.ssh/vm-key.pub.
    <vm_namespace>
    Specifies the namespace for the VM.
  9. Create a VirtualMachine CR by using the built-in VM template fedora-server-small:

    $ oc process -n openshift fedora-server-small -p NAME=<vm_name> | oc apply -n <vm_namespace> -f -

    where:

    <vm_name>
    Specifies the name of the VM.
    <vm_namespace>
    Specifies the namespace for the VM.
  10. Wait for the VM to be ready:

    $ oc wait --for=condition=Ready vmi/<vm_name> -n <vm_namespace>
  11. Patch the VM configuration accessCredentials object with the SSH public key:

    $ oc patch vm <vm_name> -n <vm_namespace> --type=merge -p '{"spec":{"template":{"spec":{"accessCredentials":[{"sshPublicKey":{"propagationMethod":{"noCloud":{}},"source":{"secret":{"secretName":"<ssh_secret_name>"}}}}]}}}}'

    where:

    <vm_name>
    Specifies the name of the VM.
    <vm_namespace>
    Specifies the namespace for the VM.
    <ssh_secret_name>
    Specifies the name of the secret that contains the SSH public key.
  12. SSH to the VM and create a test file:

    $ virtctl ssh <vm_user>@vmi/<vm_name> \
      -n <vm_namespace> \
      --identity-file=$HOME/.ssh/<vm_key_name> \
      --local-ssh-opts="-o StrictHostKeyChecking=no" \
      -c "echo 'Test file for VMFR validation - $(date)' > /home/fedora/test-vmfr-file.txt"

    where:

    <vm_user>
    Specifies the name of the VM user. For the in-built VM template, the user name is fedora.
  13. Create a Backup CR to back up the VM namespace:

    apiVersion: velero.io/v1
    kind: Backup
    metadata:
      name: <backup_name>
      namespace: openshift-adp
    spec:
      includedNamespaces:
        - <vm_namespace>
      snapshotMoveData: true
  14. Apply the Backup CR by running the following command:

    $ oc apply -f <backup_cr_filename>
  15. Verify that the backup is complete by running the following command:

    $ oc get backup.velero <backup_name> -n openshift-adp -o jsonpath='{.status.phase}'

    The output should display Completed.

  16. Create a VirtualMachineBackupsDiscovery CR to identify which backups contain the target VM:

    apiVersion: oadp.openshift.io/v1alpha1
    kind: VirtualMachineBackupsDiscovery
    metadata:
      name: <vmbd_name>
      namespace: openshift-adp
    spec:
      virtualMachineName: <vm_name>
      virtualMachineNamespace: <vm_namespace>
  17. Apply the VMBD CR by running the following command:

    $ oc apply -f <vmbd_cr_filename>
  18. Verify that the discovery is complete by running the following command:

    $ oc get vmbd <vmbd_name> -n openshift-adp

    The PHASE column should display Completed.

  19. Create a VirtualMachineFileRestore CR with SSH access to restore files from the discovered backups:

    apiVersion: oadp.openshift.io/v1alpha1
    kind: VirtualMachineFileRestore
    metadata:
      name: <vmfr_name>
      namespace: openshift-adp
    spec:
      backupsDiscoveryRef: <vmbd_name>
      fileAccess:
        ssh: {}

    The ssh: {} configuration instructs the controller to autogenerate an SSH key pair and store it in a Kubernetes secret. The default SSH username is oadp.

  20. Apply the VMFR CR by running the following command:

    $ oc apply -f <vmfr_cr_filename>
  21. Wait for the VMFR phase to complete:

    $ oc get vmfr <vmfr_name> -n openshift-adp

    The PHASE column should display Completed.

  22. Retrieve the SSH access information by running the following command:

    $ oc get vmfr <vmfr_name> -o jsonpath='{.status.fileServingInfo.ssh}' | jq

    The output includes the clusterAccess URL and the credentialsSecretRef containing the name and namespace of the auto generated SSH key secret.

  23. Retrieve the private key from the auto generated secret and save it to a file:

    $ oc get secret <secret_name> -n <secret_namespace> -o jsonpath='{.data.privateKey}' | base64 -d > id-rsa

    Replace <secret_name> and <secret_namespace> with the values from the status.fileServingInfo.ssh.credentialsSecretRef field.

  24. Copy the private key to the VM so that you can use it for scp from within the VM:

    $ virtctl scp id-rsa \
      <vm_user>@vmi/<vm_name>:/home/<vm_user>/id-rsa \
      -n <vm_namespace> \
      --identity-file=$HOME/.ssh/<vm_key_name>
  25. Update the file permissions on the private key inside the VM:

    $ virtctl ssh <vm_user>@vmi/<vm_name> \
      -n <vm_namespace> \
      --identity-file=$HOME/.ssh/<vm_key_name> \
      --local-ssh-opts="-o StrictHostKeyChecking=no" \
      -c "chmod 600 /home/<vm_user>/id-rsa"
  26. Get the name of the file server service created in the VMFR namespace:

    $ SVC=$(oc get svc -n <created_namespace> | grep fileserver | awk '{print $1}')

    Replace <created_namespace> with the value from the status.createdNamespace field of the VMFR CR.

  27. Restore the file by using scp from within the VM. The remote path format is /restores/<date>/<backup_name>/<vm_name>/<path_to_file>:

    $ virtctl ssh <vm_user>@vmi/<vm_name> \
      -n <vm_namespace> \
      --identity-file=$HOME/.ssh/<vm_key_name> \
      --local-ssh-opts="-o StrictHostKeyChecking=no" \
      -c "scp -P 2222 \
      -i /home/<vm_user>/id-rsa \
      -o StrictHostKeyChecking=no \
      -o UserKnownHostsFile=/dev/null \
      oadp@<fileserver_svc>.<created_namespace>.svc.cluster.local:<remote_path> \
      /tmp/restored.txt"

    where:

    <fileserver_svc>
    Specifies the name of the file server service you retrieved in an earlier step.
    <created_namespace>
    Specifies the namespace from the status.createdNamespace field.
    <remote_path>
    Specifies the path to the file in the format /restores/<date>/<backup_name>/<vm_name>/<path_to_file>. For example, "/restores/2026-02-10/test-backup/fedora-vm-test/home/fedora/test-vmfr-file.txt"
  28. Verify that the restored file is intact by comparing MD5 checksums:

    $ virtctl ssh <vm_user>@vmi/<vm_name> \
      -n <vm_namespace> \
      --identity-file=$HOME/.ssh/<vm_key_name> \
      --local-ssh-opts="-o StrictHostKeyChecking=no" \
      -c "echo '===MD5 Checksums===' && md5sum /home/fedora/test-vmfr-file.txt /tmp/restored.txt"

    The checksums of the original and restored files should match. You should see an output as shown in the following example:

    ===MD5 Checksums===
    92245490c552e83baf5f2a2e898d9fff  /home/fedora/test-vmfr-file.txt
    92245490c552e83baf5f2a2e898d9fff  /tmp/restored.txt
  29. After you have recovered the files, delete the VMFR CR to clean up resources:

    $ oc delete vmfr <vmfr_name> -n openshift-adp

15.5. Disaster recovery

OpenShift Virtualization supports using disaster recovery (DR) solutions to ensure that your environment can recover after a site outage. To use these methods, you must plan your OpenShift Virtualization deployment in advance.

15.5.1. About disaster recovery methods

The two primary DR methods for OpenShift Virtualization are Metropolitan Disaster Recovery (Metro-DR) and Regional-DR.

For an overview of disaster recovery (DR) concepts, architecture, and planning considerations, see the "Red Hat OpenShift Virtualization disaster recovery guide" in the Red Hat Knowledgebase.

15.5.1.1. Metro-DR

Metro-DR uses synchronous replication. It writes to storage at both the primary and secondary sites so that the data is always synchronized between sites. Because the storage provider is responsible for ensuring that the synchronization succeeds, the environment must meet the throughput and latency requirements of the storage provider.

15.5.1.2. Regional-DR

Regional-DR uses asynchronous replication. The data in the primary site is synchronized with the secondary site at regular intervals. For this type of replication, you can have a higher latency connection between the primary and secondary sites.

Define applications for disaster recovery by using VMs that Red Hat Advanced Cluster Management (RHACM) manages or discovers.

When creating an RHACM-managed application that includes a VM, you must use a GitOps workflow and create an RHACM application or ApplicationSet resource.

You can take several actions to improve your experience and chance of success when defining an RHACM-managed VM.

Use a PVC and populator to define storage for the VM
Because data volumes create persistent volume claims (PVCs) implicitly, data volumes and VMs with data volume templates do not fit as neatly into the GitOps model.
Use the import method when choosing a population source for your VM disk
Select a RHEL image from the software catalog to use the import method. Red Hat recommends using a specific version of the image rather than a floating tag for consistent results. The KubeVirt community maintains container disks for other operating systems in a Quay repository.
Use pullMethod: node
Use the pod pullMethod: node when creating a data volume from a registry source to take advantage of the OpenShift Container Platform pull secret, which is required to pull container images from the Red Hat registry.

You can configure any VM in the cluster that is not an RHACM-managed application as an RHACM-discovered application. This includes VMs imported by using the Migration Toolkit for Virtualization (MTV), VMs created by using the OpenShift Container Platform web console, or VMs created by any other means, such as the CLI.

You can take several actions to improve your experience and chance of success when defining an RHACM-discovered VM.

Protecting the VM when using MTV, the OpenShift Container Platform web console, or a custom VM

Because automatic labeling is not currently available, the application owner must manually label the components of the VM application when using MTV, the OpenShift Container Platform web console, or a custom VM.

After creating the VM, apply a common label to the following resources associated with the VM: VirtualMachine, DataVolume, PersistentVolumeClaim, Service, Route, Secret and ConfigMap. If the VM uses an instance type or preference, you must also label the ControllerRevision copy of these objects referenced by the spec or status of the VM. Do not label virtual machine instances (VMIs) or pods; OpenShift Virtualization creates and manages these automatically.

Important

You must apply the common label to everything in the namespace that you want to protect, including objects that you added to the VM that are not listed here.

Including more than the VirtualMachine object in the VM
Working VMs typically also contain data volumes, persistent volume claims (PVCs), services, routes, secrets, ConfigMap objects, and VirtualMachineSnapshot objects.
Including the VM as part of a larger logical application
This includes other pod-based workloads and VMs.

VMs typically act similarly to pod-based workloads during both relocate and failover disaster recovery flows.

15.5.3.1. Relocate

Use relocate to move an application from the primary environment to the secondary environment when the primary environment is still accessible. During relocate, the VM is gracefully terminated, any unreplicated data is synchronized to the secondary environment, and the VM starts in the secondary environment.

Because the VM terminates gracefully, there is no data loss. Therefore, the VM operating system will not perform crash recovery.

15.5.3.2. Failover

Use failover when there is a critical failure in the primary environment that makes it impractical or impossible to use relocation to move the workload to a secondary environment. When failover is executed, the storage is fenced from the primary environment, the I/O to the VM disks is abruptly halted, and the VM restarts in the secondary environment using the replicated data.

You should expect data loss due to failover. The extent of loss depends on whether you use Metro-DR, which uses synchronous replication, or Regional-DR, which uses asynchronous replication. Because Regional-DR uses snapshot-based replication intervals, the window of data loss is proportional to the replication interval length. When the VM restarts, the operating system might perform crash recovery.

You can use disaster recovery (DR) solutions that combine Red Hat Advanced Cluster Management (RHACM), Red Hat Ceph Storage, and OpenShift Data Foundation components to failover applications between sites.

OpenShift Virtualization supports the Metro-DR solution for OpenShift Data Foundation, which provides two-way synchronous data replication between managed OpenShift Virtualization clusters installed on primary and secondary sites.

Metro-DR differences

  • This synchronous solution is only available to metropolitan distance data centers with a network round-trip latency of 10 milliseconds or less.
  • Multiple disk VMs are supported.
  • To prevent data corruption, you must ensure that storage is fenced during failover.

    Tip

    Fencing means isolating a node so that workloads do not run on it.

For more information about using the Metro-DR solution for OpenShift Data Foundation with OpenShift Virtualization, see IBM’s OpenShift Data Foundation Metro-DR documentation.

OpenShift Virtualization supports the Regional-DR solution for OpenShift Data Foundation, which provides asynchronous data replication at regular intervals between managed OpenShift Virtualization clusters installed on primary and secondary sites.

Regional-DR differences

  • Regional-DR supports higher network latency between the primary and secondary sites.
  • Regional-DR uses RBD snapshots to replicate data asynchronously. Currently, your applications must be resilient to small variances between VM disks. You can prevent these variances by using single disk VMs.
  • Using the import method when selecting a population source for your VM disk is recommended. However, you can protect VMs that use cloned PVCs if you select a VolumeReplicationClass that enables image flattening. For more information, see the OpenShift Data Foundation documentation.

For more information about using the Regional-DR solution for OpenShift Data Foundation with OpenShift Virtualization, see IBM’s OpenShift Data Foundation Regional-DR documentation.

Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat Documentation

Legal Notice

Theme

© 2026 Red Hat
Back to top