Backup and restore


Red Hat build of MicroShift 4.18

Backup and restore the Red Hat build of MicroShift database

Red Hat OpenShift Documentation Team

Abstract

Learn how to backup and restore the Red Hat build of MicroShift database.

Chapter 1. Backing up and restoring MicroShift data

You can manually back up and restore the MicroShift database on all supported systems. Greenboot health checks must be completed and you must stop the MicroShift service prior to any backups.

Note

Only MicroShift data is backed up with the following procedures. Application data is not included.

  • On rpm-ostree systems, MicroShift automatically creates a backup on every start. These automatic backups are deleted and replaced with the latest backup each time the system restarts.
  • If you are using an rpm-ostree system, the data is automatically restored after Greenboot rolls the system back. This data restoration ensures that the database matches the software running on the host after the rollback is completed.
  • On other system types, you must back up and restore data manually.

1.1. Stopping the MicroShift service

Use the following procedure to stop the MicroShift service.

Prerequisites

  • The MicroShift service is running.

Procedure

  1. Enter the following command to stop the MicroShift service:

    $ sudo systemctl stop microshift
  2. Workloads deployed on MicroShift might continue running even after the MicroShift service has been stopped. Enter the following command to display running workloads:

    $ sudo crictl ps -a
  3. Enter the following commands to stop the deployed workloads:

    $ sudo systemctl stop kubepods.slice

1.2. Backing up MicroShift data manually

You can back up MicroShift data manually at any time. Back up your data before system updates to preserve it for use if an update fails or for other system trouble. Automated backups are created in the /var/lib/microshift-backups directory. You can use this directory for manually backing up and restoring data by specifying it in each command. When you create a backup, you must use the entire file path for the output file.

Prerequisites

  • You have root access to the host.
  • MicroShift is stopped.

Procedure

  1. Manually create a backup by using the parent directory and specifying a name, such as /var/lib/microshift-backups/<my_manual_backup>, by running the following command:

    $ sudo microshift backup /var/lib/microshift-backups/<my_manual_backup>

    Replace <my_manual_backup> with the backup name that you want to use.

    Example output

    ??? I1017 07:38:16.770506    5900 data_manager.go:92] "Copying data to backup directory" storage="/var/lib/microshift-backups" name="test" data="/var/lib/microshift"
    ??? I1017 07:38:16.770713    5900 data_manager.go:227] "Starting copy" cmd="/bin/cp --verbose --recursive --preserve --reflink=auto /var/lib/microshift /var/lib/microshift-backups/test"
    ??? I1017 07:38:16.776162    5900 data_manager.go:241] "Finished copy" cmd="/bin/cp --verbose --recursive --preserve --reflink=auto /var/lib/microshift /var/lib/microshift-backups/test"
    ??? I1017 07:38:16.776256    5900 data_manager.go:125] "Copied data to backup directory" backup="/var/lib/microshift-backups/test" data="/var/lib/microshift"

  2. Optional: Manually create a backup in a specific parent directory with a custom name by running the following command:

    $ sudo microshift backup /mnt/<other_backups_location>/<another_manual_backup>

    Replace <other_backups_location> with the directory you want to use and <my_manual_backup> with the backup name you want to use.

Verification

  • You can verify that the backup exists by viewing the data in the directory you chose. For example, /var/lib/microshift-backups/<my_manual_backup>/ or /mnt/<other_backups_location>/<another_manual_backup>.

1.3. Restoring MicroShift data backups manually

You can restore MicroShift data from a backup manually. Backups can be restored after updates, or after other system events that remove or damage required data. Automated backups are in the /var/lib/microshift-backups directory by default. You can use this directory for manually backing up and restoring data by specifying it in each command. When you restore a backup, you must use the entire file path.

Note

On an rpm-ostree system, MicroShift backs up and restores data automatically.

Prerequisites

  • Root access to the host.
  • Full path of the data backup file.
  • The MicroShift service is stopped.

Procedure

  1. Manually restore MicroShift data by using the full file path of the backup you want to restore by running the following command:

    $ sudo microshift restore /var/lib/microshift-backups/<my_manual_backup>

    Replace <my_manual_backup> with the backup name that you used. Optional: You can also restore automatic ostree backups using the full file path.

    Example output

    ??? I1017 07:39:52.055165    6007 data_manager.go:131] "Copying backup to data directory" storage="/var/lib/microshift-backups" name="test" data="/var/lib/microshift"
    ??? I1017 07:39:52.055243    6007 data_manager.go:154] "Renaming existing data dir" data="/var/lib/microshift" renamedTo="/var/lib/microshift.saved"
    ??? I1017 07:39:52.055326    6007 data_manager.go:227] "Starting copy" cmd="/bin/cp --verbose --recursive --preserve --reflink=auto /var/lib/microshift-backups/test /var/lib/microshift"
    ??? I1017 07:39:52.061363    6007 data_manager.go:241] "Finished copy" cmd="/bin/cp --verbose --recursive --preserve --reflink=auto /var/lib/microshift-backups/test /var/lib/microshift"
    ??? I1017 07:39:52.061404    6007 data_manager.go:175] "Removing temporary data directory" path="/var/lib/microshift.saved"
    ??? I1017 07:39:52.063745    6007 data_manager.go:180] "Copied backup to data directory" name="test" data="/var/lib/microshift"

  2. Optional. Manually restore data from a customized directory by using the full file path of the backup. Run the following command:

    $ sudo microshift restore /<mnt>/<other_backups_location>/<another_manual_backup>

    Replace <other_backups_location> with the directory you used and <my_manual_backup> with the backup name you used when creating the backup you are restoring.

  3. Restart the host. Restarting the host enables all workloads and pods to restart.

Verification

  • Use the oc get pods -A command to verify that the cluster is running, then check the restored data.

Chapter 2. Automated recovery from manual backups

You can automatically restore data from manual backups when MicroShift fails to start by configuring automatic recovery.

2.1. Modifying backup and restore commands to automate data recovery

Use automatic recovery options to store all of your backups in a single directory, then automatically select the latest one to restore. Modifying existing backup and restore commands enables you to set up automatic recovery.

The --auto-recovery option treats the PATH argument as a path to a directory that holds all the backups for automated recovery, and not just as a path to a particular backup file. You can use the --auto-recovery option with both backup and restore commands.

  • For example, if you use the automatic recovery option with restore, such as in microshift restore --auto-recovery PATH, running the modified command automatically selects and restores the most recent backup.
  • If you use the same option in the microshift backup command, such as in microshift backup --auto-recovery PATH, a new backup is created in the PATH.
  • By default, microshift restore --auto-recovery PATH creates a backup of the failed MicroShift data in PATH/failed. You can add the --dont-save-failed option to disable the creation of failed backup data.
Important

You can only use the --dont-save-failed option with the restore command.

2.2. Creating backups using the auto-recovery feature

Use the following procedure to create backups using automatic recovery options.

Note

Creating backups require stopping MicroShift, so you must determine the best time to stop MicroShift.

Prerequisites

  • You stopped MicroShift.

Procedure

  • Create and store backups in the directory you choose by running the following command:

    $ sudo microshift backup --auto-recovery <path_of_directory> 1
    1
    Replace <path_of_directory> with the path of the directory that stores backups. For example, /var/lib/microshift-auto-recovery.
    Note

    The --auto-recovery option modifies the interpretation of the PATH argument from the final backup path to a directory that holds all of the backups for automated recovery.

    Example output

    ??? I1104 09:18:52.100725    8906 system.go:58] "OSTree deployments" deployments=[{"id":"default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1","booted":true,"staged":false,"pinned":false},{"id":"default-a129624b9233fa54fe3574f1aa211bc2d85e1052b52245fe7d83f10c2f6d28e3.0","booted":false,"staged":false,"pinned":false}]
    ??? I1104 09:18:52.100895    8906 data_manager.go:83] "Copying data to backup directory" storage="/var/lib/microshift-auto-recovery" name="20241104091852_default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1" data="/var/lib/microshift"
    ??? I1104 09:18:52.102296    8906 disk_space.go:33] Calculated size of "/var/lib/microshift": 261M - increasing by 10% for safety: 287M
    ??? I1104 09:18:52.102321    8906 disk_space.go:44] Calculated available disk space for "/var/lib/microshift-auto-recovery": 1658M
    ??? I1104 09:18:52.105700    8906 atomic_dir_copy.go:66] "Made an intermediate copy" cmd="/bin/cp --verbose --recursive --preserve --reflink=auto /var/lib/microshift /var/lib/microshift-auto-recovery/20241104091852_default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1.tmp.99142"
    ??? I1104 09:18:52.105732    8906 atomic_dir_copy.go:115] "Renamed to final destination" src="/var/lib/microshift-auto-recovery/20241104091852_default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1.tmp.99142" dest="/var/lib/microshift-auto-recovery/20241104091852_default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1"
    ??? I1104 09:18:52.105749    8906 data_manager.go:120] "Copied data to backup directory" backup="/var/lib/microshift-auto-recovery/20241104091852_default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1" data="/var/lib/microshift"
    /var/lib/microshift-auto-recovery/20241104091852_default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1

Verification

  • Verify that the backup you created exists in your customized storage directory by running the following command:

    $ sudo ls -la <path_of_directory> 1
    1
    Replace <path_of_directory> with the path of the directory that stores backups. For example, /var/lib/microshift-auto-recovery.

2.3. Restoring backups using the auto-recovery feature

You can restore backups after system events that remove or damage required data. Use the following procedure to restore backups using automatic recovery. Automatic recovery selects the most recent backup and restores it. Previously restored backups that used automatic recovery are moved to your PATH/restored directory.

Prerequisites

  • You have stopped MicroShift.

Procedure

  1. Restore the latest backup from your backups directory by running the following command:

    $ sudo microshift restore --auto-recovery <path_of_directory> 1
    1
    Replace <path_of_directory> with the path of the directory that stores backups. For example, /var/lib/microshift-auto-recovery.
    Note
    • The --auto-recovery option copies the MicroShift data to /var/lib/microshift-auto-recovery/failed/ for later investigation, selects the most recent backup, and restores it.
    • The --dont-save-failed option disables the backing up of failed MicroShift data.

    Example output

    ??? I1104 09:19:28.617225    8950 state.go:80] "Read state from the disk" state={"LastBackup":"20241022101528_default-a129624b9233fa54fe3574f1aa211bc2d85e1052b52245fe7d83f10c2f6d28e3.0"}
    ??? I1104 09:19:28.617323    8950 storage.go:78] "Auto-recovery backup storage read and parsed" dirs=["20241022101255_default-a129624b9233fa54fe3574f1aa211bc2d85e1052b52245fe7d83f10c2f6d28e3.0","20241022101520_default-a129624b9233fa54fe3574f1aa211bc2d85e1052b52245fe7d83f10c2f6d28e3.0","20241022101528_default-a129624b9233fa54fe3574f1aa211bc2d85e1052b52245fe7d83f10c2f6d28e3.0","20241104091852_default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1","restored"] backups=[{"CreationTime":"2024-10-22T10:12:55Z","Version":"default-a129624b9233fa54fe3574f1aa211bc2d85e1052b52245fe7d83f10c2f6d28e3.0"},{"CreationTime":"2024-10-22T10:15:20Z","Version":"default-a129624b9233fa54fe3574f1aa211bc2d85e1052b52245fe7d83f10c2f6d28e3.0"},{"CreationTime":"2024-10-22T10:15:28Z","Version":"default-a129624b9233fa54fe3574f1aa211bc2d85e1052b52245fe7d83f10c2f6d28e3.0"},{"CreationTime":"2024-11-04T09:18:52Z","Version":"default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1"}]
    ??? I1104 09:19:28.617350    8950 storage.go:40] "Filtered list of backups - removed previously restored backup" removed="20241022101528_default-a129624b9233fa54fe3574f1aa211bc2d85e1052b52245fe7d83f10c2f6d28e3.0" newList=[{"CreationTime":"2024-10-22T10:12:55Z","Version":"default-a129624b9233fa54fe3574f1aa211bc2d85e1052b52245fe7d83f10c2f6d28e3.0"},{"CreationTime":"2024-10-22T10:15:20Z","Version":"default-a129624b9233fa54fe3574f1aa211bc2d85e1052b52245fe7d83f10c2f6d28e3.0"},{"CreationTime":"2024-11-04T09:18:52Z","Version":"default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1"}]
    ??? I1104 09:19:28.633237    8950 system.go:58] "OSTree deployments" deployments=[{"id":"default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1","booted":true,"staged":false,"pinned":false},{"id":"default-a129624b9233fa54fe3574f1aa211bc2d85e1052b52245fe7d83f10c2f6d28e3.0","booted":false,"staged":false,"pinned":false}]
    ??? I1104 09:19:28.633258    8950 storage.go:49] "Filtered list of backups by version" version="default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1" newList=[{"CreationTime":"2024-11-04T09:18:52Z","Version":"default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1"}]
    ??? I1104 09:19:28.633268    8950 restore.go:170] "Potential backups" bz=[{"CreationTime":"2024-11-04T09:18:52Z","Version":"default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1"}]
    ??? I1104 09:19:28.633277    8950 restore.go:173] "Candidate backup for restore" b={"CreationTime":"2024-11-04T09:18:52Z","Version":"default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1"}
    ??? I1104 09:19:28.634007    8950 disk_space.go:33] Calculated size of "/var/lib/microshift-auto-recovery/20241104091852_default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1": 261M - increasing by 10% for safety: 287M
    ??? I1104 09:19:28.634096    8950 disk_space.go:44] Calculated available disk space for "/var/lib": 1658M
    ??? I1104 09:19:28.634507    8950 disk_space.go:33] Calculated size of "/var/lib/microshift": 261M - increasing by 10% for safety: 287M
    ??? I1104 09:19:28.634522    8950 disk_space.go:44] Calculated available disk space for "/var/lib/microshift-auto-recovery": 1658M
    ??? I1104 09:19:28.649719    8950 system.go:58] "OSTree deployments" deployments=[{"id":"default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1","booted":true,"staged":false,"pinned":false},{"id":"default-a129624b9233fa54fe3574f1aa211bc2d85e1052b52245fe7d83f10c2f6d28e3.0","booted":false,"staged":false,"pinned":false}]
    ??? I1104 09:19:28.653880    8950 atomic_dir_copy.go:66] "Made an intermediate copy" cmd="/bin/cp --verbose --recursive --preserve --reflink=auto /var/lib/microshift /var/lib/microshift-auto-recovery/failed/20241104091928_default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1.tmp.22742"
    ??? I1104 09:19:28.657362    8950 atomic_dir_copy.go:66] "Made an intermediate copy" cmd="/bin/cp --verbose --recursive --preserve --reflink=auto /var/lib/microshift-auto-recovery/20241104091852_default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1 /var/lib/microshift.tmp.482"
    ??? I1104 09:19:28.657385    8950 state.go:40] "Saving intermediate state" state="{\"LastBackup\":\"20241104091852_default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1\"}" path="/var/lib/microshift-auto-recovery/state.json.tmp.41544"
    ??? I1104 09:19:28.662438    8950 atomic_dir_copy.go:115] "Renamed to final destination" src="/var/lib/microshift.tmp.482" dest="/var/lib/microshift"
    ??? I1104 09:19:28.662451    8950 state.go:46] "Moving state file to final path" intermediatePath="/var/lib/microshift-auto-recovery/state.json.tmp.41544" finalPath="/var/lib/microshift-auto-recovery/state.json"
    ??? I1104 09:19:28.662521    8950 atomic_dir_copy.go:115] "Renamed to final destination" src="/var/lib/microshift-auto-recovery/failed/20241104091928_default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1.tmp.22742" dest="/var/lib/microshift-auto-recovery/failed/20241104091928_default-b3442053c9ce69310cd54140d8d592234c5306e4c5132de6efe615f79c84300a.1"
    ??? I1104 09:19:28.662969    8950 atomic_dir_copy.go:115] "Renamed to final destination" src="/var/lib/microshift-auto-recovery/20241022101528_default-a129624b9233fa54fe3574f1aa211bc2d85e1052b52245fe7d83f10c2f6d28e3.0" dest="/var/lib/microshift-auto-recovery/restored/20241022101528_default-a129624b9233fa54fe3574f1aa211bc2d85e1052b52245fe7d83f10c2f6d28e3.0"
    ??? I1104 09:19:28.662983    8950 restore.go:141] "Auto-recovery restore completed".

    Important
    • The restore command does not restart MicroShift after restoration. When you execute this command, MicroShift service has already failed or you stopped it.
    • MicroShift does not monitor the disk space of any filesystem. You must ensure that your automation handles old backup removal. For example, you can add this process to the auto-recovery service or add another service that runs periodically.
  2. Restart MicroShift by running the following command:

    $ sudo systemctl restart microshift

Verification

  • Verify that MicroShift has started successfully by running the following command:

    $ oc get pods -A

    Example output

    NAMESPACE                   NAME                                                     READY   STATUS   RESTARTS  AGE
    default                     i-06166fbb376f14a8bus-west-2computeinternal-debug-qtwcr  1/1     Running  0		    46m
    kube-system                 csi-snapshot-controller-5c6586d546-lprv4                 1/1     Running  0		    51m
    openshift-dns               dns-default-45jl7                                        2/2     Running  0		    50m
    openshift-dns               node-resolver-7wmzf                                      1/1     Running  0		    51m
    openshift-ingress           router-default-78b86fbf9d-qvj9s                          1/1     Running  0		    51m
    openshift-ovn-kubernetes    ovnkube-master-5rfhh                                     4/4     Running  0		    51m
    openshift-ovn-kubernetes    ovnkube-node-gcnt6                                       1/1     Running  0		    51m
    openshift-service-ca        service-ca-bf5b7c9f8-pn6rk                               1/1     Running  0		    51m
    openshift-storage           topolvm-controller-549f7fbdd5-7vrmv                      5/5     Running  0		    51m
    openshift-storage           topolvm-node-rht2m                                       3/3     Running  0		    50m

    Note

    This example output shows basic MicroShift. If you have installed optional RPMs, the status of pods running those services is also expected to be shown in your output.

2.3.1. Using automatic recovery in RPM systems

When MicroShift enters a failed state, the systemd service starts the microshift-auto-recovery.service unit. This unit executes the auto-recovery restore process.

As a use case, consider the following example situation in which you want to automate the automatic recovery process for RPM systems that use the systemd service.

Procedure

  1. Create a directory for the microshift systemd service by running the following command:

    $ sudo mkdir -p /usr/lib/systemd/system/microshift.service.d
  2. To instruct systemd to run microshift-auto-recovery.service when the microshift.service fails, create the 10-auto-recovery.conf file by running the following command:

    $ sudo tee /usr/lib/systemd/system/microshift.service.d/10-auto-recovery.conf > /dev/null <<'EOF'
    [Unit]
    OnFailure=microshift-auto-recovery.service
    StartLimitIntervalSec=25s 1
    EOF
    1
    Increase the StartLimitInterval value from the default 10s to a larger value for slower systems. A value that is too low can result in systemd never marking the microshift systemd service as failed, which means that the OnFailure= service does not get triggered.
  3. Create the microshift-auto-recovery.service file by running the following command:

    $ sudo tee /usr/lib/systemd/system/microshift-auto-recovery.service > /dev/null <<'EOF'
    [Unit]
    Description=MicroShift auto-recovery
    
    [Service]
    Type=oneshot
    ExecStart=/usr/bin/microshift-auto-recovery
    
    [Install]
    WantedBy=multi-user.target
    EOF
  4. Create the microshift-auto-recovery script by running the following command:

    $ sudo tee /usr/bin/microshift-auto-recovery > /dev/null <<'EOF'
    #!/usr/bin/env bash
    set -xeuo pipefail
    
    # If greenboot uses a non-default file for clearing boot_counter, use boot_success instead.
    if grep -q  "/boot/grubenv" /usr/libexec/greenboot/greenboot-grub2-set-success; then
        if grub2-editenv - list | grep -q ^boot_success=0; then
            echo "Greenboot didn't decide the system is healthy after staging new deployment."
            echo "Quitting to not interfere with the process"
            exit 0
        fi
    else
        if grub2-editenv - list | grep -q ^boot_counter=; then
            echo "Greenboot didn't decide the system is healthy after staging a new deployment."
            echo "Quitting to not interfere with the process"
            exit 0
        fi
    fi
    
    /usr/bin/microshift restore --auto-recovery /var/lib/microshift-auto-recovery
    /usr/bin/systemctl reset-failed microshift
    /usr/bin/systemctl start microshift
    
    echo "DONE"
    EOF
  5. Make the script executable by running the following command:

    $ sudo chmod +x /usr/bin/microshift-auto-recovery
  6. Reload the system configuration by running the following command:

    $ sudo systemctl daemon-reload

2.3.2. Using automatic recovery with RHEL for Edge

As a use case, consider the following example situation in which you want to automate the auto-recovery process for Red Hat Enterprise Linux for Edge (RHEL for Edge) systems that use systemd in the blueprint file.

Important

You must include the entire auto-recovery process for RHEL for Edge systems that use systemd in the blueprint file.

Prerequisites

  • You installed Podman.
  • You installed the command-line composer-cli tool.

Procedure

  1. Optional: Because the composer-cli can only create files in the /etc directory, package your files into an RPM that you include the blueprint.
  2. Use the following example to create your blueprint file:

    [[customizations.directories]]
    path = "/etc/systemd/system/microshift.service.d"
    
    [[customizations.directories]]
    path = "/etc/bin"
    
    [[customizations.files]]
    path = "/etc/systemd/system/microshift.service.d/10-auto-recovery.conf"
    data = """
    [Unit]
    OnFailure=microshift-auto-recovery.service
    """
    
    [[customizations.files]]
    path = "/etc/systemd/system/microshift-auto-recovery.service"
    data = """
    [Unit]
    Description=MicroShift auto-recovery
    [Service]
    Type=oneshot
    ExecStart=/etc/bin/microshift-auto-recovery
    [Install]
    WantedBy=multi-user.target
    """
    
    [[customizations.files]]
    path = "/etc/bin/microshift-auto-recovery"
    mode = "0755"
    data = """
    #!/usr/bin/env bash
    set -xeuo pipefail
    # If greenboot uses a non-default file for clearing boot_counter, use boot_success instead.
    if grep -q  "/boot/grubenv" /usr/libexec/greenboot/greenboot-grub2-set-success; then
        if grub2-editenv - list | grep -q ^boot_success=0; then
            echo "Greenboot didn't decide the system is healthy after staging a new deployment."
            echo "Quitting to not interfere with the process"
            exit 0
        fi
    else
        if grub2-editenv - list | grep -q ^boot_counter=; then
            echo "Greenboot didn't decide the system is healthy after staging a new deployment."
            echo "Quitting to not interfere with the process"
            exit 0
        fi
    fi
    /usr/bin/microshift restore --auto-recovery /var/lib/microshift-auto-recovery
    /usr/bin/systemctl reset-failed microshift
    /usr/bin/systemctl start microshift
    echo "DONE"
    """
  3. For the next steps, see Preparing for image building.

2.3.3. Using automatic recovery in image mode for RHEL systems

Important

Image mode for RHEL is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

As a use case, consider the following example situation in which you want to automate the auto-recovery process for image mode for Red Hat Enterprise Linux (RHEL) systems that use the systemd service.

Important

You must include the entire auto-recovery process for image mode for RHEL systems that use systemd in the container file.

Prerequisites

  • You created a Containerfile as instructed in Building the bootc image.
  • You created the 10-auto-recovery.conf and microshift-auto-recovery.service files as explained in the "Using auto-recovery in RPM systems" section.

    Important

    The location of the the 10-auto-recovery.conf and microshift-auto-recovery.service must be relative to the Containerfile.

    For example, if the path to the Containerfile is /home/microshift/my-build/Containerfile, the systemd files need to be adjacent for proper embedding. The following paths are correct for this example:

    • /home/microshift/my-build/auto-rec/10-auto-recovery.conf
    • /home/microshift/my-build/auto-rec/microshift-auto-recovery.service
    • /home/microshift/my-build/auto-rec/microshift-auto-recovery
  • You created the microshift-auto-recovery script as explained in the "Using auto-recovery in RPM systems" section.

Procedure

  1. Use the following example snippet to update the container file that you use to prepare the image mode for RHEL image.

    RUN mkdir -p /usr/lib/systemd/system/microshift.service.d
    COPY ./auto-rec/10-auto-recovery.conf /usr/lib/systemd/system/microshift.service.d/10-auto-recovery.conf
    COPY ./auto-rec/microshift-auto-recovery.service /usr/lib/systemd/system/
    COPY ./auto-rec/microshift-auto-recovery /usr/bin/
    RUN chmod +x /usr/bin/microshift-auto-recovery
    Important

    Podman uses the host subscription information and repositories inside the container when building the container image. If the rhocp and fast-datapath repositories are not available on the host, the build fails.

  2. Rebuild your local bootc image by running the following image build command:

    PULL_SECRET=~/.pull-secret.json
    USER_PASSWD=<your_redhat_user_password>
    IMAGE_NAME=microshift-4.18-bootc
    
    sudo podman build --authfile "${PULL_SECRET}" -t "${IMAGE_NAME}" \
        --build-arg USER_PASSWD="${USER_PASSWD}" \
        -f Containerfile
    Note

    Secrets are used during the image build in the following ways:

    • The podman --authfile argument is required to pull the base rhel-bootc:9.4 image from the registry.redhat.io registry.
    • The build USER_PASSWD argument is used to set a password for the redhat user.

Verification

  • Verify that the local bootc image was created by running the following command:

    $ sudo podman images "${IMAGE_NAME}"

    Example output

    REPOSITORY                       TAG         IMAGE ID      CREATED        SIZE
    localhost/microshift-4.18-bootc  latest      193425283c00  2 minutes ago  2.31 GB

Legal Notice

Copyright © 2025 Red Hat, Inc.
The text of and illustrations in this document are licensed by Red Hat under a Creative Commons Attribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA is available at http://creativecommons.org/licenses/by-sa/3.0/. In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must provide the URL for the original version.
Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.
Red Hat, Red Hat Enterprise Linux, the Shadowman logo, the Red Hat logo, JBoss, OpenShift, Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries.
Linux® is the registered trademark of Linus Torvalds in the United States and other countries.
Java® is a registered trademark of Oracle and/or its affiliates.
XFS® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries.
MySQL® is a registered trademark of MySQL AB in the United States, the European Union and other countries.
Node.js® is an official trademark of Joyent. Red Hat is not formally related to or endorsed by the official Joyent Node.js open source or commercial project.
The OpenStack® Word Mark and OpenStack logo are either registered trademarks/service marks or trademarks/service marks of the OpenStack Foundation, in the United States and other countries and are used with the OpenStack Foundation's permission. We are not affiliated with, endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.
All other trademarks are the property of their respective owners.
Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

© 2024 Red Hat, Inc.