Using image mode for RHEL to build, deploy, and manage operating systems

1. Package mode: the operating system uses RPM packages and is updated by using the dnf package manager. The root filesystem is mutable. However, the operating system cannot be managed as a containerized application.
2. Image mode: a container-native approach to build, deploy, and manage RHEL. The same RPM packages are delivered as a base image and updates are deployed as a container image. The root filesystem is immutable by default, except for /etc and /var, with most content coming from the container image.

Procedure

1. Create a Containerfile:

   FROM registry.redhat.io/rhel10/rhel-bootc:latest
   RUN dnf -y install cloud-init && \
       ln -s ../cloud-init.target /usr/lib/systemd/system/default.target.wants && \
       dnf clean all

   This Containerfile example adds the cloud-init tool, so it automatically fetches SSH keys and can run scripts from the infrastructure and also gather configuration and secrets from the instance metadata. For example, you can use this container image for pre-generated AWS or KVM guest systems.

2. Build the <image> image by using Containerfile in the current directory:

   $ podman build -t quay.io/<namespace>/<image>:<tag> .

Procedure

1. Create a Containerfile:

   FROM registry.redhat.io/rhel10/rhel-bootc:latest
   # In this example, the custom operating system has its own
   # version scheme.
   # The operating system major version is copied
   # and a sub-version of it is added, which represents a point-in-time
   # snapshot of the base OS content.
   # This just changes the output of bootc status. A deeper level
   # of customization is available by also changing the contents of /usr/lib/os-release.
   # Define the custom version and release metadata
   LABEL org.opencontainers.image.version=”10.1.1”

2. Build the <image> image by using Containerfile from the current directory:

   $ podman build -t quay.io/<namespace>/<image>:<tag> .

Procedure

1. Select the image that you want to logically bound.

2. Create a Containerfile:

   $ cat Containerfile
   FROM quay.io/<namespace>/<image>:latest
   COPY ./<app_1>.image /usr/share/containers/systemd/<app_1>.image
   COPY ./<app_2>.container /usr/share/containers/systemd/<app_2>.container
   
   RUN ln -s /usr/share/containers/systemd/<app_1>.image \
   	/usr/lib/bootc/bound-images.d/<app_1>.image && \
       ln -s /usr/share/containers/systemd/<app_2>.container \
       	/usr/lib/bootc/bound-images.d/<app_2>.container

3. In the .container definition, use:

   GlobalArgs=--storage-opt=additionalimagestore=/usr/lib/bootc/storage

   In the Containerfile example, the image is selected to be logically bound by creating a symlink in the /usr/lib/bootc/bound-images.d directory pointing to either an .image or a .container file.

4. Run the bootc upgrade command.

   $ bootc upgrade

   The bootc upgrade performs the following overall steps:

   1. Fetches the new base image from the image repository. See Configuring container pull secrets.
   2. Reads the new base image root file system to discover logically bound images.
   3. Automatically pulls any discovered logically bound images defined in the new bootc image into the bootc-owned /usr/lib/bootc/storage image storage.

5. Make the bound images become available to container runtimes such as Podman. For that, you must explicitly configure bound images to point to the bootc storage as an "additional image store". For example:

   podman --storage-opt=additionalimagestore=/usr/lib/bootc/storage run <image>

   Important

   Do not attempt to globally enable the /usr/lib/bootc/storage image storage in /etc/containers/storage.conf. Only use the bootc storage for logically bound images.

   The bootc image store is owned by bootc. The logically bound images will be garbage collected when they are no longer referenced by a file in the /usr/lib/bootc/bound-images.d directory.

Procedure

1. Login to authenticate to registry.redhat.io:

   $ sudo podman login registry.redhat.io

2. Install the bootc-image-builder tool:

   $ sudo podman pull registry.redhat.io/rhel10/bootc-image-builder

Procedure

1. Optional: Create a config.toml to configure user access, for example:

   [[customizations.user]]
   name = "user"
   password = "pass"
   key = "ssh-rsa AAA ... user@email.com"
   groups = ["wheel"]

2. Before running the container, initialize the output folder. Use the -p argument to ensure that the command does not fail if the directory already exists:

   $ mkdir -p ./output

3. Run bootc-image-builder. Optionally, if you want to use user access configuration, pass the config.toml as an argument.

   1. The following example creates a public QEMU disk image (QCOW2). To build a public image, you must have a container image that is available in a remote, publicly accessible registry, for example, registry.redhat.io/rhel10/bootc-image-builder:latest. The image is available to download and use without special credentials.

      $ podman run \
          --rm \
          --privileged \
          --pull=newer \
          --security-opt label=type:unconfined_t \
          -v ./config.toml:/config.toml:ro \
          -v ./output:/output \
          registry.redhat.io/rhel10/bootc-image-builder:latest \
          --type qcow2 \
          --config /config.toml \
        quay.io/<namespace>/<image>:<tag>

   2. This example creates a private QEMU disk image (QCOW2) from a local container. To build a private image, you must have a container image on your local machine, which is not available on a public registry. The local image could be an image you built yourself using a Containerfile, an image you pulled from a private, access-controlled registry that required a login, or an image you loaded from a tar file. The bootc-image-builder finds and uses the source image from your local Podman /var/lib/containers/storage storage, which is mounted into the builder container.

      $ sudo podman run \
          --rm \
          -it \
          --privileged \
          --pull=newer \
          --security-opt label=type:unconfined_t \
          -v ./config.toml:/config.toml:ro \
          -v ./output:/output \
          -v /var/lib/containers/storage:/var/lib/containers/storage \
          registry.redhat.io/rhel10/bootc-image-builder:latest \
          --type qcow2 \
          --config /config.toml \
          quay.io/<namespace>/<image>:<tag>

      You can find the .qcow2 image in the output folder.

Procedure

1. Create a Containerfile with the following content:

   FROM registry.redhat.io/rhel10/rhel-bootc:latest
   RUN dnf -y install cloud-init open-vm-tools && \
   ln -s ../cloud-init.target /usr/lib/systemd/system/default.target.wants && \
   rm -rf /var/{cache,log} /var/lib/{dnf,rhsm} && \
   systemctl enable vmtoolsd.service

2. Build the bootc image:

   # podman build . -t localhost/rhel-bootc-vmdk

3. Before running the container, initialize the output folder. Use the -p argument to ensure that the command does not fail if the directory already exists:

   $ mkdir -p ./output

4. Create a VMDK file from the previously created bootc image. The image must be accessible from a registry, such as registry.redhat.io/rhel10/bootc-image-builder:latest.

   # podman run \
       --rm \
       --privileged \
       -v /var/lib/containers/storage:/var/lib/containers/storage \
       -v ./output:/output \
       --security-opt label=type:unconfined_t \
       --pull newer \
       registry.redhat.io/rhel10/bootc-image-builder:latest \
       --type vmdk \
       --config /config.toml \
       localhost/rhel-bootc-vmdk:latest

   A VMDK disk file for the bootc image is stored in the output/vmdk directory.

Procedure

1. Optional: Create a config.toml to configure user access, for example:

   [[customizations.user]]
   name = "user"
   password = "pass"
   key = "ssh-rsa AAA ... user@email.com"
   groups = ["wheel"]

2. Before running the container, initialize the output folder. Use the -p argument to ensure that the command does not fail if the directory already exists:

   $ mkdir -p ./output

3. Run bootc-image-builder. Optionally, if you want to use user access configuration, pass the config.toml as an argument. The image must be accessible from a registry, such as registry.redhat.io/rhel10/bootc-image-builder:latest.

   1. The following is an example of creating a gce image:

      $ podman run \
          --rm \
          --privileged \
          --pull=newer \
          --security-opt label=type:unconfined_t \
          -v ./config.toml:/config.toml:ro \
          -v ./output:/output \
          -v /var/lib/containers/storage:/var/lib/containers/storage \
          registry.redhat.io/rhel10/bootc-image-builder:latest \
          --type gce \
          --config /config.toml \
        quay.io/<namespace>/<image>:<tag>

      You can find the gce image in the output folder.

Procedure

1. Create a disk image from the bootc image.

   • Configure the user details in the Containerfile. Make sure that you assign it with sudo access.
   • Build a customized operating system image with the configured user from the Containerfile. It creates a default user without password sudo access.

2. Optional: Configure the machine image with cloud-init. See Users and groups configuration - Injecting users and SSH keys by using cloud-init. The following is an example:

   FROM registry.redhat.io/rhel10/rhel-bootc:latest
   
   RUN dnf -y install cloud-init && \
       ln -s ../cloud-init.target /usr/lib/systemd/system/default.target.wants && \
       rm -rf /var/{cache,log} /var/lib/{dnf,rhsm}

   Note

   You can also use cloud-init to add users and additional configuration by using instance metadata.

3. Build the bootc image. For example, to deploy the image to an x86_64 AWS machine, use the following commands:

   $ podman build -t quay.io/<namespace>/<image>:<tag> .
   $ podman push quay.io/<namespace>/<image>:<tag> .

4. Before running the container, initialize the output folder. Use the -p argument to ensure that the command does not fail if the directory already exists:

   $ mkdir -p ./output

5. Use the bootc-image-builder tool to create a public AMI image from the bootc container image. The image must be accessible from a registry, such as registry.redhat.io/rhel10/bootc-image-builder:latest.

   $ podman run \
     --rm \
     --privileged \
     --pull=newer \
     -v $HOME/.aws:/root/.aws:ro \
     -v /var/lib/containers/storage:/var/lib/containers/storage \
     --env AWS_PROFILE=default \
     registry.redhat.io/rhel10/bootc-image-builder:latest \
     --type ami \
     --config /config.toml \
     --aws-ami-name rhel-bootc-x86 \
     --aws-bucket rhel-bootc-bucket \
     --aws-region us-east-1 \
   quay.io/<namespace>/<image>:<tag>

   Note

   The following flags must be specified all together. If you do not specify any flag, the AMI is exported to your output directory.

   • --aws-ami-name - The name of the AMI image in AWS
   • --aws-bucket - The target S3 bucket name for intermediate storage when you are creating the AMI

   • --aws-region - The target region for AWS uploads

     The bootc-image-builder tool builds an AMI image and uploads it to your AWS S3 bucket by using your AWS credentials to push and register an AMI image after building it.

Procedure

1. Optional: Create a config.toml to configure user access, for example:

   [[customizations.user]]
   name = "user"
   password = "pass"
   key = "ssh-rsa AAA ... user@email.com"
   groups = ["wheel"]

2. Before running the container, initialize the output folder. Use the -p argument to ensure that the command does not fail if the directory already exists:

   $ mkdir -p ./output

3. Run bootc-image-builder. If you want to use user access configuration, pass the config.toml as an argument. The image must be accessible from a registry, such as registry.redhat.io/rhel10/bootc-image-builder:latest.

   $ podman run \
       --rm \
       --privileged \
       --pull=newer \
       --security-opt label=type:unconfined_t \
       -v /var/lib/containers/storage:/var/lib/containers/storage \
       -v ./config.toml:/config.toml \
       -v ./output:/output \
       registry.redhat.io/rhel10/bootc-image-builder:latest \
       --type raw \
       --config /config.toml \
     quay.io/<namespace>/<image>:<tag>

   You can find the .raw image in the output folder.

Procedure

1. Optional: Create a config.toml to which overrides the default embedded Kickstart which performs an automatic installation.

   [customizations.installer.kickstart]
   contents = """
   text --non-interactive
   zerombr
   clearpart --all --initlabel --disklabel=gpt
   autopart --noswap --type=lvm
   network --bootproto=dhcp --device=link --activate --onboot=on
   """

2. Before running the container, initialize the output folder. Use the -p argument to ensure that the command does not fail if the directory already exists:

   $ mkdir -p ./output

3. Run bootc-image-builder to create a public ISO image. If you do not want to add any configuration, omit the -v ./config.toml:/config.toml argument. The image must be accessible from a registry, such as registry.redhat.io/rhel10/bootc-image-builder:latest.

   $ podman run \
       --rm \
       --privileged \
       --pull=newer \
       --security-opt label=type:unconfined_t \
       -v /var/lib/containers/storage:/var/lib/containers/storage \
       -v ./config.toml:/config.toml:ro \
       -v ./output:/output \
       registry.redhat.io/rhel10/bootc-image-builder:latest \
       --type iso \
       --config /config.toml \
     quay.io/<namespace>/<image>:<tag>

   You can find the .iso image in the output folder.

Procedure

1. Create your Kickstart file. The following Kickstart file is an example of a fully unattended Kickstart file configuration that contains user creation, and partition instructions.

   [customizations.installer.kickstart]
   contents = """
   lang en_GB.UTF-8
   keyboard uk
   timezone CET
   
   user --name <user> --password <password> --plaintext --groups <groups>
   sshkey --username <user> ssh-<type> <public key>
   rootpw --lock
   
   zerombr
   clearpart --all --initlabel
   autopart --type=plain
   reboot --eject
   """

2. Save the Kickstart configuration in the toml format to inject the Kickstart content. For example, config.toml.

3. Run bootc-image-builder, and include the Kickstart file configuration that you want to add to the ISO build. The bootc-image-builder automatically adds the ostreecontainer command that installs the container image.

   $ sudo podman run \
       --rm \
       -it \
       --privileged \
       --pull=newer \
       --security-opt label=type:unconfined_t \
       -v /var/lib/containers/storage:/var/lib/containers/storage \
       -v ./config.toml:/config.toml \
       -v ./output:/output \
       registry.redhat.io/rhel10/bootc-image-builder:latest \
       --type iso \
       --config /config.toml \
     quay.io/<namespace>/<image>:<tag>

   You can find the .iso image in the output folder.

Procedure

1. Create a config.toml file to configure custom mount options, for example:

   [[customizations.filesystem]]
   mountpoint = "/"
   minsize = "10 GiB"
   
   [[customizations.filesystem]]
   mountpoint = "/var/data"
   minsize = "20 GiB"

2. Run bootc-image-builder, passing the config.toml as an argument.

   Note

   If you do not have the container storage mount, your image must be public.

   The following is an example of creating a public image:

   $ sudo podman run \
       --rm \
       -it \
       --privileged \
       --pull=newer \
       --security-opt label=type:unconfined_t \
       -v ./config.toml:/config.toml \
       -v ./output:/output \
       registry.redhat.io/rhel10/bootc-image-builder:latest \
       --type <image_type> \
       --config config.toml \
     quay.io/<namespace>/<image>:<tag>

Procedure

1. Create a Containerfile. For example:

   # Base image to point to your internal registry
   FROM example.com:1234/rhel10/rhel-bootc:10.2
   
   # Configure the local repo to use the files already present in the image
   # Assuming the repo data is located at /etc/pki/repos or similar inside the image
   RUN echo -e "[local-baseos]\n\
   name=Local RHEL 10 BaseOS\n\
   baseurl=file:///path/to/repo/in/image/BaseOS\n\
   enabled=1\n\
   gpgcheck=0" > /etc/yum.repos.d/local.repo
   
   # Install your required packages using the local file source
   RUN dnf install -y firewalld && \
      dnf clean all
   
   # Ensure the kernel and bootloader are present
   # In air-gapped bootc, BIB often fails because it expects to download these.
   # Pre-installing them ensures they are part of the 'bootc' transition.
   RUN dnf install -y kernel-bootc anaconda-dracut-modules && dnf clean all

2. Use the bootc-image-builder tool to transform the Containerfile into a bootable format, such as ISO, raw, QCOW2. See Creating bootc-compatible base disk images by using bootc-image-builder.

Procedure

1. Create a Containerfile:

   FROM <internal_repository>/<image>
   # Add certificate to the input set of anchors
   COPY additional-certificate-root.pem /etc/pki/ca-trust/source/anchors
   RUN update-ca-trust

2. Build the custom image:

   # podman build -t <your_image> .

3. Run the <your_image>:

   # podman run -it --rm <your_image>

Procedure

1. Export the RHEL installation ISO image to the HTTP server. The PXE boot server is now ready to serve PXE clients.
2. Boot the client and start the installation.
3. Select PXE Boot when prompted to specify a boot source. If the boot options are not displayed, press the Enter key on your keyboard or wait until the boot window opens.
4. From the Red Hat Enterprise Linux boot window, select the boot option that you want, and press Enter.
5. Start the network installation.

Procedure

1. Create a metadata.yaml file and add the following information to this file:

   instance-id: cloud-vm
   local-hostname: vmname

2. Create a userdata.yam file and add the following information to the file:

   #cloud-config
   users:
   - name: admin
     sudo: "ALL=(ALL) NOPASSWD:ALL"
     ssh_authorized_keys:
     - ssh-rsa AAA...fhHQ== your.email@example.com

   ssh_authorized_keys is your SSH public key. You can find your SSH public key in ~/.ssh/id_rsa.pub.

3. Export the metadata.yaml and userdata.yaml files to the environment, compressed with gzip, encoded in base64 as follows. You will use these files in further steps.

   export METADATA=$(gzip -c9 <metadata.yaml | { base64 -w0 2>/dev/null || base64; }) \
   USERDATA=$(gzip -c9 <userdata.yaml | { base64 -w0 2>/dev/null || base64; })

4. Launch the image on vSphere with the metadata.yaml and userdata.yaml files:

   1. Import the .vmdk image in to vSphere:

      $ govc import.vmdk ./composer-api.vmdk <foldername>

   2. Create the VM in vSphere without powering it on:

      govc vm.create \
      -net.adapter=vmxnet3 \
      -m=4096 -c=2 -g=rhel8_64Guest \
      -firmware=bios -disk="foldername/composer-api.vmdk” \
      -disk.controller=ide -on=false \
      vmname

   3. Change the VM to add ExtraConfig variables, the cloud-init config:

      govc vm.change -vm vmname \
      -e guestinfo.metadata="${METADATA}" \
      -e guestinfo.metadata.encoding="gzip+base64" \
      -e guestinfo.userdata="${USERDATA}" \
      -e guestinfo.userdata.encoding="gzip+base64"
      .. Power-on the VM:
      govc vm.power -on vmname

   4. Retrieve the VM IP address:

      HOST=$(govc vm.ip vmname)

Procedure

1. In a browser, access Service→EC2 and log in.
2. On the AWS console dashboard menu, choose the correct region. The image must have the Available status, to indicate that it was correctly uploaded.
3. On the AWS dashboard, select your image and click Launch.
4. In the new window that opens, choose an instance type according to the resources you need to start your image. Click Review and Launch.
5. Review your instance details. You can edit each section if you need to make any changes. Click Launch.
6. Before you start the instance, select a public key to access it. You can either use the key pair you already have or you can create a new key pair.

7. Click Launch Instance to start your instance. You can check the status of the instance, which displays as Initializing.

   After the instance status is Running, the Connect button becomes available.

8. Click Connect. A window appears with instructions on how to connect by using SSH.

9. Run the following command to set the permissions of your private key file so that only you can read it. See Connect to your Linux instance.

   $ chmod 400 <your-instance-name.pem>

10. Connect to your instance by using its Public DNS:

    $ ssh -i <your-instance-name.pem>ec2-user@<your-instance-IP-address>

    Note

    Your instance continues to run unless you stop it.

Procedure

1. Create a custom installer ISO disk image with bootc-image-builder. See Creating ISO images by using bootc-image-builder.
2. Copy the ISO disk image to a USB flash drive.
3. Perform a bare-metal installation by using the content in the USB stick into a disconnected environment.

Procedure

1. Create a ./config.toml. The following example shows how to add a user to the disk image.

   [[customizations.user]]
   name = "user"
   password = "pass"
   key = "ssh-rsa AAA ... user@email.com"
   groups = ["wheel"]

   • name - Mandatory. Name of the user.
   • password - Not mandatory. Nonencrypted password.
   • key - Not mandatory. Public SSH key contents.
   • groups - Not mandatory. An array of groups to add the user into.

2. Run bootc-image-builder and pass the following arguments, including the ./config.toml:

   $ sudo podman run \
       --rm \
       -it \
       --privileged \
       --pull=newer \
       --security-opt label=type:unconfined_t \
       -v ./config.toml:/config.toml \
       -v ./output:/output \
       registry.redhat.io/rhel10/bootc-image-builder:latest \
       --type qcow2 \
       --config config.toml \
       quay.io/<namespace>/<image>:<tag>

3. Launch a VM, for example, by using virt-install:

   $ sudo virt-install \
     --name bootc \
     --memory 4096 \
     --vcpus 2 \
     --disk qcow2/disk.qcow2 \
     --import \
     --os-variant rhel10

Procedure

1. After the instance starts, connect to it by using SSH using the key you selected when creating the instance:

   $ ssh -i <ssh-key-file> <cloud-user@ip>

2. Make sure that the system-reinstall-bootc subpackage is installed:

   # rpm -q system-reinstall-bootc

   If not, install the system-reinstall-bootc subpackage:

   # dnf -y install system-reinstall-bootc

3. Convert the system to use a bootc image:

   # system-reinstall-bootc <image>

   • You can use the container image from the Red Hat Ecosystem Catalog or the customized bootc image built from a Containerfile.
4. Select users to import to the bootc image by pressing the "a" key.
5. Confirm your selection twice and wait until the image is downloaded.

6. Reboot the system:

   # reboot

7. Remove the stored SSH host key for the given <ip> from your /.ssh/known_hosts file:

   # ssh-keygen -R <ip>

   The bootc system is now using a new public SSH host key. When attempting to connect to the same IP address with a different key than what is stored locally, SSH will raise a warning or refuse the connection due to a host key mismatch. Since this change is expected, the existing host key entry can be safely removed from the ~/.ssh/known_hosts file using the following command.

8. Connect to the bootc system:

   # ssh -i <ssh-key-file> root@<ip>

Procedure

1. Verify which storage devices are already connected to your system.

   $ lsblk
   NAME          MAJ:MIN     SIZE   RO TYPE  MOUNTPOINTS
   zram0           251:0       8G    0 disk  [SWAP]
   nvme0n1         259:0   476.9G    0 disk
   ├─nvme0n1p1    259:1     600M    0 part  /boot/efi
   ├─nvme0n1p2    259:2       1G    0 part  /boot
   └─nvme0n1p3    259:3   475.4G    0 part

2. Connect your external storage and run the same command. Compare the two outputs to find the name of your external storage device on your system.

   $ lsblk
   NAME        MAJ:MIN   SIZE   RO TYPE  MOUNTPOINTS
   sda             8:0   28.9G    0 disk
   └─sda1         8:1   28.9G    0 part
   zram0         251:0      8G    0 disk  [SWAP]
   nvme0n1       259:0  476.9G    0 disk
   ├─nvme0n1p1  259:1    600M    0 part  /boot/efi
   ├─nvme0n1p2  259:2      1G    0 part  /boot
   └─nvme0n1p3  259:3  475.4G    0 part

   In this case, the USB drive whose name is sda has an sda1 partition.

   The MOUNTPOINTS column lists the mount points of the partitions on your external storage. If your system automatically mounts external storage, then valid mount points already exist. However, if there are no mount points, you must mount it yourself before you can store anything on the device.

3. Create an empty directory, or use an existing one, to mount your partition:

   $ sudo mkdir /mnt/usb/

   1. Mount your device partition.

      $ sudo mount /dev/sda1 /mnt/usb

   2. Optional: Verify if the partition was correctly created:

      $ lsblk
      NAME         MAJ:MIN    SIZE   RO TYPE  MOUNTPOINTS
      sda              8:0   28.9G    0 disk
      └─sda1          8:1   28.9G    0 part  /mnt/usb
      [...]

      Your external storage device is ready for copying files onto it.

4. Copy the container stored locally to your mounted device by using the skopeo command, and adapting the paths and names of the container for your own environment:

   • For local storage:

     $ sudo skopeo copy --preserve-digests --all \
       containers-storage:localhost/rhel-container:latest \
       oci://mnt/usb/

   • For a container stored on a remote registry:

     $ sudo skopeo copy --preserve-digests --all \
       docker://quay.io/example:latest \
       oci://mnt/usb/

     Note

     Depending on the size of the container, these commands might take a few minutes to complete.

5. Unmount and eject the external storage:

   $ sudo umount /dev/sda1
   $ sudo eject /dev/sda1

6. Apply the update to the container on the offline system.
7. Plug the external storage device into your offline system. If the storage device does not mount automatically, use the mkdir and mount commands to locate the external storage and mount it.

8. Copy the container from the external device over to the offline system’s local container registry. Copy the container to the offline machine’s local container storage:

   $ skopeo copy --preserve-digests --all \
     oci://mnt/usb \
     containers-storage:rhel-update:latest

   In this case, the mount point of the external storage is the path to the OCI section, while the containers-storage section varies depending on the name and tag you want the container to have.

9. Use Podman to verify that your container is now local:

   $ podman images
   REPOSITORY			        TAG     IMAGE ID  CREATED  SIZE
   example.io/library/rhel-update   latest  cdb6d...  1 min    1.48 GB

10. Deploy the update to the container on the offline system by using bootc:

    $ bootc switch --transport containers-storage \
    example.io/library/rhel-update:latest

    1. If you cannot copy your container to local storage, use the oci transport flag and the path to your storage device instead:

       $ bootc switch --transport oci /mnt/usb

       With the --transport flag in the bootc switch command, you can specify an alternative source for the container.

       By default, bootc attempts to pull from a registry because the bootc-image-builder uses a registry to build the original image. When using bootc upgrade, you cannot specify where an update is located. By using the bootc switch and specifying that you are using local container storage, you cannot only remove the requirement of a remote registry, but also deploy updates by using this local container in the future.

       You can now successfully use the bootc upgrade, provided that your local container and the update share the same location. If you want to switch to updates on a remote repository in the future, you must use bootc switch again.

Verification

1. Ensure that the update was properly deployed:

   $ bootc status
   Staged image: containers-storage:example.io/library/rhel-update:latest
     Digest: sha256: 05b1dfa791...
     Version: 10.0 (2025-07-07 18:33:19.380715153 UTC)
   Booted Image: localhost/rhel-intel:base
     Digest: sha256: 7d6f312e09...
     Version: 10.0 (2025-06-23 15:58:12.228704562 UTC)

   The output shows your current booted image, along with any changes staged to happen. The container you used earlier is visible, but the staged changes do not happen until the next reboot. The output also confirms that updates will be pulled from your container storage.

2. Reboot the system:

   $ bootc status
   Booted image: containers-storage:example.io/library/rhel-update:latest
   	Digest: sha256: 05b1dfa791...
   	Version: 10.0 (2025-07-07 18:33:19.380715153 UTC)
   Rollback image: localhost/rhel-intel:base
   	Digest: sha256: 7d6f312e09...
   	Version: 10.0 (2025-06-23 15:58:12.228704562 UTC)

   You can verify that you have booted into the correct image:

   • The booted image is your updated image.
   • The rollback image is your previous image. You have successfully performed an offline image mode update.

Verification

1. Save and build your image.

   $ podman build -t quay.io/<namespace>/<image>:<tag> . --cap-add=all --security-opt=label=type:container_runtime_t --device /dev/fuse

2. Build <_image_> image by using the Containerfile in the current directory:

   $ podman build -t quay.io/<namespace>/<image>:<tag> .

Procedure

1. Create a 01-fips.toml to configure FIPS enablement, for example:

   # Enable FIPS
   kargs = ["fips=1"]

2. Create a Containerfile with the following instructions to enable the fips=1 kernel argument and adjust the cryptographic policies:

   FROM registry.redhat.io/rhel10/rhel-bootc:latest
   # Enable fips=1 kernel argument: https://bootc-dev.github.io/bootc/building/kernel-arguments.html
   COPY 01-fips.toml /usr/lib/bootc/kargs.d/
   # Install and enable the FIPS crypto policy
   RUN dnf install -y crypto-policies-scripts && update-crypto-policies --no-reload --set FIPS

3. Before running the container, initialize the output folder. Use the -p argument to ensure that the command does not fail if the directory already exists:

   $ mkdir -p ./output

4. Create your bootc <image> compatible base disk image by using Containerfile in the current directory:

   $ sudo podman run \
       --rm \
       -it \
       --privileged \
       --pull=newer \
       --security-opt label=type:unconfined_t \
       -v ./config.toml:/config.toml:ro \
       -v ./output:/output \
       -v /var/lib/containers/storage:/var/lib/containers/storage \
       registry.redhat.io/rhel10/bootc-image-builder:latest \
       --type iso \
       quay.io/<namespace>/<image>:<tag>

5. Enable FIPS mode during the system installation:

   1. When booting the RHEL Anaconda installer, on the installation screen, press the TAB key and add the fips=1 kernel argument.

      After the installation, the system starts in FIPS mode automatically.

Procedure

1. Create a Containerfile:

   FROM registry.redhat.io/rhel10/rhel-bootc:latest
   
   # Install OpenSCAP scanner and security content to the image
   RUN dnf install -y openscap-utils scap-security-guide && dnf clean all
   
   # Run scan and hardening
   RUN oscap-im --profile <profile_ID> /usr/share/xml/scap/ssg/content/ssg-rhel10-ds.xml
   
   # Because certain profiles prevent ssh root logins, add a separate sudo user with a password
   # Alternatively, you can add users with Kickstart, cloud-init, or other methods
   RUN useradd -G wheel -p "<password_hash>" <admin_user>

   Replace <admin_user> with the user name and <password_hash> with the hash of the selected password.

   This Containerfile performs the following tasks:

   • Installs the openscap-utils package that provides the oscap-im tool and the scap-security-guide package that provides the data streams with the Security Content Automation Protocol (SCAP) content.
   • Adds a user with sudoer privileges for profiles that prevent SSH root logins.
   • Scans and remediates the image for compliance with the selected profile.

2. Build the image by using the Containerfile in the current directory:

   $ podman build -t quay.io/<namespace>/<image>:<tag> .

Procedure

1. Create a Containerfile:

   FROM registry.redhat.io/rhel10/rhel-bootc:latest
   
   # Copy a tailoring file into the Containerfile
   COPY tailoring.xml /usr/share/
   
   # Install OpenSCAP scanner and security content to the image
   RUN dnf install -y openscap-utils scap-security-guide && dnf clean all
   
   
   # Add sudo user 'admin' with password 'admin123'.
   # The user can be used with profiles that prevent
   # ssh root logins.
   RUN useradd -G wheel -p "\$6\$Ga6Zn
   IlytrWpuCzO\$q0LqT1USHpahzUafQM9jyHCY9BiE5/ahXLNWUMiVQnFGblu0WWGZ1e6icTaCGO4GNgZNtspp1Let/qpM7FMVB0" admin
   
   # Run scan and hardening including the tailoring file
   RUN oscap-im --tailoring-file /usr/share/tailoring.xml --profile stig_customized /usr/share/xml/scap/ssg/content/ssg-rhel10-ds.xml

   This Containerfile performs the following tasks:

   • Injects the tailoring file to your image.
   • Installs the openscap-utils package that provides the oscap-im tool and the scap-security-guide package that provides the data streams with the Security Content Automation Protocol (SCAP) content.
   • Adds a user with sudoer privileges for profiles that prevent SSH root logins.
   • Scans and remediates the image for compliance with the selected profile.

2. Build the image by using the Containerfile in the current directory:

   $ podman build -t quay.io/<namespace>/<image>:<tag> .

Procedure

1. Create a transient-root-example.service service to create the directory with read an write permissions. In the following example, /etc/afs exists.

   [Unit]
   Description=Transient Root Example - Dynamic Mount Point Creation
   After=local-fs.target
   ConditionPathExists=/etc/afs
   DefaultDependencies=no
   
   [Service]
   Type=oneshot
   RemainAfterExit=yes
   MountFlags=slave
   # LIBMOUNT_FORCE_MOUNT2=always is required for mount command compatibility
   # See: https://github.com/util-linux/util-linux/issues/3171
   Environment=LIBMOUNT_FORCE_MOUNT2=always
   ExecStart=/usr/bin/bash -c "echo 'Transient root example: Detected /etc/afs, remounting root as read-write'; mount -o remount,rw /; mkdir -p /afs; echo 'Created /afs directory for AFS mount point'"
   
   [Install]
   WantedBy=multi-user.target

2. Add the following configuration to the ContainerFile:

   # Copy the systemd service unit that demonstrates dynamic mount point creation
   # The service will:
   # 1. Check for existence of /etc/afs (ConditionPathExists)
   # 2. Use MountFlags=slave for proper mount propagation
   # 3. Set LIBMOUNT_FORCE_MOUNT2=always for mount command compatibility
   # 4. Remount root as read-write and create /afs directory when triggered
   COPY transient-root-example.service /usr/lib/systemd/system/
   
   # Enable the service to start automatically on boot
   RUN systemctl enable transient-root-example.service

3. Enable the transient-ro option. During the container build process, create the following configuration to the /usr/lib/ostree/prepare-root.conf file inside the image to enable the feature:

   RUN echo -e '[root]\ntransient=true' >> /usr/lib/ostree/prepare-root.conf && \
       set -x; kver=$(cd /usr/lib/modules && echo *); dracut -vf /usr/lib/modules/$kver/initramfs.img $kver

   Note

   When making changes to the file system configuration, you must regenerate the initramfs image.

4. Build the bootc image with the transient root file system:

   $ podman build --cap-add SYS_ADMIN -t transient-root-ro:test .

   This adds the SYS_ADMIN capability that bootc might require for ostree operations during the build.

Verification

1. Verify that transient-ro = true configuration file was correctly set in the built image:

   $ podman run --rm transient-root-ro:test cat /usr/lib/ostree/prepare-root.conf

   You should see an output similar to the following:

   [root]
   transient-ro = true

2. On the running system, verify the runtime functionality:

   1. Boot the image.

   2. Create the /etc/afs test directory.

      $ sudo mkdir -p /etc/afs

   3. Restart the service to trigger dynamic mount point creation:

      $ systemctl restart transient-root-example.service

      The service remounts root as read-write and creates the /afs directory.

3. To test the functionality without a systemd service, create a test directory as root on the running system:

   $ sudo export LIBMOUNT_FORCE_MOUNT2=always
   $ sudo unshare -m -- /bin/sh -c 'mount -o remount,rw / && mkdir /transient-test'

4. Verify if the directory was created:

   $ sudo ls -d /transient-test /transient-test

5. Verify the read-only status for other processes by attempting to write to the root filesystem:

   $ touch /another-test-file
   touch: cannot touch '/another-test-file': Read-only file system

Procedure

1. Create files within /usr/lib/bootc/kargs.d with kernel arguments.

   $ sudo tee /usr/lib/bootc/kargs.d/console.kargs << EOF
   console=tty0 console=ttyS0,115200n8
   EOF

2. Fetch the container image to get the OSTree commit.

   $ podman pull quay.io/<your_org>/<your_bootc_image>:latest

3. Use the OSTree commit to return the file tree.

   # bootc install to-filesystem --karg=root=<UUID>=<uuid of /mnt> --imgref $self /mnt

4. Navigate to the /usr/lib/bootc/kargs.d kernel arguments directory.

   cd /usr/lib/bootc/kargs.d

5. Read each file within the kernel arguments directory.

   $ find /usr/lib/bootc/kargs.d -name ".kargs" -exec cat {} \;*

6. Push the contents of each kargs file into a file containing all the needed kargs.

   $ CONSOLIDATED_KARGS="/tmp/all-kargs.txt"

7. Pass the kargs to the stage() function.

   $ bootc kargs --append="$KARGS_STRING"

8. Apply the kernel arguments to switch, upgrade, or edit during operations.

   $ bootc switch --transport=registry quay.io/<your_org>/<your_bootc_image>:latest

Procedure

1. On your build host, compile your driver and build the RPM package.

   1. Verify the spec and build the package:

      $ cat SPECS/hello.spec

   2. Build the driver RPM:

      $ rpmbuild -ba SPECS/hello.spec

2. Create the Containerfile to append the driver into the operating system image. Use the rhel-bootc base image. The following is an example:

   # Copy the pre-built RPM into the build context
   COPY rpms/hello-<version>-<release>.el10.<arch>.rpm /tmp/

   Alternatively, you can fetch it from a network location.

   # Install the RPM.
   # The %post scripts in the RPM triggers 'depmod' automatically inside the image.
   RUN dnf install -y /tmp/hello-<version>-<release>.el10.<arch>.rpm && \
      	dnf clean all && \
       	rm /tmp/hello-<version>-<release>.el10.<arch>.rpm

3. Build and run your container image. See Building a container.

Verification

1. Reboot the system:

   $ sudo reboot

2. Confirm the kernel module is available on the system:

   $ sudo uname -r
   <kernel_version>
   
   $ sudo ls -l /lib/modules/<kernel_version>/extra/
   total 12
   -rwxr-xr-x. 1 root root 8512 Jan  1  1970 hello.ko

3. Ensure that the module dependencies are calculated before the image is finalized, otherwise modprobe might fail on the first boot.

   $ sudo depmod -a <kernel-version>

4. Confirm that you can load the module:

   $ sudo modprobe /lib/modules/<kernel_version>/extra/hello.ko
   $ sudo lsmod | grep hello
      hello        12288  0
   $ sudo rmmod hello
   $ sudo dmesg
    ..snip..
   [   84.738633] hello: loading out-of-tree module taints kernel.
   [   84.738684] hello: module verification failed: signature and/or required key missing -tainting kernel.
   [   84.740548] Hello, world!
   [  138.206978] Goodbye!

Procedure

1. Perform an update, switch, or rollback operation by using the bootc tool to stage the new container image. Otherwise, the soft re-executes the current userspace.

   $ sudo bootc update --soft-reboot=required --apply

2. Initiate a soft reboot, for example, while running bootc switch:

   $ sudo bootc switch --soft-reboot=required --apply quay.io/test_rh/soft-reboot:1

Procedure

1. For bootc to fetch updates from a registry that requires authentication, you must include a pull secret in a file. In the following example, the creds secret contains the registry pull secret.

   FROM quay.io/<namespace>/<image>:<tag>
   COPY containers-auth.conf /usr/lib/tmpfiles.d/link-podman-credentials.conf
   RUN --mount=type=secret,id=creds,required=true cp /run/secrets/creds /usr/lib/container-auth.json && \
     chmod 0600 /usr/lib/container-auth.json && \
     ln -sr /usr/lib/container-auth.json /etc/ostree/auth.json

2. Build an image by using podman build command:

   $ podman build --secret id=creds,src=$HOME/.docker/config.json

   Use a single pull secret for bootc and Podman by using a symlink to both locations to a common persistent file embedded in the container image, for example /usr/lib/container-auth.json.

   Injecting secrets by embedding them in a container build

   You can include secrets in the container image if the registry server is suitably protected. In some cases, embedding only bootstrap secrets into the container image is a viable pattern, especially alongside a mechanism for having a machine authenticate to a cluster. In this pattern, a provisioning tool, whether run as part of the host system or a container image, uses the bootstrap secret to inject or update other secrets, such as SSH keys, certificates, among others.

   Injecting secrets by using cloud metadata

   Most production Infrastructure as a Service (IaaS) systems support a metadata server or equivalent which can securely host secrets, particularly bootstrap secrets. Your container image can include tools such as cloud-init or ignition to fetch these secrets.

   Injecting secrets by embedding them in disk images

   You can embed bootstrap secrets only in disk images. For example, when you generate a cloud disk image from an input container image, such as AMI or OpenStack, the disk image can contain secrets that are effectively machine-local state. Rotating them requires an additional management tool or refreshing the disk images.

   Injecting secrets by using bare-metal installers

   Installer tools usually support injecting configuration through secrets.

   Injecting secrets through systemd credentials

   The systemd project has a credential concept for securely acquiring and passing credential data to systems and services, which applies in some deployment methodologies. See the systemd credentials documentation for more details.

Procedure

1. Configure a pull secret:

   %pre
   mkdir -p /etc/ostree
   cat > /etc/ostree/auth.json << 'EOF'
   {
           "auths": {
                   "quay.io": {
                           "auth": "<your secret here>"
                   }
           }
   }
   EOF
   %end

   With this configuration, the system pulls images from quay.io by using the provided authentication credentials, which are stored in /etc/ostree/auth.json.

2. Disable TLS for an insecure registry:

   %pre
   mkdir -p /etc/containers/registries.conf.d/
   cat > /etc/containers/registries.conf.d/local-registry.conf << 'EOF'
   
   [[registry]]
   location="[IP_Address]:5000"
   insecure=true
   EOF
   %end

   With this configuration, the system pulls container images from a registry that is not secured with TLS. You can use it in development or internal networks.

   You can also use %pre to:

   • Fetch data from the network by using binaries included in the installation environment, such as curl.

   • Inject trusted certificate authorities into the installation environment /etc/pki/ca-trust/source/anchors by using the update-ca-trust command.

     You can configure insecure registries similarly by modifying the /etc/containers directory. See the containers-auth.json(5) for more detailed information about the format and configurations of the auth.json file.

Procedure

1. Create a symbolic link between bootc and Podman to use a single pull secret. By creating the symbolic link, you ensure that both locations are present to a common persistent file embedded in the container image.

   FROM quay.io/<namespace>/<image>:<tag>
   COPY containers-auth.conf /usr/lib/tmpfiles.d/link-podman-credentials.conf
   RUN --mount=type=secret,id=creds,required=true cp /run/secrets/creds /usr/lib/container-auth.json && \
     chmod 0600 /usr/lib/container-auth.json && \
     ln -sr /usr/lib/container-auth.json /etc/ostree/auth.json

2. Create the containers-auth.conf file.

   # Make /run/containers/0/auth.json (a transient runtime file)
   # a symlink to our /usr/lib/container-auth.json (a persistent file)
   # which is also symlinked from /etc/ostree/auth.json.
   d /run/containers/0 0755 root root -
   L /run/user/0/containers/auth.json - - - - ../../../../usr/lib/container-auth.json

3. Build an image by using the podman build command:

   $ podman build --secret id=creds,src=$HOME/.docker/config.json

   When you run the containerfile, the following actions happen:

   • The Containerfile makes containers-auth.conf a transient runtime file.
   • It creates a symbolic link to the containers-auth.conf.

   • It also creates a persistent file, which is also symbolic linked from /etc/ostree/auth.json.

     Podman does not have system-wide credentials. Podman accepts the containers-auth locations that are underneath the following directories:

   • /run: The content of this directory vanishes on reboot, which is not required.

   • /root: Part of root home directory, which is local mutable state by default.

     To unify bootc and Podman credentials, use a single default global pull secret for both bootc and Podman. The following container build is an example to unify the bootc and the Podman credentials. The example expects a secret named creds to contain the registry pull secret to build.