Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Using image mode for RHEL to build, deploy, and manage operating systems

Red Hat Enterprise Linux 10

Using RHEL bootc images on Red Hat Enterprise Linux 10

Red Hat Customer Content Services

Abstract

By using RHEL bootc images, you can build, deploy, and manage the operating system as a container. With image mode for RHEL, you can manage your application as well as the underlying OS in a single container-native workflow.

Providing feedback on Red Hat documentation

We appreciate your feedback on our documentation. Let us know how we can improve it.

Submitting feedback through Jira (account required)

  1. Log in to the Jira website.
  2. Click Create in the top navigation bar
  3. Enter a descriptive title in the Summary field.
  4. Enter your suggestion for improvement in the Description field. Include links to the relevant parts of the documentation.
  5. Click Create at the bottom of the dialogue.

Chapter 1. Introducing image mode for RHEL

Use image mode for RHEL to build, test, and deploy operating systems by using the same tools and techniques as application containers. Image mode for RHEL is available by using the registry.redhat.io/rhel10/rhel-bootc bootc image. The RHEL bootc images differ from the existing application Universal Base Images (UBI) in that they contain additional components necessary to boot that were traditionally excluded, such as, kernel, initrd, boot loader, firmware, among others.

Note

The rhel-bootc and user-created containers based on rhel-bootc container image are subject to the Red Hat Enterprise Linux end user license agreement (EULA). You are not allowed to publicly redistribute these images.

Figure 1.1. Building, deploying, and managing operating system by using image mode for RHEL

639 RHEL Bootable Container Bifrost 0524 1

Red Hat provides bootc image for the following computer architectures:

  • AMD and Intel 64-bit architectures (x86-64-v2)
  • The 64-bit ARM architecture (ARMv8.0-A)
  • IBM Power Systems 64-bit Little Endian architecture (ppc64le)
  • IBM Z 64-bit architecture (s390x)
Warning

Anaconda may not work correctly on s390x and ppc64le architectures. For more information, see the Release Notes.

The benefits of image mode for RHEL occur across the lifecycle of a system. The following list contains some of the most important advantages:

Container images are easier to understand and use than other image formats and are fast to build
Containerfiles, also known as Dockerfiles, provide a straightforward approach to defining the content and build instructions for an image. Container images are often significantly faster to build and iterate on compared to other image creation tools.
Consolidate process, infrastructure, and release artifacts
As you distribute applications as containers, you can use the same infrastructure and processes to manage the underlying operating system.
Immutable updates
Just as containerized applications are updated in an immutable way, with image mode for RHEL, the operating system is also. You can boot into updates and roll back when needed in the same way that you use rpm-ostree systems.
Portability across hybrid cloud environments
You can use bootc images across physical, virtualized, cloud, and edge environments.

Although containers provide the foundation to build, transport, and run images, it is important to understand that after you deploy these bootc images, either by using an installation mechanism, or you convert them to a disk image, the system does not run as a container.

  • Bootc supports the following container image formats and disk image formats:
Table 1.1. bootc supported image types
Image typeTarget environment

OCI container format

Physical, virtualized, cloud, and edge environments.

ami

Amazon Machine Image.

qcow2 (default)

QEMU.

vmdk

VMDK for vSphere.

anaconda-iso

An unattended Anaconda installer that installs to the first disk found.

raw

Unformatted raw disk. Also supported in QEMU and Libvirt

vhd

VHD for Virtual PC, among others.

gce

Google Compute Engine (GCE) environment.

Containers help streamline the lifecycle of a RHEL system by offering the following possibilities:

Building container images
You can configure your operating system at a build time by modifying the Containerfile. Image mode for RHEL is available by using the registry.redhat.io/rhel10/rhel-bootc container image. You can use Podman, OpenShift Container Platform, or other standard container build tools to manage your containers and container images. You can automate the build process by using CI/CD pipelines.
Versioning, mirroring, and testing container images
You can version, mirror, introspect, and sign your derived bootc image by using any container tools such as Podman or OpenShift Container Platform.
Deploying container images to the target environment

You have several options on how to deploy your image:

  • Anaconda: is the installation program used by RHEL. You can deploy all image types to the target environment by using Anaconda and Kickstart to automate the installation process.
  • bootc-image-builder: is a containerized tool that converts the container image to different types of disk images, and optionally uploads them to an image registry or object storage.
  • bootc: is a tool responsible for fetching container images from a container registry and installing them to a system, updating the operating system, or switching from an existing ostree-based system. The RHEL bootc image contains the bootc utility by default and works with all image types. It is intended to supersede rpm-ostree.
Updating your operating system
The system supports in-place transactional updates with rollback after deployment. Automatic updates are on by default. A systemd service unit and systemd timer unit files check the container registry for updates and apply them to the system. As the updates are transactional, a reboot is required. For environments that require more sophisticated or scheduled rollouts, disable auto updates and use the bootc utility to update your operating system.

RHEL has two deployment modes. Both provide the same stability, reliability, and performance during deployment. See their differences:

  1. Package mode: the operating system uses RPM packages and is updated by using the dnf package manager. The root filesystem is mutable. However, the operating system cannot be managed as a containerized application.
  2. Image mode: a container-native approach to build, deploy, and manage RHEL. The same RPM packages are delivered as a base image and updates are deployed as a container image. The root filesystem is immutable by default, except for /etc and /var, with most content coming from the container image.

You can choose to use either the Image mode or the Package mode deployment to build, test, and share your operating system. Image mode additionally enables you to manage your operating system in the same way as any other containerized application.

1.1. Prerequisites

Chapter 2. Building and testing RHEL bootc images

You can build and test RHEL container images using Podman and Containerfiles to create, customize, and share bootable RHEL system images across environments efficiently. You can also use other tools, for example, OpenShift Container Platform. For more examples of configuring RHEL systems by using containers, see the rhel-bootc-examples repository.

Figure 2.1. Building an image by using instructions from a Containerfile, testing the container, pushing an image to a registry, and sharing it with others

639 RHEL Bootable Container Bifrost 0524 2

A general Containerfile structure is the following: 

FROM registry.redhat.io/rhel10/rhel-bootc:latest

RUN dnf -y install [software] [dependencies] && dnf clean all

ADD [application]
ADD [configuration files]

RUN [config scripts]
Copy to Clipboard

The available commands that are usable inside a Containerfile and a Dockerfile are equivalent.

However, the following commands in a Containerfile are ignored when the rhel-10-bootc image is installed to a system:

  • ENTRYPOINT and CMD (OCI: Entrypoint/Cmd): you can set CMD /sbin/init instead.
  • ENV (OCI: Env): change the systemd configuration to configure the global system environment.
  • EXPOSE (OCI: exposedPorts): it is independent of how the system firewall and network function at runtime.
  • USER (OCI: User): configure individual services inside the RHEL bootc to run as unprivileged users instead.

The rhel-10-bootc container image reuses the OCI image format.

  • The rhel-10-bootc container image ignores the container config section (Config) when it is installed to a system.
  • The rhel-10-bootc container image does not ignore the container config section (Config) when you run this image by using container runtimes such as podman or docker.

2.1. Building a container image

To build an image by using instructions from a Containerfile, use the podman build command.

Prerequisites

  • The container-tools meta-package is installed.

Procedure

  1. Create a Containerfile: 

    $ cat Containerfile
FROM registry.redhat.io/rhel10/rhel-bootc:latest
RUN dnf -y install cloud-init && \
    ln -s ../cloud-init.target /usr/lib/systemd/system/default.target.wants && \
    dnf clean all
    Copy to Clipboard

    This Containerfile example adds the cloud-init tool, so it automatically fetches SSH keys and can run scripts from the infrastructure and also gather configuration and secrets from the instance metadata. For example, you can use this container image for pre-generated AWS or KVM guest systems.

  2. Build the <image> image by using Containerfile in the current directory: 

    $ podman build -t quay.io/<namespace>/<image>:<tag> .
    Copy to Clipboard

Verification

  • List all images: 

    $ podman images
REPOSITORY                                  TAG      IMAGE ID       CREATED              SIZE
localhost/<image>                           latest   b28cd00741b3   About a minute ago   2.1 GB
    Copy to Clipboard

2.2. Running a container image

Use the podman run command to run and test your container.

Prerequisites

  • The container-tools meta-package is installed.

Procedure

  • Run the container named mybootc based on the quay.io/<namespace>/<image>:<tag> container image: 

    $ podman run -it --rm --name mybootc quay.io/<namespace>/<image>:<tag> /bin/bash
    Copy to Clipboard
    • The -i option creates an interactive session. Without the -t option, the shell stays open, but you cannot type anything to the shell.
    • The -t option opens a terminal session. Without the -i option, the shell opens and then exits.
    • The --rm option removes the quay.io/<namespace>/<image>:<tag> container image after the container exits.

Verification

  • List all running containers: 

    $ podman ps
CONTAINER ID  IMAGE                                    COMMAND          CREATED        STATUS            PORTS   NAMES
7ccd6001166e  quay.io/<namespace>/<image>:<tag>  /sbin/init  6 seconds ago  Up 5 seconds ago          mybootc
    Copy to Clipboard

2.3. Building derived bootable images by using multi-stage builds

The deployment image include only the application and its required runtime, without adding any build tools or unnecessary libraries. To achieve this, use a two-stage Containerfile: one stage for building the artifacts and another for hosting the application.

With multi-stage builds, you use multiple FROM instructions in your Containerfile. Each FROM instruction can use a different base, and each of them begins a new stage of the build. You can selectively copy artifacts from one stage to another, and exclude everything you do not need in the final image.

Multi-stage builds offer several advantages:

Smaller image size
By separating the build environment from the runtime environment, only the necessary files and dependencies are included in the final image, significantly reducing its size.
Improved security
Since build tools and unnecessary libraries are excluded from the final image, the attack surface is reduced, leading to a more secure container.
Optimized performance
A smaller image size means faster download, deployment, and startup times, improving the overall efficiency of the containerized application.
Simplified maintenance
With the build and runtime environments separated, the final image is cleaner and easier to maintain, containing only what is needed to run the application.
Cleaner builds
Multi-stage builds help avoid clutter from intermediate files, which could accumulate during the build process, ensuring that only essential artifacts make it into the final image.
Resource efficiency
The ability to build in one stage and discard unnecessary parts minimizes the use of storage and bandwidth during deployment.
Better layer caching
With clearly defined stages, Podman can efficiently cache the results of previous stages, potentially speeding up future builds.

The following Containerfile consists of two stages. The first stage is typically named builder and it compiles a golang binary. The second stage copies the binary from the first stage. The default working directory for the go-toolset builder is opt/ap-root/src. 

FROM registry.access.redhat.com/ubi10/go-toolset:latest as builder
RUN echo 'package main; import "fmt"; func main() { fmt.Println("hello world") }' > helloworld.go
RUN go build helloworld.go

FROM registry.redhat.io/rhel10/rhel-bootc:latest
COPY --from=builder /opt/app-root/src/helloworld /
CMD ["/helloworld"]
Copy to Clipboard

As a result, the final container image includes the helloworld binary but no data from the previous stage.

You can also use multi-stage builds to perform the following scenarios:

Stopping at a specific build stage
When you build your image, you can stop at a specified build stage. For example: 
$ podman build --target build -t hello .
Copy to Clipboard

For example, you can use this approach to debugging a specific build stage.

Using an external image as a stage
You can use the COPY --from instruction to copy from a separate image either using the local image name, a tag available locally or on a container registry, or a tag ID. For example: 
COPY --from=<image> <source_path> <destination_path>
Copy to Clipboard
Using a previous stage as a new stage
You can continue where a previous stage ended by using the FROM instruction. From example: 
FROM ubi10 AS stage1
[...]

FROM stage1 AS stage2
[...]

FROM ubi10 AS final-stage
[...]
Copy to Clipboard

2.4. Pushing a container image to the registry

Use the podman push command to push an image to your own, or a third party, registry and share it with others. The following procedure uses the Red Hat Quay registry.

Prerequisites

  • The container-tools meta-package is installed.
  • An image is built and available on the local system.
  • You have created the Red Hat Quay registry. For more information see Proof of Concept - Deploying Red Hat Quay.

Procedure

  • Push the quay.io/<namespace>/<image>:<tag> container image from your local storage to the registry: 

    $ podman push quay.io/<namespace>/<image>:<tag>
    Copy to Clipboard

Chapter 3. Building and managing logically bound images

Logically bound images give you support for container images that are lifecycle bound to the base bootc image. This helps combine different operational processes for applications and operating systems, and the container application images are referenced from the base image as image files or an equivalent. Consequently, you can manage multiple container images for system installations.

You can use containers for lifecycle-bound workloads, such as security agents and monitoring tools. You can also upgrade such workloads by using the bootc upgrade command.

3.1. Logically bound images

Logically bound images enable an association of the container application images to a base bootc system image. By default, application containers as executed by, for example, podman have a lifecycle independent of host upgrades; they can be added or removed at any time, and are typically fetched on demand after booting if the container image is not present.

The logically bound images offer a key benefit that the application containers bound in this way have a lifecycle tied to the host upgrade and are available before the host reboots into the new operating system. The container images bound in this way will be present as long as a bootc container references them.

The following are examples for lifecycle bound workloads which are usually not updated outside of the host:

  • Logging, for example, journald→remote log forwarder container
  • Monitoring, for example, Prometheus node_exporter
  • Configuration management agents
  • Security agents

For these types of workloads it is often important that the container start from a very early stage in the boot process before e.g. networking might be available. Logically bound images enable you to start containers (often via systemd) with the the same reliability of using ExecStart= of a binary in the base bootc image.

The term logically bound can also be contrasted with another model of physically bound images, where application container content is physically stored in the bootc container image. A key advantage for logically bound over physically bound is that tou can update the bootc system without re-downloading application container images which are not changed.

When using logically bound images, you must manage multiple container images for the system to install the logically bound images. This is an advantage and also a disadvantage. For example, for a disconnected or offline installation, you must mirror all the containers, not just one. The application images are only referenced from the base image as .image files or an equivalent.

Logically bound images installation
When you run bootc install, logically bound images must be present in the default /var/lib/containers container store. The images will be copied into the target system and present directly at boot, alongside the bootc base image.
Logically bound images lifecycle
Logically bound images are referenced by the bootable container and have guaranteed availability when the bootc based server starts. The image is always upgraded by using bootc upgrade and is available as read-only to other processes, such as Podman.
Logically bound images management on upgrades, rollbacks, and garbage collection
  • During upgrades, the logically bound images are managed exclusively by bootc.
  • During rollbacks, the logically bound images corresponding to rollback deployments are retained.
  • bootc performs garbage collection of unused bound images.

3.2. Using logically bound images

Each logically bound image is defined in a Podman Quadlet .image or .container file. To use logically bound images, follow the steps:

Prerequisites

  • The container-tools meta-package is installed.

Procedure

  1. Select the image that you want to logically bound.

  2. Create a Containerfile: 

    $ cat Containerfile
FROM quay.io/<namespace>/<image>:latest
COPY ./<app_1>.image /usr/share/containers/systemd/<app_1>.image
COPY ./<app_2>.container /usr/share/containers/systemd/<app_2>.container

RUN ln -s /usr/share/containers/systemd/<app_1>.image \
	/usr/lib/bootc/bound-images.d/<app_1>.image && \
    ln -s /usr/share/containers/systemd/<app_2>.container \
    	/usr/lib/bootc/bound-images.d/<app_2>.container
    Copy to Clipboard

  3. In the .container definition, use: 

    GlobalArgs=--storage-opt=additionalimagestore=/usr/lib/bootc/storage
    Copy to Clipboard

    In the Containerfile example, the image is selected to be logically bound by creating a symlink in the /usr/lib/bootc/bound-images.d directory pointing to either an .image or a .container file.

  4. Run the bootc upgrade command. 

    $ bootc upgrade
    Copy to Clipboard

    The bootc upgrade performs the following overall steps:

    1. Fetches the new base image from the image repository. See Configuring container pull secrets.
    2. Reads the new base image root file system to discover logically bound images.
    3. Automatically pulls any discovered logically bound images defined in the new bootc image into the bootc-owned /usr/lib/bootc/storage image storage.

  5. Make the bound images become available to container runtimes such as Podman. For that, you must explicitly configure bound images to point to the bootc storage as an "additional image store". For example: 

    podman --storage-opt=additionalimagestore=/usr/lib/bootc/storage run <image>
    Copy to Clipboard
Important

Do not attempt to globally enable the /usr/lib/bootc/storage image storage in /etc/containers/storage.conf. Only use the bootc storage for logically bound images.

The bootc image store is owned by bootc. The logically bound images will be garbage collected when they are no longer referenced by a file in the /usr/lib/bootc/bound-images.d directory.

Chapter 4. Creating bootc compatible base disk images with bootc-image-builder

The bootc-image-builder is a containerized tool to create disk images from bootc images. You can use the images that you build to deploy disk images in different environments, such as the edge, server, and clouds.

4.1. Introducing image mode for RHEL for bootc-image-builder

With the bootc-image-builder tool, you can convert bootc images into disk images for a variety of different platforms and formats. Converting bootc images into disk images is equivalent to installing a bootc. After you deploy these disk images to the target environment, you can update them directly from the container registry.

Note

Building base disk images which come from private registries by using bootc-image-builder is not supported in this release.

The bootc-image-builder tool supports generating the following image types:

  • Disk image formats, such as ISO, suitable for disconnected installations.

  • Virtual disk images formats, such as:

    • QEMU copy-on-write (QCOW2)
    • Amazon Machine Image (AMI)/ — Raw
    • Virtual Machine Image (VMI)

Deploying from a container image is beneficial when you run VMs or servers because you can achieve the same installation result. That consistency extends across multiple different image types and platforms when you build them from the same container image. Consequently, you can minimize the effort in maintaining operating system images across platforms. You can also update systems that you deploy from these disk images by using the bootc tool, instead of re-creating and uploading new disk images with bootc-image-builder.

Note

Generic base container images do not include any default passwords or SSH keys. Also, the disk images that you create by using the bootc-image-builder tool do not contain the tools that are available in common disk images, such as cloud-init. These disk images are transformed container images only.

Although you can deploy a rhel-10-bootc image directly, you can also create your own customized images that are derived from this bootc image. The bootc-image-builder tool takes the rhel-10-bootc OCI container image as an input.

4.2. Installing bootc-image-builder

The bootc-image-builder is intended to be used as a container and it is not available as an RPM package in RHEL. To access it, follow the procedure.

Prerequisites

  • The container-tools meta-package is installed. The meta-package contains all container tools, such as Podman, Buildah, and Skopeo.
  • You are authenticated to registry.redhat.io. For details, see Red Hat Container Registry Authentication.

Procedure

  1. Login to authenticate to registry.redhat.io: 

    $ sudo podman login registry.redhat.io
    Copy to Clipboard

  2. Install the bootc-image-builder tool: 

    $ sudo podman pull registry.redhat.io/rhel10/bootc-image-builder
    Copy to Clipboard

Verification

  • List all images pulled to your local system: 

    $ sudo podman images
REPOSITORY                                    TAG         IMAGE ID      CREATED       SIZE
registry.redhat.io/rhel10/bootc-image-builder  latest      b361f3e845ea  24 hours ago  676 MB
    Copy to Clipboard

4.3. Creating QCOW2 images by using bootc-image-builder

Build a RHEL bootc image into a QEMU Disk Images (QCOW2) image for the architecture that you are running the commands on.

The RHEL base image does not include a default user. Optionally, you can inject a user configuration with the --config option to run the bootc-image-builder container. Alternatively, you can configure the base image with cloud-init to inject users and SSH keys on first boot. See Users and groups configuration - Injecting users and SSH keys by using cloud-init.

Prerequisites

  • You have Podman installed on your host machine.
  • You have virt-install installed on your host machine.
  • You have root access to run the bootc-image-builder tool, and run the containers in --privileged mode, to build the images.

Procedure

  1. Optional: Create a config.toml to configure user access, for example: 

    [[customizations.user]]
name = "user"
password = "pass"
key = "ssh-rsa AAA ... user@email.com"
groups = ["wheel"]
    Copy to Clipboard

  2. Run bootc-image-builder. Optionally, if you want to use user access configuration, pass the config.toml as an argument.

    Note

    If you do not have the container storage mount and --local image options, your image must be public.

    1. The following is an example of creating a public QCOW2 image: 

      $ sudo podman run \
    --rm \
    -it \
    --privileged \
    --pull=newer \
    --security-opt label=type:unconfined_t \
    -v ./config.toml:/config.toml \
    -v ./output:/output \
    registry.redhat.io/rhel10/bootc-image-builder:latest \
    --type qcow2 \
    --config /config.toml \
  quay.io/<namespace>/<image>:<tag>
      Copy to Clipboard

    2. The following is an example of creating a private QCOW2 image: 

      $ sudo podman run \
    --rm \
    -it \
    --privileged \
    --pull=newer \
    --security-opt label=type:unconfined_t \
    -v $(pwd)/config.toml:/config.toml:ro \
    -v $(pwd)/output:/output \
    -v /var/lib/containers/storage:/var/lib/containers/storage \
    registry.redhat.io/rhel10/bootc-image-builder:latest \
    --local
    --type qcow2 \
    quay.io/<namespace>/<image>:<tag>
      Copy to Clipboard

      You can find the .qcow2 image in the output folder.

4.4. Creating VMDK images by using bootc-image-builder

Create a Virtual Machine Disk (VMDK) from a bootc image and use it within VMware’s virtualization platforms, such as vSphere, or use the Virtual Machine Disk (VMDK) in VirtualBox.

Prerequisites

  • You have Podman installed on your host machine.
  • You have authenticated to the Red Hat Registry by using the podman login registry.redhat.io.
  • You have pulled the rhel10/bootc-image-builder container image.

Procedure

  1. Create a Containerfile with the following content: 

    FROM registry.redhat.io/rhel10/rhel-bootc:latest
RUN dnf -y install cloud-init open-vm-tools && \
ln -s ../cloud-init.target /usr/lib/systemd/system/default.target.wants && \
rm -rf /var/{cache,log} /var/lib/{dnf,rhsm} && \
systemctl enable vmtoolsd.service
    Copy to Clipboard

  2. Build the bootc image: 

    # podman build . -t localhost/rhel-bootc-vmdk
    Copy to Clipboard

  3. Create a VMDK file from the previously created bootc image: 

    # podman run \
--rm \
-it \
--privileged \
-v /var/lib/containers/storage:/var/lib/containers/storage \
-v ./output:/output \
--security-opt label=type:unconfined_t \
--pull newer \
registry.redhat.io/rhel10/bootc-image-builder:latest
--local \
--type vmdk \
localhost/rhel-bootc-vmdk:latest
    Copy to Clipboard

    The --local option uses the local container storage to source the originating image to produce the VMDK instead of a remote repository.

A VMDK disk file for the bootc image is stored in the output/vmdk directory.

4.5. Creating GCE images by using bootc-image-builder

Build a RHEL bootc image into a GCE image for the architecture that you are running the commands on. The RHEL base image does not include a default user. Optionally, you can inject a user configuration with the --config option to run the bootc-image-builder container. Alternatively, you can configure the base image with cloud-init to inject users and SSH keys on first boot. See Users and groups configuration - Injecting users and SSH keys by using cloud-init.

Prerequisites

  • You have Podman installed on your host machine.
  • You have root access to run the bootc-image-builder tool, and run the containers in --privileged mode, to build the images.

Procedure

  1. Optional: Create a config.toml to configure user access, for example: 

    [[customizations.user]]
name = "user"
password = "pass"
key = "ssh-rsa AAA ... user@email.com"
groups = ["wheel"]
    Copy to Clipboard

  2. Run bootc-image-builder. Optionally, if you want to use user access configuration, pass the config.toml as an argument.

    Note

    If you do not have the container storage mount and --local image options, your image must be public.

    1. The following is an example of creating a gce image: 

      $ sudo podman run \
    --rm \
    -it \
    --privileged \
    --pull=newer \
    --security-opt label=type:unconfined_t \
    -v ./config.toml:/config.toml \
    -v ./output:/output \
    registry.redhat.io/rhel10/bootc-image-builder:latest \
    --type gce \
    --config /config.toml \
  quay.io/<namespace>/<image>:<tag>
      Copy to Clipboard

      You can find the gce image in the output folder.

4.6. Creating AMI images by using bootc-image-builder and uploading it to AWS

Create an Amazon Machine Image (AMI) from a bootc image and use it to launch an Amazon Web Service EC2 (Amazon Elastic Compute Cloud) instance.

Prerequisites

  • You have Podman installed on your host machine.
  • You have an existing AWS S3 bucket within your AWS account.
  • You have root access to run the bootc-image-builder tool, and run the containers in --privileged mode, to build the images.
  • You have the vmimport service role configured on your account to import an AMI into your AWS account.

Procedure

  1. Create a disk image from the bootc image.

    • Configure the user details in the Containerfile. Make sure that you assign it with sudo access.
    • Build a customized operating system image with the configured user from the Containerfile. It creates a default user with passwordless sudo access.

  2. Optional: Configure the machine image with cloud-init. See Users and groups configuration - Injecting users and SSH keys by using cloud-init. The following is an example: 

    FROM registry.redhat.io/rhel10/rhel-bootc:latest

RUN dnf -y install cloud-init && \
    ln -s ../cloud-init.target /usr/lib/systemd/system/default.target.wants && \
    rm -rf /var/{cache,log} /var/lib/{dnf,rhsm}
    Copy to Clipboard
    Note

    You can also use cloud-init to add users and additional configuration by using instance metadata.

  3. Build the bootc image. For example, to deploy the image to an x86_64 AWS machine, use the following commands: 

    $ podman build -t quay.io/<namespace>/<image>:<tag> .
$ podman push quay.io/<namespace>/<image>:<tag> .
    Copy to Clipboard

  4. Use the bootc-image-builder tool to create an AMI from the bootc container image. 

    $ sudo podman run \
  --rm \
  -it \
  --privileged \
  --pull=newer \
  -v $HOME/.aws:/root/.aws:ro \
  --env AWS_PROFILE=default \
  registry.redhat.io/rhel10/bootc-image-builder:latest \
  --type ami \
  --aws-ami-name rhel-bootc-x86 \
  --aws-bucket rhel-bootc-bucket \
  --aws-region us-east-1 \
quay.io/<namespace>/<image>:<tag>
    Copy to Clipboard
    Note

    The following flags must be specified all together. If you do not specify any flag, the AMI is exported to your output directory.

    • --aws-ami-name - The name of the AMI image in AWS
    • --aws-bucket - The target S3 bucket name for intermediate storage when you are creating the AMI

    • --aws-region - The target region for AWS uploads

      The bootc-image-builder tool builds an AMI image and uploads it to your AWS s3 bucket by using your AWS credentials to push and register an AMI image after building it.

Next steps

Additional resources

4.7. Creating raw disk images by using bootc-image-builder

You can convert a bootc image to a raw image with an MBR or GPT partition table by using bootc-image-builder. The RHEL base image does not include a default user, so optionally, you can inject a user configuration with the --config option to run the bootc-image-builder container. Alternatively, you can configure the base image with cloud-init to inject users and SSH keys on first boot. See Users and groups configuration - Injecting users and SSH keys by using cloud-init.

Prerequisites

  • You have Podman installed on your host machine.
  • You have root access to run the bootc-image-builder tool, and run the containers in --privileged mode, to build the images.
  • You have pulled your target container image in the container storage.

Procedure

  1. Optional: Create a config.toml to configure user access, for example: 

    [[customizations.user]]
name = "user"
password = "pass"
key = "ssh-rsa AAA ... user@email.com"
groups = ["wheel"]
    Copy to Clipboard

  2. Run bootc-image-builder. If you want to use user access configuration, pass the config.toml as an argument: 

    $ sudo podman run \
    --rm \
    -it \
    --privileged \
    --pull=newer \
    --security-opt label=type:unconfined_t \
    -v /var/lib/containers/storage:/var/lib/containers/storage \
    -v ./config.toml:/config.toml \
    -v ./output:/output \
    registry.redhat.io/rhel10/bootc-image-builder:latest \
    --local \
    --type raw \
    --config /config.toml \
  quay.io/<namespace>/<image>:<tag>
    Copy to Clipboard

    You can find the .raw image in the output folder.

4.8. Creating ISO images by using bootc-image-builder

You can use bootc-image-builder to create an ISO from which you can perform an offline deployment of a bootable container.

Prerequisites

  • You have Podman installed on your host machine.
  • Your host system is subscribed or you have injected repository configuration by using bind mounts to ensure the image build process can fetch RPMs.
  • You have root access to run the bootc-image-builder tool, and run the containers in --privileged mode, to build the images.

Procedure

  1. Optional: Create a config.toml to which overrides the default embedded Kickstart which performs an automatic installation. 

    [customizations.installer.kickstart]
contents = """
text --non-interactive
zerombr
clearpart --all --initlabel --disklabel=gpt
autopart --noswap --type=lvm
network --bootproto=dhcp --device=link --activate --onboot=on
"""
    Copy to Clipboard

  2. Run bootc-image-builder. If you do not want to add any configuration, omit the -v $(pwd)/config.toml:/config.toml argument. 

    $ sudo podman run \
    --rm \
    -it \
    --privileged \
    --pull=newer \
    --security-opt label=type:unconfined_t \
    -v /var/lib/containers/storage:/var/lib/containers/storage \
    -v $(pwd)/config.toml:/config.toml \
    -v $(pwd)/output:/output \
    registry.redhat.io/rhel10/bootc-image-builder:latest \
    --type iso \
    --config /config.toml \
  quay.io/<namespace>/<image>:<tag>
    Copy to Clipboard

    You can find the .iso image in the output folder.

4.9. Using bootc-image-builder to build ISO images with a Kickstart file

You can use a Kickstart file to configure various parts of the installation process, such as setting up users, customizing partitioning, and adding an SSH key. You can include the Kickstart file in an ISO build to configure any part of the installation process, except the deployment of the base image. For ISOs with bootc container base images, you can use a Kickstart file to configure anything except the ostreecontainer command.

For example, you can use a Kickstart to perform either a partial installation, a full installation, or even omit the user creation. Use bootc-image-builder to build an ISO image that contains the custom Kickstart to configure your installation process.

Prerequisites

  • You have Podman installed on your host machine.
  • You have root access to run the bootc-image-builder tool, and run the containers in --privileged mode, to build the images.

Procedure

  1. Create your Kickstart file. The following Kickstart file is an example of a fully unattended Kickstart file configuration that contains user creation, and partition instructions. 

    [customizations.installer.kickstart]
contents = """
lang en_GB.UTF-8
keyboard uk
timezone CET

user --name <user> --password <password> --plaintext --groups <groups>
sshkey --username <user> ssh-<type> <public key>
rootpw --lock

zerombr
clearpart --all --initlabel
autopart --type=plain
reboot --eject
"""
    Copy to Clipboard
  2. Save the Kickstart configuration in the toml format to inject the Kickstart content. For example, config.toml.

  3. Run bootc-image-builder, and include the Kickstart file configuration that you want to add to the ISO build. The bootc-image-builder automatically adds the ostreecontainer command that installs the container image. 

    $ sudo podman run \
    --rm \
    -it \
    --privileged \
    --pull=newer \
    --security-opt label=type:unconfined_t \
    -v /var/lib/containers/storage:/var/lib/containers/storage \
    -v $(pwd)/config.toml:/config.toml \
    -v $(pwd)/output:/output \
    registry.redhat.io/rhel10/bootc-image-builder:latest \
    --type iso \
    --config /config.toml \
  quay.io/<namespace>/<image>:<tag>
    Copy to Clipboard

    You can find the .iso image in the output folder.

4.10. Building disk images of image-mode RHEL with advanced partitioning

Create image mode disk images with advanced partitioning by using bootc-image-builder. The image mode disk images that you create of image mode RHEL with custom mount points, include custom mount options, LVM-based partitions and LVM-based SWAP. With that you can, for example, change the size of the / and the /boot directories by using a config.toml file. When installing the RHEL image mode on bare-metal machines, you can benefit from all partitioning features available on Anaconda.

Prerequisites

  • You have Podman installed on your host machine.
  • You have virt-install installed on your host machine.
  • You have root access to run the bootc-image-builder tool, and run the containers in --privileged mode, to build the images.

Procedure

  1. Create a config.toml to configure custom mount options, for example: 

    [[customizations.filesystem]]
mountpoint = "/"
minsize = "10 GiB"

[[customizations.filesystem]]
mountpoint = "/var/data"
minsize = "20 GiB"
    Copy to Clipboard

  2. Run bootc-image-builder, passing the config.toml as an argument.

    Note

    If you do not have the container storage mount and --local image options, your image must be public.

    The following is an example of creating a public image: 

    $ sudo podman run \
    --rm \
    -it \
    --privileged \
    --pull=newer \
    --security-opt label=type:unconfined_t \
    -v ./config.toml:/config.toml \
    -v ./output:/output \
    registry.redhat.io/rhel10/bootc-image-builder:latest \
    --type <image_type> \
    --config config.toml \
  quay.io/<namespace>/<image>:<tag>
    Copy to Clipboard

Next steps

Additional resources

4.11. Verification and troubleshooting

If you have any issues configuring the requirements for your AWS image, see the following documentation
For more details on users, groups, SSH keys, and secrets, see

Chapter 5. Best practices for running containers using local sources

You can access content hosted in an internal registry that requires a custom Transport Layer Security (TLS) root certificate, when running RHEL bootc images.

There are two options available to install content to a container by using only local resources:

  • Bind mounts: Use for example -v /etc/pki:/etc/pki to override the container’s store with the host’s.
  • Derived image: Create a new container image with your custom certificates by building it using a Containerfile.

You can use the same techniques to run a bootc-image-builder container or a bootc container when appropriate.

5.1. Importing custom certificate to a container by using bind mounts

Use bound mounts to override the container’s store with the host’s.

Procedure

  • Run bootc-image-builder and use a bind mount, for example -v /etc/pki:/etc/pki, to override the container’s store with the host’s: 

    # podman run \
  --rm \
  -it \
  --privileged \
  --pull=newer \
  --security-opt label=type:unconfined_t \
  -v $(pwd)/output:/output \
  -v /etc/pki:/etc/pki \
  registry.redhat.io/rhel10/bootc-image-builder:latest \
  --type iso \
  --config /config.toml \
  quay.io/<namespace>/<image>:<tag>
    Copy to Clipboard

Verification

  • The disk image build process should now be able to access internal certificates.

5.2. Importing custom certificates to an image by using Containerfile

Include instructions to install custom certificate roots with a Containerfile.

Procedure

  1. Create a Containerfile: 

    FROM <internal_repository>/<image>
# Add certificate to the input set of anchors
COPY additional-certificate-root.pem /etc/pki/ca-trust/source/anchors
RUN update-ca-trust
    Copy to Clipboard

  2. Build the custom image: 

    # podman build -t <your_image> .
    Copy to Clipboard

  3. Run the <your_image>: 

    # podman run -it --rm <your_image>
    Copy to Clipboard

Verification

  • Verify your certificate is in the generated merged store: 

    # cat etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
...
    Copy to Clipboard

Chapter 6. Deploying the RHEL bootc images

You can deploy the rhel-bootc container image by using the following different mechanisms.

  • Anaconda
  • bootc-image-builder
  • bootc install

The following bootc image types are available:

  • Disk images that you generated by using the bootc image-builder such as:

    • QCOW2 (QEMU copy-on-write, virtual disk)
    • Raw (Mac Format)
    • AMI (Amazon Cloud)
    • ISO: Unattended installation method, by using an USB Sticks or Install-on-boot.

After you have created a layered image that you can deploy, there are several ways that the image can be installed to a host:

  • You can use RHEL installer and Kickstart to install the layered image to a bare metal system, by using the following mechanisms:

    • Deploy by using USB
    • PXE
  • You can also use bootc-image-builder to convert the container image to a bootc image and deploy it to a bare metal or to a cloud environment.

The installation method happens only one time. After you deploy your image, any future updates will apply directly from the container registry as the updates are published.

Figure 6.1. Deploying a bootc image by using a basic build installer bootc install, or deploying a container image by using Anaconda and Kickstart

639 RHEL Bootable Container Bifrost 0524 3

Figure 6.2. Using bootc-image-builder to create disk images from bootc images and deploying disk images in different environments, such as the edge, servers, and clouds by using Anaconda, bootc-image-builder or bootc install

639 RHEL Bootable Container Bifrost 0524 4

6.1. Deploying a container image by using KVM with a QCOW2 disk image

After creating a QEMU disk image from a RHEL bootc image by using the bootc-image-builder tool, you can use a virtualization software to boot it.

Prerequisites

Procedure

  • By using libvirt, create a virtual machine (VM) with the disk image that you previously created from the container image. For more details, see Creating virtual machines by using the command line.

    • The following example uses virt-install to create a VM. Replace <qcow2/disk.qcow2> with the path to your QCOW2 file: 

      $ sudo virt-install \
  --name bootc \
  --memory 4096 \
  --vcpus 2 \
  --disk <qcow2/disk.qcow2> \
  --import
      Copy to Clipboard

Verification

Next steps

6.2. Deploying a container image and creating a RHEL virtual machine in vSphere

After creating a Virtual Machine Disk (VMDK) from a RHEL bootc image by using the bootc-image-builder tool, you can deploy it to VMware vSphere by using the vSphere GUI client. The deployment creates a VM which can be customized further before booting.

Prerequisites

  • You created a container image. See Creating QCOW2 images by using bootc-image-builder.
  • You pushed the container image to an accessible repository.

  • You configured the govc VMware CLI tool client. To use the govc VMware CLI tool client, you must set the following values in the environment:

    • GOVC_URL
    • GOVC_DATACENTER
    • GOVC_FOLDER
    • GOVC_DATASTORE
    • GOVC_RESOURCE_POOL
    • GOVC_NETWORK

Procedure

  1. Create a metadata.yaml file and add the following information to this file: 

    instance-id: cloud-vm
local-hostname: vmname
    Copy to Clipboard

  2. Create a userdata.yam file and add the following information to the file: 

    #cloud-config
users:
- name: admin
  sudo: "ALL=(ALL) NOPASSWD:ALL"
  ssh_authorized_keys:
  - ssh-rsa AAA...fhHQ== your.email@example.com
    Copy to Clipboard

    ssh_authorized_keys is your SSH public key. You can find your SSH public key in ~/.ssh/id_rsa.pub.

  3. Export the metadata.yaml and userdata.yaml files to the environment, compressed with gzip, encoded in base64 as follows. You will use these files in further steps. 

    export METADATA=$(gzip -c9 <metadata.yaml | { base64 -w0 2>/dev/null || base64; }) \
USERDATA=$(gzip -c9 <userdata.yaml | { base64 -w0 2>/dev/null || base64; })
    Copy to Clipboard

  4. Launch the image on vSphere with the metadata.yaml and userdata.yaml files:

    1. Import the .vmdk image in to vSphere: 

      $ govc import.vmdk ./composer-api.vmdk <_foldername_>
      Copy to Clipboard

    2. Create the VM in vSphere without powering it on: 

      govc vm.create \
-net.adapter=vmxnet3 \
-m=4096 -c=2 -g=rhel8_64Guest \
-firmware=bios -disk=”foldername/composer-api.vmdk” \
-disk.controller=ide -on=false \
vmname
      Copy to Clipboard

    3. Change the VM to add ExtraConfig variables, the cloud-init config: 

      govc vm.change -vm vmname \
-e guestinfo.metadata="${METADATA}" \
-e guestinfo.metadata.encoding="gzip+base64" \
-e guestinfo.userdata="${USERDATA}" \
-e guestinfo.userdata.encoding="gzip+base64"
.. Power-on the VM:
govc vm.power -on vmname
      Copy to Clipboard

    4. Retrieve the VM IP address: 

      HOST=$(govc vm.ip vmname)
      Copy to Clipboard

Verification

  • Connect to the VM in which you are running the container image. See Connecting to virtual machines for more details.

    1. Use SSH to log in to the VM, using the user-data specified in cloud-init file configuration: 

      $ ssh admin@HOST
      Copy to Clipboard

6.3. Deploying a container image to AWS with an AMI disk image

After using the bootc-image-builder tool to create an AMI from a bootc image, and uploading it to a AWS s3 bucket, you can deploy a container image to AWS with the AMI disk image.

Prerequisites

Procedure

  1. In a browser, access Service→EC2 and log in.
  2. On the AWS console dashboard menu, choose the correct region. The image must have the Available status, to indicate that it was correctly uploaded.
  3. On the AWS dashboard, select your image and click Launch.
  4. In the new window that opens, choose an instance type according to the resources you need to start your image. Click Review and Launch.
  5. Review your instance details. You can edit each section if you need to make any changes. Click Launch.
  6. Before you start the instance, select a public key to access it. You can either use the key pair you already have or you can create a new key pair.

  7. Click Launch Instance to start your instance. You can check the status of the instance, which displays as Initializing.

    After the instance status is Running, the Connect button becomes available.

  8. Click Connect. A window appears with instructions on how to connect by using SSH.

  9. Run the following command to set the permissions of your private key file so that only you can read it. See Connect to your Linux instance. 

    $ chmod 400 <your-instance-name.pem>
    Copy to Clipboard

  10. Connect to your instance by using its Public DNS: 

    $ ssh -i <your-instance-name.pem>ec2-user@<your-instance-IP-address>
    Copy to Clipboard
Note

Your instance continues to run unless you stop it.

Verification

After launching your image, you can:

  • Try to connect to http://<your_instance_ip_address> in a browser.
  • Check if you are able to perform any action while connected to your instance by using SSH.

6.4. Deploying a container image from the network by using Anaconda and Kickstart

You can deploy an ISO image by using Anaconda and Kickstart to install your container image. The installable boot ISO already contains the ostreecontainer Kickstart file configured that you can use to provision your custom container image.

Prerequisites

  • You have downloaded a RHEL 10 Boot ISO for your architecture from Red Hat.

Procedure

  1. Create an ostreecontainer Kickstart file to fetch the image from the network. For example: 

    # Basic setup
text
network --bootproto=dhcp --device=link --activate
# Basic partitioning
clearpart --all --initlabel --disklabel=gpt
reqpart --add-boot
part / --grow --fstype xfs

# Reference the container image to install - The kickstart
# has no %packages section. A container image is being installed.
ostreecontainer --url quay.io/<namespace>/<image>:<tag> . bootc-image-builder:latest

firewall --disabled
services --enabled=sshd

# Only inject a SSH key for root
rootpw --iscrypted locked
sshkey --username root "<your-key>"
reboot
    Copy to Clipboard

  2. Boot a system by using the RHEL 10 Boot ISO installation media.

    1. Append the Kickstart file with the following to the kernel argument: 

      inst.ks=http://<path_to_your_kickstart>
      Copy to Clipboard
  3. Press CTRL+X to boot the system.

6.5. Deploying a custom ISO container image in disconnected environments

By using using bootc-image-builder to convert a bootc image to an ISO image, you create a system similar to the RHEL ISOs available for download, except that your container image content is embedded in the ISO disk image. You do not need to have access to the network during installation. You can install the ISO disk image that you created from bootc-image-builder to a bare metal system.

Prerequisites

  • You have created a customized container image.

Procedure

  1. Create a custom installer ISO disk image with bootc-image-builder. See Creating ISO images by using bootc-image-builder.
  2. Copy the ISO disk image to a USB flash drive.
  3. Perform a bare-metal installation by using the content in the USB stick into a disconnected environment.

6.6. Deploying an ISO bootc image over PXE boot

You can use a network installation to deploy the RHEL ISO image over PXE boot to run your ISO bootc image.

Prerequisites

Procedure

  1. Export the RHEL installation ISO image to the HTTP server. The PXE boot server is now ready to serve PXE clients.
  2. Boot the client and start the installation.
  3. Select PXE Boot when prompted to specify a boot source. If the boot options are not displayed, press the Enter key on your keyboard or wait until the boot window opens.
  4. From the Red Hat Enterprise Linux boot window, select the boot option that you want, and press Enter.
  5. Start the network installation.

Next steps

6.7. Injecting configuration in the resulting disk images with bootc-image-builder

You can inject configuration into a custom image by using a build config, that is, a .toml or a .json file with customizations for the resulting image. The `build config file is mapped into the container directory to /config.toml. The following example shows how to add a user to the resulting disk image:

Procedure

  1. Create a ./config.toml. The following example shows how to add a user to the disk image. 

    [[customizations.user]]
name = "user"
password = "pass"
key = "ssh-rsa AAA ... user@email.com"
groups = ["wheel"]
    Copy to Clipboard
    • name - Mandatory. Name of the user.
    • password - Not mandatory. Nonencrypted password.
    • key - Not mandatory. Public SSH key contents.
    • groups - Not mandatory. An array of groups to add the user into.

  2. Run bootc-image-builder and pass the following arguments, including the ./config.toml: 

    $ sudo podman run \
    --rm \
    -it \
    --privileged \
    --pull=newer \
    --security-opt label=type:unconfined_t \
    -v $(pwd)/config.toml:/config.toml \
    -v $(pwd)/output:/output \
    registry.redhat.io/rhel10/bootc-image-builder:latest \
    --type qcow2 \
    --config config.toml \
    quay.io/<namespace>/<image>:<tag>
    Copy to Clipboard

  3. Launch a VM, for example, by using virt-install: 

    $ sudo virt-install \
  --name bootc \
  --memory 4096 \
  --vcpus 2 \
  --disk qcow2/disk.qcow2 \
  --import \
  --os-variant rhel10
    Copy to Clipboard

Verification

  • Access the system with SSH: 

    # ssh -i /<path_to_private_ssh-key> <user1>_@_<ip-address>
    Copy to Clipboard

Next steps

  • After you deploy your container image, you can make updates to the image and push the changes to a registry. See Managing RHEL bootable images.

6.8. Deploying a container image to bare metal by using bootc install

You can perform a bare-metal installation to a device by using a RHEL ISO image. Bootc contains a basic build installer and it is available as the following methods: bootc install to-disk or bootc install to-filesystem.

  • bootc install to-disk: By using this method, you do not need to perform any additional steps to deploy the container image, because the container images include a basic installer.
  • bootc install to-filesystem: By using this method, you can configure a target device and root filesystem by using a tool of your choice, for example, LVM.

Prerequisites

  • You have downloaded a RHEL 10 Boot ISO from Red Hat for your architecture. See Downloading RHEL boot images.
  • You have created a configuration file.

Procedure

  • Inject a configuration into the running ISO image.

    • By using bootc install to-disk: 

      $ podman run \
--rm --privileged \
--pid=host
-v /dev:/dev \
-v /var/lib/containers:/var/lib/containers \
--security-opt label=type:unconfined_t
<image>
bootc install to-disk <path-to-disk>
      Copy to Clipboard

    • By using bootc install to-filesystem: 

      $ podman run \
--rm --privileged \
--pid=host
-v /:/target \
-v /dev:/dev \
-v /var/lib/containers:/var/lib/containers \
--security-opt label=type:unconfined_t
<image>
bootc install to-filesystem <path-to-disk>
      Copy to Clipboard

6.9. Deploying a container image by using a single command

The system-reinstall-bootc command provides an interactive CLI that wraps the bootc install to-existing root command. You can deploy a container image into a RHEL cloud instance by using a signal command. The system-reinstall-bootc command performs the following actions:

  • Pull the supplied image to set up SSH keys or access the system.
  • Run the bootc install to-existing-root command with all the bind mounts and SSH keys configured.

The following procedure deploys a bootc image to a new RHEL 10 instance on AWS. When launching the instance, make sure to select your SSH key, or create a new one. Otherwise, the default instance configuration settings can be used.

Prerequisites

  • Red Hat Account or Access to Red Hat RPMS
  • A package-based RHEL (9.6 / 10.0 or greater) virtual system running in an AWS environment.
  • Ability and permissions to SSH into the package system and make "destructive changes."

Procedure

  1. After the instance starts, connect to it by using SSH using the key you selected when creating the instance: 

    $ ssh -i <ssh-key-file> <cloud-user@ip>
    Copy to Clipboard

  2. Make sure that the system-reinstall-bootc subpackage is installed: 

    # rpm -q system-reinstall-bootc
    Copy to Clipboard

    If not, install the system-reinstall-bootc subpackage: 

    # dnf -y install system-reinstall-bootc
    Copy to Clipboard

  3. Convert the system to use a bootc image: 

    # system-reinstall-bootc <image>
    Copy to Clipboard
    • You can use the container image from the Red Hat Ecosystem Catalog or the customized bootc image built from a Containerfile.
  4. Select users to import to the bootc image by pressing the "a" key.
  5. Confirm your selection twice and wait until the image is downloaded.

  6. Reboot the system: 

    # reboot
    Copy to Clipboard

  7. Remove the stored SSH host key for the given <ip> from your /.ssh/known_hosts file: 

    # ssh-keygen -R <ip>
    Copy to Clipboard

    The bootc system is now using a new public SSH host key. When attempting to connect to the same IP address with a different key than what is stored locally, SSH will raise a warning or refuse the connection due to a host key mismatch. Since this change is expected, the existing host key entry can be safely removed from the ~/.ssh/known_hosts file using the following command.

  8. Connect to the bootc system: 

    # ssh -i <ssh-key-file> root@<ip>
    Copy to Clipboard

Verification

  • Confirm that the system OS has changed: 

    # bootc status
    Copy to Clipboard

6.10. Advanced installation with to-filesystem

The bootc install contains two subcommands: bootc install to-disk and bootc install to-filesystem.

  • The bootc-install-to-filesystem performs installation to the target filesystem.

  • The bootc install to-disk subcommand consists of a set of opinionated lower level tools that you can also call independently. The command consist of the following tools:

    • mkfs.$fs /dev/disk
    • mount /dev/disk /mnt
    • bootc install to-filesystem --karg=root=UUID=<uuid of /mnt> --imgref $self /mnt

Chapter 7. Creating bootc images from scratch

With bootc images from scratch, you can have control over the underlying image content, and tailor your system environment to your requirements.

You can use the bootc-base-imagectl command to create a bootc image from scratch by using an existing bootc base image as a build environment, providing greater control over the content included in the build process. This process takes the user RPMs as input, so you need to rebuild the image if the RPMs change.

The custom base derives from the base container, and does not automatically consume changes to the default base image unless you make them part of a container pipeline.

You can use the bootc-base-imagectl rechunk subcommand on any bootc container image.

If you want to perform kernel management, you do not need to create a bootc image from scratch. See Managing kernel arguments in bootc systems.

7.1. Using pinned content to build images

To ensure the base image version contains a set of packages at exactly specific versions, for example, defined by a lockfile, or an rpm-md or yum repository, you can use several tools to manage snapshots of rpm-md or yum repository repositories.

With the bootc image from scratch feature, you can configure and override package information in source RPM repositories, while referencing mirrored, pinned, or snapshotted repository content. Consequently, you gain control over package versions and their dependencies.

For example, you might want to gain control over package versions and their dependencies in the following situations:

  • You need to use a specific package version because of strict certification and compliance requirements.
  • You need to use specific software versions to support critical dependencies.

Prerequisites

  • A standard bootc base image.

Procedure

  • The following example creates a bootc image from scratch with pinned content: 

    # Begin with a standard bootc base image that serves as a "builder" for our custom image.
FROM registry.redhat.io/rhel10/rhel-bootc:latest
# Configure and override source RPM repositories, if necessary. The following step is required when referencing specific content views or target mirrored/snapshotted/pinned versions of content.
RUN rm -vf /etc/yum.repos.d
COPY mypinnedcontent.repo /etc/yum.repos
# Add additional repositories to apply customizations to the image. However, referencing a custom manifest in this step is not currently supported without forking the code.
# Build the root file system by using the specified repositories and non-RPM content from the "builder" base image.
# If no repositories are defined, the default build will be used. You can modify the scope of packages in the base image by changing the manifest between the "standard" and "minimal" sets.
RUN /usr/libexec/bootc-base-imagectl build-rootfs --manifest=standard /target-rootfs
# Create a new, empty image from scratch.
FROM scratch
# Copy the root file system built in the previous step into this image.
COPY --from=builder /target-rootfs/ /
# Apply customizations to the image. This syntax uses "heredocs" https://www.docker.com/blog/introduction-to-heredocs-in-dockerfiles/ to pass multi-line arguments in a more readable format.
RUN <<EORUN
# Set pipefail to display failures within the heredoc and avoid false-positive successful builds.
set -xeuo pipefail
# Install necessary packages, run scripts, etc.
dnf -y install NetworkManager emacs
# Remove leftover build artifacts from installing packages in the final built image.
dnf clean all
rm /var/{log,cache,lib}/* -rf
EORUN
# Define required labels for this bootc image to be recognized as such.
LABEL containers.bootc 1
LABEL ostree.bootable 1
# Optional labels that only apply when running this image as a container. These keep the default entry point running under systemd.
STOPSIGNAL SIGRTMIN+3
CMD ["/sbin/init"]
# Run the bootc linter to avoid encountering certain bugs and maintain content quality. Place this command last in your Containerfile.
RUN bootc container lint
    Copy to Clipboard

Verification

  1. Save and build your image. 

    $ podman build -t quay.io/<namespace>/<image>:<tag> . --cap-add=all --security-opt=label=type:container_runtime_t --device /dev/fuse
    Copy to Clipboard

  2. Build <_image_> image by using the Containerfile in the current directory: 

    $ podman build -t quay.io/<namespace>/<image>:<tag> .
    Copy to Clipboard

7.2. Building a base image up from minimal

Previously, you could build just a standard image by using image mode for RHEL. The standard image is roughly a headless server-oriented installation, although you can use it for desktops as well, and includes many opinionated packages for networking, CLI tool, among others.

You now have the option to generate from the standard image a new minimal image which only starts from bootc, kernel, and dnf. This image can then be extended further in a multi-stage build. At the current time the minimal image is not shipped pre-built in the registry.

The base images include the /usr/libexec/bootc-base-imagectl tool that enables you to generate a custom base image. By using the tool, you can build a root file system that is based on the RPM packages that you selected in the base image.

Prerequisites

  • A standard bootc base image.

Procedure

  • The following example creates a custom minimal base image: 

    # Begin with a standard bootc base image that is reused as a "builder" for the custom image.
FROM registry.redhat.io/rhel10/rhel-bootc:latest
# Configure and override source RPM repositories, if necessary. This step is not required when building up from minimal unless referencing specific content views or target mirrored/snapshotted/pinned versions of content.
# Add additional repositories to apply customizations to the image. However, referencing a custom manifest in this step is not currently supported without forking the code.
# Build the root file system by using the specified repositories and non-RPM content from the "builder" base image.
# If no repositories are defined, the default build will be used. You can modify the scope of packages in the base image by changing the manifest between the "standard" and "minimal" sets.
RUN /usr/libexec/bootc-base-imagectl build-rootfs --manifest=minimal /target-rootfs
# Create a new, empty image from scratch.
FROM scratch
# Copy the root file system built in the previous step into this image.
COPY --from=builder /target-rootfs/ /
# Apply customizations to the image. This syntax uses "heredocs" https://www.docker.com/blog/introduction-to-heredocs-in-dockerfiles/ to pass multi-line arguments in a more readable format.
RUN <<EORUN
# Set pipefail to display failures within the heredoc and avoid false-positive successful builds.
set -xeuo pipefail
# Install required packages for our custom bootc image.
# Note that using a minimal manifest means we need to add critical components specific to our use case and environment.
dnf -y install NetworkManager openssh-server
# Remove package caches
dnf clean all
# Clean up all logs and caches
rm /var/{log,cache,lib}/* -rf
# Run the bootc linter to perform build-time verification. Keep this as the last command in your build instructions.
bootc container lint
# Close the shell command.
EORUN
# Define required labels for this bootc image to be recognized as such.
LABEL containers.bootc 1
LABEL ostree.bootable 1
# Optional labels that only apply when running this image as a container. These keep the default entry point running under systemd.
STOPSIGNAL SIGRTMIN+3
CMD ["/sbin/init"]
    Copy to Clipboard

7.3. Building required privileges

Generating a root filesystem from scratch requires the inner build process to use some nested containerization (such as mount namespacing) that are not enabled by default by many container build tools.

Prerequisites

  • In this example using podman, the container-tools meta-package is installed.

Procedure

  • Generate a new root file system, providing these arguments at a minimum to podman build: 

    --cap-add=all --security-opt=label=type:container_runtime_t --device /dev/fuse
    Copy to Clipboard

7.4. Generating your bootc images from scratch

Create bootc images from scratch from a custom RHEL bootc default base container image to get a small root content set.

Prerequisites

  • The container-tools metapackage is installed.

Procedure

  • Create a Containerfile. The following is an example: 

    # The following example reuses the default base image as a "builder" image. Optionally,  you can use the commented instructions to configure or override the RPM repositories in /etc/yum.repos.d to, for example, refer to pinned versions
FROM registry.redhat.io/rhel10/rhel-bootc:latest
# RUN rm -rf /etc/yum.repos.d/*
# COPY mycustom.repo /etc/yum.repos.d
RUN /usr/libexec/bootc-base-imagectl build-rootfs --manifest=minimal /target-rootfs
# Create a new, empty image from scratch.
FROM scratch
# Copy the root file system built in the previous step into this image.
COPY --from=builder /target-rootfs/ /
# You can make arbitrary changes such as copying the systemd units and other tweaks from the baseconfig container image. This example uses the heredocs syntax, to improve and make it easy to add complex instructions, and install critical components
RUN <<EORUN
set -xeuo pipefail
# Install networking support and SSH which are not in minimal
dnf -y install NetworkManager openssh-server
dnf clean all
rm /var/{log,cache,lib}/* -rf
bootc container lint
EORUN
# This label is required
LABEL containers.bootc 1
LABEL ostree.bootable 1
# These labels are optional but useful if you want to keep the default of running under systemd when run as a container image.
STOPSIGNAL SIGRTMIN+3
CMD ["/sbin/init"]
    Copy to Clipboard

Next steps

  • After creating your Containerfile, you get an image with a single tar file large layer. Every change, such as pushing to the registry, pulling for clients, results in copying the single large tar file, and increases the container image size. You can optimize the container image that you created for a smaller version.

7.5. Optimizing container images to a smaller version

You can use the bootc-base-imagectl rechunk subcommand to optimize an input container image into a new image with the same filesystem tree, but split into content-addressed reproducible layers, with precomputed SELinux labeling.

This provides better network efficiency (for both pushes and pulls) since layers that did not change across an image build can be reused without causing a transfer.

The rechunk operation works on an image produced by the default mode of creating new images FROM <rhel-bootc>, but is especially useful in combination with the scratch builds that output only a single large tar layer. Without rechunk every change to the input, for example a kernel update, will result in a new layer including the entire contents of the bootc image. This new layer must then be pushed, stored by registries, and pulled by clients.

The bootc-base-imagectl is shipped as part of the bootc images and is intended to be run inside a container, but requires mapping the host containers-storage into the container to execute.

Prerequisites

  • You have a previously-built base image.

Procedure

  • Run the following command to rechunk your base image. 

    $ sudo podman run --rm --privileged -v /var/lib/containers:/var/lib/containers \
      registry.redhat.io/rhel10/rhel-bootc:latest \
      /usr/libexec/bootc-base-imagectl rechunk \
          quay.io/exampleos/rhel-bootc:single \
          quay.io/exampleos/rhel-bootc:chunked
    Copy to Clipboard

Chapter 8. Enabling FIPS mode in a bootc image build

The Federal Information Processing Standard (FIPS) 140 defines requirements for cryptographic modules. To fulfill these requirements, you must enable FIPS mode. You can enable FIPS mode during the bootc container image build.

8.1. Creating a bootable disk image for a FIPS-enabled system

You can create a disk image and enable FIPS mode when performing an Anaconda installation. You must add the fips=1 kernel argument when booting the disk image.

Prerequisites

  • You have Podman installed on your host machine.
  • You have virt-install installed on your host machine.
  • You have root access to run the bootc-image-builder tool, and run the containers in --privileged mode, to build the images.

Procedure

  1. Create a 01-fips.toml to configure FIPS enablement, for example: 

    # Enable FIPS
kargs = ["fips=1"]
    Copy to Clipboard

  2. Create a Containerfile with the following instructions to enable the fips=1 kernel argument and adjust the cryptographic policies: 

    FROM registry.redhat.io/rhel10/rhel-bootc:latest
# Enable fips=1 kernel argument: https://bootc-dev.github.io/bootc/building/kernel-arguments.html
COPY 01-fips.toml /usr/lib/bootc/kargs.d/
# Install and enable the FIPS crypto policy
RUN dnf install -y crypto-policies-scripts && update-crypto-policies --no-reload --set FIPS
    Copy to Clipboard

  3. Create your bootc <image> compatible base disk image by using Containerfile in the current directory: 

    $ sudo podman run \
    --rm \
    -it \
    --privileged \
    --pull=newer \
    --security-opt label=type:unconfined_t \
    -v $(pwd)/config.toml:/config.toml:ro \
    -v $(pwd)/output:/output \
    -v /var/lib/containers/storage:/var/lib/containers/storage \
    registry.redhat.io/rhel10/bootc-image-builder:latest \
    --local
    --type qcow2 \
    --type iso \
    quay.io/<namespace>/<image>:<tag>
    Copy to Clipboard

  4. Enable FIPS mode during the system installation:

    1. When booting the RHEL Anaconda installer, on the installation screen, press the TAB key and add the fips=1 kernel argument.

      After the installation, the system starts in FIPS mode automatically.

Verification

  • After login in to the system, check that FIPS mode is enabled: 

    $ cat /proc/sys/crypto/fips_enabled
1
$ update-crypto-policies --show
FIPS
    Copy to Clipboard

Chapter 9. Security hardening and compliance of bootable images

Image mode for RHEL provides security compliance features and supports workloads that require compliant configuration. However, the process of hardening systems and verifying compliance status is different than in package mode.

The key part of using Image mode for RHEL is creating a bootable container image. The deployed system mirrors the image. Therefore, the built image must contain all packages and configuration settings that are required by the security policy.

Important

When a bootable image is run as a container, some of the hardening configuration is not in effect. To get a system that is fully configured in accordance with the security profile, you must boot the image in a bare metal or virtual machine instead of running as a container. Main differences of a container deployment include the following:

  • Systemd services that are required by security profiles do not run on containers because systemd is not running in the container. Therefore, the container cannot comply with the related policy requirements.
  • Other services cannot run in containers, although they are configured correctly. This means that oscap reports them as correctly configured, even if they are not running.
  • Configurations defined by the compliance profile are not enforcing. Requests from other packages or installation prescripts can change the compliance state. Always check the compliance of the installed product and alter your Containerfile to fit your requirements.

9.1. Building hardened bootable images

You can build hardened bootable images more easily by including the oscap-im tool in the Containerfile that you use to build your bootable container image.

Although oscap-im can consume any SCAP content, the SCAP source data streams shipped in scap-security-guide are specifically adjusted and tested to be compatible with bootable containers.

Prerequisites

Procedure

  1. Create a Containerfile: 

    FROM registry.redhat.io/rhel10/rhel-bootc:latest

# Install OpenSCAP scanner and security content to the image
RUN dnf install -y openscap-utils scap-security-guide && dnf clean all

# Run scan and hardening
RUN oscap-im --profile <profile_ID> /usr/share/xml/scap/ssg/content/ssg-rhel10-ds.xml

# Because certain profiles prevent ssh root logins, add a separate sudo user with a password
# Alternatively, you can add users with Kickstart, cloud-init, or other methods
RUN useradd -G wheel -p "<password_hash>" <admin_user>
    Copy to Clipboard

    Replace <admin_user> with the user name and <password_hash> with the hash of the selected password.

    This Containerfile performs the following tasks:

    • Installs the openscap-utils package that provides the oscap-im tool and the scap-security-guide package that provides the data streams with the Security Content Automation Protocol (SCAP) content.
    • Adds a user with sudoer privileges for profiles that prevent SSH root logins.
    • Scans and remediates the image for compliance with the selected profile.

  2. Build the image by using the Containerfile in the current directory: 

    $ podman build -t quay.io/<namespace>/<image>:<tag> .
    Copy to Clipboard

Verification

  • List all images: 

    $ podman images
REPOSITORY                                  TAG      IMAGE ID       CREATED              SIZE
quay.io/<namespace>/<image>                 <tag>   b28cd00741b3   About a minute ago   2.1 GB
    Copy to Clipboard

Next steps

  • You can deploy hardened bootable images by using any of the normal bootable image deployment methods. For more information, see Deploying the RHEL bootc images.

    The deployment method, however, can affect the compliance state of the target system.

  • You can verify the compliance of a running system in Image Mode RHEL by using the oscap tool with the same syntax and usage as in package mode RHEL. For more information, see Configuration compliance scanning.

9.2. Customizing hardened bootable images

You can apply a customized profile to a bootable image by using the oscap-im tool. You can customize a security profile by changing parameters in certain rules, for example, minimum password length, removing rules that you cover in a different way, and selecting additional rules, to implement internal policies. You cannot define new rules by customizing a profile.

Prerequisites

Procedure

  1. Create a Containerfile: 

    FROM registry.redhat.io/rhel10/rhel-bootc:latest

# Copy a tailoring file into the Containerfile
COPY tailoring.xml /usr/share/

# Install OpenSCAP scanner and security content to the image
RUN dnf install -y openscap-utils scap-security-guide && dnf clean all


# Add sudo user 'admin' with password 'admin123'.
# The user can be used with profiles that prevent
# ssh root logins.
RUN useradd -G wheel -p "\$6\$Ga6Zn
IlytrWpuCzO\$q0LqT1USHpahzUafQM9jyHCY9BiE5/ahXLNWUMiVQnFGblu0WWGZ1e6icTaCGO4GNgZNtspp1Let/qpM7FMVB0" admin

# Run scan and hardening including the tailoring file
RUN oscap-im --tailoring-file /usr/share/tailoring.xml --profile stig_customized /usr/share/xml/scap/ssg/content/ssg-rhel10-ds.xml
    Copy to Clipboard

This Containerfile performs the following tasks:

  • Injects the tailoring file to your image.
  • Installs the openscap-utils package that provides the oscap-im tool and the scap-security-guide package that provides the data streams with the Security Content Automation Protocol (SCAP) content.
  • Adds a user with sudoer privileges for profiles that prevent SSH root logins.

  • Scans and remediates the image for compliance with the selected profile.

    1. Build the image by using the Containerfile in the current directory: 

      $ podman build -t quay.io/<namespace>/<image>:<tag> .
      Copy to Clipboard

Verification

  • List all images: 

    $ podman images
REPOSITORY                                  TAG      IMAGE ID       CREATED              SIZE
quay.io/<namespace>/<image>                 <tag>   b28cd00741b3   About a minute ago   2.1 GB
    Copy to Clipboard

Next steps

  • You can deploy hardened bootable images by using any of the normal bootable image deployment methods. For more information, see Deploying the RHEL bootc images.

    The deployment method, however, can affect the compliance state of the target system.

    Note

    Some customizations performed during the deployment, in blueprint for bootc-image-builder or in Kickstart for Anaconda, can interfere with the configuration present in the container image. Do not use customizations that conflict with the security policy requirements.

  • You can verify the compliance of a running system in Image Mode RHEL by using the oscap tool with the same syntax and usage as in package mode RHEL. For more information, see Configuration compliance scanning.

Chapter 10. Managing RHEL bootc images

After installing and deploying RHEL bootc images, you can perform management operations on your container images, such as changing or updating the systems. The system supports in-place transactional updates with rollback after deployment.

This kind of management, also known as Day 2 management baseline, consists of transactionally fetching new operating system updates from a container registry and booting the system into them, while supporting manual, or automated rollbacks in case of failures.

You can also rely on automatic updates, that are turned on by default. The systemd service unit and the systemd timer unit files check the container registry for updates and apply them to the system. You can trigger an update process with different events, such as updating an application. There are automation tools watching these updates and then triggering the CI/CD pipelines. A reboot is required, because the updates are transactional. For environments that require more sophisticated or scheduled rollouts, you must disable auto updates and use the bootc utility to update your operating system.

See Day 2 operations support for more details.

Note

The rhel-bootc images are rebuilt whenever their underlying inputs, such as RPM packages, are updated. These rebuilds occur at least monthly, or more frequently if critical updates are released. As a user, you maintain full control over when to push the update images. A newly published base image does not trigger automatic rebuilds or redeployments of your custom images. You configure the update cadence and only push changes as required.

Figure 10.1. Manually updating an installed operating system, changing the container image reference or rolling back changes if needed

Bootc update diagram

10.1. Switching the container image reference

You can change the container image reference used for upgrades by using the bootc switch command. For example, you can switch from the stage to the production tag. The bootc switch command performs the same operations as the bootc upgrade command and additionally changes the container image reference.

To manually switch an existing ostree-based container image reference, use the bootc switch command.

Prerequisites

  • A booted system using bootc.

Procedure

  • Run the following command: 

    $ sudo bootc switch [--apply] quay.io/<namespace>/<image>:<tag>
    Copy to Clipboard

    Optionally, you can use the --apply option when you want to automatically take actions, such as rebooting if the system has changed.

Note

The bootc switch command has the same effect as bootc upgrade. The only difference is the container image reference is changed. This allows preserving the existing states in /etc and /var, for example, host SSH keys and home directories.

10.2. Adding or changing content in the bootc image initramfs

The rhel10/rhel-bootc image uses the dracut infrastructure to build an initial RAM disk, the initrd during the image build time. A default initrd is built and included in the /usr/lib/modules/$kver/initramfs.img location inside the container image.

You can use a drop-in configuration file to extend the dracut configuration (for example, /usr/lib/dracut/dracut.conf.d/<50-custom-added-modules.conf>) and rerun dracut to thus re-create initrd with the content (modules) you want to add.

Procedure

  • Re-create the initrd as part of a container build: 

    FROM <baseimage>
COPY <50-custom-added-modules>.conf /usr/lib/dracut/dracut.conf.d
RUN set -x; kver=$(cd /usr/lib/modules && echo *); dracut -vf /usr/lib/modules/$kver/initramfs.img $kver
    Copy to Clipboard
    Note

    It does not currently work to simply run dracut. By default dracut will operate on the running kernel version, and the initramfs must explicitly be written to the /usr/lib/modules directory and not /boot.

10.3. Performing manual updates from an installed operating system

Installing image mode for RHEL is a one time task. You can perform any other management task, such as changing or updating the system, by pushing the changes to the container registry.

When using image mode for RHEL, you can choose to perform manual updates for your systems. Manual updates are also useful if you have an automated way to perform updates, for example, by using Ansible. Because the automatic updates are enabled by default, to perform manual updates you must turn the automatic updates off. You can do this by choosing one of the following options:

  • Running the bootc upgrade command
  • Modifying the systemd timer file

10.4. Turning off automatic updates

To perform manual updates you must turn off automatic updates. You can do this by disabling the timer of the container build, by using one of the following options described in the procedure.

Procedure

  • Disable the timer of a container build.

    • By running the systemctl mask command: 

      $ systemctl mask bootc-fetch-apply-updates.timer
      Copy to Clipboard

    • By modifying the systemd timer file. Use systemd "drop-ins" to override the timer. In the following example, updates are scheduled for once a week.

      1. Create an updates.conf file with the following content: 

        [Timer]
# Clear previous timers
OnBootSec= OnBootSec=1w OnUnitInactiveSec=1w
        Copy to Clipboard

      2. Add you file to the directory that you created: 

        $ mkdir -p /usr/lib/systemd/system/bootc-fetch-apply-updates.timer.d
$ cp updates.conf /usr/lib/systemd/system/bootc-fetch-apply-updates.timer.d
        Copy to Clipboard

10.5. Manually updating an installed operating system

To manually fetch updates from a registry and boot the system into the new updates, use bootc upgrade. This command fetches the transactional in-place updates from the installed operating system to the container image registry. The command queries the registry and queues an updated container image for the next boot. It stages the changes to the base image, while not changing the running system by default.

Procedure

  • Run the following command: 

    $ bootc upgrade [--apply]
    Copy to Clipboard

    The apply argument is optional and you can use it when you want to automatically take actions, such as rebooting if the system has changed.

Note

The bootc upgrade and bootc update commands are aliases.

10.6. Performing rollbacks from a updated operating system

You can roll back to a previous boot entry to revert changes by using the bootc rollback command. This command changes the boot loader entry ordering by making the deployment under rollback queued for the next boot. The current deployment then becomes the rollback. Any staged changes, such as a queued upgrade that was not applied, are discarded.

After a rollback completes, the system reboots and the update timer run within 1 to 3 hours which automatically update and reboot your system to the image you just rolled back from.

Warning

If you perform a rollback, the system will automatically update again unless you turn off auto-updates. See Turning off automatic updates.

Note

When performing a rollback, for example, by using the bootc rollback command, changes made to files in the /etc directory do not carry over to the rolled-back deployment. Instead, the files in the /etc directory revert to the state they were in during the previous deployment.

The bootc rollback command reorders existing deployments but does not create new ones. The /etc directory is merged when new deployments are created.

To preserve a modified /etc file for use after a rollback, copy it to a directory under /var, such as /var/home/<user> for a specific <user>, or under /var/root/, for the root user. These directories are unaffected by rollbacks, as they store user content.

When returning to the original state, either through a temporary rollback or another bootc rollback, the /etc directory reverts to its state from the original deployment.

Alternatively, if the issue you are rolling back does not involve configuration files in the /etc directory and you want to revert to an older deployment, use the bootc switch command. This command performs the necessary /etc merge and deploy the previous version of the software.

Prerequisites

  • You performed an update to the system.

Procedure

  • Run the following command: 

    $ bootc rollback [-h|--help] [-V|--version]
    Copy to Clipboard

Verification

  • Use systemd journal to check the logged message for the detected rollback invocation. 

    $ journalctl -b
    Copy to Clipboard

    You can see a log similar to: 

    MESSAGE_ID=26f3b1eb24464d12aa5e7b544a6b5468
    Copy to Clipboard

10.7. Deploying updates to system groups

You can change the configuration of your operating system by modifying the Containerfile. Then you can build and push your container image to the registry. When you next boot your operating system, an update will be applied.

You can also change the container image source by using the bootc switch command. The container registry is the source of truth. See Switching the container image reference.

Usually, when deploying updates to system groups, you can use a central management service to provide a client to be installed on each system which connects to the central service. Often, the management service requires the client to perform a one time registration. The following is an example on how to deploy updates to system groups. You can modify it to create a persistent systemd service, if required.

Note

For clarity reasons, the Containerfile in the example is not optimized. For example, a better optimization to avoid creating multiple layers in the image is by invoking RUN a single time.

You can install a client into a image mode for RHEL image and run it at startup to register the system.

Prerequisites

  • The management-client handles future connections to the server, by using a cron job or a separate systemd service.

Procedure

  • Create a management service with the following characteristics. It determines when to upgrade the system.

    1. Disable bootc-fetch-apply-updates.timer if it is included in the base image.
    2. Install the client by using dnf, or some other method that applies for your client.
    3. Inject the credentials for the management service into the image.

10.8. Checking inventory health

Health checks are one of the Day 2 Operations. You can manually check the system health of the container images and events that are running inside the container.

You can set health checks by creating the container on the command line. You can display the health check status of a container by using the podman inspect or podman ps commands.

You can monitor and print events that occur in Podman by using the podman events command. Each event includes a timestamp, a type, a status, a name, if applicable, and an image, if applicable.

For more information about health checks and events, see the Monitoring containers chapter.

10.9. Automation and GitOps

You can automate the building process by using CI/CD (Continuous Integration and Continuous Delivery) pipelines so that an update process can be triggered by events, such as updating an application. You can use automation tools that track these updates and trigger the CI/CD pipelines. The pipeline keeps the systems up to date by using the transactional background operating system updates.

For more details on resources to create image mode for RHEL instances, check the specific implementations available to create image mode for RHEL instances:RHEL Image Mode CI/CD.

Chapter 11. Managing kernel arguments in bootc systems

You can use bootc to configure kernel arguments. By default, bootc uses the boot loader configuration files that are stored in /boot/loader/entries. This directory defines arguments provided to the Linux kernel. The set of kernel arguments is machine-specific state, but you can also manage the kernel arguments by using container updates. The boot loader menu entries are shared between multiple operating systems and boot loaders are installed on one device.

Note

Currently, the boot loader entries are written by an OSTree backend.

11.1. How to add support to inject kernel arguments with bootc

The bootc tool uses generic operating system kernels. You can add support to inject kernel arguments by adding a custom configuration, in the TOML format, in /usr/lib/bootc/kargs.d. For example: 

# /usr/lib/bootc/kargs.d/10-example.toml
kargs = ["mitigations=auto,nosmt"]
Copy to Clipboard

You can also make these kernel arguments architecture-specific by using the match-architectures key. For example: 

# /usr/lib/bootc/kargs.d/00-console.toml
kargs = ["console=ttyS0,114800n8"]
match-architectures = ["x86_64"]
Copy to Clipboard

11.2. How to modify kernel arguments by using bootc install configs

You can use bootc install to add kernel arguments during the install time in the following ways:

  • Adding kernel arguments into the container image.
  • Adding kernel arguments by using the bootc install --karg command.

You can use the kernel arguments on Day 2 operations, by adding the arguments and applying them on a switch, upgrade, or edit. Adding kernel arguments and using it for Day 2 operations involves the following high-level steps:

  1. Create files within /usr/lib/bootc/kargs.d with kernel arguments.
  2. Fetch the container image to get the OSTree commit.
  3. Use the OSTree commit to return the file tree.
  4. Navigate to /usr/lib/bootc/kargs.d.
  5. Read each file within the directory.
  6. Push the contents of each kargs file into a file containing all the needed kargs.
  7. Pass the kargs to the stage() function.
  8. Apply these arguments to switch, upgrade, or edit.

11.3. How to inject kernel arguments in the Containerfile

To add kernel arguments into a container image, use a Containerfile. The following is an example: 

FROM registry.redhat.io/rhel10/rhel-bootc:latest

RUN mkdir -p /usr/lib/bootc/kargs.d
RUN cat <<EOF >> /usr/lib/bootc/kargs.d/console.toml
kargs = ["console=ttyS0,114800n8"]
match-architectures = ["x86_64"]
EOF

RUN cat <<EOF >> /usr/lib/bootc/kargs.d/01-mitigations.toml
kargs = ["mitigations=on", "systemd.unified_cgroup_hierarchy=0"]
match-architectures = ["x86_64", "aarch64"]
EOF
Copy to Clipboard

11.4. How to inject kernel arguments at installation time

You can use boot install with the --karg to inject kernel arguments during installation time. As a result, the kernel arguments become machine-local state.

For example, to inject kernel arguments, use the following command: 

# bootc install to-filesystem --karg
Copy to Clipboard
Note

Currently, bootc does not have an API to manipulate kernel arguments. This is only supported by rpm-ostree , by using the rpm-ostree kargs command.

11.5. How to add install-time kernel arguments with bootc-image-builder

The bootc-image-builder tool supports the customizations.kernel.append during install-time.

To add the kernel arguments with bootc-image-builder, use the following customization: 

{
  "customizations": {
    "kernel": {
      "append": "mitigations=auto,nosmt"
    }
  }
}
Copy to Clipboard

11.6. About changing kernel arguments post-install with kargs.d

The changes that you make to kargs.d files and include in a container build are applied after the installation, and the difference between the set of kernel arguments is applied to the current boot loader configuration. This preserves any machine-local kernel arguments. You can use any tool to edit the /boot/loader/entries files, which are in a standardized format. The /boot file has read-only access to limit the set of tools that can write to this filesystem.

11.7. How to edit kernel arguments in bootc systems

To perform machine local changes, you also can edit kernel arguments on a bootc system or an rpm-ostree system, by using the rpm-ostree kargs command. The changes are made through the user/lib/bootc/kargs.d path, which also handles "Day 2" changes, besides the first boot changes.

The following are the options that you can use to add, modify or remove kernel arguments.

rpm-ostree kargs

--append=KEY=VALUE
Appends a kernel argument. It is useful with, for example, console= that can be used multiple times. You can use an empty value for an argument.
--replace=KEY=VALUE=NEWVALUE
Replaces an existing kernel argument. You can replace an argument with KEY=VALUE only if one value already exists for that argument.
--delete=KEY=VALUE
Deletes a specific kernel key-value pair argument or an entire argument with a single key-value pair.
--append-if-missing=KEY=VALUE
Appends a kernel argument. Does nothing if the key is already present.
--delete-if-present=KEY=VALUE
Deletes a specific kernel key-value pair argument. Does nothing if the key is missing.
--editor
Uses an editor to modify the kernel arguments.

For more information, check the help: 

# rpm-ostree kargs --help
Copy to Clipboard

The following is an example: 

# rpm-ostree kargs --append debug
Staging deployment... done
Freed: 40.1 MB (pkgcache branches: 0)
Changes queued for next boot. Run "systemctl reboot" to start a reboot
Copy to Clipboard

Chapter 12. Managing file systems in image mode for RHEL

Image mode uses OSTree as a backend, and by default, enables composefs for storage. That means you can easily install third-party content in derived container images that write into /opt for example, because the /opt and /usr local paths are plain directories, and not symbolic links into /var.

Warning

When you install third-party content to /opt, the third-party components might also attempt to write to subdirectories within /opt during runtime, what can create potential conflicts.

12.1. Understanding physical and logical root with /sysroot

When a bootc system is fully booted, it is similar to an environment created by chroot, that is, the operating system changes the apparent root directory for the current running process and its children. The physical host root filesystem is mounted at /sysroot. The chroot filesystem is called a deployment root.

The remaining filesystem paths are part of a deployment root which is used as a final target for the system boot. The system uses the ostree=kernel argument to find the deployment root.

/usr
It is preferred to keep all operating system content in /usr, but not strictly required. Directories such as /bin will work as symbolic links to /usr/bin. This layout creates a separation of operating system and host specific resources.
Note

When composefs is enabled, /usr is not different from /. Both directories are part of the same immutable image, so you do not need to perform a full UsrMove with a bootc system.

/usr/local
With bootc systems, you can create custom container images as the default entrypoint. Because of that, the base images configure /usr/local to be a regular directory, that is, the default directory.

The default filesystem layout has both /opt and /usr/local as regular directories, that is, they writable at build time and immutable at runtime. This differs from RHEL CoreOS, for example, which makes these symlinks into /var.

/etc

The /etc directory contains mutable persistent state by default, but it supports enabling the etc.transient config option. When the directory is in mutable persistent state, it performs a 3-way merge across upgrades:

  • Uses the new default /etc as a base
  • Applies the diff between current and previous /etc to the new /etc directory
  • Retains locally modified files that are different from the default /usr/etc of the same deployment in /etc.

The ostree-finalize-staged.service executes these tasks during shutdown time, before creating the new boot loader entry.

This happens because many components of a Linux system ship default configuration files in the /etc directory. Even if the default package does not ship it, by default the software only checks for config files in /etc. Non bootc image based update systems with no distinct versions of /etc are populated only during the installation time, and will not be changed at any point after installation. This causes the /etc system state to be influenced by the initial image version and can lead to problems to apply a change, for example, to /etc/sudoers.conf, and requires external intervention. For more details about file configuration, see Building and testing RHEL bootc images.

/var
For example, there is only one /var directory. By default, the files and data placed in the /var directory are persistent until explicitly deleted, and available across different sessions and system restarts. You can turn the /var partition or its subdirectories into a mount point, such as a temporary file system (TMPFS) or a network mount point. If you do not make a distinct partition for /var, the system performs a bind mount, and creates a single, shared and persistent /ostree/deploy/$stateroot/var in the /var directory, so that both directories share the same data across deployments.

By default, the content in /var acts as a volume, that is, the content from the container image is copied during the initial installation time, and is not updated thereafter.

The /var and the /etc directories are different. You can use /etc for relatively small configuration files, and the expected configuration files are often bound to the operating system binaries in /usr. The /var directory has arbitrarily large data, for example, system logs, databases, and by default, do not roll back if the operating system state is rolled back.

For example, making an update such as dnf downgrade postgresql should not affect the physical database in /var/lib/postgres. Similarly, making a bootc update or bootc rollback do not affect this application data.

Having /var separate also makes it work cleanly to stage new operating system updates before applying them, that is, updates are downloaded and ready, but only take effect on reboot. The same applies for Docker volume, as it decouples the application code from its data.

You can use this case if you want applications to have a pre-created directory structure, for example, /var/lib/postgresql. Use systemd tmpfiles.d for this. You can also use StateDirectory=<directory> in units.

Other directories
There is no support to ship content in /run, /proc or other API Filesystems in container images. Apart from that, other top level directories such as /usr, and /opt, are lifecycled with the container image.
/opt
Because bootc uses composefs, the /opt directory is read-only, alongside other top level directories such as /usr.

When a software needs to write to its own directory in /opt/exampleapp, a common pattern is to use a symbolic link to redirect to, for example, /var for operations such as log files: 

RUN rmdir /opt/exampleapp/logs && ln -sr /var/log/exampleapp /opt/exampleapp/logs
Copy to Clipboard

Optionally, you can configure the systemd unit to launch the service to do these mounts dynamically. For example: 

BindPaths=/var/log/exampleapp:/opt/exampleapp/logs
Copy to Clipboard
Enabling transient root
To enable a software to transiently (until the next reboot) write to all top-level directories, including /usr and /opt, with symlinks to /var for content that should persist, you can enable transient root. To enable a fully transient writable rootfs by default, set the following option in /usr/lib/ostree/prepare-root.conf. 
[root]
transient = true
Copy to Clipboard

This enables a software to transiently write to /opt, with symlinks to /var for content that must persist.

12.2. Version selection and bootup

Image mode for RHEL uses GRUB by default, with exception to s390x architectures. Each version of image mode for RHEL currently available on a system has a menu entry.

The menu entry references an OSTree deployment which consists of a Linux kernel, an initramfs and a hash linking to an OSTree commit, that you can pass by using the ostree=kernel argument.

During bootup, OSTree reads the kernel argument to determine which deployment to use as the root filesystem. Each update or change to the system, such as package installation, addition of kernel arguments, creates a new deployment.

This enables rolling back to a previous deployment if the update causes problems.

Chapter 13. Appendix: Managing users, groups, SSH keys, and secrets in image mode for RHEL

Learn more about users, groups, SSH keys, and secrets management in image mode for RHEL.

13.1. Users and groups configuration

Image mode for RHEL is a generic operating system update and configuration mechanism. You cannot use it to configure users or groups. The only exception is the bootc install command that has the --root-ssh-authorized-keys option.

Users and groups configuration for generic base images
Usually, the distribution base images do not have any configuration. Do not encrypt passwords and SSH keys with publicly-available private keys in generic images because of security risks.
Injecting SSH keys through systemd credentials
You can use systemd to inject a root password or SSH authorized_keys file in some environments. For example, use System Management BIOS (SMBIOS) to inject SSH keys system firmware. You can configure this in local virtualization environments, such as qemu.
Injecting users and SSH keys by using cloud-init
Many Infrastructure as a service (IaaS) and virtualization systems use metadata servers that are commonly processed by software such as cloud-init or ignition. See AWS instance metadata. The base image you are using might include cloud-init or Ignition, or you can install it in your own derived images. In this model, the SSH configuration is managed outside of the bootc image.
Adding users and credentials by using container or unit custom logic
Systems such as cloud-init are not privileged. You can inject any logic you want to manage credentials in the way you want to launch a container image, for example, by using a systemd unit. To manage the credentials, you can use a custom network-hosted source, for example, FreeIPA.
Adding users and credentials statically in the container build

In package-oriented systems, you can use the derived build to inject users and credentials by using the following command: 

RUN useradd someuser
Copy to Clipboard

You can find issues in the default shadow-utils implementation of useradd: Users and groups IDs are allocated dynamically, and this can cause drift.

User and group home directories and /var directory

For systems configured with persistent /home → /var/home, any changes to /var made in the container image after initial installation will not be applied on subsequent updates.

For example, if you inject /var/home/someuser/.ssh/authorized_keys into a container build, existing systems do not get the updated authorized_keys file.

Using DynamicUser=yes for systemd units

Use the systemd DynamicUser=yes option where possible for system users.

This is significantly better than the pattern of allocating users or groups at package install time, because it avoids potential UID or GID drift.

Using systemd-sysusers

Use systemd-sysusers, for example, in your derived build. For more information, see the systemd -sysusers documentation. 

COPY mycustom-user.conf /usr/lib/sysusers.d
Copy to Clipboard

The sysusers tool makes changes to the traditional /etc/passwd file as necessary during boot time. If /etc is persistent, this can avoid UID or GID drift. It means that the UID or GID allocation depends on how a specific machine was upgraded over time.

Using systemd JSON user records
See JSON user records systemd documentation. Unlike sysusers, the canonical state for these users lives in /usr. If a subsequent image drops a user record, then it also vanishes from the system.
Using nss-altfiles

With nss-altfiles, you can remove the systemd JSON user records. It splits system users into /usr/lib/passwd and /usr/lib/group, aligning with the way the OSTree project handles the 3 way merge for /etc as it relates to /etc/passwd. Currently, if the /etc/passwd file is modified in any way on the local system, then subsequent changes to /etc/passwd in the container image are not applied.

Base images built by rpm-ostree have nns-altfiles enabled by default.

Also, base images have a system users pre-allocated and managed by the NSS file to avoid UID or GID drift.

In a derived container build, you can also append users to /usr/lib/passwd, for example. Use sysusers.d or DynamicUser=yes.

Machine-local state for users

The filesystem layout depends on the base image.

By default, the user data is stored in both /etc, /etc/passwd, /etc/shadow and groups, and /home, depending on the base image. However, the generic base images have to both be machine-local persistent state. In this model /home is a symlink to /var/home/user.

Injecting users and SSH keys at system provisioning time

For base images where /etc and /var are configured to persist by default, you can inject users by using installers such as Anaconda or Kickstart.

Typically, generic installers are designed for one time bootstrap. Then, the configuration becomes a mutable machine-local state that you can change in Day 2 operations, by using some other mechanism.

You can use the Anaconda installer to set the initial password. However, changing this initial password requires a different in-system tool, such as passwd.

These flows work equivalently in a bootc-compatible system, to support users directly installing generic base images, without requiring changes to the different in-system tool.

Transient home directories

Many operating system deployments minimize persistent, mutable, and executable state. This can damage user home directories.

The /home directory can be set as tmpfs, to ensure that user data is cleared across reboots. This approach works especially well when combined with a transient /etc directory.

To set up the user’s home directory to, for example, inject SSH authorized_keys or other files, use the systemd tmpfiles.d snippets: 

f~ /home/user/.ssh/authorized_keys 600 user user - <base64 encoded data>
Copy to Clipboard

SSH is embedded in the image as: /usr/lib/tmpfiles.d/<username-keys.conf. Another example is a service embedded in the image that can fetch keys from the network and write them. This is the pattern used by cloud-init.

UID and GID drift
The /etc/passwd and similar files are a mapping between names and numeric identifiers. When the mapping is dynamic and mixed with "stateless" container image builds, it can cause issues. Each container image build might result in the UID changing due to RPM installation ordering or other reasons. This can be a problem if that user maintains a persistent state. To handle such cases, convert it to use sysusers.d or use DynamicUser=yes.

13.2. Injecting secrets in image mode for RHEL

Image mode for RHEL does not have an opinionated mechanism for secrets. You can inject container pull secrets in your system for some cases, for example:

  • For bootc to fetch updates from a registry that requires authentication, you must include a pull secret in a file. In the following example, the creds secret contains the registry pull secret. 

    FROM registry.redhat.io/rhel10/bootc-image-builder:latest
COPY containers-auth.conf /usr/lib/tmpfiles.d/link-podman-credentials.conf
RUN --mount=type=secret,id=creds,required=true cp /run/secrets/creds /usr/lib/container-auth.json && \
    chmod 0600 /usr/lib/container-auth.json && \
    ln -sr /usr/lib/container-auth.json /etc/ostree/auth.json
    Copy to Clipboard

    To build it, run podman build --secret id=creds,src=$HOME/.docker/config.json. Use a single pull secret for bootc and Podman by using a symlink to both locations to a common persistent file embedded in the container image, for example /usr/lib/container-auth.json.

  • For Podman to fetch container images, include a pull secret to /etc/containers/auth.json. With this configuration, the two stacks share the /usr/lib/container-auth.json file.

    Injecting secrets by embedding them in a container build
    You can include secrets in the container image if the registry server is suitably protected. In some cases, embedding only bootstrap secrets into the container image is a viable pattern, especially alongside a mechanism for having a machine authenticate to a cluster. In this pattern, a provisioning tool, whether run as part of the host system or a container image, uses the bootstrap secret to inject or update other secrets, such as SSH keys, certificates, among others.
    Injecting secrets by using cloud metadata
    Most production Infrastructure as a Service (IaaS) systems support a metadata server or equivalent which can securely host secrets, particularly bootstrap secrets. Your container image can include tools such as cloud-init or ignition to fetch these secrets.
    Injecting secrets by embedding them in disk images
    You can embed bootstrap secrets only in disk images. For example, when you generate a cloud disk image from an input container image, such as AMI or OpenStack, the disk image can contain secrets that are effectively machine-local state. Rotating them requires an additional management tool or refreshing the disk images.
    Injecting secrets by using bare-metal installers
    Installer tools usually support injecting configuration through secrets.
    Injecting secrets through systemd credentials
    The systemd project has a credential concept for securely acquiring and passing credential data to systems and services, which applies in some deployment methodologies. See the systemd credentials documentation for more details.

13.3. Injecting pull secrets for registries and disabling TLS

You can configure container images, pull secrets, and disable TLS for a registry within a system. These actions enable containerized environments to pull images from private or insecure registries.

You can include container pull secrets and other configuration to access a registry inside the base image. However, when installing by using Anaconda, the installation environment might need a duplicate copy of "bootstrap" configuration to access the targeted registry when fetching over the network.

To perform arbitrary changes to the installation environment before the target bootc container image is fetched, you can use the Anaconda %pre command.

See the containers-auth.json(5) for more detailed information about format and configurations of the auth.json file.

Procedure

  1. Configure a pull secret: 

    %pre
mkdir -p /etc/ostree
cat > /etc/ostree/auth.json << 'EOF'
{
        "auths": {
                "quay.io": {
                        "auth": "<your secret here>"
                }
        }
}
EOF
%end
    Copy to Clipboard

    With this configuration, the system pulls images from quay.io using the provided authentication credentials, which are stored in /etc/ostree/auth.json.

  2. Disable TLS for an insecure registry: 

    %pre
mkdir -p /etc/containers/registries.conf.d/
cat > /etc/containers/registries.conf.d/local-registry.conf << 'EOF'

[[registry]]
location="[IP_Address]:5000"
insecure=true
EOF
%end
    Copy to Clipboard

    With this configuration, the system pulls container images from a registry that is not secured with TLS. You can use it in development or internal networks.

You can also use %pre to:

  • Fetch data from the network by using binaries included in the installation environment, such as curl.
  • Inject trusted certificate authorities into the installation environment /etc/pki/ca-trust/source/anchors by using the update-ca-trust command.

You can configure insecure registries similarly by modifying the /etc/containers directory.

13.4. Configuring container pull secrets

To be able to fetch container images, you must configure a host system with a "pull secret", which includes the host updates itself. See the appendix about Injecting secrets in image mode for RHEL documentation for more details.

You can configure the container pull secrets to an image already built. If you use an external installer such as Anaconda for bare metal, or bootc-image-builder, you must configure the systems with any applicable pull secrets.

The host bootc updates write the configuration to the /etc/ostree/auth.json file, which is shared with rpm-ostree.

Podman does not have system wide credentials. Podman accepts the containers-auth locations that are underneath the following directories:

  • /run: The content of this directory vanishes on reboot, which is not desired.
  • /root: Part of root home directory, which is local mutable state by default.

To unify bootc and Podman credentials, use a single default global pull secret for both bootc and Podman. The following container build is an example to unify the bootc and the Podman credentials. The example expects a secret named creds to contain the registry pull secret to build.

Procedure

  1. Create a symbolic link between bootc and Podman to use a single pull secret. By creating the symbolic link, you ensure that both locations are present to a common persistent file embedded in the container image.

  2. Create the /usr/lib/container-auth.json file. 

    FROM quay.io/<namespace>/<image>:<tag>
COPY containers-auth.conf /usr/lib/tmpfiles.d/link-podman-credentials.conf
RUN --mount=type=secret,id=creds,required=true cp /run/secrets/creds /usr/lib/container-auth.json && \
    chmod 0600 /usr/lib/container-auth.json && \
    ln -sr /usr/lib/container-auth.json /etc/ostree/auth.json
    Copy to Clipboard

    When you run the containerfile, the following actions happen:

    • The Containerfile makes /run/containers/0/auth.json a transient runtime file.
    • It creates a symbolic link to the /usr/lib/container-auth.json.
    • It also creates a persistent file, which is also symbolic linked from /etc/ostree/auth.json.

Chapter 14. Appendix: System configuration

14.1. Transient runtime reconfiguration

You can perform a dynamic reconfiguration in the base image configuration. For example, you can run the firewall-cmd --permanent command to achieve persistent changes across a reboot.

Warning

The /etc directory is persistent by default. If you perform changes made by using tools, for example firewall-cmd --permanent, the contents of the /etc on the system can differ from the one described in the container image.

In the default configuration, first make the changes in the base image, then queue the changes without restarting running systems, and then simultaneously write to apply the changes to existing systems only in memory.

You can configure the /etc directory to be transient by using bind mounts. In this case, the etc directory is a part of the machine’s local root filesystem. For example, if you inject static IP addresses by using Anaconda Kickstart, they persist across upgrades.

A 3-way merge is applied across upgrades and each "deployment" has its own copy of /etc.

The /run directory
The /run directory is an API filesystem that is defined to be deleted when the system is restarted. Use the /run directory for transient files.
Dynamic reconfiguration models
In the Pull model, you can include code directly embedded in your base image or a privileged container that contacts the remote network server for configuration, and subsequently launch additional container images, by using the Podman API.

In the Push model, some workloads are implemented by tools such as Ansible.

systemd
You can use systemd units for dynamic transient reconfiguration by writing to /run/systemd directory. For example, the systemctl edit --runtime myservice.service dynamically changes the configuration of the myservice.service unit, without persisting the changes.
NetworkManager
Use a /run/NetworkManager/conf.d directory for applying temporary network configuration. Use the nmcli connection modify --temporary command to write changes only in memory. Without the --temporary option, the command writes persistent changes.
Podman
Use the podman run --rm command to automatically remove the container when it exits. Without the --rm option, the podman run command creates a container that persists across system reboots.

14.2. Using DNF

The rhel10/rhel-bootc container image includes dnf. There are several use cases:

Using dnf as a part of a container build
You can use the RUN dnf install directive in the Containerfile.
Using dnf at runtime
Warning

The functionality depends on the dnf version. You might get an error: error: can’t create transaction lock on /usr/share/rpm/.rpm.lock (Read-only file system).

You can use the bootc-usr-overlay command to create a writable overlay filesystem for /usr directory. The dnf install writes to this overlay. You can use this feature for installing debugging tools. Note that changes will be lost on reboot.

Configuring storage

The supported storage technologies are the following:

  • xfs/ext4
  • Logical volume management (LVM)
  • Linux Unified Key Setup (LUKS)

You can add other storage packages to the host system.

  • Storage with bootc-image-builder You can use the bootc-image-builder tool to create a disk image. The available configuration for partitioning and layout is relatively fixed. The default filesystem type is derived from the container image’s bootc install configuration.
  • Storage with bootc install You can use the bootc install to-disk command for flat storage configurations and bootc install to-filesytem command for more advanced installations. For more information see Advanced installation with to-filesystem.

14.3. Network configuration

The default images include the NetworkManager dynamic network control and configuration system, and bootc attempts to connect by using DHCP on every interface with a cable plugged in. You can apply a temporary network configuration, by setting up the /run/NetworkManager/conf.d directory.

However, if you need to use static addressing or more complex networking such as VLANs, bonds, bridges, teams, among others, you can use different ways. Regardless of the way you choose to configure networking, it results as a configuration for NetworkManager, which takes the form of NetworkManager keyfiles.

Host Network Configuration options
Complex networking configuration often also requires per-machine state. You can generate machine-specific container images that have, for example, static IP addressing included. You can also include code to generate network configuration from inside the image by inspecting the MAC address of the host.
Network configuration options available

The following are the available options for configuring static IP, and how the configuration should be done:

  • By using a Containerfile: Create a container image with static IP or include code to generate network configuration from inside the image based on MAC address.
  • By using Anaconda: You can use an Anaconda Kickstart to configure networking, including Wi-Fi, for bare-metal installations. The configuration is stored by default in /etc/NetworkManager/system-connections/, and is inherently per-machine state.
  • By using kernel arguments: Add kernel parameters on first boot to define networking configuration. On the first boot of a machine, enter kernel arguments that define networking configuration. The kernel arguments are mostly defined in the dracut.cmdline man page. You can apply these kernel arguments on first boot by using different methods. When using bootc install, you can also set per-machine kernel arguments by using --karg.
  • By using NetworkManager key files: nmcli or nm-initrd-generator

Generating a NetworkManager keyfiles by using nmcli

The nmcli NetworkManager command line tool provides an offline mode that does not communicate with the NetworkManager daemon and just writes the keyfile content to standard output.

  • Run the nmcli tool for each connection profile you want to create: 

    $ nmcli --offline connection add \
        type ethernet ifname enp1s0 \
        ipv4.method manual ipv4.addresses 192.0.0.1/24 \
        ipv6.method disabled

[connection]
id=ethernet-enp1s0
uuid=ff242096-f803-425f-9a77-4c3ec92686bd
type=ethernet
interface-name=enp1s0

[ethernet]

[ipv4]
address1=192.0.0.1/24
method=manual

[ipv6]
addr-gen-mode=default
method=disabled
[proxy]
    Copy to Clipboard

See the settings man page for a list of the properties that can be specified by using nmcli. Bash autocompletion is available.

Generating NetworkManager Keyfiles by using nm-initrd-generator

NetworkManager contains the nm-initrd-generator tool, that can generate keyfiles from dracut kernel argument syntax. You can use the tool to either convert from kernel arguments to keyfiles or to just quickly generate some keyfiles giving a small amount of input and then modify some more detailed settings.

  • Generate keyfiles for a bond by using nm-initrd-generator: 

    $ podman run --rm -ti quay.io/<namespace>/<image>:<tag> /usr/libexec/nm-initrd-generator -s -- "ip=bond0:dhcp" "bond=bond0:ens2,ens3:mode=active-backup,miimon=100" "nameserver=8.8.8.8"

* Connection 'bond0' *

[connection]
id=bond0
uuid=643c17b5-b364-4137-b273-33f450a45476
type=bond
interface-name=bond0
multi-connect=1
permissions=

[ethernet]
mac-address-blacklist=

[bond]
miimon=100
mode=active-backup

[ipv4]
dns=8.8.8.8;
dns-search=
may-fail=false
method=auto

[ipv6]
addr-gen-mode=eui64
dns-search=
method=auto

[proxy]

* Connection 'ens3' *

[connection]
id=ens3
uuid=b42cc917-fd87-47df-9ac2-34622ecddd8c
type=ethernet
interface-name=ens3
master=643c17b5-b364-4137-b273-33f450a45476
multi-connect=1
permissions=
slave-type=bond

[ethernet]
mac-address-blacklist=

* Connection 'ens2' *

[connection]
id=ens2
uuid=e111bb4e-3ee3-4612-afc2-1d2dfff97671
type=ethernet
interface-name=ens2
master=643c17b5-b364-4137-b273-33f450a45476
multi-connect=1
permissions=
slave-type=bond

[ethernet]
mac-address-blacklist=
    Copy to Clipboard

The command generates three keyfiles for each interface: bond0, ens3, and ens2. You can use the generated output, add more settings or modify existing settings, and then commit the files into a container image.

Configuring a Static IP

  • You can use the following dracut kernel arguments:

    Template: 

ip=${ip}::${gateway}:${netmask}:${hostname}:${interface}:none:${nameserver}
Copy to Clipboard

Example: 

ip=10.10.10.10::10.10.10.1:255.255.255.0:myhostname:ens2:none:8.8.8.8
Copy to Clipboard

Writing configuration embedded in container images

Store the NetworkManager configuration embedded in container images in /usr/lib/NetworkManager/system-connections/ because this form is part of the immutable image state. You can also write configuration to /etc/NetworkManager/system-connections/ as part of the container image. The default OSTree 3-way merge, that is, using the old default configuration, the active /etc system, and the new default configuration, applies with any machine-specific configuration.

The keyfiles must have the 600 root-only access permissions, otherwise NetworkManager ignores them.

Disabling automatic configuration of Ethernet devices

By default, NetworkManager attempts to autoconfigure by using the DHCP or SLAAC addresses on every interface with a cable plugged in. In some network environments this might not be desirable. For that, it is possible to change the NetworkManager behavior by adding a configuration file, such as /usr/lib/NetworkManager/conf.d/noauto.conf.

  • Disable the NetworkManager autoconfiguration of Ethernet devices 

    [main]
# Do not do automatic (DHCP or SLAAC) configuration on ethernet devices
# with no other matching connections.
no-auto-default=*
    Copy to Clipboard

14.4. Setting a hostname

To set a custom hostname for your system, modify the /etc/hostname file. You can set the hostname by using Anaconda, or with a privileged container.

Once you boot a system, you can verify the hostname by using the hostnamectl command.

14.5. Proxied Internet Access

If you are deploying to an environment requiring internet access by using a proxy, you need to configure services so that they can access resources as intended.

This is done by defining a single file with required environment variables in your configuration, and to reference this by using systemd drop-in unit files for all such services.

Defining common proxy environment variables
This common file has to be subsequently referenced explicitly by each service that requires internet access. 
# /etc/example-proxy.env
https_proxy="http://example.com:8080"
all_proxy="http://example.com:8080"
http_proxy="http://example.com:8080"
HTTP_PROXY="http://example.com:8080"
HTTPS_PROXY="http://example.com:8080"
no_proxy="*.example.com,127.0.0.1,0.0.0.0,localhost"
Copy to Clipboard
Defining drop-in units for core services
The bootc and podman tools commonly need proxy configuration. At the current time, bootc does not always run as a systemd unit. 
# /usr/lib/systemd/system/bootc-fetch-apply-updates.service.d/99-proxy.conf
[Service]
EnvironmentFile=/etc/example-proxy.env
Copy to Clipboard
Defining proxy use for podman systemd units

Using the Podman systemd configuration, similarly add EnvironmentFile=/etc/example-proxy.env.

You can set the configuration for proxy and environment settings of podman and containers in the /etc/containers/containers.conf configuration file as a root user or in the $HOME/.config/containers/containers.conf configuration file as a non-root user.

Chapter 15. Appendix: Getting the source code of container images

You can find the source code for bootc image in the Red Hat Ecosystem Catalog.

Procedure

  1. Access the Red Hat Ecosystem Catalog and search for rhel-bootc.
  2. In the Get this image tab, click Get the source and follow the instructions.

  3. After you extract the content, the input RPM package list and other content resources are available in the extra_src_dir directory.

    The .tar files are snapshots of the input git repository, and contain YAML files with the package lists.

Chapter 16. Appendix: Contributing to the upstream projects

You can contribute to the following upstream bootc projects:

Legal Notice

Copyright © 2025 Red Hat, Inc.
The text of and illustrations in this document are licensed by Red Hat under a Creative Commons Attribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA is available at http://creativecommons.org/licenses/by-sa/3.0/. In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must provide the URL for the original version.
Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.
Red Hat, Red Hat Enterprise Linux, the Shadowman logo, the Red Hat logo, JBoss, OpenShift, Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries.
Linux® is the registered trademark of Linus Torvalds in the United States and other countries.
Java® is a registered trademark of Oracle and/or its affiliates.
XFS® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries.
MySQL® is a registered trademark of MySQL AB in the United States, the European Union and other countries.
Node.js® is an official trademark of Joyent. Red Hat is not formally related to or endorsed by the official Joyent Node.js open source or commercial project.
The OpenStack® Word Mark and OpenStack logo are either registered trademarks/service marks or trademarks/service marks of the OpenStack Foundation, in the United States and other countries and are used with the OpenStack Foundation's permission. We are not affiliated with, endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.
All other trademarks are the property of their respective owners.
Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat