Chapter 6. Introduction to reproducible container builds

• Reduced registry storage: When an image is updated, only changed layers are stored. Reproducible builds ensure identical images from the same source code by preventing non-deterministic factors (timestamps, file order, metadata) from causing changes, thus avoiding additional storage.
• Efficient and smaller update payloads: For container minor updates, for example, security patches, only the changed layer needs to be downloaded, not the whole image. Reproducible builds also ensure that only affected layers change with source updates, unlike non-reproducible builds where small code changes can alter multiple layers.
• Faster downloads: Container reproducible builds optimize both build systems and end users by enabling faster downloads through efficient caching, and reduced network traffic.

Konflux

• Enhanced software supply chain integrity: Reproducible container builds enhance Konflux’s mission of delivering a secure and transparent software supply chain. Konflux uses reproducible builds to verify that a built container image derives exactly from its source code. Any third party can rebuild the container from the same inputs and verify that the output is bit-for-bit identical. Also, RHEL reproducible container builds protect against "in-transit" vulnerabilities, where an attacker can compromise a distribution mirror or inject malicious code into the build process. Konflux can prove that the released binary matches its source, mitigating attacks on its own build infrastructure.
• Improved compliance and transparency: Konflux enforces SLSA security policies. It verifies the origin and provenance of reproducible RHEL images, simplifying compliance. Konflux uses Tekton Chains to create an immutable, signed attestation that documents the entire build process. RHEL’s reproducible container builds add a foundational layer of trust to this attestation by ensuring the base image is trustworthy and verifiably built.
• Development and security workflows: Reproducible builds guarantee consistent container image digests across multiple runs, simplifying testing and debugging. Konflux leverages this to efficiently scan and update vulnerable packages in RHEL containers. Konflux uses verified attestations to automatically block non-compliant builds and enforce security policies without reducing flexibility.

Bootc

• Verifiable supply chain and enhanced security: Reproducible RHEL container builds enhance rhel-bootc by creating a more secure, reliable, and transparent build process for bootable OS images. You can verify that a specific bootc image was built from its claimed source code, which makes it more difficult for attackers to inject malicious code into a container image by compromising a build pipeline.

• Streamlined CI/CD and GitOps Workflows: You can use reproducibility to manage their entire OS configuration and application stack using Git-based workflows (GitOps). A change to the Containerfile, guarantees a consistent bootable image across all environments. Reproducible builds form a cornerstone of automated CI/CD pipelines.

RHEL AI

• Reproducible container builds are critical for RHEL AI because they provide the foundational consistency, security, and efficiency AI model development and deployment need. RHEL AI delivers a bootable container image, which means it manages the operating system itself as a container artifact. Reproducibility ensures that this base AI environment is always consistent and trustworthy.

• Fixed timestamps: Timestamps cause major irreproducibility. By default, file creation and modification times reflect when someone adds a file to a container layer, which is never the same twice. Buildah allows you to zero out these timestamps or set them to a specific, fixed value.

  • -–rewrite-timestamp: This option timestamps the contents of layers to be no later than the --source-date-epoch. Also, controls the created timestamp of an image and the timestamps of files within its layers, primarily to achieve deterministic builds.
  • --source-date-epoch: This option is more flexible option than the older --timestamp option, allowing you to define a specific, reproducible timestamp for all files in the image layer. It affects creation and history dates in image metadata. You can set it by using CLI flag, environment variable, or as a build-arg. When the flag is set, declared ARGs are exposed in the environment for RUN instructions and get static hostname. Also, the container ID field is cleared in the committed image.

Procedure

• To set these options when running your Buildah command. For example, to build an image from a Containerfile and force all timestamps to a specific point in time:

  Use a specific, immutable image.
  FROM registry.access.redhat.com/ubi9/ubi:9.4 AS builder
  
  # Set the SOURCE_DATE_EPOCH for deterministic timestamps
  ARG SOURCE_DATE_EPOCH
  ENV SOURCE_DATE_EPOCH=${SOURCE_DATE_EPOCH:-1}
  
  # Build the image using the build-arg and rewrite-timestamp options
  buildah bud --build-arg --source-date-epoch=${SOURCE_DATE_EPOCH} \
    --rewrite-timestamp \
    -f Containerfile \
    -t my-reproducible-image .

  Copy to Clipboard Toggle word wrap

• Run the podman build command with a consistent timestamp to create the reproducible image:

  # Set a consistent timestamp using the last Git commit date
  export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)

  Copy to Clipboard Toggle word wrap

• Build the image with the specified timestamp:

  podman build --source-date-epoch=${SOURCE_DATE_EPOCH} --rewrite-timestamp -t my-reproducible-app .

  Copy to Clipboard Toggle word wrap

Verification

• After building, run the reproducible command again. If the build is truly reproducible, the buildah inspect command should show the same image digest.

  buildah bud --build-arg --source-date-epoch=${SOURCE_DATE_EPOCH} \ --rewrite-timestamp \ -f Containerfile \ -t my-reproducible-image-2 .

  Copy to Clipboard Toggle word wrap

• Compare the digests:

  buildah inspect --format '{{.Digest}}' my-reproducible-image
  buildah inspect --format '{{.Digest}}' my-reproducible-image-2

  Copy to Clipboard Toggle word wrap