Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 4. Virtual builds with Red Hat Quay on OpenShift Container Platform

To build container images in virtualized environments with {product-title} on Red Hat Quay on OpenShift Container Platform, you can use virtual builds. Virtual builds run in unprivileged containers and provide isolation for your build processes.

With virtual builds, the build manager first creates the Job Object resource. Then, the Job Object creates a pod using the quay-builder-image. The quay-builder-image contains the quay-builder binary and the Podman service. The created pod runs as unprivileged. The quay-builder binary then builds the image while communicating status and retrieving build information from the build manager.

4.1. Virtual builds limitations
Copy link

The following limitations apply to the virtual builds feature:

  • Running virtual builds with Red Hat Quay on OpenShift Container Platform in an unprivileged context might cause some commands that were working under the previous build strategy to fail. Attempts to change the build strategy could potentially cause performance issues and reliability with the build.
  • Running virtual builds directly in a container does not have the same isolation as using virtual machines. Changing the build environment might also cause builds that were previously working to fail.

To configure virtual builds for Red Hat Quay on OpenShift Container Platform with {product-title}, you can create a build project, configure service accounts, and update your configuration file.

Note
  • If you are using Amazon Web Service (AWS) S3 storage, you must modify your storage bucket in the AWS console, prior to running builders. See "Modifying your AWS S3 storage bucket" in the following section for the required parameters.
  • If you are using a Google Cloud Platform (GCP) object bucket, you must configure cross-origin resource sharing (CORS) to enable virtual builds.

Prerequisites

  • You have an OpenShift Container Platform cluster provisioned with the Red Hat Quay Operator running.
  • You have set the tls component to unmanaged and uploaded custom SSL/TLS certificates to the Red Hat Quay Operator. For more information, see SSL and TLS for Red Hat Quay.
  • You have configured the OpenShift Container Platform TLS component for builds.
  • You are logged into OpenShift Container Platform as a cluster administrator.

Procedure

  1. Create a new project where your virtual builders will be run, for example, virtual-builders, by running the following command: 

    $ oc new-project virtual-builders
    Copy to Clipboard Toggle word wrap

  2. Create a ServiceAccount in the project that will be used to run builds by entering the following command: 

    $ oc create sa -n virtual-builders quay-builder
    Copy to Clipboard Toggle word wrap

    Example output

    serviceaccount/quay-builder created
    Copy to Clipboard Toggle word wrap

  3. Provide the created service account with editing permissions so that it can run a build: 

    $ oc adm policy -n virtual-builders add-role-to-user edit system:serviceaccount:virtual-builders:quay-builder
    Copy to Clipboard Toggle word wrap

    Example output

    clusterrole.rbac.authorization.k8s.io/edit added: "system:serviceaccount:virtual-builders:quay-builder"
    Copy to Clipboard Toggle word wrap

  4. Grant the builder worker anyuid scc permissions by entering the following command. This requires cluster administrator privileges, which is required because builders must run as the Podman user for unprivileged or rootless builds to work. 

    $ oc adm policy -n virtual-builders add-scc-to-user anyuid -z quay-builder
    Copy to Clipboard Toggle word wrap

    Example output

    clusterrole.rbac.authorization.k8s.io/system:openshift:scc:anyuid added: "quay-builder"
    Copy to Clipboard Toggle word wrap

  5. Obtain the token for the builder service account by entering the following command: 

    $ oc create token quay-builder -n virtual-builders
    Copy to Clipboard Toggle word wrap
    Note

    When the token expires you will need to request a new token. Optionally, you can also add a custom expiration. For example, specify --duration 20160m to retain the token for two weeks.

    Example output

    eyJhbGciOiJSUzI1NiIsImtpZCI6IldfQUJkaDVmb3ltTHZ0dGZMYjhIWnYxZTQzN2dJVEJxcDJscldSdEUtYWsifQ...
    Copy to Clipboard Toggle word wrap

  6. Determine the builder route by entering the following command: 

    $ oc get route -n quay-enterprise
    Copy to Clipboard Toggle word wrap

    Example output

    NAME: example-registry-quay-builder
HOST/PORT: example-registry-quay-builder-quay-enterprise.apps.stevsmit-cluster-new.gcp.quaydev.org
PATH:
SERVICES: example-registry-quay-app
PORT: grpc
TERMINATION: passthrough/Redirect
WILDCARD: None
    Copy to Clipboard Toggle word wrap

  7. Generate a self-signed SSL/TlS certificate with the .crt extension by entering the following command: 

    $ oc extract cm/kube-root-ca.crt -n openshift-apiserver
    Copy to Clipboard Toggle word wrap

    Example output

    ca.crt
    Copy to Clipboard Toggle word wrap

  8. Rename the ca.crt file to build-cluster.crt by entering the following command: 

    $ mv ca.crt build-cluster.crt
    Copy to Clipboard Toggle word wrap

  9. Update the config.yaml file of your Red Hat Quay on OpenShift Container Platform deployment to include an appropriate virtual builds configuration by using the OpenShift Container Platform web console.

    1. Click Operators Installed Operators Red Hat Quay Quay Registry.
    2. Click the name of your registry, for example, example-registry.
    3. Under Config Bundle Secret, click the name of your configuration bundle, for example, extra-ca-certificate-config-bundle-secret.
    4. Click Actions Edit Secret.

    5. Add an appropriate virtual builds configuration using the following as a reference: 

      FEATURE_USER_INITIALIZE: true
BROWSER_API_CALLS_XHR_ONLY: false
SUPER_USERS:
- <superusername>
FEATURE_USER_CREATION: false
FEATURE_QUOTA_MANAGEMENT: true
FEATURE_BUILD_SUPPORT: True
BUILDMAN_HOSTNAME: <sample_build_route>
BUILD_MANAGER:
  - ephemeral
  - ALLOWED_WORKER_COUNT: 1
    ORCHESTRATOR_PREFIX: buildman/production/
    JOB_REGISTRATION_TIMEOUT: 3600
    ORCHESTRATOR:
      REDIS_HOST: <sample_redis_hostname>
      REDIS_PASSWORD: ""
      REDIS_SSL: false
      REDIS_SKIP_KEYSPACE_EVENT_SETUP: false
    EXECUTORS:
      - EXECUTOR: kubernetesPodman
        NAME: openshift
        BUILDER_NAMESPACE: <sample_builder_namespace>
        SETUP_TIME: 180
        MINIMUM_RETRY_THRESHOLD: 0
        BUILDER_CONTAINER_IMAGE: quay.io/projectquay/quay-builder:{producty}
        # Kubernetes resource options
        K8S_API_SERVER: <sample_k8s_api_server>
        K8S_API_TLS_CA: <sample_crt_file>
        VOLUME_SIZE: 8G
        KUBERNETES_DISTRIBUTION: openshift
        CONTAINER_MEMORY_LIMITS: 1G
        CONTAINER_CPU_LIMITS: 300m
        CONTAINER_MEMORY_REQUEST: 1G
        CONTAINER_CPU_REQUEST: 300m
        NODE_SELECTOR_LABEL_KEY: ""
        NODE_SELECTOR_LABEL_VALUE: ""
        SERVICE_ACCOUNT_NAME: <sample_service_account_name>
        SERVICE_ACCOUNT_TOKEN: <sample_account_token>
        HTTP_PROXY: <http://10.0.0.1:80>
        HTTPS_PROXY: <http://10.0.0.1:80>
        NO_PROXY: <hostname.example.com>
      Copy to Clipboard Toggle word wrap

      where:

      BUILDMAN_HOSTNAME:: Specifies the hostname of the Red Hat Quay registry. Obtain this by running the following command: $ oc get route quayregistry-quay-builder -n ${QUAY_PROJECT} -o jsonpath='{.spec.host}'.

      REDIS_HOST:: Specifies the hostname for your Redis service.

      BUILDER_NAMESPACE:: Specifies the name of your bare metal builds namespace. This example used bare-metal-builder.

      K8S_API_SERVER:: Specifies the K8S_API_SERVER is obtained by running $ oc cluster-info.

      K8S_API_TLS_CA:: Specifies the name of your custom CA cert, for example, K8S_API_TLS_CA: /conf/stack/extra_ca_certs/build-cluster.crt.

      CONTAINER_MEMORY_LIMITS:: Specifies the memory limit for your container. Defaults to 5120Mi if left unspecified.

      CONTAINER_CPU_LIMITS:: Specifies the CPU limit for your container. Defaults to 1000m if left unspecified.

      CONTAINER_MEMORY_REQUEST:: Specifies the memory request for your container. Defaults to 3968Mi if left unspecified.

      CONTAINER_CPU_REQUEST:: Specifies the CPU request for your container. Defaults to 500m if left unspecified.

      SERVICE_ACCOUNT_TOKEN:: Specifies the token for your service account. Obtain this by running $ oc create sa.

      SSH_AUTHORIZED_KEYS:: Specifies the SSH authorized keys for your build environment. This key, or keys, should correspond to the private key that an admin or developer will use to SSH into the build worker for debugging purposes. This key can be obtained by establishing an SSH connection to the remote host using a specific SSH key and port. For example: $ ssh -i /path/to/ssh/key/set/in/ssh_authorized_keys -p 9999 core@localhost.

      Example virtual builds configuration

      FEATURE_USER_INITIALIZE: true
BROWSER_API_CALLS_XHR_ONLY: false
SUPER_USERS:
- quayadmin
FEATURE_USER_CREATION: false
FEATURE_QUOTA_MANAGEMENT: true
FEATURE_BUILD_SUPPORT: True
BUILDMAN_HOSTNAME: example-registry-quay-builder-quay-enterprise.apps.docs.quayteam.org:443
BUILD_MANAGER:
  - ephemeral
  - ALLOWED_WORKER_COUNT: 1
    ORCHESTRATOR_PREFIX: buildman/production/
    JOB_REGISTRATION_TIMEOUT: 3600
    ORCHESTRATOR:
      REDIS_HOST: example-registry-quay-redis
      REDIS_PASSWORD: ""
      REDIS_SSL: false
      REDIS_SKIP_KEYSPACE_EVENT_SETUP: false
    EXECUTORS:
      - EXECUTOR: kubernetesPodman
        NAME: openshift
        BUILDER_NAMESPACE: virtual-builders
        SETUP_TIME: 180
        MINIMUM_RETRY_THRESHOLD: 0
        BUILDER_CONTAINER_IMAGE: quay.io/projectquay/quay-builder:{producty}
        # Kubernetes resource options
        K8S_API_SERVER: api.docs.quayteam.org:6443
        K8S_API_TLS_CA: /conf/stack/extra_ca_certs/build-cluster.crt
        VOLUME_SIZE: 8G
        KUBERNETES_DISTRIBUTION: openshift
        CONTAINER_MEMORY_LIMITS: 1G
        CONTAINER_CPU_LIMITS: 300m
        CONTAINER_MEMORY_REQUEST: 1G
        CONTAINER_CPU_REQUEST: 300m
        NODE_SELECTOR_LABEL_KEY: ""
        NODE_SELECTOR_LABEL_VALUE: ""
        SERVICE_ACCOUNT_NAME: quay-builder
        SERVICE_ACCOUNT_TOKEN: "eyJhbGciOiJSUzI1NiIsImtpZCI6IldfQUJkaDVmb3ltTHZ0dGZMYjhIWnYxZTQzN2dJVEJxcDJscldSdEUtYWsifQ"
        HTTP_PROXY: <http://10.0.0.1:80>
        HTTPS_PROXY: <http://10.0.0.1:80>
        NO_PROXY: <hostname.example.com>
      Copy to Clipboard Toggle word wrap

    6. Click Save on the Edit Secret page.
  10. Restart your Red Hat Quay on OpenShift Container Platform registry with the new configuration.

4.2.1. Modifying your AWS S3 storage bucket
Copy link

To enable builds with AWS S3 storage in {product-title}, you can configure cross-origin resource sharing (CORS) settings in your S3 bucket. This allows build workers to access and store build artifacts in your S3 bucket.

Procedure

  1. Log in to your AWS console at s3.console.aws.com.
  2. In the search bar, search for S3 and then click S3.
  3. Click the name of your bucket, for example, myawsbucket.
  4. Click the Permissions tab.

  5. Under Cross-origin resource sharing (CORS), include the following parameters: 

      [
      {
          "AllowedHeaders": [
              "Authorization"
          ],
          "AllowedMethods": [
              "GET"
          ],
          "AllowedOrigins": [
              "*"
          ],
          "ExposeHeaders": [],
          "MaxAgeSeconds": 3000
      },
      {
          "AllowedHeaders": [
              "Content-Type",
              "x-amz-acl",
              "origin"
          ],
          "AllowedMethods": [
              "PUT"
          ],
          "AllowedOrigins": [
              "*"
          ],
          "ExposeHeaders": [],
          "MaxAgeSeconds": 3000
      }
  ]
    Copy to Clipboard Toggle word wrap

To enable virtual builds with Google Cloud Platform storage in {product-title}, you can configure cross-origin resource sharing (CORS) settings in your GCP bucket. This allows build workers to upload Dockerfiles and access build artifacts.

Note

Currently, modifying your Google Cloud Platform object bucket is not supported on IBM Power and IBM Z.

Procedure

  1. Use the following reference to create a JSON file for your specific CORS needs. For example: 

    $ cat gcp_cors.json
    Copy to Clipboard Toggle word wrap

    Example output

    [
    {
      "origin": ["*"],
      "method": ["GET"],
      "responseHeader": ["Authorization"],
      "maxAgeSeconds": 3600
    },
    {
      "origin": ["*"],
      "method": ["PUT"],
      "responseHeader": [
              "Content-Type",
              "x-goog-acl",
              "origin"],
      "maxAgeSeconds": 3600
    }
]
    Copy to Clipboard Toggle word wrap

  2. Enter the following command to update your GCP storage bucket: 

    $ gcloud storage buckets update gs://<bucket_name> --cors-file=./gcp_cors.json
    Copy to Clipboard Toggle word wrap

    Example output

    Updating
  Completed 1
    Copy to Clipboard Toggle word wrap

  3. You can display the updated CORS configuration of your GCP bucket by running the following command: 

    $ gcloud storage buckets describe gs://<bucket_name>  --format="default(cors)"
    Copy to Clipboard Toggle word wrap

    Example output

    cors:
- maxAgeSeconds: 3600
  method:
  - GET
  origin:
  - '*'
  responseHeader:
  - Authorization
- maxAgeSeconds: 3600
  method:
  - PUT
  origin:
  - '*'
  responseHeader:
  - Content-Type
  - x-goog-acl
  - origin
    Copy to Clipboard Toggle word wrap

Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2026 Red HatBack to top