SupportConsoleDevelopersStart a trialContact

Chapter 4. Virtual builds with Red Hat Quay on OpenShift Container Platform

Virtual builds in Red Hat Quay run build workers in unprivileged containers on Red Hat Quay on OpenShift Container Platform. This approach provides process isolation without requiring dedicated virtual machines for each build.

With virtual builds, the build manager first creates the Job Object resource. Then, the Job Object creates a pod using the quay-builder-image. The quay-builder-image contains the quay-builder binary and the Podman service. The created pod runs as unprivileged. The quay-builder binary then builds the image while communicating status and retrieving build information from the build manager.

4.1. Virtual builds limitations
Copy link

The following limitations apply to the virtual builds feature:

  • Running virtual builds with Red Hat Quay on OpenShift Container Platform in an unprivileged context might cause some commands that were working under the previous build strategy to fail. Attempts to change the build strategy could potentially cause performance issues and reliability with the build.
  • Running virtual builds directly in a container does not have the same isolation as using virtual machines. Changing the build environment might also cause builds that were previously working to fail.

To configure virtual builds for Red Hat Quay on OpenShift Container Platform with Red Hat Quay, you can create a build project, configure service accounts, and update your configuration file.

Note
  • If you are using Amazon Web Service (AWS) S3 storage, you must modify your storage bucket in the AWS console, prior to running builders. See "Modifying your AWS S3 storage bucket" in the following section for the required parameters.
  • If you are using a Google Cloud Platform (GCP) object bucket, you must configure cross-origin resource sharing (CORS) to enable virtual builds.

Prerequisites

  • You have an OpenShift Container Platform cluster provisioned with the Red Hat Quay Operator running.
  • You have set the tls component to unmanaged and uploaded custom SSL/TLS certificates to the Red Hat Quay Operator. For more information, see SSL and TLS for Red Hat Quay.
  • You have configured the OpenShift Container Platform TLS component for builds.
  • You are logged into OpenShift Container Platform as a cluster administrator.

Procedure

  1. Create a new project where your virtual builders will be run, for example, virtual-builders, by running the following command: 

    $ oc new-project virtual-builders

  2. Create a ServiceAccount in the project that will be used to run builds by entering the following command: 

    $ oc create sa -n virtual-builders quay-builder

    Example output

    serviceaccount/quay-builder created

  3. Provide the created service account with editing permissions so that it can run a build: 

    $ oc adm policy -n virtual-builders add-role-to-user edit system:serviceaccount:virtual-builders:quay-builder

    Example output

    clusterrole.rbac.authorization.k8s.io/edit added: "system:serviceaccount:virtual-builders:quay-builder"

  4. Grant the builder worker anyuid scc permissions by entering the following command. This requires cluster administrator privileges, which is required because builders must run as the Podman user for unprivileged or rootless builds to work. 

    $ oc adm policy -n virtual-builders add-scc-to-user anyuid -z quay-builder

    Example output

    clusterrole.rbac.authorization.k8s.io/system:openshift:scc:anyuid added: "quay-builder"

  5. Obtain the token for the builder service account by entering the following command: 

    $ oc create token quay-builder -n virtual-builders
    Note

    When the token expires you will need to request a new token. Optionally, you can also add a custom expiration. For example, specify --duration 20160m to retain the token for two weeks.

    Example output

    eyJhbGciOiJSUzI1NiIsImtpZCI6IldfQUJkaDVmb3ltTHZ0dGZMYjhIWnYxZTQzN2dJVEJxcDJscldSdEUtYWsifQ...

  6. Determine the builder route by entering the following command: 

    $ oc get route -n quay-enterprise

    Example output

    NAME: example-registry-quay-builder
HOST/PORT: example-registry-quay-builder-quay-enterprise.apps.stevsmit-cluster-new.gcp.quaydev.org
PATH:
SERVICES: example-registry-quay-app
PORT: grpc
TERMINATION: passthrough/Redirect
WILDCARD: None

  7. Generate a self-signed SSL/TLS certificate with the .crt extension by entering the following command: 

    $ oc extract cm/kube-root-ca.crt -n openshift-apiserver

    Example output

    ca.crt

  8. Rename the ca.crt file to build-cluster.crt by entering the following command: 

    $ mv ca.crt build-cluster.crt

  9. Update the config.yaml file of your Red Hat Quay on OpenShift Container Platform deployment to include an appropriate virtual builds configuration by using the OpenShift Container Platform web console.

    1. Click Operators Installed Operators Red Hat Quay Quay Registry.
    2. Click the name of your registry, for example, example-registry.
    3. Under Config Bundle Secret, click the name of your configuration bundle, for example, extra-ca-certificate-config-bundle-secret.
    4. Click Actions Edit Secret.

    5. Add an appropriate virtual builds configuration using the following as a reference: 

      FEATURE_USER_INITIALIZE: true
BROWSER_API_CALLS_XHR_ONLY: false
SUPER_USERS:
- <superusername>
FEATURE_USER_CREATION: false
FEATURE_QUOTA_MANAGEMENT: true
FEATURE_BUILD_SUPPORT: True
BUILDMAN_HOSTNAME: <sample_build_route>
BUILD_MANAGER:
  - ephemeral
  - ALLOWED_WORKER_COUNT: 1
    ORCHESTRATOR_PREFIX: buildman/production/
    JOB_REGISTRATION_TIMEOUT: 3600
    ORCHESTRATOR:
      REDIS_HOST: <sample_redis_hostname>
      REDIS_PASSWORD: ""
      REDIS_SSL: false
      REDIS_SKIP_KEYSPACE_EVENT_SETUP: false
    EXECUTORS:
      - EXECUTOR: kubernetesPodman
        NAME: openshift
        BUILDER_NAMESPACE: <sample_builder_namespace>
        SETUP_TIME: 180
        MINIMUM_RETRY_THRESHOLD: 0
        BUILDER_CONTAINER_IMAGE: quay.io/projectquay/quay-builder:{producty}
        # Kubernetes resource options
        K8S_API_SERVER: <sample_k8s_api_server>
        K8S_API_TLS_CA: <sample_crt_file>
        VOLUME_SIZE: 8G
        KUBERNETES_DISTRIBUTION: openshift
        CONTAINER_MEMORY_LIMITS: 1G
        CONTAINER_CPU_LIMITS: 300m
        CONTAINER_MEMORY_REQUEST: 1G
        CONTAINER_CPU_REQUEST: 300m
        NODE_SELECTOR_LABEL_KEY: ""
        NODE_SELECTOR_LABEL_VALUE: ""
        SERVICE_ACCOUNT_NAME: <sample_service_account_name>
        SERVICE_ACCOUNT_TOKEN: <sample_account_token>
        HTTP_PROXY: <http://10.0.0.1:80>
        HTTPS_PROXY: <http://10.0.0.1:80>
        NO_PROXY: <hostname.example.com>

      where:

      BUILDMAN_HOSTNAME
      Specifies that the build route is obtained by running $ oc get route -n with the namespace of your Red Hat Quay on OpenShift Container Platform deployment. A port must be provided at the end of the route, and it should use the following format: [quayregistry-cr-name]-quay-builder-[ocp-namespace].[ocp-domain-name]:443.
      BUILD_MANAGER.JOB_REGISTRATION_TIMEOUT
      Specifies that you might receive the following error when set too low: failed to register job to build manager: rpc error: code = Unauthenticated desc = Invalid build token: Signature has expired. This parameter should be set to at least 240.
      BUILD_MANAGER.ORCHESTRATOR.REDIS_HOST
      Specifies that you must update this field accordingly if your Redis host has a password or SSL/TLS certificates.
      BUILD_MANAGER.EXECUTORS.BUILDER_NAMESPACE
      Specifies the name of your virtual builds namespace. This example used virtual-builders.
      BUILD_MANAGER.EXECUTORS.K8S_API_SERVER
      Specifies the value obtained by running $ oc cluster-info.
      BUILD_MANAGER.EXECUTORS.K8S_API_TLS_CA
      Specifies that you must manually create and add your custom CA cert, for example, K8S_API_TLS_CA: /conf/stack/extra_ca_certs/build-cluster.crt.
      BUILD_MANAGER.EXECUTORS.CONTAINER_MEMORY_LIMITS
      Specifies the memory limit. Defaults to 5120Mi if left unspecified.
      BUILD_MANAGER.EXECUTORS.CONTAINER_CPU_LIMITS
      Specifies that for virtual builds, you must ensure that there are enough resources in your cluster. Defaults to 1000m if left unspecified.
      BUILD_MANAGER.EXECUTORS.CONTAINER_MEMORY_REQUEST
      Specifies the memory request. Defaults to 3968Mi if left unspecified.
      BUILD_MANAGER.EXECUTORS.CONTAINER_CPU_REQUEST
      Specifies the CPU request. Defaults to 500m if left unspecified.
      BUILD_MANAGER.EXECUTORS.SERVICE_ACCOUNT_TOKEN
      Specifies the token obtained when running $ oc create sa.

      Example virtual builds configuration

      FEATURE_USER_INITIALIZE: true
BROWSER_API_CALLS_XHR_ONLY: false
SUPER_USERS:
- quayadmin
FEATURE_USER_CREATION: false
FEATURE_QUOTA_MANAGEMENT: true
FEATURE_BUILD_SUPPORT: True
BUILDMAN_HOSTNAME: example-registry-quay-builder-quay-enterprise.apps.docs.quayteam.org:443
BUILD_MANAGER:
  - ephemeral
  - ALLOWED_WORKER_COUNT: 1
    ORCHESTRATOR_PREFIX: buildman/production/
    JOB_REGISTRATION_TIMEOUT: 3600
    ORCHESTRATOR:
      REDIS_HOST: example-registry-quay-redis
      REDIS_PASSWORD: ""
      REDIS_SSL: false
      REDIS_SKIP_KEYSPACE_EVENT_SETUP: false
    EXECUTORS:
      - EXECUTOR: kubernetesPodman
        NAME: openshift
        BUILDER_NAMESPACE: virtual-builders
        SETUP_TIME: 180
        MINIMUM_RETRY_THRESHOLD: 0
        BUILDER_CONTAINER_IMAGE: quay.io/projectquay/quay-builder:{producty}
        # Kubernetes resource options
        K8S_API_SERVER: api.docs.quayteam.org:6443
        K8S_API_TLS_CA: /conf/stack/extra_ca_certs/build-cluster.crt
        VOLUME_SIZE: 8G
        KUBERNETES_DISTRIBUTION: openshift
        CONTAINER_MEMORY_LIMITS: 1G
        CONTAINER_CPU_LIMITS: 300m
        CONTAINER_MEMORY_REQUEST: 1G
        CONTAINER_CPU_REQUEST: 300m
        NODE_SELECTOR_LABEL_KEY: ""
        NODE_SELECTOR_LABEL_VALUE: ""
        SERVICE_ACCOUNT_NAME: quay-builder
        SERVICE_ACCOUNT_TOKEN: "eyJhbGciOiJSUzI1NiIsImtpZCI6IldfQUJkaDVmb3ltTHZ0dGZMYjhIWnYxZTQzN2dJVEJxcDJscldSdEUtYWsifQ"
        HTTP_PROXY: <http://10.0.0.1:80>
        HTTPS_PROXY: <http://10.0.0.1:80>
        NO_PROXY: <hostname.example.com>

    6. Click Save on the Edit Secret page.
  10. Restart your Red Hat Quay on OpenShift Container Platform registry with the new configuration.

4.2.1. Modifying your AWS S3 storage bucket
Copy link

To enable builds with AWS S3 storage in Red Hat Quay, you can configure cross-origin resource sharing (CORS) settings in your S3 bucket. This allows build workers to access and store build artifacts in your S3 bucket.

Procedure

  1. Log in to your AWS console at s3.console.aws.com.
  2. In the search bar, search for S3 and then click S3.
  3. Click the name of your bucket, for example, myawsbucket.
  4. Click the Permissions tab.

  5. Under Cross-origin resource sharing (CORS), include the following parameters: 

      [
      {
          "AllowedHeaders": [
              "Authorization"
          ],
          "AllowedMethods": [
              "GET"
          ],
          "AllowedOrigins": [
              "*"
          ],
          "ExposeHeaders": [],
          "MaxAgeSeconds": 3000
      },
      {
          "AllowedHeaders": [
              "Content-Type",
              "x-amz-acl",
              "origin"
          ],
          "AllowedMethods": [
              "PUT"
          ],
          "AllowedOrigins": [
              "*"
          ],
          "ExposeHeaders": [],
          "MaxAgeSeconds": 3000
      }
  ]

To enable virtual builds with Google Cloud Platform storage in Red Hat Quay, you can configure cross-origin resource sharing (CORS) settings in your GCP bucket. This allows build workers to upload Dockerfiles and access build artifacts.

Note

Currently, modifying your Google Cloud Platform object bucket is not supported on IBM Power and IBM Z.

Procedure

  1. Use the following reference to create a JSON file for your specific CORS needs. For example: 

    $ cat gcp_cors.json

    Example output

    [
    {
      "origin": ["*"],
      "method": ["GET"],
      "responseHeader": ["Authorization"],
      "maxAgeSeconds": 3600
    },
    {
      "origin": ["*"],
      "method": ["PUT"],
      "responseHeader": [
              "Content-Type",
              "x-goog-acl",
              "origin"],
      "maxAgeSeconds": 3600
    }
]

  2. Enter the following command to update your GCP storage bucket: 

    $ gcloud storage buckets update gs://<bucket_name> --cors-file=./gcp_cors.json

    Example output

    Updating
  Completed 1

  3. You can display the updated CORS configuration of your GCP bucket by running the following command: 

    $ gcloud storage buckets describe gs://<bucket_name>  --format="default(cors)"

    Example output

    cors:
- maxAgeSeconds: 3600
  method:
  - GET
  origin:
  - '*'
  responseHeader:
  - Authorization
- maxAgeSeconds: 3600
  method:
  - PUT
  origin:
  - '*'
  responseHeader:
  - Content-Type
  - x-goog-acl
  - origin

Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Legal Notice

Theme

© 2026 Red HatBack to top