Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 4. Virtual builds with Red Hat Quay on OpenShift Container Platform

The procedures in this section explain how to create an environment for bare metal builds for Red Hat Quay on OpenShift Container Platform.

Virtual builds can be run on virtualized machines with Red Hat Quay on OpenShift Container Platform. With this method, the build manager first creates the Job Object resource. Then, the Job Object creates a pod using the quay-builder-image. The quay-builder-image contains the quay-builder binary and the Podman service. The created pod runs as unprivileged. The quay-builder binary then builds the image while communicating status and retrieving build information from the build manager.

4.1. Virtual builds limitations
Copy link

The following limitations apply to the virtual builds feature:

  • Running virtual builds with Red Hat Quay on OpenShift Container Platform in an unprivileged context might cause some commands that were working under the previous build strategy to fail. Attempts to change the build strategy could potentially cause performance issues and reliability with the build.
  • Running virtual builds directly in a container does not have the same isolation as using virtual machines. Changing the build environment might also cause builds that were previously working to fail.

The procedures in this section explain how to create an environment for virtual builds for Red Hat Quay on OpenShift Container Platform.

Note
  • If you are using Amazon Web Service (AWS) S3 storage, you must modify your storage bucket in the AWS console, prior to running builders. See "Modifying your AWS S3 storage bucket" in the following section for the required parameters.
  • If you are using a Google Cloud Platform (GCP) object bucket, you must configure cross-origin resource sharing (CORS) to enable virtual builds.

Prerequisites

  • You have an OpenShift Container Platform cluster provisioned with the Red Hat Quay Operator running.
  • You have set the tls component to unmanaged and uploaded custom SSL/TLS certificates to the Red Hat Quay Operator. For more information, see SSL and TLS for Red Hat Quay.
  • You have configured the OpenShift Container Platform TLS component for builds.
  • You are logged into OpenShift Container Platform as a cluster administrator.

Procedure

  1. Create a new project where your virtual builders will be run, for example, virtual-builders, by running the following command: 

    $ oc new-project virtual-builders
    Copy to Clipboard Toggle word wrap

  2. Create a ServiceAccount in the project that will be used to run builds by entering the following command: 

    $ oc create sa -n virtual-builders quay-builder
    Copy to Clipboard Toggle word wrap

    Example output

    serviceaccount/quay-builder created
    Copy to Clipboard Toggle word wrap

  3. Provide the created service account with editing permissions so that it can run a build: 

    $ oc adm policy -n virtual-builders add-role-to-user edit system:serviceaccount:virtual-builders:quay-builder
    Copy to Clipboard Toggle word wrap

    Example output

    clusterrole.rbac.authorization.k8s.io/edit added: "system:serviceaccount:virtual-builders:quay-builder"
    Copy to Clipboard Toggle word wrap

  4. Grant the builder worker anyuid scc permissions by entering the following command. This requires cluster administrator privileges, which is required because builders must run as the Podman user for unprivileged or rootless builds to work. 

    $ oc adm policy -n virtual-builders add-scc-to-user anyuid -z quay-builder
    Copy to Clipboard Toggle word wrap

    Example output

    clusterrole.rbac.authorization.k8s.io/system:openshift:scc:anyuid added: "quay-builder"
    Copy to Clipboard Toggle word wrap

  5. Obtain the token for the builder service account by entering the following command: 

    $ oc create token quay-builder -n virtual-builders
    Copy to Clipboard Toggle word wrap
    Note

    When the token expires you will need to request a new token. Optionally, you can also add a custom expiration. For example, specify --duration 20160m to retain the token for two weeks.

    Example output

    eyJhbGciOiJSUzI1NiIsImtpZCI6IldfQUJkaDVmb3ltTHZ0dGZMYjhIWnYxZTQzN2dJVEJxcDJscldSdEUtYWsifQ...
    Copy to Clipboard Toggle word wrap

  6. Determine the builder route by entering the following command: 

    $ oc get route -n quay-enterprise
    Copy to Clipboard Toggle word wrap

    Example output

    NAME: example-registry-quay-builder
HOST/PORT: example-registry-quay-builder-quay-enterprise.apps.stevsmit-cluster-new.gcp.quaydev.org
PATH:
SERVICES: example-registry-quay-app
PORT: grpc
TERMINATION: passthrough/Redirect
WILDCARD: None
    Copy to Clipboard Toggle word wrap

  7. Generate a self-signed SSL/TlS certificate with the .crt extension by entering the following command: 

    $ oc extract cm/kube-root-ca.crt -n openshift-apiserver
    Copy to Clipboard Toggle word wrap

    Example output

    ca.crt
    Copy to Clipboard Toggle word wrap

  8. Rename the ca.crt file to build-cluster.crt by entering the following command: 

    $ mv ca.crt build-cluster.crt
    Copy to Clipboard Toggle word wrap

  9. Update the config.yaml file of your Red Hat Quay on OpenShift Container Platform deployment to include an appropriate virtual builds configuration by using the OpenShift Container Platform web console.

    1. Click Operators Installed Operators Red Hat Quay Quay Registry.
    2. Click the name of your registry, for example, example-registry.
    3. Under Config Bundle Secret, click the name of your configuration bundle, for example, extra-ca-certificate-config-bundle-secret.
    4. Click Actions Edit Secret.

    5. Add an appropriate virtual builds configuration using the following as a reference: 

      FEATURE_USER_INITIALIZE: true
BROWSER_API_CALLS_XHR_ONLY: false
SUPER_USERS:
- <superusername>
FEATURE_USER_CREATION: false
FEATURE_QUOTA_MANAGEMENT: true
FEATURE_BUILD_SUPPORT: True
BUILDMAN_HOSTNAME: <sample_build_route>
      1 
      
BUILD_MANAGER:
  - ephemeral
  - ALLOWED_WORKER_COUNT: 1
    ORCHESTRATOR_PREFIX: buildman/production/
    JOB_REGISTRATION_TIMEOUT: 3600
      2 
      
    ORCHESTRATOR:
      REDIS_HOST: <sample_redis_hostname>
      3 
      
      REDIS_PASSWORD: ""
      REDIS_SSL: false
      REDIS_SKIP_KEYSPACE_EVENT_SETUP: false
    EXECUTORS:
      - EXECUTOR: kubernetesPodman
        NAME: openshift
        BUILDER_NAMESPACE: <sample_builder_namespace>
      4 
      
        SETUP_TIME: 180
        MINIMUM_RETRY_THRESHOLD: 0
        BUILDER_CONTAINER_IMAGE: quay.io/projectquay/quay-builder:{producty}
        # Kubernetes resource options
        K8S_API_SERVER: <sample_k8s_api_server>
      5 
      
        K8S_API_TLS_CA: <sample_crt_file>
      6 
      
        VOLUME_SIZE: 8G
        KUBERNETES_DISTRIBUTION: openshift
        CONTAINER_MEMORY_LIMITS: 1G
      7 
      
        CONTAINER_CPU_LIMITS: 300m
      8 
      
        CONTAINER_MEMORY_REQUEST: 1G
      9 
      
        CONTAINER_CPU_REQUEST: 300m
      10 
      
        NODE_SELECTOR_LABEL_KEY: ""
        NODE_SELECTOR_LABEL_VALUE: ""
        SERVICE_ACCOUNT_NAME: <sample_service_account_name>
        SERVICE_ACCOUNT_TOKEN: <sample_account_token>
      11 
      
        HTTP_PROXY: <http://10.0.0.1:80>
        HTTPS_PROXY: <http://10.0.0.1:80>
        NO_PROXY: <hostname.example.com>
      Copy to Clipboard Toggle word wrap
      1
      The build route is obtained by running $ oc get route -n with the namespace of your Red Hat Quay on OpenShift Container Platform deployment. A port must be provided at the end of the route, and it should use the following format: [quayregistry-cr-name]-quay-builder-[ocp-namespace].[ocp-domain-name]:443.
      2
      If the JOB_REGISTRATION_TIMEOUT parameter is set too low, you might receive the following error: failed to register job to build manager: rpc error: code = Unauthenticated desc = Invalid build token: Signature has expired. This parameter should be set to at least 240.
      3
      If your Redis host has a password or SSL/TLS certificates, you must update this field accordingly.
      4
      Set to match the name of your virtual builds namespace. This example used virtual-builders.
      5
      The K8S_API_SERVER is obtained by running $ oc cluster-info.
      6
      You must manually create and add your custom CA cert, for example, K8S_API_TLS_CA: /conf/stack/extra_ca_certs/build-cluster.crt.
      7
      Defaults to 5120Mi if left unspecified.
      8
      For virtual builds, you must ensure that there are enough resources in your cluster. Defaults to 1000m if left unspecified.
      9
      Defaults to 3968Mi if left unspecified.
      10
      Defaults to 500m if left unspecified.
      11
      Obtained when running $ oc create sa.

      Example virtual builds configuration

      FEATURE_USER_INITIALIZE: true
BROWSER_API_CALLS_XHR_ONLY: false
SUPER_USERS:
- quayadmin
FEATURE_USER_CREATION: false
FEATURE_QUOTA_MANAGEMENT: true
FEATURE_BUILD_SUPPORT: True
BUILDMAN_HOSTNAME: example-registry-quay-builder-quay-enterprise.apps.docs.quayteam.org:443
BUILD_MANAGER:
  - ephemeral
  - ALLOWED_WORKER_COUNT: 1
    ORCHESTRATOR_PREFIX: buildman/production/
    JOB_REGISTRATION_TIMEOUT: 3600
    ORCHESTRATOR:
      REDIS_HOST: example-registry-quay-redis
      REDIS_PASSWORD: ""
      REDIS_SSL: false
      REDIS_SKIP_KEYSPACE_EVENT_SETUP: false
    EXECUTORS:
      - EXECUTOR: kubernetesPodman
        NAME: openshift
        BUILDER_NAMESPACE: virtual-builders
        SETUP_TIME: 180
        MINIMUM_RETRY_THRESHOLD: 0
        BUILDER_CONTAINER_IMAGE: quay.io/projectquay/quay-builder:{producty}
        # Kubernetes resource options
        K8S_API_SERVER: api.docs.quayteam.org:6443
        K8S_API_TLS_CA: /conf/stack/extra_ca_certs/build-cluster.crt
        VOLUME_SIZE: 8G
        KUBERNETES_DISTRIBUTION: openshift
        CONTAINER_MEMORY_LIMITS: 1G
        CONTAINER_CPU_LIMITS: 300m
        CONTAINER_MEMORY_REQUEST: 1G
        CONTAINER_CPU_REQUEST: 300m
        NODE_SELECTOR_LABEL_KEY: ""
        NODE_SELECTOR_LABEL_VALUE: ""
        SERVICE_ACCOUNT_NAME: quay-builder
        SERVICE_ACCOUNT_TOKEN: "eyJhbGciOiJSUzI1NiIsImtpZCI6IldfQUJkaDVmb3ltTHZ0dGZMYjhIWnYxZTQzN2dJVEJxcDJscldSdEUtYWsifQ"
        HTTP_PROXY: <http://10.0.0.1:80>
        HTTPS_PROXY: <http://10.0.0.1:80>
        NO_PROXY: <hostname.example.com>
      Copy to Clipboard Toggle word wrap

    6. Click Save on the Edit Secret page.
  10. Restart your Red Hat Quay on OpenShift Container Platform registry with the new configuration.

4.2.1. Modifying your AWS S3 storage bucket
Copy link

If you are using AWS S3 storage, you must change your storage bucket in the AWS console prior to starting a build.

Procedure

  1. Log in to your AWS console at s3.console.aws.com.
  2. In the search bar, search for S3 and then click S3.
  3. Click the name of your bucket, for example, myawsbucket.
  4. Click the Permissions tab.

  5. Under Cross-origin resource sharing (CORS), include the following parameters: 

      [
      {
          "AllowedHeaders": [
              "Authorization"
          ],
          "AllowedMethods": [
              "GET"
          ],
          "AllowedOrigins": [
              "*"
          ],
          "ExposeHeaders": [],
          "MaxAgeSeconds": 3000
      },
      {
          "AllowedHeaders": [
              "Content-Type",
              "x-amz-acl",
              "origin"
          ],
          "AllowedMethods": [
              "PUT"
          ],
          "AllowedOrigins": [
              "*"
          ],
          "ExposeHeaders": [],
          "MaxAgeSeconds": 3000
      }
  ]
    Copy to Clipboard Toggle word wrap
Note

Currently, modifying your Google Cloud Platform object bucket is not supported on IBM Power and IBM Z.

Use the following procedure to configure cross-origin resource sharing (CORS) for virtual builders. Without CORS configuration, uploading a build Dockerfile fails.

Procedure

  1. Use the following reference to create a JSON file for your specific CORS needs. For example: 

    $ cat gcp_cors.json
    Copy to Clipboard Toggle word wrap

    Example output

    [
    {
      "origin": ["*"],
      "method": ["GET"],
      "responseHeader": ["Authorization"],
      "maxAgeSeconds": 3600
    },
    {
      "origin": ["*"],
      "method": ["PUT"],
      "responseHeader": [
              "Content-Type",
              "x-goog-acl",
              "origin"],
      "maxAgeSeconds": 3600
    }
]
    Copy to Clipboard Toggle word wrap

  2. Enter the following command to update your GCP storage bucket: 

    $ gcloud storage buckets update gs://<bucket_name> --cors-file=./gcp_cors.json
    Copy to Clipboard Toggle word wrap

    Example output

    Updating
  Completed 1
    Copy to Clipboard Toggle word wrap

  3. You can display the updated CORS configuration of your GCP bucket by running the following command: 

    $ gcloud storage buckets describe gs://<bucket_name>  --format="default(cors)"
    Copy to Clipboard Toggle word wrap

    Example output

    cors:
- maxAgeSeconds: 3600
  method:
  - GET
  origin:
  - '*'
  responseHeader:
  - Authorization
- maxAgeSeconds: 3600
  method:
  - PUT
  origin:
  - '*'
  responseHeader:
  - Content-Type
  - x-goog-acl
  - origin
    Copy to Clipboard Toggle word wrap

Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat