Chapter 6. Automatically building Dockerfiles with Build workers

download PDF

Red Hat Quay supports building Dockerfiles using a set of worker nodes on OpenShift or Kubernetes. Build triggers, such as GitHub webhooks can be configured to automatically build new versions of your repositories when new code is committed. This document will walk you through enabling builds with your Red Hat Quay installation and setting up one or more OpenShift/K8s clusters to accept builds from Red Hat Quay. With Red Hat Quay 3.4, the underlying Build Manager has been completely re-written as part of Red Hat Quay’s migration from Python 2 to Python 3. As a result, builder nodes are now dynamically created as Kubernetes Jobs versus builder nodes that ran continuously in Red Hat Quay 3.3 and earlier. This greatly simplifies how Red Hat Quay manages builds and provides the same mechanism utilizes to handle thousands of container image builds daily. Customers who are currently running static (“Enterprise” builders under Red Hat Quay 3.3) will be required to migrate to a Kubernetes-based build mechanism.

6.1. Architecture Overview

The Red Hat Quay Build system is designed for scalability (since it is used to host all builds at The Build Manager component of Red Hat Quay provides an orchestration layer that tracks build requests and ensures that a Build Executor (OpenShift/K8s cluster) will carry out each request. Each build is handled by a Kubernetes Job which launches a small virtual machine to completely isolate and contain the image build process. This ensures that container builds do not affect each other or the underlying build system. Multiple Executors can be configured to ensure that builds are performed even in the event of infrastructure failures. Red Hat Quay will automatically send builds to a different Executor if it detects that one Executor is having difficulties.


The upstream version of Red Hat Quay provides instructions on how to configure an AWS/EC2 based Executor. This configuration is not supported for Red Hat Quay customers.

6.1.1. Build manager

The build manager is responsible for the lifecycle of scheduled build. Operations requiring updating the build queue, build phase and running jobs’ status is handled by the build manager.

6.1.2. Build workers’ control plane

Build jobs are run on separate worker nodes, and are scheduled on separate control planes (executor). Currently, Red Hat Quay supports running jobs on AWS and Kubernetes. Builds are executed using On AWS, builds are scheduled on EC2 instances. On k8s, the builds are scheduled as job resources.

6.1.3. Orchestrator

The orchestrator is used to store the state of currently running build jobs, and publish events for the build manager to consume. e.g expiry events. Currently, the supported orchestrator backend is Redis.

6.2. OpenShift Requirements

Red Hat Quay builds are supported on Kubernetes and OpenShift 4.5 and higher. A bare metal (non-virtualized) worker node is required since build pods require the ability to run kvm virtualization. Each build is done in an ephemeral virtual machine to ensure complete isolation and security while the build is running. In addition, your OpenShift cluster should permit the ServiceAccount associated with Red Hat Quay builds to run with the necessary SecurityContextConstraint to support privileged containers.

6.3. Orchestrator Requirements

The Red Hat Quay builds need access to a Redis instance to track build status information. It is acceptable to use the same Redis instance already deployed with your Red Hat Quay installation. All build queues are managed in the Red Hat Quay database so there is no need for a highly available Redis instance.

6.4. Setting Up Red Hat Quay Builders With OpenShift

6.4.1. Prepare OpenShift for Red Hat Quay Builds

There are several actions that are needed on an OpenShift cluster before it can accept builds from Red Hat Quay.

  1. Create a project where builds will be run (e.g. ‘builder’)

    $ oc new-project builder
  2. Create a ServiceAccount in this Project that will be used to run builds. Ensure that it has sufficient privileges to create Jobs and Pods. Copy the ServiceAccount’s token for use later.

    $ oc create sa -n builder quay-builder
    $ oc policy add-role-to-user -n builder edit system:serviceaccount:builder:quay-builder
    $ oc sa get-token -n builder quay-builder
  3. Identify the URL for the OpenShift cluster’s API server. This can be found from the OpenShift Console.
  4. Identify a worker node label to be used when scheduling build Jobs. Because build pods need to run on bare metal worker nodes, typically these are identified with specific labels. Check with your cluster administrator to determine exactly which node label should be used.
  5. If the cluster is using a self-signed certificate, get the kube apiserver’s CA to add to Red Hat Quay’s extra certs.

    1. Get the name of the secret containing the CA:

      $ oc get sa openshift-apiserver-sa --namespace=openshift-apiserver -o json | jq '.secrets[] | select(.name | contains("openshift-apiserver-sa-token"))'.name
    2. Get the ca.crt key value from the secret in the Openshift console. The value should begin with “-----BEGIN CERTIFICATE-----”
    3. Import the CA in Red Hat Quay using the ConfigTool. Ensure the name of this file matches K8S_API_TLS_CA.
  6. Create the necessary security contexts/role bindings for the ServiceAccount:
kind: SecurityContextConstraints
  name: quay-builder
priority: null
readOnlyRootFilesystem: false
requiredDropCapabilities: null
  type: RunAsAny
  type: RunAsAny
- '*'
  type: RunAsAny
- '*'
allowHostDirVolumePlugin: true
allowHostIPC: true
allowHostNetwork: true
allowHostPID: true
allowHostPorts: true
allowPrivilegeEscalation: true
allowPrivilegedContainer: true
- '*'
- '*'
defaultAddCapabilities: null
  type: RunAsAny
kind: Role
  name: quay-builder-scc
  namespace: builder
- apiGroups:
  - quay-builder
  - securitycontextconstraints
  - use
kind: RoleBinding
  name: quay-builder-scc
  namespace: builder
- kind: ServiceAccount
  name: quay-builder
  kind: Role
  name: quay-builder-scc

6.4.2. Enable Builders and add Build Configuration to Red Hat Quay’s Configuration Bundle

  1. Ensure that you’ve got Builds enabled in your Red Hat Quay configuration.
  1. Add the following to your Red Hat Quay configuration bundle, replacing each value with a value specific to your installation.

Currently only the Build feature itself can be enabled via the Red Hat Quay Config Tool. The actual configuration of the Build Manager and Executors must be done manually in the config.yaml file.

- ephemeral
  ORCHESTRATOR_PREFIX: buildman/production/
    REDIS_HOST: quay-redis-host
    REDIS_PASSWORD: quay-redis-password
    REDIS_SSL: true
  - EXECUTOR: kubernetes
    K8S_API_TLS_CA: /conf/stack/extra_ca_cert_build_cluster.crt
    NODE_SELECTOR_LABEL_VALUE: n1-standard-4
    QUAY_USERNAME: quay-username
    QUAY_PASSWORD: quay-password
    WORKER_IMAGE: <registry>/quay-quay-builder
    WORKER_TAG: some_tag
    BUILDER_VM_CONTAINER_IMAGE: <registry>/quay-quay-builder-qemu-rhcos:v3.4.0
    SETUP_TIME: 180
    - ssh-rsa 12345
    - ssh-rsa 67890

Each configuration field is explained below.

Defines how many Build Workers are instantiated per Red Hat Quay Pod. Typically this is ‘1’.
Defines a unique prefix to be added to all Redis keys (useful to isolate Orchestrator values from other Redis keys).
Hostname for your Redis service.
Password to authenticate into your Redis service.
Defines whether or not your Redis connection uses SSL.
By default, Red Hat Quay does not set up the keyspace events required for key events at runtime. To do so, set REDIS_SKIP_KEYSPACE_EVENT_SETUP to false.
Starts a definition of an Executor of this type. Valid values are ‘kubernetes’ and ‘ec2’
Kubernetes namespace where Red Hat Quay builds will take place
Hostname for API Server of OpenShift cluster where builds will take place
The filepath in the Quay container of the build cluster’s CA certificate for the Quay app to trust when making API calls.
Indicates which type of Kubernetes is being used. Valid values are ‘openshift’ and ‘k8s’.
Define the resource requests and limits for each build pod.
Defines the node selector label name/value pair where build Pods should be scheduled.
Specifies whether the builder should run docker or podman. Customers using Red Hat’s quay-builder image should set this to podman.
Defines the Service Account name/token that will be used by build Pods.
Defines the registry credentials needed to pull the Red Hat Quay build worker image that is specified in the WORKER_IMAGE field. Customers should provide a Red Hat Service Account credential as defined in the section "Creating Registry Service Accounts" against in the article at
Image reference for the Red Hat Quay builder image.
Tag for the builder image desired. The latest version is v3.4.0.
The full reference to the container image holding the internal VM needed to run each Red Hat Quay build (
Specifies the number of seconds at which a build times out if it has not yet registered itself with the Build Manager (default is 500 seconds). Builds that time out are attempted to be restarted three times. If the build does not register itself after three attempts it is considered failed.
This setting is used with multiple Executors; it indicates how many retries are attempted to start a build before a different Executor is chosen. Setting to 0 means there are no restrictions on how many tries the build job needs to have. This value should be kept intentionally small (three or less) to ensure failovers happen quickly in the event of infrastructure failures. E.g Kubernetes is set as the first executor and EC2 as the second executor. If we want the last attempt to run a job to always be executed on EC2 and not Kubernetes, we would set the Kubernetes executor’s MINIMUM_RETRY_THRESHOLD to 1 and EC2’s MINIMUM_RETRY_THRESHOLD to 0 (defaults to 0 if not set). In this case, kubernetes’ MINIMUM_RETRY_THRESHOLD > retries_remaining(1) would evaluate to False, thus falling back to the second executor configured
List of ssh keys to bootstrap in the ignition config. This allows other keys to be used to ssh into the EC2 instance or QEMU VM

6.5. OpenShift Routes Limitation


This section only applies if you are using the Quay Operator on OpenShift with managed route component.

Due to a limitation of OpenShift Routes to only be able to serve traffic to a single port, additional steps are required to set up builds. Ensure that your kubectl or oc CLI tool is configured to work with the cluster where the Quay Operator is installed and that your QuayRegistry exists (not necessarily the same as the bare metal cluster where your builders run).

  • Ensure that HTTP/2 ingress is enabled on the OpenShift cluster by following these steps.
  • The Quay Operator will create a Route which directs gRPC traffic to the build manager server running inside the existing Quay pod(s). If you want to use a custom hostname (such as a subdomain like, ensure that you create a CNAME record with your DNS provider which points to the status.ingress[0].host of the created Route:

    $ kubectl get -n <namespace> route <quayregistry-name>-quay-builder -o jsonpath={.status.ingress[0].host}
  • Using the OpenShift UI or CLI, update the Secret referenced by spec.configBundleSecret of the QuayRegistry with the build cluster CA certificate (name the key extra_ca_cert_build_cluster.cert), and update the config.yaml entry with the correct values referenced in the builder config above (depending on your build executor) along with the BUILDMAN_HOSTNAME field:

    - ephemeral
      ORCHESTRATOR_PREFIX: buildman/production/
        REDIS_HOST: quay-redis-host
        REDIS_PASSWORD: quay-redis-password
        REDIS_SSL: true
      - EXECUTOR: kubernetes
        BUILDER_NAMESPACE: builder
        BUILDMAN_HOSTNAME: <build-manager-hostname>

The extra configuration field is explained below:

The externally accessible server hostname which the build jobs use to communicate back to the build manager. Default is the same as SERVER_HOSTNAME. For OpenShift Route, it is either status.ingress[0].host or the CNAME entry if using a custom hostname. BUILDMAN_HOSTNAME needs to include the port number, e.g somehost:443 for Openshift Route, as the gRPC client used to communicate with the build manager does not infer any port if omitted.

6.6. Troubleshooting Builds

The builder instances started by the build manager are ephemeral. This means that they will either get shut down by Red Hat Quay} on timeouts/failure or garbage collected by the control plane (EC2/K8s). This means that in order to get the builder logs, one needs to do so while the builds are running.

6.6.1. DEBUG config flag

A DEBUG flag can be set in order to prevent the builder instances from getting cleaned up after completion/failure. To do so, in the desired executor configuration, set DEBUG to true. For example:

    - EXECUTOR: ec2
      DEBUG: true
    - EXECUTOR: kubernetes
      DEBUG: true

When set to true, DEBUG will prevent the build nodes from shutting down after the quay-builder service is done or fails, and will prevent the build manager from cleaning up the instances (terminating EC2 instances or deleting k8s jobs). This will allow debugging builder node issues, and should not be set in a production environment. The lifetime service will still exist. i.e The instance will still shutdown after approximately 2 hours (EC2 instances will terminate, k8s jobs will complete) Setting DEBUG will also affect ALLOWED_WORKER_COUNT, as the unterminated instances/jobs will still count towards the total number of running workers. This means the existing builder workers will need to manually be deleted if ALLOWED_WORKER_COUNT is reached to be able to schedule new builds.

Use the followings steps:

  1. The guest VM forwards its SSH port (22) to its host’s (the pod) port 2222. Port forward the builder pod’s port 2222 to a port on localhost. e.g

    $ kubectl port-forward <builder pod> 9999:2222
  2. SSH into the VM running inside the container using a key set from SSH_AUTHORIZED_KEYS:

    $ ssh -i /path/to/ssh/key/set/in/ssh_authorized_keys -p 9999 core@localhost
  3. Get the quay-builder service logs:

    $ systemctl status quay-builder
    $ journalctl -f -u quay-builder
    • Step 2-3 can also be done in a single SSH command:

      $ ssh -i /path/to/ssh/key/set/in/ssh_authorized_keys -p 9999 core@localhost ‘systemctl status quay-builder’
      $ ssh -i /path/to/ssh/key/set/in/ssh_authorized_keys -p 9999 core@localhost ‘journalctl -f -u quay-builder’

6.7. Setting up GitHub builds (optional)

If your organization plans to have builds be conducted via pushes to GitHub (or GitHub Enterprise), continue with Creating an OAuth application in GitHub.

Red Hat logoGithubRedditYoutubeTwitter


Try, buy, & sell


About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

© 2024 Red Hat, Inc.