Chapter 6. Automatically building Dockerfiles with Build workers
Red Hat Quay supports building Dockerfiles using a set of worker nodes on OpenShift or Kubernetes. Build triggers, such as GitHub webhooks can be configured to automatically build new versions of your repositories when new code is committed. This document will walk you through enabling builds with your Red Hat Quay installation and setting up one or more OpenShift/K8s clusters to accept builds from Red Hat Quay. With Red Hat Quay 3.4, the underlying Build Manager has been completely re-written as part of Red Hat Quay’s migration from Python 2 to Python 3. As a result, builder nodes are now dynamically created as Kubernetes Jobs versus builder nodes that ran continuously in Red Hat Quay 3.3 and earlier. This greatly simplifies how Red Hat Quay manages builds and provides the same mechanism quay.io utilizes to handle thousands of container image builds daily. Customers who are currently running static (“Enterprise” builders under Red Hat Quay 3.3) will be required to migrate to a Kubernetes-based build mechanism.
6.1. Architecture Overview
The Red Hat Quay Build system is designed for scalability (since it is used to host all builds at quay.io). The Build Manager component of Red Hat Quay provides an orchestration layer that tracks build requests and ensures that a Build Executor (OpenShift/K8s cluster) will carry out each request. Each build is handled by a Kubernetes Job which launches a small virtual machine to completely isolate and contain the image build process. This ensures that container builds do not affect each other or the underlying build system. Multiple Executors can be configured to ensure that builds are performed even in the event of infrastructure failures. Red Hat Quay will automatically send builds to a different Executor if it detects that one Executor is having difficulties.
The upstream version of Red Hat Quay provides instructions on how to configure an AWS/EC2 based Executor. This configuration is not supported for Red Hat Quay customers.
6.1.1. Build manager
The build manager is responsible for the lifecycle of scheduled build. Operations requiring updating the build queue, build phase and running jobs’ status is handled by the build manager.
6.1.2. Build workers’ control plane
Build jobs are run on separate worker nodes, and are scheduled on separate control planes (executor). Currently, Red Hat Quay supports running jobs on AWS and Kubernetes. Builds are executed using quay.io/quay/quay-builder. On AWS, builds are scheduled on EC2 instances. On k8s, the builds are scheduled as job resources.
6.1.3. Orchestrator
The orchestrator is used to store the state of currently running build jobs, and publish events for the build manager to consume. e.g expiry events. Currently, the supported orchestrator backend is Redis.
6.2. OpenShift Requirements
Red Hat Quay builds are supported on Kubernetes and OpenShift 4.5 and higher. A bare metal (non-virtualized) worker node is required since build pods require the ability to run kvm virtualization. Each build is done in an ephemeral virtual machine to ensure complete isolation and security while the build is running. In addition, your OpenShift cluster should permit the ServiceAccount associated with Red Hat Quay builds to run with the necessary SecurityContextConstraint to support privileged containers.
6.3. Orchestrator Requirements
The Red Hat Quay builds need access to a Redis instance to track build status information. It is acceptable to use the same Redis instance already deployed with your Red Hat Quay installation. All build queues are managed in the Red Hat Quay database so there is no need for a highly available Redis instance.
6.4. Setting Up Red Hat Quay Builders With OpenShift
6.4.1. OpenShift TLS component
The tls
component allows you to control TLS configuration.
Red Hat Quay 3.7 does not support builders when the TLS component is managed by the Operator.
If you set tls
to unmanaged
, you supply your own ssl.cert
and ssl.key
files. In this instance, if you want your cluster to support builders, you must add both the Quay route and the builder route name to the SAN list in the cert, or alternatively use a wildcard. To add the builder route, use the following format:
[quayregistry-cr-name]-quay-builder-[ocp-namespace].[ocp-domain-name]
6.4.2. Prepare OpenShift for Red Hat Quay Builds
There are several actions that are needed on an OpenShift cluster before it can accept builds from Red Hat Quay.
Create a project where builds will be run (e.g. ‘builder’)
$ oc new-project builder
Create a
ServiceAccount
in thisProject
that will be used to run builds. Ensure that it has sufficient privileges to createJobs
andPods
. Copy theServiceAccount
’s token for use later.$ oc create sa -n builder quay-builder $ oc policy add-role-to-user -n builder edit system:serviceaccount:builder:quay-builder $ oc sa get-token -n builder quay-builder
- Identify the URL for the OpenShift cluster’s API server. This can be found from the OpenShift Console.
-
Identify a worker node label to be used when scheduling build
Jobs
. Because build pods need to run on bare metal worker nodes, typically these are identified with specific labels. Check with your cluster administrator to determine exactly which node label should be used. If the cluster is using a self-signed certificate, get the kube apiserver’s CA to add to Red Hat Quay’s extra certs.
Get the name of the secret containing the CA:
$ oc get sa openshift-apiserver-sa --namespace=openshift-apiserver -o json | jq '.secrets[] | select(.name | contains("openshift-apiserver-sa-token"))'.name
-
Get the
ca.crt
key value from the secret in the Openshift console. The value should begin with “-----BEGIN CERTIFICATE-----” -
Import the CA in Red Hat Quay using the ConfigTool. Ensure the name of this file matches
K8S_API_TLS_CA
.
-
Create the necessary security contexts/role bindings for the
ServiceAccount
:
apiVersion: security.openshift.io/v1 kind: SecurityContextConstraints metadata: name: quay-builder priority: null readOnlyRootFilesystem: false requiredDropCapabilities: null runAsUser: type: RunAsAny seLinuxContext: type: RunAsAny seccompProfiles: - '*' supplementalGroups: type: RunAsAny volumes: - '*' allowHostDirVolumePlugin: true allowHostIPC: true allowHostNetwork: true allowHostPID: true allowHostPorts: true allowPrivilegeEscalation: true allowPrivilegedContainer: true allowedCapabilities: - '*' allowedUnsafeSysctls: - '*' defaultAddCapabilities: null fsGroup: type: RunAsAny --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: quay-builder-scc namespace: builder rules: - apiGroups: - security.openshift.io resourceNames: - quay-builder resources: - securitycontextconstraints verbs: - use --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: quay-builder-scc namespace: builder subjects: - kind: ServiceAccount name: quay-builder roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: quay-builder-scc
6.4.3. Enable Builders and add Build Configuration to Red Hat Quay’s Configuration Bundle
- Ensure that you’ve got Builds enabled in your Red Hat Quay configuration.
FEATURE_BUILD_SUPPORT: True
- Add the following to your Red Hat Quay configuration bundle, replacing each value with a value specific to your installation.
Currently only the Build feature itself can be enabled via the Red Hat Quay Config Tool. The actual configuration of the Build Manager and Executors must be done manually in the config.yaml file.
BUILD_MANAGER: - ephemeral - ALLOWED_WORKER_COUNT: 1 ORCHESTRATOR_PREFIX: buildman/production/ ORCHESTRATOR: REDIS_HOST: quay-redis-host REDIS_PASSWORD: quay-redis-password REDIS_SSL: true REDIS_SKIP_KEYSPACE_EVENT_SETUP: false EXECUTORS: - EXECUTOR: kubernetes BUILDER_NAMESPACE: builder K8S_API_SERVER: api.openshift.somehost.org:6443 K8S_API_TLS_CA: /conf/stack/extra_ca_certs/build_cluster.crt VOLUME_SIZE: 8G KUBERNETES_DISTRIBUTION: openshift CONTAINER_MEMORY_LIMITS: 5120Mi CONTAINER_CPU_LIMITS: 1000m CONTAINER_MEMORY_REQUEST: 3968Mi CONTAINER_CPU_REQUEST: 500m NODE_SELECTOR_LABEL_KEY: beta.kubernetes.io/instance-type NODE_SELECTOR_LABEL_VALUE: n1-standard-4 CONTAINER_RUNTIME: podman SERVICE_ACCOUNT_NAME: ***** SERVICE_ACCOUNT_TOKEN: ***** QUAY_USERNAME: quay-username QUAY_PASSWORD: quay-password WORKER_IMAGE: <registry>/quay-quay-builder WORKER_TAG: some_tag BUILDER_VM_CONTAINER_IMAGE: <registry>/quay-quay-builder-qemu-rhcos:v3.4.0 SETUP_TIME: 180 MINIMUM_RETRY_THRESHOLD: 0 SSH_AUTHORIZED_KEYS: - ssh-rsa 12345 someuser@email.com - ssh-rsa 67890 someuser2@email.com
Each configuration field is explained below.
- ALLOWED_WORKER_COUNT
- Defines how many Build Workers are instantiated per Red Hat Quay Pod. Typically this is ‘1’.
- ORCHESTRATOR_PREFIX
- Defines a unique prefix to be added to all Redis keys (useful to isolate Orchestrator values from other Redis keys).
- REDIS_HOST
- Hostname for your Redis service.
- REDIS_PASSWORD
- Password to authenticate into your Redis service.
- REDIS_SSL
- Defines whether or not your Redis connection uses SSL.
- REDIS_SKIP_KEYSPACE_EVENT_SETUP
-
By default, Red Hat Quay does not set up the keyspace events required for key events at runtime. To do so, set REDIS_SKIP_KEYSPACE_EVENT_SETUP to
false
. - EXECUTOR
- Starts a definition of an Executor of this type. Valid values are ‘kubernetes’ and ‘ec2’
- BUILDER_NAMESPACE
- Kubernetes namespace where Red Hat Quay builds will take place
- K8S_API_SERVER
- Hostname for API Server of OpenShift cluster where builds will take place
- K8S_API_TLS_CA
-
The filepath in the
Quay
container of the build cluster’s CA certificate for the Quay app to trust when making API calls. - KUBERNETES_DISTRIBUTION
- Indicates which type of Kubernetes is being used. Valid values are ‘openshift’ and ‘k8s’.
- CONTAINER_*
- Define the resource requests and limits for each build pod.
- NODE_SELECTOR_*
- Defines the node selector label name/value pair where build Pods should be scheduled.
- CONTAINER_RUNTIME
-
Specifies whether the builder should run
docker
orpodman
. Customers using Red Hat’squay-builder
image should set this topodman
. - SERVICE_ACCOUNT_NAME/SERVICE_ACCOUNT_TOKEN
- Defines the Service Account name/token that will be used by build Pods.
- QUAY_USERNAME/QUAY_PASSWORD
- Defines the registry credentials needed to pull the Red Hat Quay build worker image that is specified in the WORKER_IMAGE field. Customers should provide a Red Hat Service Account credential as defined in the section "Creating Registry Service Accounts" against registry.redhat.io in the article at https://access.redhat.com/RegistryAuthentication.
- WORKER_IMAGE
- Image reference for the Red Hat Quay builder image. registry.redhat.io/quay/quay-builder
- WORKER_TAG
- Tag for the builder image desired. The latest version is v3.4.0.
- BUILDER_VM_CONTAINER_IMAGE
-
The full reference to the container image holding the internal VM needed to run each Red Hat Quay build (
registry.redhat.io/quay/quay-builder-qemu-rhcos:v3.4.0
). - SETUP_TIME
- Specifies the number of seconds at which a build times out if it has not yet registered itself with the Build Manager (default is 500 seconds). Builds that time out are attempted to be restarted three times. If the build does not register itself after three attempts it is considered failed.
- MINIMUM_RETRY_THRESHOLD
-
This setting is used with multiple Executors; it indicates how many retries are attempted to start a build before a different Executor is chosen. Setting to 0 means there are no restrictions on how many tries the build job needs to have. This value should be kept intentionally small (three or less) to ensure failovers happen quickly in the event of infrastructure failures. You must specify a value for this setting. E.g Kubernetes is set as the first executor and EC2 as the second executor. If we want the last attempt to run a job to always be executed on EC2 and not Kubernetes, we would set the Kubernetes executor’s
MINIMUM_RETRY_THRESHOLD
to 1 and EC2’sMINIMUM_RETRY_THRESHOLD
to 0 (defaults to 0 if not set). In this case, kubernetes’MINIMUM_RETRY_THRESHOLD
> retries_remaining(1) would evaluate to False, thus falling back to the second executor configured - SSH_AUTHORIZED_KEYS
- List of ssh keys to bootstrap in the ignition config. This allows other keys to be used to ssh into the EC2 instance or QEMU VM
6.5. OpenShift Routes Limitation
This section only applies if you are using the Quay Operator on OpenShift with managed route
component.
Due to a limitation of OpenShift Routes
to only be able to serve traffic to a single port, additional steps are required to set up builds. Ensure that your kubectl
or oc
CLI tool is configured to work with the cluster where the Quay Operator is installed and that your QuayRegistry
exists (not necessarily the same as the bare metal cluster where your builders run).
- Ensure that HTTP/2 ingress is enabled on the OpenShift cluster by following these steps.
The Quay Operator will create a
Route
which directs gRPC traffic to the build manager server running inside the existing Quay pod(s). If you want to use a custom hostname (such as a subdomain likebuilder.registry.example.com
), ensure that you create a CNAME record with your DNS provider which points to thestatus.ingress[0].host
of the createdRoute
:$ kubectl get -n <namespace> route <quayregistry-name>-quay-builder -o jsonpath={.status.ingress[0].host}
Using the OpenShift UI or CLI, update the
Secret
referenced byspec.configBundleSecret
of theQuayRegistry
with the build cluster CA certificate (name the keyextra_ca_cert_build_cluster.cert
), and update theconfig.yaml
entry with the correct values referenced in the builder config above (depending on your build executor) along with theBUILDMAN_HOSTNAME
field:BUILDMAN_HOSTNAME: <build-manager-hostname> BUILD_MANAGER: - ephemeral - ALLOWED_WORKER_COUNT: 1 ORCHESTRATOR_PREFIX: buildman/production/ JOB_REGISTRATION_TIMEOUT: 600 ORCHESTRATOR: REDIS_HOST: quay-redis-host REDIS_PASSWORD: quay-redis-password REDIS_SSL: true REDIS_SKIP_KEYSPACE_EVENT_SETUP: false EXECUTORS: - EXECUTOR: kubernetes BUILDER_NAMESPACE: builder ...
The extra configuration field is explained below:
- BUILDMAN_HOSTNAME
-
The externally accessible server hostname which the build jobs use to communicate back to the build manager. Default is the same as
SERVER_HOSTNAME
. For OpenShiftRoute
, it is eitherstatus.ingress[0].host
or the CNAME entry if using a custom hostname.BUILDMAN_HOSTNAME
needs to include the port number, e.gsomehost:443
for Openshift Route, as the gRPC client used to communicate with the build manager does not infer any port if omitted.
6.6. Troubleshooting Builds
The builder instances started by the build manager are ephemeral. This means that they will either get shut down by Red Hat Quay} on timeouts/failure or garbage collected by the control plane (EC2/K8s). This means that in order to get the builder logs, one needs to do so while the builds are running.
6.6.1. DEBUG config flag
A DEBUG flag can be set in order to prevent the builder instances from getting cleaned up after completion/failure. To do so, in the desired executor configuration, set DEBUG to true. For example:
EXECUTORS: - EXECUTOR: ec2 DEBUG: true ... - EXECUTOR: kubernetes DEBUG: true ...
When set to true, DEBUG will prevent the build nodes from shutting down after the quay-builder service is done or fails, and will prevent the build manager from cleaning up the instances (terminating EC2 instances or deleting k8s jobs). This will allow debugging builder node issues, and should not be set in a production environment. The lifetime service will still exist. i.e The instance will still shutdown after approximately 2 hours (EC2 instances will terminate, k8s jobs will complete) Setting DEBUG will also affect ALLOWED_WORKER_COUNT, as the unterminated instances/jobs will still count towards the total number of running workers. This means the existing builder workers will need to manually be deleted if ALLOWED_WORKER_COUNT is reached to be able to schedule new builds.
Use the followings steps:
The guest VM forwards its SSH port (22) to its host’s (the pod) port 2222. Port forward the builder pod’s port 2222 to a port on localhost. e.g
$ kubectl port-forward <builder pod> 9999:2222
SSH into the VM running inside the container using a key set from SSH_AUTHORIZED_KEYS:
$ ssh -i /path/to/ssh/key/set/in/ssh_authorized_keys -p 9999 core@localhost
Get the quay-builder service logs:
$ systemctl status quay-builder $ journalctl -f -u quay-builder
Step 2-3 can also be done in a single SSH command:
$ ssh -i /path/to/ssh/key/set/in/ssh_authorized_keys -p 9999 core@localhost ‘systemctl status quay-builder’ $ ssh -i /path/to/ssh/key/set/in/ssh_authorized_keys -p 9999 core@localhost ‘journalctl -f -u quay-builder’
6.7. Setting up GitHub builds (optional)
If your organization plans to have builds be conducted via pushes to GitHub (or GitHub Enterprise), continue with Creating an OAuth application in GitHub.