Chapter 3. Bare metal builds with Red Hat Quay on OpenShift Container Platform

The procedures in this section explain how to create an environment for bare metal builds for Red Hat Quay on OpenShift Container Platform.

3.1. Configuring bare metal builds for Red Hat Quay on OpenShift Container Platform

Use the following procedure to configure bare metal builds for Red Hat Quay on OpenShift Container Platform.

Note

If you are using the Red Hat Quay Operator on OpenShift Container Platform with a managed route component in your QuayRegistry CRD, see "Red Hat Quay on OpenShift Container Platform builds limitations with self-managed routes".

Prerequisites

You have an OpenShift Container Platform cluster provisioned with the Red Hat Quay Operator running.
You have set the tls component to unmanaged and uploaded custom SSL/TLS certificates to the Red Hat Quay Operator. For more information, see SSL and TLS for Red Hat Quay.
You are logged into OpenShift Container Platform as a cluster administrator.

Procedure

Enter the following command to create a project where Builds will be run, for example, bare-metal-builder:
```
$ oc new-project bare-metal-builder
```
Create a new ServiceAccount in the the bare-metal-builder namespace by entering the following command:
```
$ oc create sa -n bare-metal-builder quay-builder
```

Enter the following command to grant a user the edit role within the bare-metal-builder namespace:

$ oc policy add-role-to-user -n bare-metal-builder edit system:serviceaccount:bare-metal-builder:quay-builder

Enter the following command to retrieve a token associated with the quay-builder service account in the bare-metal-builder namespace. This token is used to authenticate and interact with the OpenShift Container Platform cluster’s API server.
1. If your OpenShift Container Platform cluster is version 4.11+, enter the following command:
```
oc create token quay-builder  -n bare-metal-builder --duration 24h
```
2. If your OpenShift Container Platform cluster is earlier than version 4.11, for example, version 4.10, enter the following command:
```
$ oc sa get-token -n bare-metal-builder quay-builder
```
Identify the URL for the OpenShift Container Platform cluster’s API server. This can be found in the OpenShift Container Platform web console.
Identify a worker node label to be used when scheduling build jobs. Because build pods must run on bare metal worker nodes, typically these are identified with specific labels.
Check with your cluster administrator to determine exactly which node label should be used.
Obtain the Kube API Server’s certificate authority (CA) to add to Red Hat Quay’s extra certificates.
1. On OpenShift Container Platform versions 4.15+, enter the following commands to obtain the name of the secret containing the CA:
```
$ oc extract cm/kube-root-ca.crt -n openshift-apiserver
```
```
$ mv ca.crt build_cluster.crt
```
2. On OpenShift Container Platform versions earlier than 4.15, for example, 4.14, enter the following command:
```
$ oc get sa openshift-apiserver-sa --namespace=openshift-apiserver -o json | jq '.secrets[] | select(.name | contains("openshift-apiserver-sa-token"))'.name
```
3. Obtain the ca.crt key value from the secret in the OpenShift Container Platform Web Console. The value begins with "-----BEGIN CERTIFICATE-----"`.
4. Import the CA to Red Hat Quay. Ensure that the name of this file matches the K8S_API_TLS_CA field used in Step 9.

Create the following SecurityContextConstraints resource for the ServiceAccount:

apiVersion: security.openshift.io/v1
kind: SecurityContextConstraints
metadata:
  name: quay-builder
priority: null
readOnlyRootFilesystem: false
requiredDropCapabilities: null
runAsUser:
  type: RunAsAny
seLinuxContext:
  type: RunAsAny
seccompProfiles:
- '*'
supplementalGroups:
  type: RunAsAny
volumes:
- '*'
allowHostDirVolumePlugin: true
allowHostIPC: true
allowHostNetwork: true
allowHostPID: true
allowHostPorts: true
allowPrivilegeEscalation: true
allowPrivilegedContainer: true
allowedCapabilities:
- '*'
allowedUnsafeSysctls:
- '*'
defaultAddCapabilities: null
fsGroup:
  type: RunAsAny
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: quay-builder-scc
  namespace: bare-metal-builder
rules:
- apiGroups:
  - security.openshift.io
  resourceNames:
  - quay-builder
  resources:
  - securitycontextconstraints
  verbs:
  - use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: quay-builder-scc
  namespace: bare-metal-builder
subjects:
- kind: ServiceAccount
  name: quay-builder
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: quay-builder-scc

Update the config.yaml file of your Red Hat Quay on OpenShift Container Platform deployment to include an appropriate bare metal builds configuration by using the OpenShift Container Platform web console.
1. Click Operators Installed Operators Red Hat Quay Quay Registry.
2. Click the name of your registry, for example, example-registry.
3. Under Config Bundle Secret, click the name of your configuration bundle, for example, extra-ca-certificate-config-bundle-secret.
4. Click Actions Edit Secret.
5. Add the following information to your Red Hat Quay config.yaml file, replacing each value with information that is relevant to your specific installation:
```
FEATURE_USER_INITIALIZE: true
BROWSER_API_CALLS_XHR_ONLY: false
SUPER_USERS:
- <superusername>
FEATURE_USER_CREATION: false
FEATURE_QUOTA_MANAGEMENT: true
FEATURE_BUILD_SUPPORT: True
BUILDMAN_HOSTNAME: ${BUILDMAN_HOSTNAME}:443 1
BUILD_MANAGER:
- ephemeral
- ALLOWED_WORKER_COUNT: 10
  ORCHESTRATOR_PREFIX: buildman/production/
    ORCHESTRATOR:
      REDIS_HOST: <sample_redis_hostname> 2
      REDIS_PASSWORD: ""
      REDIS_SSL: false
      REDIS_SKIP_KEYSPACE_EVENT_SETUP: false
  EXECUTORS:
  - EXECUTOR: kubernetes
    BUILDER_NAMESPACE: <sample_builder_namespace> 3
    K8S_API_SERVER: <sample_k8s_api_server> 4
    K8S_API_TLS_CA: <sample_crt_file> 5
    VOLUME_SIZE: 8G
    KUBERNETES_DISTRIBUTION: openshift
    CONTAINER_MEMORY_LIMITS: 1G 6
    CONTAINER_CPU_LIMITS: 300m 7
    CONTAINER_MEMORY_REQUEST: 1G 8
    CONTAINER_CPU_REQUEST: 300m 9
    NODE_SELECTOR_LABEL_KEY: beta.kubernetes.io/instance-type
    NODE_SELECTOR_LABEL_VALUE: n1-standard-4
    CONTAINER_RUNTIME: podman
    SERVICE_ACCOUNT_NAME: <sample_service_account_name>
    SERVICE_ACCOUNT_TOKEN: <sample_account_token> 10
    QUAY_USERNAME: <quay_username>
    QUAY_PASSWORD: <quay_password>
    WORKER_IMAGE: <registry>/quay-quay-builder
    WORKER_TAG: <some_tag>
    BUILDER_VM_CONTAINER_IMAGE: quay.io/quay/quay-builder-qemu-fedoracoreos:latest
    SETUP_TIME: 180
    MINIMUM_RETRY_THRESHOLD: 0
    SSH_AUTHORIZED_KEYS: 11
    - <ssh-rsa 12345 someuser@email.com>
    - <ssh-rsa 67890 someuser2@email.com>
    HTTP_PROXY: <http://10.0.0.1:80>
    HTTPS_PROXY: <http://10.0.0.1:80>
    NO_PROXY: <hostname.example.com>
```
  1
  Obtained by running the following command: $ oc get route quayregistry-quay-builder -n ${QUAY_PROJECT} -o jsonpath='{.spec.host}'.
  2
  The hostname for your Redis service.
  3
  Set to match the name of your bare metal builds namespace. This example used bare-metal-builder.
  4
  The K8S_API_SERVER is obtained by running $ oc cluster-info.
  5
  You must manually create and add your custom CA cert, for example, K8S_API_TLS_CA: /conf/stack/extra_ca_certs/build-cluster.crt.
  6
  Defaults to 5120Mi if left unspecified.
  7
  Defaults to 1000m if left unspecified.
  8
  Defaults to 3968Mi if left unspecified.
  9
  Defaults to 500m if left unspecified.
  10
  Obtained when running $ oc create sa.
  11
  Allows public SSH keys to be added to the build environment for remote troubleshooting access. This key, or keys, should correspond to the private key that an admin or developer will use to SSH into the build worker for debugging purposes. This key can be obtained by establishing an SSH connection to the remote host using a specific SSH key and port. For example: $ ssh -i /path/to/ssh/key/set/in/ssh_authorized_keys -p 9999 core@localhost.
Restart your Red Hat Quay registry to enable the builds feature.

3.1.1. Red Hat Quay on OpenShift Container Platform builds limitations with self-managed routes

The following limitations apply when you are using the Red Hat Quay Operator on OpenShift Container Platform with a managed route component:

Currently, OpenShift Container Platform routes are only able to serve traffic to a single port. Additional steps are required to set up Red Hat Quay Builds.
Ensure that your kubectl or oc CLI tool is configured to work with the cluster where the Red Hat Quay Operator is installed and that your QuayRegistry exists; the QuayRegistry does not have to be on the same bare metal cluster where builders run.
Ensure that HTTP/2 ingress is enabled on the OpenShift cluster by following these steps.
The Red Hat Quay Operator creates a Route resource that directs gRPC traffic to the Build manager server running inside of the existing Quay pod, or pods. If you want to use a custom hostname, or a subdomain like <builder-registry.example.com>, ensure that you create a CNAME record with your DNS provider that points to the status.ingress[0].host of the create Route resource. For example:
```
$ kubectl get -n <namespace> route <quayregistry-name>-quay-builder -o jsonpath={.status.ingress[0].host}
```
Using the OpenShift Container Platform UI or CLI, update the Secret referenced by spec.configBundleSecret of the QuayRegistry with the build cluster CA certificate. Name the key extra_ca_cert_build_cluster.cert. Update the config.yaml file entry with the correct values referenced in the build configuration that you created when you configured Red Hat Quay builds, and add the BUILDMAN_HOSTNAME CONFIGURATION FIELD:
```
BUILDMAN_HOSTNAME: <build-manager-hostname> 1
BUILD_MANAGER:
- ephemeral
- ALLOWED_WORKER_COUNT: 1
  ORCHESTRATOR_PREFIX: buildman/production/
  JOB_REGISTRATION_TIMEOUT: 600
  ORCHESTRATOR:
    REDIS_HOST: <quay_redis_host
    REDIS_PASSWORD: <quay_redis_password>
    REDIS_SSL: true
    REDIS_SKIP_KEYSPACE_EVENT_SETUP: false
  EXECUTORS:
  - EXECUTOR: kubernetes
    BUILDER_NAMESPACE: builder
    ...
```
1
The externally accessible server hostname which the build jobs use to communicate back to the build manager. Default is the same as SERVER_HOSTNAME. For an OpenShift route resource, it is either status.ingress[0].host or the CNAME entry if using a custom hostname. BUILDMAN_HOSTNAME must include the port number, for example, somehost:443 for an OpenShift Container Platform route resource, as the gRPC client used to communicate with the build manager does not infer any port if omitted.

Chapter 3. Bare metal builds with Red Hat Quay on OpenShift Container Platform

3.1. Configuring bare metal builds for Red Hat Quay on OpenShift Container Platform

3.1.1. Red Hat Quay on OpenShift Container Platform builds limitations with self-managed routes

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

Making open source more inclusive

About Red Hat

Red Hat legal and privacy links

Red Hat legal and privacy links