Chapter 8. Troubleshooting Builds
The builder instances started by the build manager are ephemeral. This means that they will either get shut down by Red Hat Quay on timeouts or failure, or garbage collected by the control plane (EC2/K8s). In order to obtain the builds logs, you must do so while the builds are running.
8.1. DEBUG config flag
The DEBUG flag can be set to true in order to prevent the builder instances from getting cleaned up after completion or failure. For example:
EXECUTORS:
- EXECUTOR: ec2
DEBUG: true
...
- EXECUTOR: kubernetes
DEBUG: true
...
When set to true, the debug feature prevents the build nodes from shutting down after the quay-builder service is done or fails. It also prevents the build manager from cleaning up the instances by terminating EC2 instances or deleting Kubernetes jobs. This allows debugging builder node issues.
Debugging should not be set in a production cycle. The lifetime service still exists; for example, the instance still shuts down after approximately two hours. When this happens, EC2 instances are terminated and Kubernetes jobs are completed.
Enabling debug also affects the ALLOWED_WORKER_COUNT because the unterminated instances and jobs still count toward the total number of running workers. As a result, the existing builder workers must be manually deleted if ALLOWED_WORKER_COUNT is reached to be able to schedule new builds.
8.2. Troubleshooting OpenShift Container Platform and Kubernetes Builds
Use the following procedure to troubleshooting OpenShift Container Platform Kubernetes Builds.
Procedure
Create a port forwarding tunnel between your local machine and a pod running with either an OpenShift Container Platform cluster or a Kubernetes cluster by entering the following command:
$ oc port-forward <builder_pod> 9999:2222
Establish an SSH connection to the remote host using a specified SSH key and port, for example:
$ ssh -i /path/to/ssh/key/set/in/ssh_authorized_keys -p 9999 core@localhost
Obtain the
quay-builderservice logs by entering the following commands:$ systemctl status quay-builder
$ journalctl -f -u quay-builder
