Chapter 23. Configuring for Google Compute Engine
You can configure OpenShift Container Platform to access an existing Google Compute Engine (GCE) infrastructure, including using GCE volumes as persistent storage for application data.
23.1. Before you begin
23.1.1. Configuring authorization for Google Cloud Platform
Roles
Configuring GCP for OpenShift Container Platform requires the following GCP role:
| Needed for creating service accounts, cloud storage, instances, images, templates, Cloud DNS entries, and to deploy load balancers and health checks. |
delete
permissions might also be required if the user is expected to redeploy the environment during testing phases.
You can also create a service account to avoid using personal users when deploying GCP objects.
See the Understanding roles section of the GCP documentation for more information, including steps for how to configure roles.
Scopes and service accounts
GCP uses scopes to determine if an authenticated identity is authorized to perform operations within a resource. For example, if application A with a read-only scope access token can only read, while application B with a read-write scope access token can read and modify data.
The scopes are defined at the GCP API level as https://www.googleapis.com/auth/compute.readonly
.
You can specify scopes using the --scopes=[SCOPE,…]
option when creating instances, or you can use the --no-scopes
option to create the instance without scopes if you don’t want the instance accessing the GCP API.
See the Scopes section of the GCP documentation for more information.
All GCP projects include a default [PROJECT_NUMBER]-compute@developer.gserviceaccount.com
service account with project editor permissions.
By default, a newly created instance is automatically enabled to run as the default service account with the following access scopes:
- https://www.googleapis.com/auth/devstorage.read_only
- https://www.googleapis.com/auth/logging.write
- https://www.googleapis.com/auth/monitoring.write
- https://www.googleapis.com/auth/pubsub
- https://www.googleapis.com/auth/service.management.readonly
- https://www.googleapis.com/auth/servicecontrol
- https://www.googleapis.com/auth/trace.append
- https://www.googleapis.com/auth/bigquery
- https://www.googleapis.com/auth/cloud-platform
- https://www.googleapis.com/auth/compute.readonly
- https://www.googleapis.com/auth/compute
- https://www.googleapis.com/auth/datastore
- https://www.googleapis.com/auth/logging.write
- https://www.googleapis.com/auth/monitoring
- https://www.googleapis.com/auth/monitoring.write
- https://www.googleapis.com/auth/servicecontrol
- https://www.googleapis.com/auth/service.management.readonly
- https://www.googleapis.com/auth/sqlservice.admin
- https://www.googleapis.com/auth/devstorage.full_control
- https://www.googleapis.com/auth/devstorage.read_only
- https://www.googleapis.com/auth/devstorage.read_write
- https://www.googleapis.com/auth/taskqueue
- https://www.googleapis.com/auth/userinfo.email
You can specify another service account with the --service-account=SERVICE_ACCOUNT
option when creating the instance, or explicitly disabling service accounts for the instance using the --no-service-account
option using the gcloud
CLI.
See the Creating a new service account section of the GCP documentation for more information.
23.1.2. Google Compute Engine objects
Integrating OpenShift Container Platform with Google Compute Engine (GCE) requires the following components or services.
- A GCP project
- A GCP project is the base level organizing entity that forms the basis for creating, enabling, and using all GCP services. This includes managing APIs, enabling billing, adding and removing collaborators, and managing permissions.
See the project resource section in the GCP documentation for more information.
Project IDs are unique identifiers, and project IDs must be unique across all of Google Cloud Engine. This means you cannot use myproject
as a project ID if someone else has created a project with that ID before.
- Billing
- You cannot create new resources unless billing is attached to an account. The new project can be linked to an existing project or new information can be entered.
See Create, Modify, or Close Your Billing Account in the GCP documentation for more information.
- Cloud identity and access management
- Deploying OpenShift Container Platform requires the proper permissions. A user must be able to create service accounts, cloud storage, instances, images, templates, Cloud DNS entries, and deploy load balancers and health checks. Delete permissions are also helpful in order to be able to redeploy the environment while testing.
You can create service accounts with specific permissions, then use them to deploy infrastructure components instead of regular users. You can also create roles to limit access to different users or service accounts.
GCP instances use service accounts to allow applications to call GCP APIs. For example, OpenShift Container Platform node hosts can call the GCP disk API to provide a persistent volume to an application.
Access control to the various infrastructure, service resources, and fine-grained roles are available using the IAM service. For more information, see the Access cloud overview section of the GCP documentation.
- SSH keys
- GCP injects SSH public keys as authorized keys so you can log in using SSH in the created instances. You can configure the SSH keys per instance or per project.
You can use existing SSH keys. GCP metadata can help with storing the SSH keys that are injected at boot time in the instances to allow SSH access.
See the Metadata section of the GCP documentation for more information.
- GCP regions and zones
- GCP has a global infrastructure that covers regions and availability zones. While deploying OpenShift Container Platform in GCP on different zones can help avoid single-point-of-failures, there are some caveats regarding storage.
GCP disks are created within a zone. Therefore, if a OpenShift Container Platform node host goes down in zone "A" and the pods move to zone "B", the persistent storage cannot be attached to those pods because the disks are in a different zone.
Deploying a single zone of multizone OpenShift Container Platform environment is an important decision to make before installing OpenShift Container Platform. If deploying a multizone environment, the recommended setup is to use three different zones in a single region.
See the GCP documentation on regions and zones and the Kubernetes documentation on multiple zones for more information.
- External IP address
- So that GCP instances can communicate with the Internet, you must attach an external IP address to the instance. Also, an external IP address is required to communicate with instances deployed in GCP from outside the Virtual Private Cloud (VPC) Network.
Requiring an External IP address
for internet access is a limitation of the provider. You can configure firewall rules to block incoming external traffic in instances if not needed.
See the GCP documentation on external IP address for more information.
- Cloud DNS
- GCP cloud DNS is a DNS service used to publish domain names to the global DNS using GCP DNS servers.
The public cloud DNS zone requires a domain name that you purchased either through Google’s "Domains" service or through a third-party provider. When you create the zone, you must add the name servers provided by Google to the registrar.
See the GCP documentation on Cloud DNS for more information.
GCP VPC networks have an internal DNS service that automatically resolves internal host names.
The internal fully qualified domain name (FQDN) for an instance follows the [HOST_NAME].c.[PROJECT_ID].internal
format.
See the GCP documentation on Internal DNS for more information.
- Load balancing
- The GCP load balancing service enables the distribution of traffic across multiple instances in the GCP cloud.
There are five types of Load Balancing:
HTTPS and TCP proxy load balancing are the only options for using HTTPS health checks for master nodes, which checks the status of /healthz.
Because HTTPS load balancing requires a custom certificate, this implementation uses TCP Proxy load balancing to simplify the process.
See the GCP documentation on Load balancing for more information.
- Instances sizes
- A successful OpenShift Container Platform environment requires some minimum hardware requirements:
Role | Size |
---|---|
Master |
|
Node |
|
GCP allows you to create custom instance sizes to fit different requirements. See Creating an Instance with a Custom Machine Type for more information, or see Machine types and OpenShift Container Platform Minimum Hardware Requirements for more information about instance sizes.
- Storage Options
By default, each GCP instance has a small root persistent disk that contains the operating system. When applications running on the instance require more storage space, you can add additional storage options to the instance:
- Standard persistent disks
- SSD persistent disks
- Local SSDs
- Cloud storage buckets
For more information, see the GCP documentation on Storage options.
23.2. Configuring OpenShift Container Platform for GCE
You can configure OpenShift Container Platform for GCE in two ways:
23.2.1. Option 1: Configuring OpenShift Container Platform for GCP using Ansible
You can configure OpenShift Container Platform for Google Compute Platform (GCP) by modifying the Ansible inventory file at installation time or after installation.
Procedure
At minimum, you must define the
openshift_cloudprovider_kind
,openshift_gcp_project
andopenshift_gcp_prefix
parameters, as well as the optionalopenshift_gcp_multizone
for multizone deployments andopenshift_gcp_network_name
if you are not using the default network name.Add the following section to the Ansible inventory file at installation to configure your OpenShift Container Platform environment for GCP:
[OSEv3:vars] openshift_cloudprovider_kind=gce openshift_gcp_project=<projectid> 1 openshift_gcp_prefix=<uid> 2 openshift_gcp_multizone=False 3 openshift_gcp_network_name=<network name> 4
- 1
- Provide the GCP project ID where the existing instances are running. This ID is generated when you create the project in the Google Cloud Platform Console.
- 2
- Provide a unique string to identify each OpenShift Container Platform cluster. This must be unique across GCP.
- 3
- Optionally, set to
True
to trigger multizone deployments on GCP. Set toFalse
by default. - 4
- Optionally, provide the network name if not using
default
network.
Installing with Ansible also creates and configures the following files to fit your GCP environment:
- /etc/origin/cloudprovider/gce.conf
- /etc/origin/master/master-config.yaml
- /etc/origin/node/node-config.yaml
-
If you are running load balancer services using GCP, the Compute Engine VM node instances require the
ocp
suffix. For example, if the value of theopenshift_gcp_prefix
parameter is set tomycluster
, you must tag the nodes withmyclusterocp
. See Adding and Removing Network Tags for more information on how to add network tags to Compute Engine VM instances. Optionally, you can configure multizone support.
The cluster installation process configures single-zone support by default, but you can configure for multiple zones to avoid single-point-of-failures.
Because GCP disks are created within a zone, deploying OpenShift Container Platform in GCP on different zones can cause problems with storage. If an OpenShift Container Platform node host goes down in zone "A" and the pods move to zone "B", the persistent storage cannot be attached to those pods because the disks are now in a different zone. See Multiple zone limitations in the Kubernetes documentation for more information.
To enable multizone support using the Ansible inventory file, add the following parameter:
[OSEv3:vars] openshift_gcp_multizone=true
To return to single-zone support, set the
openshift_gcp_multizone
value tofalse
and rerun the Ansible inventory file.
23.2.2. Option 2: Manually configuring OpenShift Container Platform for GCE
23.2.2.1. Manually configuring master hosts for GCE
Perform the following procedure on all master hosts.
Procedure
Add the GCE parameters to the
apiServerArguments
andcontrollerArguments
sections of the master configuration file at/etc/origin/master/master-config.yaml
by default:apiServerArguments: cloud-provider: - "gce" cloud-config: - "/etc/origin/cloudprovider/gce.conf" controllerArguments: cloud-provider: - "gce" cloud-config: - "/etc/origin/cloudprovider/gce.conf"
When you configure OpenShift Container Platform for GCP using Ansible, the /etc/origin/cloudprovider/gce.conf file is created automatically. Because you are manually configuring OpenShift Container Platform for GCP, you must create the file and enter the following:
[Global] project-id = <project-id> 1 network-name = <network-name> 2 node-tags = <node-tags> 3 node-instance-prefix = <instance-prefix> 4 multizone = true 5
- 1
- Provide the GCP project ID where the existing instances are running.
- 2
- Provide the network name if not using the default.
- 3
- Provide the tag for the GCP nodes. Must contain
ocp
as a suffix. For example, if the value of thenode-instance-prefix
parameter is set tomycluster
, the nodes must be tagged withmyclusterocp
. - 4
- Provide a unique string to identify your OpenShift Container Platform cluster.
- 5
- Set to
true
to trigger multizone deployments on GCP. Set toFalse
by default.
The cluster installation process configures single-zone support by default.
Deploying OpenShift Container Platform in GCP on different zones can be helpful to avoid single-point-of-failures, but can cause problems with storage. This is because GCP disks are created within a zone. If an OpenShift Container Platform node host goes down in zone "A" and the pods should be moved to zone "B", the persistent storage cannot be attached to those pods, because the disks are now in a different zone. See Multiple zone limitations in the Kubernetes documentation for more information.
ImportantFor running load balancer services using GCP, the Compute Engine VM node instances require the
ocp
suffix:<openshift_gcp_prefix>ocp
. For example, if the value of theopenshift_gcp_prefix
parameter is set tomycluster
, you must tag the nodes withmyclusterocp
. See Adding and Removing Network Tags for more information on how to add network tags to Compute Engine VM instances.Restart the OpenShift Container Platform host services:
# master-restart api # master-restart controllers # systemctl restart atomic-openshift-node
To return to single-zone support, set the multizone
value to false
and restart the master and node host services.
23.2.2.2. Manually configuring node hosts for GCE
Perform the following on all node hosts.
Procedure
Edit the appropriate node configuration map and update the contents of the
kubeletArguments
section:kubeletArguments: cloud-provider: - "gce" cloud-config: - "/etc/origin/cloudprovider/gce.conf"
ImportantThe
nodeName
must match the instance name in GCP in order for the cloud provider integration to work properly. The name must also be RFC1123 compliant.Restart the OpenShift Container Platform services on all nodes.
# systemctl restart atomic-openshift-node
23.2.3. Configuring the OpenShift Container Platform registry for GCP
Google Cloud Platform (GCP) provides object cloud storage that OpenShift Container Platform can use to store container images using the OpenShift Container Platform container image registry.
For more information, see Cloud Storage in the GCP documentation.
Prerequisites
You must create the bucket to host the registry images before the installation. The following commands create a regional bucket using the configured service account:
gsutil mb -c regional -l <region> gs://ocp-registry-bucket cat <<EOF > labels.json { "ocp-cluster": "mycluster" } EOF gsutil label set labels.json gs://ocp-registry-bucket rm -f labels.json
A bucket’s data is automatically encrypted using a Google-managed key by default. To specify a different key to encrypt the data, see the Data Encryption Options available in GCP.
See the Creating storage buckets documentation for more information.
Procedure
To configure the Ansible inventory file for the registry to use a Google Cloud Storage (GCS) bucket:
[OSEv3:vars] # GCP Provider Configuration openshift_hosted_registry_storage_provider=gcs openshift_hosted_registry_storage_kind=object openshift_hosted_registry_replicas=1 1 openshift_hosted_registry_storage_gcs_bucket=<bucket_name> 2 openshift_hosted_registry_storage_gcs_keyfile=<bucket_keyfile> 3 openshift_hosted_registry_storage_gcs_rootdirectory=<registry_directory> 4
For more information, see Cloud Storage in the GCP documentation.
23.2.3.1. Manually configuring OpenShift Container Platform registry for GCP
To use GCP object storage, edit the registry’s configuration file and mount to the registry pod.
See the Google Cloud Storage Driver documentation for more information about storage driver configuration files.
Procedure
Export the current /etc/registry/config.yml file:
$ oc get secret registry-config \ -o jsonpath='{.data.config\.yml}' -n default | base64 -d \ >> config.yml.old
Create a new configuration file from the old /etc/registry/config.yml file:
$ cp config.yml.old config.yml
Edit the file to include the GCP parameters. Specify the bucket and keyfile in the
storage
section of a registry’s configuration file:storage: delete: enabled: true cache: blobdescriptor: inmemory gcs: bucket: ocp-registry 1 keyfile: mykeyfile 2
Delete the
registry-config
secret:$ oc delete secret registry-config -n default
Recreate the secret to reference the updated configuration file:
$ oc create secret generic registry-config \ --from-file=config.yml -n default
Redeploy the registry to read the updated configuration:
$ oc rollout latest docker-registry -n default
23.2.3.1.1. Verify the registry is using GCP object storage
To verify if the registry is using GCP bucket storage:
Procedure
After a successful registry deployment using GCP storage, the registry
deploymentconfig
does not show any information if the registry is using anemptydir
instead of GCP bucket storage:$ oc describe dc docker-registry -n default ... Mounts: ... /registry from registry-storage (rw) Volumes: registry-storage: Type: EmptyDir 1 ...
- 1
- The temporary directory that shares a pod’s lifetime.
Check if the /registry mountpoint is empty. This is the volume GCP storage will use:
$ oc exec \ $(oc get pod -l deploymentconfig=docker-registry \ -o=jsonpath='{.items[0].metadata.name}') -i -t -- ls -l /registry total 0
If it is empty, it is because the GCP bucket configuration is performed in the
registry-config
secret:$ oc describe secret registry-config Name: registry-config Namespace: default Labels: <none> Annotations: <none> Type: Opaque Data ==== config.yml: 398 bytes
The installer creates a config.yml file with the desired configuration using the extended registry capabilities as seen in Storage in the installation documentation. To view the configuration file, including the
storage
section where the storage bucket configuration is stored:$ oc exec \ $(oc get pod -l deploymentconfig=docker-registry \ -o=jsonpath='{.items[0].metadata.name}') \ cat /etc/registry/config.yml version: 0.1 log: level: debug http: addr: :5000 storage: delete: enabled: true cache: blobdescriptor: inmemory gcs: bucket: ocp-registry auth: openshift: realm: openshift middleware: registry: - name: openshift repository: - name: openshift options: pullthrough: True acceptschema2: True enforcequota: False storage: - name: openshift
Or you can view the secret:
$ oc get secret registry-config -o jsonpath='{.data.config\.yml}' | base64 -d version: 0.1 log: level: debug http: addr: :5000 storage: delete: enabled: true cache: blobdescriptor: inmemory gcs: bucket: ocp-registry auth: openshift: realm: openshift middleware: registry: - name: openshift repository: - name: openshift options: pullthrough: True acceptschema2: True enforcequota: False storage: - name: openshift
You can verify that any image push was successful by viewing Storage in the GCP console, then clicking Browser and selecting the bucket, or by running the
gsutil
command:$ gsutil ls gs://ocp-registry/ gs://ocp-registry/docker/ $ gsutil du gs://ocp-registry/ 7660385 gs://ocp-registry/docker/registry/v2/blobs/sha256/03/033565e6892e5cc6dd03187d00a4575720a928db111274e0fbf31b410a093c10/data 7660385 gs://ocp-registry/docker/registry/v2/blobs/sha256/03/033565e6892e5cc6dd03187d00a4575720a928db111274e0fbf31b410a093c10/ 7660385 gs://ocp-registry/docker/registry/v2/blobs/sha256/03/ ...
If using an emptyDir
volume, the /registry
mountpoint looks similar to the following:
$ oc exec \ $(oc get pod -l deploymentconfig=docker-registry \ -o=jsonpath='{.items[0].metadata.name}') -i -t -- df -h /registry Filesystem Size Used Avail Use% Mounted on /dev/sdc 30G 226M 30G 1% /registry $ oc exec \ $(oc get pod -l deploymentconfig=docker-registry \ -o=jsonpath='{.items[0].metadata.name}') -i -t -- ls -l /registry total 0 drwxr-sr-x. 3 1000000000 1000000000 22 Jun 19 12:24 docker
23.2.4. Configuring OpenShift Container Platform to use GCP storage
OpenShift Container Platform can use GCP storage using persistent volumes mechanisms. OpenShift Container Platform creates the disk in GCP and attaches the disk to the correct instance.
GCP disks are ReadWriteOnce
access mode, which means the volume can be mounted as read-write by a single node. See the Access modes section of the Architecture guide for more information.
Procedure
OpenShift Container Platform creates the following
storageclass
when you use thegce-pd
provisioner and if you use theopenshift_cloudprovider_kind=gce
andopenshift_gcp_*
variables in the Ansible inventory. Otherwise, if you configured OpenShift Container Platform without using Ansible and thestorageclass
has not been created at installation time, you can create it manually:$ oc get --export storageclass standard -o yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: annotations: storageclass.kubernetes.io/is-default-class: "true" creationTimestamp: null name: standard selfLink: /apis/storage.k8s.io/v1/storageclasses/standard parameters: type: pd-standard provisioner: kubernetes.io/gce-pd reclaimPolicy: Delete
After you request a PV and using the storageclass shown in the previous step, OpenShift Container Platform creates disks in the GCP infrastructure. To verify that the disks were created:
$ gcloud compute disks list | grep kubernetes kubernetes-dynamic-pvc-10ded514-7625-11e8-8c52-42010af00003 us-west1-b 10 pd-standard READY
23.2.5. About Red Hat OpenShift Container Storage
Red Hat OpenShift Container Storage (RHOCS) is a provider of agnostic persistent storage for OpenShift Container Platform either in-house or in hybrid clouds. As a Red Hat storage solution, RHOCS is completely integrated with OpenShift Container Platform for deployment, management, and monitoring regardless if it is installed on OpenShift Container Platform (converged) or with OpenShift Container Platform (independent). OpenShift Container Storage is not limited to a single availability zone or node, which makes it likely to survive an outage. You can find complete instructions for using RHOCS in the RHOCS3.11 Deployment Guide.
23.3. Using the GCP external load balancer as a service
You can configure OpenShift Container Platform to use the GCP load balancer by exposing services externally using a LoadBalancer
service. OpenShift Container Platform creates the load balancer in GCP and creates the necessary firewall rules.
Procedure
Create a new application:
$ oc new-app openshift/hello-openshift
Expose the load balancer service:
$ oc expose dc hello-openshift --name='hello-openshift-external' --type='LoadBalancer'
This command creates a
LoadBalancer
service similar to the following example:apiVersion: v1 kind: Service metadata: labels: app: hello-openshift name: hello-openshift-external spec: externalTrafficPolicy: Cluster ports: - name: port-1 nodePort: 30714 port: 8080 protocol: TCP targetPort: 8080 - name: port-2 nodePort: 30122 port: 8888 protocol: TCP targetPort: 8888 selector: app: hello-openshift deploymentconfig: hello-openshift sessionAffinity: None type: LoadBalancer
To verify that the service has been created:
$ oc get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE hello-openshift ClusterIP 172.30.62.10 <none> 8080/TCP,8888/TCP 20m hello-openshift-external LoadBalancer 172.30.147.214 35.230.97.224 8080:31521/TCP,8888:30843/TCP 19m
The
LoadBalancer
type and External IP values indicate that the service is using GCP load balancers to expose the application.
OpenShift Container Platform creates the required objects in the GCP infrastructure such as:
Firewall rules:
$ gcloud compute firewall-rules list | grep k8s k8s-4612931a3a47c204-node-http-hc my-net INGRESS 1000 tcp:10256 k8s-fw-a1a8afaa7762811e88c5242010af0000 my-net INGRESS 1000 tcp:8080,tcp:8888
NoteThese firewall rules are applied to instances tagged with
<openshift_gcp_prefix>ocp
. For example, if the value of theopenshift_gcp_prefix
parameter is set tomycluster
, you must tag the nodes withmyclusterocp
. See Adding and Removing Network Tags for more information on how to add network tags to Compute Engine VM instances.Health checks:
$ gcloud compute http-health-checks list | grep k8s k8s-4612931a3a47c204-node 10256 /healthz
A load balancer:
$ gcloud compute target-pools list | grep k8s a1a8afaa7762811e88c5242010af0000 us-west1 NONE k8s-4612931a3a47c204-node $ gcloud compute forwarding-rules list | grep a1a8afaa7762811e88c5242010af0000 a1a8afaa7762811e88c5242010af0000 us-west1 35.230.97.224 TCP us-west1/targetPools/a1a8afaa7762811e88c5242010af0000
To verify that the load balancer is properly configured, run the following command from an external host:
$ curl 35.230.97.224:8080 Hello OpenShift!