Chapter 3. Customer Cloud Subscriptions on GCP
OpenShift Dedicated provides a Customer Cloud Subscription (CCS) model that allows Red Hat to deploy and manage clusters in a customer’s existing Google Cloud Platform (GCP) account.
3.1. Understanding Customer Cloud Subscriptions on GCP
Red Hat OpenShift Dedicated provides a Customer Cloud Subscription (CCS) model that allows Red Hat to deploy and manage OpenShift Dedicated into a customer’s existing Google Cloud Platform (GCP) account. Red Hat requires several prerequisites be met in order to provide this service.
Red Hat recommends the usage of GCP project, managed by the customer, to organize all of your GCP resources. A project consists of a set of users and APIs, as well as billing, authentication, and monitoring settings for those APIs.
It is recommended for the OpenShift Dedicated cluster using a CCS model to be hosted in a GCP project within a GCP organization. The Organization resource is the root node of the GCP resource hierarchy and all resources that belong to an organization are grouped under the organization node. Customers have the choice of using service account keys or Workload Identity Federation when creating the roles and credentials necessary to access Google Cloud resources within a GCP project.
3.2. Customer requirements
OpenShift Dedicated clusters using a Customer Cloud Subscription (CCS) model on Google Cloud Platform (GCP) must meet several prerequisites before they can be deployed.
3.2.1. Account
- The customer ensures that Google Cloud limits are sufficient to support OpenShift Dedicated provisioned within the customer-provided GCP account.
- The customer-provided GCP account should be in the customer’s Google Cloud Organization.
- The customer-provided GCP account must not be transferable to Red Hat.
- The customer may not impose GCP usage restrictions on Red Hat activities. Imposing restrictions severely hinders Red Hat’s ability to respond to incidents.
- Red Hat deploys monitoring into GCP to alert Red Hat when a highly privileged account, such as a root account, logs into the customer-provided GCP account.
The customer can deploy native GCP services within the same customer-provided GCP account.
NoteCustomers are encouraged, but not mandated, to deploy resources in a Virtual Private Cloud (VPC) separate from the VPC hosting OpenShift Dedicated and other Red Hat supported services.
3.2.2. Access requirements
To appropriately manage the OpenShift Dedicated service, Red Hat must have the
AdministratorAccesspolicy applied to the administrator role at all times.NoteThis policy only provides Red Hat with permissions and capabilities to change resources in the customer-provided GCP account.
- Red Hat must have GCP console access to the customer-provided GCP account. This access is protected and managed by Red Hat.
- The customer must not utilize the GCP account to elevate their permissions within the OpenShift Dedicated cluster.
- Actions available in the OpenShift Cluster Manager must not be directly performed in the customer-provided GCP account.
3.2.3. Support requirements
- Red Hat recommends that the customer have at least Enhanced Support from GCP.
- Red Hat has authority from the customer to request GCP support on their behalf.
- Red Hat has authority from the customer to request GCP resource limit increases on the customer-provided account.
- Red Hat manages the restrictions, limitations, expectations, and defaults for all OpenShift Dedicated clusters in the same manner, unless otherwise specified in this requirements section.
3.2.4. Security requirements
- The customer-provided IAM credentials must be unique to the customer-provided GCP account and must not be stored anywhere in the customer-provided GCP account.
- Volume snapshots will remain within the customer-provided GCP account and customer-specified region.
To manage, monitor, and troubleshoot OpenShift Dedicated clusters, Red Hat must have direct access to the cluster’s API server. You must not restrict or otherwise prevent Red Hat’s access to the OpenShift Dedicated cluster’s API server.
NoteSRE uses various methods to access clusters, depending on network configuration. Access to private clusters is restricted to Red Hat trusted IP addresses only. These access restrictions are managed automatically by Red Hat.
- OpenShift Dedicated requires egress access to certain endpoints over the internet. Only clusters deployed with Private Service Connect can use a firewall to control egress traffic. For additional information, see the GCP firewall prerequisites section.
3.3. Required customer procedure
The Customer Cloud Subscription (CCS) model allows Red Hat to deploy and manage OpenShift Dedicated into a customer’s Google Cloud Platform (GCP) project. Red Hat requires several prerequisites to provide these services.
The following requirements in this topic apply to OpenShift Dedicated on Google Cloud Platform (GCP) clusters created using both the service account and Workload Identity Federation authentication type. For additional requirements that apply to the service account authentication type only, see Service account authentication type procedure. For additional requirements that apply to the Workload Identity Federation authentication type only, see Workload Identity Federation authentication type procedure.
To use OpenShift Dedicated in your GCP project, the following GCP organizational policy constraints cannot be in place:
-
constraints/iam.allowedPolicyMemberDomains(This policy constraint is supported only if Red Hat’sDIRECTORY_CUSTOMER_ID C02k0l5e8is included in the allow list. Use this policy constraint with caution). -
constraints/compute.restrictLoadBalancerCreationForTypes -
constraints/compute.requireShieldedVm(This policy constraint is supported only if the cluster is installed with "Enable Secure Boot support for Shielded VMs" selected during the initial cluster creation). -
constraints/compute.vmExternalIpAccess(This policy constraint is supported only after installation).
Procedure
- Create a Google Cloud project to host the OpenShift Dedicated cluster.
Enable the following required APIs in the project that hosts your OpenShift Dedicated cluster:
Table 3.1. Required API services API service Console service name Purpose deploymentmanager.googleapis.comUsed for automated deployment and management of infrastructure resources.
compute.googleapis.comUsed for creating and managing virtual machines, firewalls, networks, persistent disk volumes, and load balancers.
cloudresourcemanager.googleapis.comUsed for getting projects, getting or setting an IAM policy for projects, validating required permissions, and tagging.
dns.googleapis.comUsed for creating DNS zones and managing DNS records for the cluster domains.
iamcredentials.googleapis.comUsed for creating short-lived credentials for impersonating IAM service accounts.
iam.googleapis.comUsed for managing the IAM configuration for the cluster.
servicemanagement.googleapis.comUsed indirectly to fetch quota information for GCP resources.
serviceusage.googleapis.comUsed for determining what services are available in the customer’s Google Cloud account.
storage-api.googleapis.comUsed for accessing Cloud Storage for the image registry, ignition, and cluster backups (if applicable).
storage-component.googleapis.comUsed for managing Cloud Storage for the image registry, ignition, and cluster backups (if applicable).
orgpolicy.googleapis.comUsed to identify governance rules applied to customer’s Google Cloud that might impact cluster creation or management.
iap.googleapis.com[*]Used in emergency situations to troubleshoot cluster nodes that are otherwise inaccessible.
This API is required for clusters deployed with Private Service Connect.
3.3.1. Service account authentication type procedure
Besides the required customer procedures listed in Required customer procedure, there are other specific actions that you must take when creating an OpenShift Dedicated cluster on Google Cloud Platform (GCP) using a service account as the authentication type.
Procedure
To ensure that Red Hat can perform necessary actions, you must create an
osd-ccs-adminIAM service account user within the GCP project.The following roles must be granted to the service account:
Table 3.2. Required roles Role Console role name Compute Admin
roles/compute.adminDNS Administrator
roles/dns.adminOrganization Policy Viewer
roles/orgpolicy.policyViewerService Management Administrator
roles/servicemanagement.adminService Usage Admin
roles/serviceusage.serviceUsageAdminStorage Admin
roles/storage.adminCompute Load Balancer Admin
roles/compute.loadBalancerAdminRole Viewer
roles/viewerRole Administrator
roles/iam.roleAdminSecurity Admin
roles/iam.securityAdminService Account Key Admin
roles/iam.serviceAccountKeyAdminService Account Admin
roles/iam.serviceAccountAdminService Account User
roles/iam.serviceAccountUser-
Create the service account key for the
osd-ccs-adminIAM service account. Export the key to a file namedosServiceAccount.json; this JSON file will be uploaded in Red Hat OpenShift Cluster Manager when you create your cluster.
3.3.2. Workload Identity Federation authentication type procedure
Besides the required customer procedures listed in Required customer procedure, there are other specific actions that you must take when creating an OpenShift Dedicated cluster on Google Cloud Platform (GCP) using Workload Identity Federation as the authentication type.
Procedure
Assign the following roles to the service account of the user implementing the Workload Identity Federation authentication type:
Table 3.3. Required roles Role Console role name Role purpose Role Administrator
roles/iam.roleAdminRequired by the GCP client in the OCM CLI for creating custom roles.
Service Account Admin
roles/iam.serviceAccountAdminRequired to pre-create the services account required by the OSD deployer, support and operators.
Workload Identity Pool Admin
roles/iam.workloadIdentityPoolAdminRequired to create and configure the workload identity pool.
Project IAM Admin
roles/resourcemanager.projectIamAdminRequired for assigning roles to the service account and giving permissions to those roles that are necessary to perform operations on cloud resources.
Install the OpenShift Cluster Manager API command-line interface (
ocm).To use the OCM CLI, you must authenticate against your Red Hat OpenShift Cluster Manager account. This is accomplished with the OpenShift Cluster Manager API token.
You can obtain your token here.
To authenticate against your Red Hat OpenShift Cluster Manager account, run the following command:
$ ocm login --token <token> 1- 1
- Replace
<token>with your OpenShift Cluster Manager API token.
ImportantOpenShift Cluster Manager API command-line interface (
ocm) is a Technology Preview feature only. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.- Install the gcloud CLI.
- Authenticate the gcloud CLI with the Application Default Credentials (ADC).
3.4. Red Hat managed Google Cloud resources
Red Hat is responsible for creating and managing the following IAM Google Cloud Platform (GCP) resources.
The IAM service account and roles and IAM group and roles topics are only applicable to clusters created using the service account authentication type.
3.4.1. IAM service account and roles
The osd-managed-admin IAM service account is created immediately after taking control of the customer-provided GCP account. This is the user that will perform the OpenShift Dedicated cluster installation.
The following roles are attached to the service account:
| Role | Console role name | Description |
|---|---|---|
| Compute Admin |
| Provides full control of all Compute Engine resources. |
| DNS Administrator |
| Provides read-write access to all Cloud DNS resources. |
| Security Admin |
| Security admin role, with permissions to get and set any IAM policy. |
| Storage Admin |
| Grants full control of objects and buckets. When applied to an individual bucket, control applies only to the specified bucket and objects within the bucket. |
| Service Account Admin |
| Create and manage service accounts. |
| Service Account Key Admin |
| Create and manage (and rotate) service account keys. |
| Service Account User |
| Run operations as the service account. |
| Role Administrator |
| Provides access to all custom roles in the project. |
3.4.2. IAM group and roles
The sd-sre-platform-gcp-access Google group is granted access to the GCP project to allow Red Hat Site Reliability Engineering (SRE) access to the console for emergency troubleshooting purposes.
-
For information regarding the roles within the
sd-sre-platform-gcp-accessgroup that are specific to clusters created when using the Workload Identity Federation (WIF) authentication type, see managed-cluster-config. - For information about creating a cluster using the Workload Identity Federation authentication type, see Additional resources.
The following roles are attached to the group:
| Role | Console role name | Description |
|---|---|---|
| Compute Admin |
| Provides full control of all Compute Engine resources. |
| Editor |
| Provides all viewer permissions, plus permissions for actions that modify state. |
| Organization Policy Viewer |
| Provides access to view Organization Policies on resources. |
| Project IAM Admin |
| Provides permissions to administer IAM policies on projects. |
| Quota Administrator |
| Provides access to administer service quotas. |
| Role Administrator |
| Provides access to all custom roles in the project. |
| Service Account Admin |
| Create and manage service accounts. |
| Service Usage Admin |
| Ability to enable, disable, and inspect service states, inspect operations, and consume quota and billing for a consumer project. |
| Tech Support Editor |
| Provides full read-write access to technical support cases. |
3.5. Provisioned GCP Infrastructure
This is an overview of the provisioned Google Cloud Platform (GCP) components on a deployed OpenShift Dedicated cluster. For a more detailed listing of all provisioned GCP components, see the OpenShift Container Platform documentation.
3.5.1. Compute instances
GCP compute instances are required to deploy the control plane and data plane functions of OpenShift Dedicated in GCP. Instance types might vary for control plane and infrastructure nodes depending on worker node count.
Single availability zone
- 2 infra nodes (custom machine type: 4 vCPU and 32 GB RAM)
- 3 control plane nodes (custom machine type: 8 vCPU and 32 GB RAM)
- 2 worker nodes (custom machine type: 4 vCPU and 16 GB RAM)
Multiple availability zones
- 3 infra nodes (custom machine type: 4 vCPU and 32 GB RAM)
- 3 control plane nodes (custom machine type: 8 vCPU and 32 GB RAM)
- 3 worker nodes (custom machine type: 4 vCPU and 16 GB RAM)
3.5.2. Storage
Infrastructure volumes:
- 300 GB SSD persistent disk (deleted on instance deletion)
- 110 GB Standard persistent disk (kept on instance deletion)
Worker volumes:
- 300 GB SSD persistent disk (deleted on instance deletion)
Control plane volumes:
- 350 GB SSD persistent disk (deleted on instance deletion)
3.5.3. VPC
- Subnets: One master subnet for the control plane workloads and one worker subnet for all others.
- Router tables: One global route table per VPC.
- Internet gateways: One internet gateway per cluster.
- NAT gateways: One master NAT gateway and one worker NAT gateway per cluster.
3.5.4. Services
The following services must be enabled on a GCP CCS cluster:
-
deploymentmanager -
compute -
cloudapis -
cloudresourcemanager -
dns -
iamcredentials -
iam -
servicemanagement -
serviceusage -
storage-api -
storage-component -
orgpolicy -
networksecurity
3.6. GCP account limits
The OpenShift Dedicated cluster uses a number of Google Cloud Platform (GCP) components, but the default quotas do not affect your ability to install an OpenShift Dedicated cluster.
A standard OpenShift Dedicated cluster uses the following resources. Note that some resources are required only during the bootstrap process and are removed after the cluster deploys.
| Service | Component | Location | Total resources required | Resources removed after bootstrap |
|---|---|---|---|---|
| Service account | IAM | Global | 5 | 0 |
| Firewall Rules | Compute | Global | 11 | 1 |
| Forwarding Rules | Compute | Global | 2 | 0 |
| In-use global IP addresses | Compute | Global | 4 | 1 |
| Health checks | Compute | Global | 3 | 0 |
| Images | Compute | Global | 1 | 0 |
| Networks | Compute | Global | 2 | 0 |
| Static IP addresses | Compute | Region | 4 | 1 |
| Routers | Compute | Global | 1 | 0 |
| Routes | Compute | Global | 2 | 0 |
| Subnetworks | Compute | Global | 2 | 0 |
| Target Pools | Compute | Global | 3 | 0 |
| CPUs | Compute | Region | 28 | 4 |
| Persistent Disk SSD (GB) | Compute | Region | 896 | 128 |
If any of the quotas are insufficient during installation, the installation program displays an error that states both which quota was exceeded and the region.
Be sure to consider your actual cluster size, planned cluster growth, and any usage from other clusters that are associated with your account. The CPU, Static IP addresses, and Persistent Disk SSD (Storage) quotas are the ones that are most likely to be insufficient.
If you plan to deploy your cluster in one of the following regions, you will exceed the maximum storage quota and are likely to exceed the CPU quota limit:
- asia-east2
- asia-northeast2
- asia-south1
- australia-southeast1
- europe-north1
- europe-west2
- europe-west3
- europe-west6
- northamerica-northeast1
- southamerica-east1
- us-west2
You can increase resource quotas from the GCP console, but you might need to file a support ticket. Be sure to plan your cluster size early so that you can allow time to resolve the support ticket before you install your OpenShift Dedicated cluster.
3.7. GCP firewall prerequisites
If you are using a firewall to control egress traffic from OpenShift Dedicated on Google Cloud Platform (GCP), you must configure your firewall to grant access to certain domains and port combinations listed in the tables below. OpenShift Dedicated requires this access to provide a fully managed OpenShift service.
Only OpenShift Dedicated on Google Cloud Platform (GCP) clusters deployed with Private Service Connect can use a firewall to control egress traffic.
Procedure
Add the following URLs that are used to install and download packages and tools to an allowlist:
Domain Port Function registry.redhat.io443
Provides core container images.
quay.io443
Provides core container images.
cdn01.quay.iocdn02.quay.iocdn03.quay.iocdn04.quay.iocdn05.quay.iocdn06.quay.io443
Provides core container images.
sso.redhat.com443
Required. The https://console.redhat.com/openshift site uses authentication from sso.redhat.com to download the pull secret and use Red Hat SaaS solutions to facilitate monitoring of your subscriptions, cluster inventory, chargeback reporting, and so on.
quayio-production-s3.s3.amazonaws.com443
Provides core container images.
pull.q1w2.quay.rhcloud.com443
Provides core container images.
registry.access.redhat.com443
Hosts all the container images that are stored on the Red Hat Ecosytem Catalog. Additionally, the registry provides access to the
odoCLI tool that helps developers build on OpenShift and Kubernetes.registry.connect.redhat.com443
Required for all third-party images and certified Operators.
console.redhat.com443
Required. Allows interactions between the cluster and Red Hat OpenShift Cluster Manager to enable functionality, such as scheduling upgrades.
sso.redhat.com443
The
https://console.redhat.com/openshiftsite uses authentication fromsso.redhat.com.catalog.redhat.com443
The
registry.access.redhat.comandhttps://registry.redhat.iosites redirect throughcatalog.redhat.com.Add the following telemetry URLs to an allowlist:
Domain Port Function cert-api.access.redhat.com443
Required for telemetry.
api.access.redhat.com443
Required for telemetry.
infogw.api.openshift.com443
Required for telemetry.
console.redhat.com443
Required for telemetry and Red Hat Insights.
observatorium-mst.api.openshift.com443
Required for managed OpenShift-specific telemetry.
observatorium.api.openshift.com443
Required for managed OpenShift-specific telemetry.
NoteManaged clusters require the enabling of telemetry to allow Red Hat to react more quickly to problems, better support the customers, and better understand how product upgrades impact clusters. For more information about how remote health monitoring data is used by Red Hat, see About remote health monitoring in the Additional resources section.
Add the following OpenShift Dedicated URLs to an allowlist:
Domain Port Function mirror.openshift.com443
Used to access mirrored installation content and images. This site is also a source of release image signatures.
api.openshift.com443
Used to check if updates are available for the cluster.
Add the following site reliability engineering (SRE) and management URLs to an allowlist:
Domain Port Function api.pagerduty.com443
This alerting service is used by the in-cluster alertmanager to send alerts notifying Red Hat SRE of an event to take action on.
events.pagerduty.com443
This alerting service is used by the in-cluster alertmanager to send alerts notifying Red Hat SRE of an event to take action on.
api.deadmanssnitch.com443
Alerting service used by OpenShift Dedicated to send periodic pings that indicate whether the cluster is available and running.
nosnch.in443
Alerting service used by OpenShift Dedicated to send periodic pings that indicate whether the cluster is available and running.
*.osdsecuritylogs.splunkcloud.comOR
inputs1.osdsecuritylogs.splunkcloud.cominputs2.osdsecuritylogs.splunkcloud.cominputs4.osdsecuritylogs.splunkcloud.cominputs5.osdsecuritylogs.splunkcloud.cominputs6.osdsecuritylogs.splunkcloud.cominputs7.osdsecuritylogs.splunkcloud.cominputs8.osdsecuritylogs.splunkcloud.cominputs9.osdsecuritylogs.splunkcloud.cominputs10.osdsecuritylogs.splunkcloud.cominputs11.osdsecuritylogs.splunkcloud.cominputs12.osdsecuritylogs.splunkcloud.cominputs13.osdsecuritylogs.splunkcloud.cominputs14.osdsecuritylogs.splunkcloud.cominputs15.osdsecuritylogs.splunkcloud.com9997
Used by the
splunk-forwarder-operatoras a logging forwarding endpoint to be used by Red Hat SRE for log-based alerting.http-inputs-osdsecuritylogs.splunkcloud.com443
Used by the
splunk-forwarder-operatoras a logging forwarding endpoint to be used by Red Hat SRE for log-based alerting.sftp.access.redhat.com(Recommended)22
The SFTP server used by
must-gather-operatorto upload diagnostic logs to help troubleshoot issues with the cluster.Add the following URLs for the Google Cloud Platform (GCP) API endpoints to an allowlist:
Domain Port Function accounts.google.com443
Used to access your GCP account.
*.googleapis.comOR
storage.googleapis.comiam.googleapis.comserviceusage.googleapis.comcloudresourcemanager.googleapis.comcompute.googleapis.comoauth2.googleapis.comdns.googleapis.comiamcredentials.googleapis.com443
Used to access GCP services and resources. Review Cloud Endpoints in the GCP documentation to determine the endpoints to allow for your APIs.
NoteRequired Google APIs can be exposed using the Private Google Access restricted virtual IP (VIP), with the exception of the Service Usage API (serviceusage.googleapis.com). To circumvent this, you must expose the Service Usage API using the Private Google Access private VIP.
3.8. Additional resources
- About remote health monitoring
- For more information about creating an OpenShift Dedicated cluster with the Workload Identity Federation (WIF) authentication type, see Creating a WIF configuration.
- For more information about the specific roles and permissions that are specific to clusters created when using the Workload Identity Federation (WIF) authentication type, see managed-cluster-config.
