Chapter 6. Creating ROSA with HCP clusters with external authentication

You can create Red Hat OpenShift Service on AWS (ROSA) with hosted control planes (HCP) clusters that use an external OpenID Connect (OIDC) identity provider to issue tokens for authentication, replacing the built-in OpenShift OAuth server. While the built-in OpenShift OAuth server supports integration with a variety of identity providers, including external OIDC identity providers, it is limited to the capabilities of the OAuth server itself. You can directly integrate external OIDC identity providers with ROSA with HCP clusters in order to facilitate machine-to-machine workflows, such as CLI, and provide additional capabilities which are not available when using the built-in OpenShift OAuth server.

Important

Since it is not possible to upgrade or convert existing ROSA clusters to a hosted control planes architecture, you must create a new cluster to use ROSA with HCP functionality. You also cannot convert a cluster that was created to use external authentication providers to use the internal OAuth2 server. You must also create a new cluster.

Important

Sharing VPCs across multiple AWS accounts is not currently supported for ROSA with HCP. Do not install a ROSA with HCP cluster into subnets shared from another AWS account. See "Are multiple ROSA clusters in a single VPC supported?" for more information.

Note

ROSA with HCP clusters only support Security Token Service (STS) authentication.

Further reading

See the AWS documentation for information about Getting started with ROSA with HCP using the ROSA CLI in auto mode.

6.1. ROSA with HCP Prerequisites

To create a ROSA with HCP cluster, you must have completed the following steps:

6.2. Creating a ROSA with HCP cluster that uses external authentication providers

Use the --external-auth-providers-enabled flag in the ROSA CLI to create a cluster that uses an external authentication service.

Note

When creating a ROSA with HCP cluster, the default machine Classless Inter-Domain Routing (CIDR) is 10.0.0.0/16. If this does not correspond to the CIDR range for your VPC subnets, add --machine-cidr <address_block> to the following commands.

Procedure

If you used the OIDC_ID, SUBNET_IDS, and OPERATOR_ROLES_PREFIX variables to prepare your environment, you can continue to use those variables when creating your cluster. For example, run the following command:

$ rosa create cluster --hosted-cp --subnet-ids=$SUBNET_IDS \
   --oidc-config-id=$OIDC_ID --cluster-name=<cluster_name> \
   --operator-roles-prefix=$OPERATOR_ROLES_PREFIX \
   --external-auth-providers-enabled

If you did not set environmental variables, run the following command:

$ rosa create cluster --cluster-name=<cluster_name> --sts --mode=auto \
    --hosted-cp --operator-roles-prefix <operator-role-prefix> \
    --oidc-config-id <ID-of-OIDC-configuration> \
    --external-auth-providers-enabled \
    --subnet-ids=<public-subnet-id>,<private-subnet-id>

Verification

Verify that your external authentication is enabled in the cluster details by running the following command:

$ rosa describe cluster --cluster=<cluster_name>

Name:                       rosa-ext-test
Display Name:               rosa-ext-test
ID:                         <cluster_id>
External ID:                <cluster_ext_id>
Control Plane:              ROSA Service Hosted
OpenShift Version:          4.18.0
Channel Group:              stable
DNS:                        <dns>
AWS Account:                <AWS_id>
AWS Billing Account:        <AWS_id>
API URL:                    <ocm_api>
Console URL:
Region:                     us-east-1
Availability:
 - Control Plane:           MultiAZ
 - Data Plane:              SingleAZ

Nodes:
 - Compute (desired):       2
 - Compute (current):       0
Network:
 - Type:                    OVNKubernetes
 - Service CIDR:            <service_cidr>
 - Machine CIDR:            <machine_cidr>
 - Pod CIDR:                <pod_cidr>
 - Host Prefix:             /23
 - Subnets:                 <subnet_ids>
EC2 Metadata Http Tokens:   optional
Role (STS) ARN:             arn:aws:iam::<AWS_id>:role/<account_roles_prefix>-HCP-ROSA-Installer-Role
Support Role ARN:           arn:aws:iam::<AWS_id>:role/<account_roles_prefix>-HCP-ROSA-Support-Role
Instance IAM Roles:
 - Worker:                  arn:aws:iam::<AWS_id>:role/<account_roles_prefix>-HCP-ROSA-Worker-Role
Operator IAM Roles:
 - arn:aws:iam::<AWS_id>:role/<operator_roles_prefix>-openshift-cloud-network-config-controller-clo
 - arn:aws:iam::<AWS_id>:role/<operator_roles_prefix>-kube-system-capa-controller-manager
 - arn:aws:iam::<AWS_id>:role/<operator_roles_prefix>-kube-system-control-plane-operator
 - arn:aws:iam::<AWS_id>:role/<operator_roles_prefix>-kube-system-kms-provider
 - arn:aws:iam::<AWS_id>:role/<operator_roles_prefix>-kube-system-kube-controller-manager
 - arn:aws:iam::<AWS_id>:role/<operator_roles_prefix>-openshift-image-registry-installer-cloud-cred
 - arn:aws:iam::<AWS_id>:role/<operator_roles_prefix>-openshift-ingress-operator-cloud-credentials
 - arn:aws:iam::<AWS_id>:role/<operator_roles_prefix>-openshift-cluster-csi-drivers-ebs-cloud-crede
Managed Policies:           Yes
State:                      ready
Private:                    No
Created:                    Mar 29 2024 14:25:52 UTC
User Workload Monitoring:   Enabled
Details Page:               https://<url>
OIDC Endpoint URL:          https://<endpoint> (Managed)
Audit Log Forwarding:       Disabled
External Authentication:    Enabled 1

1: The External Authentication flag is enabled, and you can now create an external authentication provider.

6.3. Creating an external authentication provider

After you have created a ROSA with HCP cluster with the enabled option for external authentication providers, you must create a provider using the ROSA CLI.

Note

Similar to the rosa create|delete|list idp[s] command in the ROSA CLI, you cannot edit an existing identity provider that you created using rosa create external-auth-provider. Instead, you must delete the external authentication provider and create a new one.

The following table shows the possible CLI flags you can use when creating your external authentication provider:

CLI Flag	Description
`--cluster`	The name or the ID of your cluster.
`--name`	A name that is used to refer to the external authentication provider.
`--console-client-secret`	This string is the client secret that is used to associate your account with the application. If you do not include the client secret, this command uses a public OIDC OAuthClient.
`--issuer-audiences`	This is a comma-separated list of token audiences.
`--issuer-url`	The URL of the token issuer.
`--claim-mapping-username-claim`	The name of the claim that should be used to construct user names for the cluster identity.
`--claim-mapping-groups-claim`	The name of the claim that should be used to construct group names for the cluster identity.

Procedure

To use the interactive command interface, run the following commands:
```
$ rosa create external-auth-provider -c <cluster_name>
```
```
I: Enabling interactive mode
? Name: 1
? Issuer audiences: 2
? The serving url of the token issuer: 3
? CA file path (optional): 4
? Claim mapping username: 5
? Claim mapping groups: 6
? Claim validation rule (optional): 7
? Console client id (optional): 8
```
1
The name of your external authentication provider. This name should be a lower-case with numbers and dashes.
2
The audience IDs that this authentication provider issues tokens for.
3
The issuer’s URL that serves the token.
4
Optional: The certificate file to use when making requests.
5
The name of the claim that is used to construct the user names for cluster identity, such as using email.
6
The method with which to transform the ID token into a cluster identity, such as using groups.
7
Optional: The rules that help validate token claims which authenticate your users. This field should be formatted as :<required_value>.
8
Optional: The application or client ID that your app registration uses for the console.

You can include the required IDs to create your external authentication provider with the following command:

rosa create external-auth-provider --cluster=<cluster_id> \
    --name=<provider_name> --issuer-url=<issuing_url> \
    --issuer-audiences=<audience_id> \
    --claim-mapping-username-claim=email \
    --claim-mapping-groups-claim=groups \
    --console-client-id=<client_id_for_app_registration> \
    --console-client-secret=<client_secret>

Example output

I: Successfully created an external authentication provider for cluster '<cluster_id>'

Verification

To verify your external authentication provider, run one of the following options:

List the external authentication configuration on a specified cluster with the following command:
```
$ rosa list external-auth-provider -c <cluster_name>
```
Example output
The following example shows a configured Microsoft Entra ID external authentication provider:
```
NAME        ISSUER URL
m-entra-id  https://login.microsoftonline.com/<group_id>/v2.0
```

Display the external authentication configuration on a specified cluster by using the following command:

$ rosa describe external-auth-provider \
    -c <cluster_name> --name <name_of_external_authentication>

Example output

ID:                          ms-entra-id
Cluster ID:                  <cluster_id>
Issuer audiences:
                             - <audience_id>
Issuer Url:                  https://login.microsoftonline.com/<group_id>/v2.0
Claim mappings group:        groups
Claim mappings username:     email

Additional resources

For more information about configuring Entra ID for your IDP, see What is Microsoft Entra ID? in the Azure documentation or the Configuring Microsoft Entra ID (formerly Azure Active Directory) as an identity provider tutorial section of the documentation.

6.4. Creating a break glass credential for a ROSA with HCP cluster

As a ROSA with HCP cluster owner, you can use the break glass credential to create temporary administrative client credentials to access your clusters that are configured with custom OpenID Connect (OIDC) token issuers. Creating a break glass credential generates a new cluster-admin kubeconfig file. The kubeconfig file contains information about the cluster that the CLI uses to connect a client to the correct cluster and API server. You can use the newly generated kubeconfig file to allow access to the ROSA with HCP cluster.

Prerequisites

You have created a ROSA with HCP cluster with external authentication enabled. For more information, see Creating a ROSA with HCP with HCP cluster that uses external authentication providers.
You have created an external authentication provider. For more information, see Creating an external authentication provider.
You have an account with cluster admin permissions.

Procedure

Create a break glass credential by using one of the following commands:
- To create a break glass credential by using the interactive command interface to interactively specify custom settings, run the following command:
```
$ rosa create break-glass-credential -c <cluster_name> -i 1
```
  1
  Replace <cluster_name> with the name of your cluster.
  This command starts an interactive CLI process:
  Example output
```
I: Enabling interactive mode
? Username (optional): 1
? Expiration duration (optional): 2
I: Successfully created a break glass credential for cluster 'ac-hcp-test'.
```
  1
  If left blank, the value in the username will have a randomly generated username value.
  2
  The minimum validity of the break glass credential is 10 minutes, and the maximum validity is 24 hours. If left blank, the expiration duration value defaults to 24 hours.
- To create a break glass credential for cluster called mycluster with specified values:
```
$ rosa create break-glass-credential -c mycluster --username test-username --expiration 1h
```
List the break glass credential IDs, status, and associated users that are available for a cluster called mycluster by running the following command:
```
$ rosa list break-glass-credential -c mycluster
```
Example output
```
ID                                USERNAME    STATUS
2a7jli9n4phe6c02ul7ti91djtv2o51d  test-user   issued
```
Note
You can also view the credentials in a JSON output by adding the -o json argument to the command.
To view the status of a break glass credential, run the following command, replacing <break_glass_credential_id> with the break glass credential ID:
```
$ rosa describe break-glass-credential <break_glass_credential_id> -c <cluster_name>
```
Example output
```
ID:                                    2a7jli9n4phe6c02ul7ti91djtv2o51d
Username:                              test-user
Expire at:                             Dec 28 2026 10:23:05 EDT
Status:                                issued
```
The following is a list of possible Status field values:
- issued The break glass credential has been issued and is ready to use.
- expired The break glass credential has expired and can no longer be used.
- failed The break glass credential has failed to create. In this case, you receive a service log detailing the failure. For more information about service logs, see Accessing the service logs for Red Hat OpenShift Service on AWS clusters. For steps to contact Red Hat Support for assistance, see Getting support.
- awaiting_revocation The break glass credential is currently being revoked, meaning it cannot be used.
- revoked The break glass credential has been revoked and can no longer be used.

To retrieve the kubeconfig, run the following commands:

Create a kubeconfigs directory:
```
$ mkdir ~/kubeconfigs
```

Export the newly generated kubeconfig file, replacing <cluster_name> with the name of your cluster:

$ export CLUSTER_NAME=<cluster_name> && export KUBECONFIG=~/kubeconfigs/break-glass-${CLUSTER_NAME}.kubeconfig

View the kubeconfig:

$ rosa describe break-glass-credential <break_glass_credential_id> -c mycluster --kubeconfig

Example output

apiVersion: v1
clusters:
- cluster:
    server: <server_url>
  name: cluster
contexts:
- context:
    cluster: cluster
    namespace: default
    user: test-username
  name: admin
current-context: admin
kind: Config
preferences: {}
users:
- name: test-user
  user:
    client-certificate-data: <client-certificate-data> 1
    client-key-data: <client-key-data> 2

1: The client-certificate contains a certificate for the user signed by the Kubernetes certificate authorities (CA).
2: The client-key contains the key that signed the client certificate.

Optional: To save the kubeconfig, run the following command :

$ rosa describe break-glass-credential <break_glass_credential_id> -c mycluster --kubeconfig > $KUBECONFIG

Additional resources

For more information about creating a ROSA with HCP cluster with external authentication enabled, see Creating a ROSA with HCP cluster that uses external authentication providers.

6.5. Accessing a ROSA with HCP cluster by using a break glass credential

Use the new kubeconfig from the break glass credential to gain temporary admin access to a ROSA with HCP cluster.

Prerequisites

You have access to a ROSA with HCP cluster with external authentication enabled. For more information, see Creating a ROSA with HCP cluster that uses external authentication providers.
You have installed the oc and the kubectl CLIs.
You have configured the new kubeconfig. For more information, see Creating a break glass credential for a ROSA with HCP cluster.

Procedure

Access the details for the cluster:

$ rosa describe break-glass-credential <break_glass_credential_id> -c <cluster_name>  --kubeconfig > $KUBECONFIG

List the nodes from the cluster:

$ oc get nodes

Example output

NAME                        STATUS   ROLES   AGE   VERSION
ip-10-0-0-27.ec2.internal   Ready    worker  8m    v1.28.7+f1b5f6c
ip-10-0-0-67.ec2.internal   Ready    worker  9m    v1.28.7+f1b5f6c

Verify you have the correct credentials:

$ kubectl auth whoami

Example output

ATTRIBUTE    VALUE
Username     system:customer-break-glass:test-user
Groups       [system:masters system:authenticated]

Apply the ClusterRoleBinding for the groups defined in the external OIDC provider. The ClusterRoleBinding maps the rosa-hcp-admins group that is created in Microsoft Entra ID to a group in the ROSA with HCP cluster.
```
$ oc apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: rosa-hcp-admins
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: f715c264-ab90-45d5-8a29-2e91a609a895
EOF
```
Example output
```
clusterrolebinding.rbac.authorization.k8s.io/rosa-hcp-admins created
```
Note
After the ClusterRoleBinding has been applied, the ROSA with HCP cluster is configured, and the rosa CLI and the Red Hat Hybrid Cloud Console are authenticated through the external OpenID Connect (OIDC) provider. You can now start assigning roles and deploying applications on the cluster.

Additional resources

For more information about cluster role binding, see Using RBAC to define and apply permissions.

6.6. Revoking a break glass credential for a ROSA with HCP cluster

You can revoke access to any break glass credentials that you have provisioned at any time by using the revoke break-glass-credentials command.

Prerequisites

You have created a break glass credential.
You are the cluster owner.

Procedure

Revoke the break glass credentials for a ROSA with HCP cluster by running the following command.
Important
Running this command will revoke access for all break glass credentials related to the cluster.
```
$ rosa revoke break-glass-credentials -c <cluster_name> 1
```
1
Replace <cluster_name> with the name of your cluster.
Example output
```
? Are you sure you want to revoke all the break glass credentials on cluster 'my-cluster'?: Yes
I: Successfully requested revocation for all break glass credentials from cluster 'my-cluster'
```

Verification

The revocation process can take several minutes. You can verify that the break glass credentials for your clusters have been revoked by running one of the following commands:

List all break glass credentials and check the status of each:

$ rosa list break-glass-credential -c <cluster_name>

Example output

ID                                USERNAME    STATUS
2330dbs0n8m3chkkr25gkkcd8pnj3lk2  test-user   awaiting_revocation

You can also verify the status by checking the individual credential:

$ rosa describe break-glass-credential <break_glass_credential_id> -c <cluster_name>

Example output

ID:                                    2330dbs0n8m3chkkr25gkkcd8pnj3lk2
Username:                              test-user
Expire at:                             Dec 28 2026 10:23:05 EDT
Status:                                issued
Revoked at:                            Dec 27 2026 15:30:33 EDT

6.7. Deleting an external authentication provider

Delete external authentication providers by using the ROSA CLI.

Procedure

Display your external authentication provider on your cluster by running the following command:

$ rosa list external-auth-provider -c <cluster_name>

Example output

NAME        ISSUER URL
entra-test  https://login.microsoftonline.com/<group_id>/v2.0

Delete the external authentication provider by running the following command:

$ rosa delete external-auth-provider <name_of_provider> -c <cluster_name>

Example output

? Are you sure you want to delete external authentication provider entra-test on cluster rosa-ext-test? Yes
I: Successfully deleted external authentication provider 'entra-test' from cluster 'rosa-ext-test'

Verification

Query for any external authentication providers on your cluster by running the following command:

$ rosa list external-auth-provider -c <cluster_name>

Example output

E: there are no external authentication providers for this cluster

6.8. Additional resources

For details about optionally setting an Operator role name prefix, see About custom Operator IAM role prefixes.
For information about the prerequisites to installing ROSA with STS, see AWS prerequisites for ROSA with STS.
For details about using the auto and manual modes to create the required STS resources, see Understanding the auto and manual deployment modes.
For more information about using OpenID Connect (OIDC) identity providers in AWS IAM, see Creating OpenID Connect (OIDC) identity providers in the AWS documentation.
For more information about troubleshooting ROSA cluster installations, see Troubleshooting ROSA with HCP cluster installations.
For steps to contact Red Hat Support for assistance, see Getting support for Red Hat OpenShift Service on AWS.

Chapter 6. Creating ROSA with HCP clusters with external authentication

6.1. ROSA with HCP Prerequisites

6.2. Creating a ROSA with HCP cluster that uses external authentication providers

6.3. Creating an external authentication provider

6.4. Creating a break glass credential for a ROSA with HCP cluster

6.5. Accessing a ROSA with HCP cluster by using a break glass credential

6.6. Revoking a break glass credential for a ROSA with HCP cluster

6.7. Deleting an external authentication provider

6.8. Additional resources

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

Making open source more inclusive

About Red Hat

Red Hat legal and privacy links

Red Hat legal and privacy links