Search
SupportConsoleDevelopersStart a trialContact

Chapter 13. Managing machines with the Cluster API

download PDF

13.1. About the Cluster API

Important

Managing machines with the Cluster API is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

The Cluster API is an upstream project that is integrated into OpenShift Container Platform as a Technology Preview for Amazon Web Services (AWS), Google Cloud Platform (GCP), and VMware vSphere.

13.1.1. Cluster API overview

You can use the Cluster API to create and manage compute machine sets and compute machines in your OpenShift Container Platform cluster. This capability is in addition or an alternative to managing machines with the Machine API.

For OpenShift Container Platform 4.17 clusters, you can use the Cluster API to perform node host provisioning management actions after the cluster installation finishes. This system enables an elastic, dynamic provisioning method on top of public or private cloud infrastructure.

With the Cluster API Technology Preview, you can create compute machines and compute machine sets on OpenShift Container Platform clusters for supported providers. You can also explore the features that are enabled by this implementation that might not be available with the Machine API.

13.1.1.1. Cluster API benefits

By using the Cluster API, OpenShift Container Platform users and developers gain the following advantages:

  • The option to use upstream community Cluster API infrastructure providers that might not be supported by the Machine API.
  • The opportunity to collaborate with third parties who maintain machine controllers for infrastructure providers.
  • The ability to use the same set of Kubernetes tools for infrastructure management in OpenShift Container Platform.
  • The ability to create compute machine sets by using the Cluster API that support features that are not available with the Machine API.

13.1.1.2. Cluster API limitations

Using the Cluster API to manage machines is a Technology Preview feature and has the following limitations:

  • To use this feature, you must enable the TechPreviewNoUpgrade feature set.

    Important

    Enabling this feature set cannot be undone and prevents minor version updates.

  • Only Amazon Web Services (AWS), Google Cloud Platform (GCP), and VMware vSphere clusters can use the Cluster API.
  • You must manually create the primary resources that the Cluster API requires. For more information, see "Getting started with the Cluster API".
  • You cannot use the Cluster API to manage control plane machines.
  • Migration of existing compute machine sets created by the Machine API to Cluster API compute machine sets is not supported.
  • Full feature parity with the Machine API is not available.

  • For clusters that use the Cluster API, OpenShift CLI (oc) commands prioritize Cluster API objects over Machine API objects. This behavior impacts any oc command that acts upon any object that is represented in both the Cluster API and the Machine API.

    For more information and a workaround for this issue, see "Referencing the intended objects when using the CLI" in the troubleshooting content.

Additional resources

13.1.2. Cluster API architecture

The OpenShift Container Platform integration of the upstream Cluster API is implemented and managed by the Cluster CAPI Operator. The Cluster CAPI Operator and its operands are provisioned in the openshift-cluster-api namespace, in contrast to the Machine API, which uses the openshift-machine-api namespace.

13.1.2.1. The Cluster CAPI Operator

The Cluster CAPI Operator is an OpenShift Container Platform Operator that maintains the lifecycle of Cluster API resources. This Operator is responsible for all administrative tasks related to deploying the Cluster API project within an OpenShift Container Platform cluster.

If a cluster is configured correctly to allow the use of the Cluster API, the Cluster CAPI Operator installs the Cluster API components on the cluster.

For more information, see the "Cluster CAPI Operator" entry in the Cluster Operators reference content.

Additional resources

13.1.2.2. Cluster API primary resources

The Cluster API consists of the following primary resources. For the Technology Preview of this feature, you must create these resources manually in the openshift-cluster-api namespace.

Cluster
A fundamental unit that represents a cluster that is managed by the Cluster API.
Infrastructure
A provider-specific resource that defines properties that are shared by all the compute machine sets in the cluster, such as the region and subnets.
Machine template
A provider-specific template that defines the properties of the machines that a compute machine set creates.
Machine set

A group of machines.

Compute machine sets are to machines as replica sets are to pods. To add machines or scale them down, change the replicas field on the compute machine set custom resource to meet your compute needs.

With the Cluster API, a compute machine set references a Cluster object and a provider-specific machine template.

Machine

A fundamental unit that describes the host for a node.

The Cluster API creates machines based on the configuration in the machine template.

13.2. Getting started with the Cluster API

Important

Managing machines with the Cluster API is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

For the Cluster API Technology Preview, you must create the primary resources that the Cluster API requires manually.

13.2.1. Creating the Cluster API primary resources

To create the Cluster API primary resources, you must obtain the cluster ID value, which you use for the <cluster_name> parameter in the cluster resource manifest.

13.2.1.1. Obtaining the cluster ID value

You can find the cluster ID value by using the OpenShift CLI (oc).

Prerequisites

  • You have deployed an OpenShift Container Platform cluster.
  • You have access to the cluster using an account with cluster-admin permissions.
  • You have installed the OpenShift CLI (oc).

Procedure

  • Obtain the value of the cluster ID by running the following command: 

    $  oc get infrastructure cluster \
   -o jsonpath='{.status.infrastructureName}'

You can create the Cluster API primary resources manually by creating YAML manifest files and applying them with the OpenShift CLI (oc).

13.2.1.2. Creating the Cluster API cluster resource

You can create the cluster resource by creating a YAML manifest file and applying it with the OpenShift CLI (oc).

Prerequisites

  • You have deployed an OpenShift Container Platform cluster.
  • You have enabled the use of the Cluster API.
  • You have access to the cluster using an account with cluster-admin permissions.
  • You have installed the OpenShift CLI (oc).
  • You have the cluster ID value.

Procedure

  1. Create a YAML file similar to the following. This procedure uses <cluster_resource_file>.yaml as an example file name. 

    apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <cluster_name> 1
  namespace: openshift-cluster-api
spec:
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: <infrastructure_kind> 2
    name: <cluster_name>
    namespace: openshift-cluster-api
    1
    Specify the cluster ID as the name of the cluster.
    2
    Specify the infrastructure kind for the cluster. The following values are valid:
    • AWSCluster: The cluster is running on Amazon Web Services (AWS).
    • GCPCluster: The cluster is running on Google Cloud Platform (GCP).
    • VSphereCluster: The cluster is running on VMware vSphere.

  2. Create the cluster CR by running the following command: 

    $ oc create -f <cluster_resource_file>.yaml

Verification

  • Confirm that the cluster CR exists by running the following command: 

    $ oc get cluster

    Example output

    NAME             PHASE          AGE    VERSION
<cluster_name>   Provisioning   4h6m

    The cluster resource is ready when the value of PHASE is Provisioned.

Additional resources

13.2.1.3. Creating a Cluster API infrastructure resource

You can create a provider-specific infrastructure resource by creating a YAML manifest file and applying it with the OpenShift CLI (oc).

Prerequisites

  • You have deployed an OpenShift Container Platform cluster.
  • You have enabled the use of the Cluster API.
  • You have access to the cluster using an account with cluster-admin permissions.
  • You have installed the OpenShift CLI (oc).
  • You have the cluster ID value.
  • You have created and applied the cluster resource.

Procedure

  1. Create a YAML file similar to the following. This procedure uses <infrastructure_resource_file>.yaml as an example file name. 

    apiVersion: infrastructure.cluster.x-k8s.io/<version> 1
kind: <infrastructure_kind> 2
metadata:
  name: <cluster_name> 3
  namespace: openshift-cluster-api
spec: 4
    1
    The apiVersion varies by platform. For more information, see the sample Cluster API infrastructure resource YAML for your provider. The following values are valid:
    • infrastructure.cluster.x-k8s.io/v1beta2: The version that Amazon Web Services (AWS) clusters use.
    • infrastructure.cluster.x-k8s.io/v1beta1: The version that Google Cloud Platform (GCP) and VMware vSphere clusters use.
    2
    Specify the infrastructure kind for the cluster. This value must match the value for your platform. The following values are valid:
    • AWSCluster: The cluster is running on AWS.
    • GCPCluster: The cluster is running on GCP.
    • VSphereCluster: The cluster is running on vSphere.
    3
    Specify the name of the cluster.
    4
    Specify the details for your environment. These parameters are provider specific. For more information, see the sample Cluster API infrastructure resource YAML for your provider.

  2. Create the infrastructure CR by running the following command: 

    $ oc create -f <infrastructure_resource_file>.yaml

Verification

  • Confirm that the infrastructure CR is created by running the following command: 

    $ oc get <infrastructure_kind>

    where <infrastructure_kind> is the value that corresponds to your platform.

    Example output

    NAME             CLUSTER          READY
<cluster_name>   <cluster_name>   true

    Note

    This output might contain additional columns that are specific to your cloud provider.

Additional resources

13.2.1.4. Creating a Cluster API machine template

You can create a provider-specific machine template resource by creating a YAML manifest file and applying it with the OpenShift CLI (oc).

Prerequisites

  • You have deployed an OpenShift Container Platform cluster.
  • You have enabled the use of the Cluster API.
  • You have access to the cluster using an account with cluster-admin permissions.
  • You have installed the OpenShift CLI (oc).
  • You have created and applied the cluster and infrastructure resources.

Procedure

  1. Create a YAML file similar to the following. This procedure uses <machine_template_resource_file>.yaml as an example file name. 

    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: <machine_template_kind> 1
metadata:
  name: <template_name> 2
  namespace: openshift-cluster-api
spec:
  template:
    spec: 3
    1
    Specify the machine template kind. This value must match the value for your platform. The following values are valid:
    • AWSMachineTemplate: The cluster is running on Amazon Web Services (AWS).
    • GCPMachineTemplate: The cluster is running on Google Cloud Platform (GCP).
    • VSphereMachineTemplate: The cluster is running on VMware vSphere.
    2
    Specify a name for the machine template.
    3
    Specify the details for your environment. These parameters are provider specific. For more information, see the sample Cluster API machine template YAML for your provider.

  2. Create the machine template CR by running the following command: 

    $ oc create -f <machine_template_resource_file>.yaml

Verification

  • Confirm that the machine template CR is created by running the following command: 

    $ oc get <machine_template_kind>

    where <machine_template_kind> is the value that corresponds to your platform.

    Example output

    NAME              AGE
<template_name>   77m

Additional resources

13.2.1.5. Creating a Cluster API compute machine set

You can create compute machine sets that use the Cluster API to dynamically manage the machine compute resources for specific workloads of your choice.

Prerequisites

  • You have deployed an OpenShift Container Platform cluster.
  • You have enabled the use of the Cluster API.
  • You have access to the cluster using an account with cluster-admin permissions.
  • You have installed the OpenShift CLI (oc).
  • You have created the cluster, infrastructure, and machine template resources.

Procedure

  1. Create a YAML file similar to the following. This procedure uses <machine_set_resource_file>.yaml as an example file name. 

    apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineSet
metadata:
  name: <machine_set_name> 1
  namespace: openshift-cluster-api
spec:
  clusterName: <cluster_name> 2
  replicas: 1
  selector:
    matchLabels:
      test: example
  template:
    metadata:
      labels:
        test: example
    spec: 3
# ...
    1
    Specify a name for the compute machine set.
    2
    Specify the name of the cluster.
    3
    Specify the details for your environment. These parameters are provider specific. For more information, see the sample Cluster API compute machine set YAML for your provider.

  2. Create the compute machine set CR by running the following command: 

    $ oc create -f <machine_set_resource_file>.yaml

  3. Confirm that the compute machine set CR is created by running the following command: 

    $ oc get machineset -n openshift-cluster-api 1
    1
    Specify the openshift-cluster-api namespace.

    Example output

    NAME                 CLUSTER          REPLICAS   READY   AVAILABLE   AGE   VERSION
<machine_set_name>   <cluster_name>   1          1       1           17m

    When the new compute machine set is available, the REPLICAS and AVAILABLE values match. If the compute machine set is not available, wait a few minutes and run the command again.

Verification

  • To verify that the compute machine set is creating machines according to your required configuration, review the lists of machines and nodes in the cluster by running the following commands:

    • View the list of Cluster API machines: 

      $ oc get machine -n openshift-cluster-api 1
      1
      Specify the openshift-cluster-api namespace.

      Example output

      NAME                             CLUSTER          NODENAME                                 PROVIDERID      PHASE     AGE     VERSION
<machine_set_name>-<string_id>   <cluster_name>   <ip_address>.<region>.compute.internal   <provider_id>   Running   8m23s

    • View the list of nodes: 

      $ oc get node

      Example output

      NAME                                       STATUS   ROLES    AGE     VERSION
<ip_address_1>.<region>.compute.internal   Ready    worker   5h14m   v1.28.5
<ip_address_2>.<region>.compute.internal   Ready    master   5h19m   v1.28.5
<ip_address_3>.<region>.compute.internal   Ready    worker   7m      v1.28.5

Additional resources

13.3. Managing machines with the Cluster API

Important

Managing machines with the Cluster API is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

13.3.1. Modifying a Cluster API machine template

You can update the machine template resource for your cluster by modifying the YAML manifest file and applying it with the OpenShift CLI (oc).

Prerequisites

  • You have deployed an OpenShift Container Platform cluster that uses the Cluster API.
  • You have access to the cluster using an account with cluster-admin permissions.
  • You have installed the OpenShift CLI (oc).

Procedure

  1. List the machine template resource for your cluster by running the following command: 

    $ oc get <machine_template_kind> 1
    1
    Specify the value that corresponds to your platform. The following values are valid:
    • AWSMachineTemplate: The cluster is running on Amazon Web Services (AWS).
    • GCPMachineTemplate: The cluster is running on Google Cloud Platform (GCP).
    • VSphereMachineTemplate: The cluster is running on VMware vSphere.

    Example output

    NAME              AGE
<template_name>   77m

  2. Write the machine template resource for your cluster to a file that you can edit by running the following command: 

    $ oc get <template_name> -o yaml > <template_name>.yaml

    where <template_name> is the name of the machine template resource for your cluster.

  3. Make a copy of the <template_name>.yaml file with a different name. This procedure uses <modified_template_name>.yaml as an example file name.

  4. Use a text editor to make changes to the <modified_template_name>.yaml file that defines the updated machine template resource for your cluster. When editing the machine template resource, observe the following:

    • The parameters in the spec stanza are provider specific. For more information, see the sample Cluster API machine template YAML for your provider.

    • You must use a value for the metadata.name parameter that differs from any existing values.

      Important

      For any Cluster API compute machine sets that reference this template, you must update the spec.template.spec.infrastructureRef.name parameter to match the metadata.name value in the new machine template resource.

  5. Apply the machine template CR by running the following command: 

    $ oc apply -f <modified_template_name>.yaml 1
    1
    Use the edited YAML file with a new name.

Next steps

  • For any Cluster API compute machine sets that reference this template, update the spec.template.spec.infrastructureRef.name parameter to match the metadata.name value in the new machine template resource. For more information, see "Modifying a compute machine set by using the CLI."

Additional resources

13.3.2. Modifying a compute machine set by using the CLI

You can modify the configuration of a compute machine set, and then propagate the changes to the machines in your cluster by using the CLI.

By updating the compute machine set configuration, you can enable features or change the properties of the machines it creates. When you modify a compute machine set, your changes only apply to compute machines that are created after you save the updated MachineSet custom resource (CR). The changes do not affect existing machines.

Note

Changes made in the underlying cloud provider are not reflected in the Machine or MachineSet CRs. To adjust instance configuration in cluster-managed infrastructure, use the cluster-side resources.

You can replace the existing machines with new ones that reflect the updated configuration by scaling the compute machine set to create twice the number of replicas and then scaling it down to the original number of replicas.

If you need to scale a compute machine set without making other changes, you do not need to delete the machines.

Note

By default, the OpenShift Container Platform router pods are deployed on compute machines. Because the router is required to access some cluster resources, including the web console, do not scale the compute machine set to 0 unless you first relocate the router pods.

The output examples in this procedure use the values for an AWS cluster.

Prerequisites

  • Your OpenShift Container Platform cluster uses the Cluster API.
  • You are logged in to the cluster as an administrator by using the OpenShift CLI (oc).

Procedure

  1. List the compute machine sets in your cluster by running the following command: 

    $ oc get machinesets.cluster.x-k8s.io -n openshift-cluster-api

    Example output

    NAME                          CLUSTER             REPLICAS   READY   AVAILABLE   AGE   VERSION
<compute_machine_set_name_1>  <cluster_name>      1          1       1           26m
<compute_machine_set_name_2>  <cluster_name>      1          1       1           26m

  2. Edit a compute machine set by running the following command: 

    $ oc edit machinesets.cluster.x-k8s.io <machine_set_name> \
  -n openshift-cluster-api

  3. Note the value of the spec.replicas field, because you need it when scaling the machine set to apply the changes. 

    apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineSet
metadata:
  name: <machine_set_name>
  namespace: openshift-cluster-api
spec:
  replicas: 2 1
# ...
    1
    The examples in this procedure show a compute machine set that has a replicas value of 2.
  4. Update the compute machine set CR with the configuration options that you want and save your changes.

  5. List the machines that are managed by the updated compute machine set by running the following command: 

    $ oc get machines.cluster.x-k8s.io \
  -n openshift-cluster-api \
  -l cluster.x-k8s.io/set-name=<machine_set_name>

    Example output for an AWS cluster

    NAME                        CLUSTER          NODENAME                                    PROVIDERID                              PHASE           AGE     VERSION
<machine_name_original_1>   <cluster_name>   <original_1_ip>.<region>.compute.internal   aws:///us-east-2a/i-04e7b2cbd61fd2075   Running         4h
<machine_name_original_2>   <cluster_name>   <original_2_ip>.<region>.compute.internal   aws:///us-east-2a/i-04e7b2cbd61fd2075   Running         4h

  6. For each machine that is managed by the updated compute machine set, set the delete annotation by running the following command: 

    $ oc annotate machines.cluster.x-k8s.io/<machine_name_original_1> \
  -n openshift-cluster-api \
  cluster.x-k8s.io/delete-machine="true"

  7. To create replacement machines with the new configuration, scale the compute machine set to twice the number of replicas by running the following command: 

    $ oc scale --replicas=4 \1
  machinesets.cluster.x-k8s.io <machine_set_name> \
  -n openshift-cluster-api
    1
    The original example value of 2 is doubled to 4.

  8. List the machines that are managed by the updated compute machine set by running the following command: 

    $ oc get machines.cluster.x-k8s.io \
  -n openshift-cluster-api \
  -l cluster.x-k8s.io/set-name=<machine_set_name>

    Example output for an AWS cluster

    NAME                        CLUSTER          NODENAME                                    PROVIDERID                              PHASE           AGE     VERSION
<machine_name_original_1>   <cluster_name>   <original_1_ip>.<region>.compute.internal   aws:///us-east-2a/i-04e7b2cbd61fd2075   Running         4h
<machine_name_original_2>   <cluster_name>   <original_2_ip>.<region>.compute.internal   aws:///us-east-2a/i-04e7b2cbd61fd2075   Running         4h
<machine_name_updated_1>    <cluster_name>   <updated_1_ip>.<region>.compute.internal    aws:///us-east-2a/i-04e7b2cbd61fd2075   Provisioned     55s
<machine_name_updated_2>    <cluster_name>   <updated_2_ip>.<region>.compute.internal    aws:///us-east-2a/i-04e7b2cbd61fd2075   Provisioning    55s

    When the new machines are in the Running phase, you can scale the compute machine set to the original number of replicas.

  9. To remove the machines that were created with the old configuration, scale the compute machine set to the original number of replicas by running the following command: 

    $ oc scale --replicas=2 \1
  machinesets.cluster.x-k8s.io <machine_set_name> \
  -n openshift-cluster-api
    1
    The original example value of 2.

Verification

  • To verify that a machine created by the updated machine set has the correct configuration, examine the relevant fields in the CR for one of the new machines by running the following command: 

    $ oc describe machines.cluster.x-k8s.io <machine_name_updated_1> \
  -n openshift-cluster-api

  • To verify that the compute machines without the updated configuration are deleted, list the machines that are managed by the updated compute machine set by running the following command: 

    $ oc get machines.cluster.x-k8s.io \
  -n openshift-cluster-api \
  cluster.x-k8s.io/set-name=<machine_set_name>

    Example output while deletion is in progress for an AWS cluster

    NAME                        CLUSTER          NODENAME                                    PROVIDERID                              PHASE      AGE     VERSION
<machine_name_original_1>   <cluster_name>   <original_1_ip>.<region>.compute.internal   aws:///us-east-2a/i-04e7b2cbd61fd2075   Running    18m
<machine_name_original_2>   <cluster_name>   <original_2_ip>.<region>.compute.internal   aws:///us-east-2a/i-04e7b2cbd61fd2075   Running    18m
<machine_name_updated_1>    <cluster_name>   <updated_1_ip>.<region>.compute.internal    aws:///us-east-2a/i-04e7b2cbd61fd2075   Running    18m
<machine_name_updated_2>    <cluster_name>   <updated_2_ip>.<region>.compute.internal    aws:///us-east-2a/i-04e7b2cbd61fd2075   Running    18m

    Example output when deletion is complete for an AWS cluster

    NAME                        CLUSTER          NODENAME                                    PROVIDERID                              PHASE      AGE     VERSION
<machine_name_updated_1>    <cluster_name>   <updated_1_ip>.<region>.compute.internal    aws:///us-east-2a/i-04e7b2cbd61fd2075   Running    18m
<machine_name_updated_2>    <cluster_name>   <updated_2_ip>.<region>.compute.internal    aws:///us-east-2a/i-04e7b2cbd61fd2075   Running    18m

Additional resources

13.4. Cluster API configuration

Important

Managing machines with the Cluster API is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

The following example YAML files show how to make the Cluster API primary resources work together and configure settings for the machines that they create that are appropriate for your environment.

13.4.1. Sample YAML for a Cluster API cluster resource

The cluster resource defines the name and infrastructure provider for the cluster and is managed by the Cluster API. This resource has the same structure for all providers. 

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: <cluster_name> 1
  namespace: openshift-cluster-api
spec:
  controlPlaneEndpoint: 2
    host: <control_plane_endpoint_address>
    port: 6443
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: <infrastructure_kind> 3
    name: <cluster_name>
    namespace: openshift-cluster-api
1
Specify the name of the cluster.
2
Specify the IP address of the control plane endpoint and the port used to access it.
3
Specify the infrastructure kind for the cluster. Valid values are:
  • AWSCluster: The cluster is running on Amazon Web Services.
  • GCPCluster: The cluster is running on Google Cloud Platform.
  • VSphereCluster: The cluster is running on VMware vSphere.

13.4.2. Provider-specific configuration options

The remaining Cluster API resources are provider-specific. For provider-specific configuration options for your cluster, see the following resources:

13.5. Configuration options for Cluster API machines

13.5.1. Cluster API configuration options for Amazon Web Services

Important

Managing machines with the Cluster API is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

You can change the configuration of your Amazon Web Services (AWS) Cluster API machines by updating values in the Cluster API custom resource manifests.

13.5.1.1. Sample YAML for configuring Amazon Web Services clusters

The following example YAML files show configurations for an Amazon Web Services cluster.

13.5.1.1.1. Sample YAML for a Cluster API infrastructure resource on Amazon Web Services

The infrastructure resource is provider-specific and defines properties that are shared by all the compute machine sets in the cluster, such as the region and subnets. The compute machine set references this resource when creating machines. 

apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: AWSCluster 1
metadata:
  name: <cluster_name> 2
  namespace: openshift-cluster-api
spec:
  controlPlaneEndpoint: 3
    host: <control_plane_endpoint_address>
    port: 6443
  region: <region> 4
1
Specify the infrastructure kind for the cluster. This value must match the value for your platform.
2
Specify the cluster ID as the name of the cluster.
3
Specify the address of the control plane endpoint and the port to use to access it.
4
Specify the AWS region.
13.5.1.1.2. Sample YAML for a Cluster API machine template resource on Amazon Web Services

The machine template resource is provider-specific and defines the basic properties of the machines that a compute machine set creates. The compute machine set references this template when creating machines. 

apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: AWSMachineTemplate 1
metadata:
  name: <template_name> 2
  namespace: openshift-cluster-api
spec:
  template:
    spec: 3
      uncompressedUserData: true
      iamInstanceProfile: # ...
      instanceType: m5.large
      ignition:
        storageType: UnencryptedUserData
        version: "3.2"
      ami:
        id: # ...
      subnet:
        filters:
        - name: tag:Name
          values:
          - # ...
      additionalSecurityGroups:
      - filters:
        - name: tag:Name
          values:
          - # ...
1
Specify the machine template kind. This value must match the value for your platform.
2
Specify a name for the machine template.
3
Specify the details for your environment. The values here are examples.
13.5.1.1.3. Sample YAML for a Cluster API compute machine set resource on Amazon Web Services

The compute machine set resource defines additional properties of the machines that it creates. The compute machine set also references the infrastructure resource and machine template when creating machines. 

apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineSet
metadata:
  name: <machine_set_name> 1
  namespace: openshift-cluster-api
spec:
  clusterName: <cluster_name> 2
  replicas: 1
  selector:
    matchLabels:
      test: example
  template:
    metadata:
      labels:
        test: example
    spec:
      bootstrap:
         dataSecretName: worker-user-data 3
      clusterName: <cluster_name>
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
        kind: AWSMachineTemplate 4
        name: <template_name> 5
1
Specify a name for the compute machine set.
2
Specify the cluster ID as the name of the cluster.
3
For the Cluster API Technology Preview, the Operator can use the worker user data secret from the openshift-machine-api namespace.
4
Specify the machine template kind. This value must match the value for your platform.
5
Specify the machine template name.

13.5.2. Cluster API configuration options for Google Cloud Platform

Important

Managing machines with the Cluster API is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

You can change the configuration of your Google Cloud Platform (GCP) Cluster API machines by updating values in the Cluster API custom resource manifests.

13.5.2.1. Sample YAML for configuring Google Cloud Platform clusters

The following example YAML files show configurations for a Google Cloud Platform cluster.

13.5.2.1.1. Sample YAML for a Cluster API infrastructure resource on Google Cloud Platform

The infrastructure resource is provider-specific and defines properties that are shared by all the compute machine sets in the cluster, such as the region and subnets. The compute machine set references this resource when creating machines. 

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: GCPCluster 1
metadata:
  name: <cluster_name> 2
spec:
  controlPlaneEndpoint: 3
    host: <control_plane_endpoint_address>
    port: 6443
  network:
    name: <cluster_name>-network
  project: <project> 4
  region: <region> 5
1
Specify the infrastructure kind for the cluster. This value must match the value for your platform.
2
Specify the cluster ID as the name of the cluster.
3
Specify the IP address of the control plane endpoint and the port used to access it.
4
Specify the GCP project name.
5
Specify the GCP region.
13.5.2.1.2. Sample YAML for a Cluster API machine template resource on Google Cloud Platform

The machine template resource is provider-specific and defines the basic properties of the machines that a compute machine set creates. The compute machine set references this template when creating machines. 

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: GCPMachineTemplate 1
metadata:
  name: <template_name> 2
  namespace: openshift-cluster-api
spec:
  template:
    spec: 3
      rootDeviceType: pd-ssd
      rootDeviceSize: 128
      instanceType: n1-standard-4
      image: projects/rhcos-cloud/global/images/rhcos-411-85-202203181601-0-gcp-x86-64
      subnet: <cluster_name>-worker-subnet
      serviceAccounts:
        email: <service_account_email_address>
        scopes:
          - https://www.googleapis.com/auth/cloud-platform
      additionalLabels:
        kubernetes-io-cluster-<cluster_name>: owned
      additionalNetworkTags:
        - <cluster_name>-worker
      ipForwarding: Disabled
1
Specify the machine template kind. This value must match the value for your platform.
2
Specify a name for the machine template.
3
Specify the details for your environment. The values here are examples.
13.5.2.1.3. Sample YAML for a Cluster API compute machine set resource on Google Cloud Platform

The compute machine set resource defines additional properties of the machines that it creates. The compute machine set also references the infrastructure resource and machine template when creating machines. 

apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineSet
metadata:
  name: <machine_set_name> 1
  namespace: openshift-cluster-api
spec:
  clusterName: <cluster_name> 2
  replicas: 1
  selector:
    matchLabels:
      test: example
  template:
    metadata:
      labels:
        test: example
    spec:
      bootstrap:
         dataSecretName: worker-user-data 3
      clusterName: <cluster_name>
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
        kind: GCPMachineTemplate 4
        name: <template_name> 5
      failureDomain: <failure_domain> 6
1
Specify a name for the compute machine set.
2
Specify the cluster ID as the name of the cluster.
3
For the Cluster API Technology Preview, the Operator can use the worker user data secret from the openshift-machine-api namespace.
4
Specify the machine template kind. This value must match the value for your platform.
5
Specify the machine template name.
6
Specify the failure domain within the GCP region.

13.5.3. Cluster API configuration options for VMware vSphere

Important

Managing machines with the Cluster API is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

You can change the configuration of your VMware vSphere Cluster API machines by updating values in the Cluster API custom resource manifests.

13.5.3.1. Sample YAML for configuring VMware vSphere clusters

The following example YAML files show configurations for a VMware vSphere cluster.

13.5.3.1.1. Sample YAML for a Cluster API infrastructure resource on VMware vSphere

The infrastructure resource is provider-specific and defines properties that are shared by all the compute machine sets in the cluster, such as the region and subnets. The compute machine set references this resource when creating machines. 

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: VSphereCluster 1
metadata:
  name: <cluster_name> 2
spec:
  controlPlaneEndpoint: 3
    host: <control_plane_endpoint_address>
    port: 6443
  identityRef:
    kind: Secret
    name: <cluster_name>
  server: <vsphere_server> 4
1
Specify the infrastructure kind for the cluster. This value must match the value for your platform.
2
Specify the cluster ID as the name of the cluster.
3
Specify the IP address of the control plane endpoint and the port used to access it.
4
Specify the vSphere server for the cluster. You can find this value on an existing vSphere cluster by running the following command: 
$ oc get infrastructure cluster \
  -o jsonpath="{.spec.platformSpec.vsphere.vcenters[0].server}"
13.5.3.1.2. Sample YAML for a Cluster API machine template resource on VMware vSphere

The machine template resource is provider-specific and defines the basic properties of the machines that a compute machine set creates. The compute machine set references this template when creating machines. 

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: VSphereMachineTemplate 1
metadata:
  name: <template_name> 2
  namespace: openshift-cluster-api
spec:
  template:
    spec: 3
      template: <vm_template_name> 4
      server: <vcenter_server_ip> 5
      diskGiB: 128
      cloneMode: linkedClone 6
      datacenter: <vcenter_data_center_name> 7
      datastore: <vcenter_datastore_name> 8
      folder: <vcenter_vm_folder_path> 9
      resourcePool: <vsphere_resource_pool> 10
      numCPUs: 4
      memoryMiB: 16384
      network:
        devices:
        - dhcp4: true
          networkName: "<vm_network_name>" 11
1
Specify the machine template kind. This value must match the value for your platform.
2
Specify a name for the machine template.
3
Specify the details for your environment. The values here are examples.
4
Specify the vSphere VM template to use, such as user-5ddjd-rhcos.
5
Specify the vCenter server IP or fully qualified domain name.
6
Specify the type of VM clone to use. The following values are valid:
  • fullClone
  • linkedClone

When using the linkedClone type, the disk size matches the clone source instead of using the diskGiB value. For more information, see the vSphere documentation about VM clone types.

7
Specify the vCenter data center to deploy the compute machine set on.
8
Specify the vCenter datastore to deploy the compute machine set on.
9
Specify the path to the vSphere VM folder in vCenter, such as /dc1/vm/user-inst-5ddjd.
10
Specify the vSphere resource pool for your VMs.
11
Specify the vSphere VM network to deploy the compute machine set to. This VM network must be where other compute machines reside in the cluster.
13.5.3.1.3. Sample YAML for a Cluster API compute machine set resource on VMware vSphere

The compute machine set resource defines additional properties of the machines that it creates. The compute machine set also references the infrastructure resource and machine template when creating machines. 

apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineSet
metadata:
  name: <machine_set_name> 1
  namespace: openshift-cluster-api
spec:
  clusterName: <cluster_name> 2
  replicas: 1
  selector:
    matchLabels:
      test: example
  template:
    metadata:
      labels:
        test: example
    spec:
      bootstrap:
         dataSecretName: worker-user-data 3
      clusterName: <cluster_name>
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
        kind: VSphereMachineTemplate 4
        name: <template_name> 5
      failureDomain: 6
        - name: <failure_domain_name>
          region: <region_a>
          zone: <zone_a>
          server: <vcenter_server_name>
          topology:
            datacenter: <region_a_data_center>
            computeCluster: "</region_a_data_center/host/zone_a_cluster>"
            resourcePool: "</region_a_data_center/host/zone_a_cluster/Resources/resource_pool>"
            datastore: "</region_a_data_center/datastore/datastore_a>"
            networks:
            - port-group
1
Specify a name for the compute machine set.
2
Specify the cluster ID as the name of the cluster.
3
For the Cluster API Technology Preview, the Operator can use the worker user data secret from the openshift-machine-api namespace.
4
Specify the machine template kind. This value must match the value for your platform.
5
Specify the machine template name.
6
Specify the failure domain configuration details.
Note

Using multiple regions and zones on a vSphere cluster that uses the Cluster API is not a validated configuration.

13.6. Troubleshooting clusters that use the Cluster API

Important

Managing machines with the Cluster API is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Use the information in this section to understand and recover from issues you might encounter. Generally, troubleshooting steps for problems with the Cluster API are similar to those steps for problems with the Machine API.

The Cluster CAPI Operator and its operands are provisioned in the openshift-cluster-api namespace, whereas the Machine API uses the openshift-machine-api namespace. When using oc commands that reference a namespace, be sure to reference the correct one.

13.6.1. Referencing the intended objects when using the CLI

For clusters that use the Cluster API, OpenShift CLI (oc) commands prioritize Cluster API objects over Machine API objects.

This behavior impacts any oc command that acts upon any object that is represented in both the Cluster API and the Machine API. This explanation uses the oc delete machine command, which deletes a machine, as an example.

Cause

When you run an oc command, oc communicates with the Kube API server to determine which objects to act upon. The Kube API server uses the first installed custom resource definition (CRD) it encounters alphabetically when an oc command is run.

CRDs for Cluster API objects are in the cluster.x-k8s.io group, while CRDs for Machine API objects are in the machine.openshift.io group. Because the letter c precedes the letter m alphabetically, the Kube API server matches on the Cluster API object CRD. As a result, the oc command acts upon Cluster API objects.

Consequences

Due to this behavior, the following unintended outcomes can occur on a cluster that uses the Cluster API:

  • For namespaces that contain both types of objects, commands such as oc get machine return only Cluster API objects.
  • For namespaces that contain only Machine API objects, commands such as oc get machine return no results.
Workaround

You can ensure that oc commands act on the type of objects you intend by using the corresponding fully qualified name.

Prerequisites

  • You have access to the cluster using an account with cluster-admin permissions.
  • You have installed the OpenShift CLI (oc).

Procedure

  • To delete a Machine API machine, use the fully qualified name machine.machine.openshift.io when running the oc delete machine command: 

    $ oc delete machine.machine.openshift.io <machine_name>

  • To delete a Cluster API machine, use the fully qualified name machine.cluster.x-k8s.io when running the oc delete machine command: 

    $ oc delete machine.cluster.x-k8s.io <machine_name>
Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

© 2024 Red Hat, Inc.