Chapter 5. Manage compute nodes using Red Hat build of Karpenter

Procedure

1. Download the latest Red Hat build of Karpenter IAM policy:

   $ curl -o autonode-policy.json https://raw.githubusercontent.com/openshift/managed-cluster-config/refs/heads/master/resources/sts/hypershift/openshift_hcp_karpenter_controller_credentials_policy.json

2. Export your cluster name and ID to environment variables:

   $ export CLUSTER_NAME=<cluster_name>
   $ export CLUSTER_ID=$(rosa describe cluster -c "$CLUSTER_NAME" -o json | jq -r '.id')
   $ echo $CLUSTER_NAME $CLUSTER_ID

3. Create the IAM policy:

   $ POLICY_ARN=$(aws iam create-policy \
     --policy-name rosa-karpenter-controller-role-${CLUSTER_NAME} \
     --policy-document file://autonode-policy.json \
     --query 'Policy.Arn' \
     --output text)

4. Create the trust policy for the Karpenter service account:

   $ cat > trust-policy.json <<EOF
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Principal": {
                   "Federated": "arn:aws:iam::$(aws sts get-caller-identity --query Account --output text):oidc-provider/$(rosa describe cluster -c $CLUSTER_NAME -o json | jq -r .aws.sts.oidc_endpoint_url | sed 's|https://||')"
               },
               "Action": "sts:AssumeRoleWithWebIdentity",
               "Condition": {
                   "StringEquals": {
                       "$(rosa describe cluster -c $CLUSTER_NAME -o json | jq -r .aws.sts.oidc_endpoint_url | sed 's|https://||'):sub": "system:serviceaccount:kube-system:karpenter"
                   }
               }
           }
       ]
   }
   EOF

5. Create the IAM role and attach the policy:

   $ ROLE_ARN=$(aws iam create-role \
     --role-name rosa-karpenter-controller-role-${CLUSTER_NAME} \
     --assume-role-policy-document file://trust-policy.json \
     --query 'Role.Arn' \
     --output text)
   
   $ aws iam attach-role-policy \
     --role-name rosa-karpenter-controller-role-${CLUSTER_NAME} \
     --policy-arn $POLICY_ARN

   Verification

   Verify that policies are present by running:

Procedure

1. Export your cluster name and ID to environment variables:

   $ export CLUSTER_NAME=<cluster_name>
   $ export AWS_REGION=us-east-2
   $ export CLUSTER_ID=$(rosa describe cluster -c "$CLUSTER_NAME" -o json | jq -r '.id')
   $ echo $CLUSTER_NAME $CLUSTER_ID

   Important

   Ensure that your AWS client is using the region where your cluster is deployed. In this example, us-east-2 is used.

2. Find your cluster’s security group by running the following command:

   $ SECURITY_GROUP_ID=$(aws ec2 describe-security-groups --filters "Name=tag:Name,Values=$CLUSTER_ID-default-sg" | jq -r .SecurityGroups[0].GroupId)

3. Tag the security group with the Red Hat build of Karpenter auto-discovery tags by running the following command:

   $ aws ec2 create-tags --resources "$SECURITY_GROUP_ID" --tags Key="karpenter.sh/discovery",Value="$CLUSTER_ID"

Procedure

1. Export your cluster name and ID to environment variables:

   $ export CLUSTER_NAME=<cluster_name>
   $ export CLUSTER_ID=$(rosa describe cluster -c "$CLUSTER_NAME" -o json | jq -r '.id')
   $ echo $CLUSTER_NAME $CLUSTER_ID

2. Wait for the cluster to become ready:

   $ rosa describe cluster -c $CLUSTER_ID | grep -i State

   Example output

   State:                      ready

3. Ensure that your Red Hat build of Karpenter IAM role is correctly set:

   $ ROLE_ARN=$(aws iam get-role --role-name rosa-karpenter-controller-role-${CLUSTER_NAME} --query 'Role.Arn' --output text)

4. Enable Red Hat build of Karpenter:

   $ rosa edit cluster -c $CLUSTER_ID \
     --autonode=enabled \
     --autonode-iam-role-arn=$ROLE_ARN

5. If you do not already have cluster admin access, create a cluster admin user:

   $ rosa create admin -c $CLUSTER_ID

6. Log in to the cluster using the credentials from the previous command:

   $ oc login <api_url> --username cluster-admin --password <password>

7. Verify that the Red Hat build of Karpenter custom resource definitions (CRDs) are present:

   $ oc get ec2nodeclass

   Note

   The node pool manifest uses the EC2NodeClass resource.

   Example output

   NAME      READY   AGE
   default   True    5m

   $ oc get openshiftec2nodeclass

   Note

   The OpenShiftEC2NodeClass resource is Red Hat’s wrapper to communicate with the EC2NodeClass resource.

   Example output

   NAME      READY
   default   True

Procedure

1. Export your AWS ID:

   $ export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)

2. Log in to the AWS console.
3. In the AWS console, navigate to IAM > Roles.

4. On your Red Hat build of Karpenter Amazon Resource Name (ARN), update the trust policy to include the following policy specifications:

   Note

   To access this ARN, run:

   $ echo $ROLE_ARN

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Principal": {
                   "Federated": "arn:aws:iam::<aws_account_id>:oidc-provider/<oidc-endpoint-url>"
               },
               "Action": "sts:AssumeRoleWithWebIdentity",
               "Condition": {
                   "StringEquals": {
                       "<oidc-endpoint-url>:sub": "system:serviceaccount:kube-system:karpenter"
                   }
               }
           }
       ]
   }

   where:

   <aws_account_id>

   Specifies your AWS Account ID.

   <oidc-endpoint-url>

   Specifies the OIDC endpoint URL that you acquired.

5. In OpenShift Cluster Manager, select your cluster from the cluster list.
6. On the cluster details screen, select the Edit button next to the status for Red Hat build of Karpenter.
7. On the Edit Autonode settings dialog box, toggle Enable Autonode.
8. Add your Red Hat build of Karpenter IAM role ARN to the field in this dialog box.
9. Select Save to save your configurations and close the Edit Autonode settings box.

Procedure

1. Create a node pool manifest:

   $ cat > nodepool.yaml <<'EOF'
   apiVersion: karpenter.sh/v1
   kind: NodePool
   metadata:
     name: default-np
   spec:
     template:
       metadata:
         labels:
           autonode: "true"
       spec:
         requirements:
         - key: node.kubernetes.io/instance-type
           operator: In
           values:
           - c5.xlarge
         - key: karpenter.sh/capacity-type
           operator: In
           values: ["on-demand"]
         nodeClassRef:
           group: karpenter.k8s.aws
           kind: EC2NodeClass
           name: default
   EOF

   where:

   nodeClassRef.kind

   Required field that must use the EC2NodeClass type.

   spec.labels

   Optional field that you can use to place pods by using labels.

   Note

   For a list of all requirements available under spec.requirements, see the Additional resources.

2. Apply the node pool:

   $ oc apply -f nodepool.yaml

3. Verify the node pool is ready:

   $ oc get nodepool

   Example output

   NAME         NODECLASS   NODES   READY   AGE
   default-np   default     0       True    3s