Chapter 13. Expanding single-node OpenShift clusters with GitOps ZTP

Procedure

1. Check the daemon node selector settings of the PTP Operator on one of the spoke clusters:

   $ oc get ptpoperatorconfig/default -n openshift-ptp -ojsonpath='{.spec}' | jq

   Example output for PTP Operator

   {"daemonNodeSelector":{"node-role.kubernetes.io/master":""}} 1

   1

   If the node selector is set to master, the spoke was deployed with the version of the GitOps ZTP plugin that requires changes.

2. Check the daemon node selector settings of the SR-IOV Operator on one of the spoke clusters:

   $  oc get sriovoperatorconfig/default -n \
   openshift-sriov-network-operator -ojsonpath='{.spec}' | jq

   Example output for SR-IOV Operator

   {"configDaemonNodeSelector":{"node-role.kubernetes.io/worker":""},"disableDrain":false,"enableInjector":true,"enableOperatorWebhook":true} 1

   1

   If the node selector is set to master, the spoke was deployed with the version of the GitOps ZTP plugin that requires changes.

3. In the group policy, add the following complianceType and spec entries:

   spec:
       - fileName: PtpOperatorConfig.yaml
         policyName: "config-policy"
         complianceType: mustonlyhave
         spec:
           daemonNodeSelector:
             node-role.kubernetes.io/worker: ""
       - fileName: SriovOperatorConfig.yaml
         policyName: "config-policy"
         complianceType: mustonlyhave
         spec:
           configDaemonNodeSelector:
             node-role.kubernetes.io/worker: ""

   Important

   Changing the daemonNodeSelector field causes temporary PTP synchronization loss and SR-IOV connectivity loss.

4. Commit the changes in Git, and then push to the Git repository being monitored by the GitOps ZTP ArgoCD application.

Procedure

1. Create the following PolicyGenerator CR:

   apiVersion: policy.open-cluster-management.io/v1
   kind: PolicyGenerator
   metadata:
       name: example-sno-workers
   placementBindingDefaults:
       name: example-sno-workers-placement-binding
   policyDefaults:
       namespace: example-sno
       placement:
           labelSelector:
               matchExpressions:
                   - key: sites
                     operator: In
                     values:
                       - example-sno 1
       remediationAction: inform
       severity: low
       namespaceSelector:
           exclude:
               - kube-*
           include:
               - '*'
       evaluationInterval:
           compliant: 10m
           noncompliant: 10s
   policies:
       - name: example-sno-workers-config-policy
         policyAnnotations:
           ran.openshift.io/ztp-deploy-wave: "10"
         manifests:
           - path: source-crs/MachineConfigGeneric.yaml 2
             patches:
               - metadata:
                   labels:
                       machineconfiguration.openshift.io/role: worker 3
                   name: enable-workload-partitioning
                 spec:
                   config:
                       storage:
                           files:
                               - contents:
                                   source: data:text/plain;charset=utf-8;base64,W2NyaW8ucnVudGltZS53b3JrbG9hZHMubWFuYWdlbWVudF0KYWN0aXZhdGlvbl9hbm5vdGF0aW9uID0gInRhcmdldC53b3JrbG9hZC5vcGVuc2hpZnQuaW8vbWFuYWdlbWVudCIKYW5ub3RhdGlvbl9wcmVmaXggPSAicmVzb3VyY2VzLndvcmtsb2FkLm9wZW5zaGlmdC5pbyIKcmVzb3VyY2VzID0geyAiY3B1c2hhcmVzIiA9IDAsICJjcHVzZXQiID0gIjAtMyIgfQo=
                                 mode: 420
                                 overwrite: true
                                 path: /etc/crio/crio.conf.d/01-workload-partitioning
                                 user:
                                   name: root
                               - contents:
                                   source: data:text/plain;charset=utf-8;base64,ewogICJtYW5hZ2VtZW50IjogewogICAgImNwdXNldCI6ICIwLTMiCiAgfQp9Cg==
                                 mode: 420
                                 overwrite: true
                                 path: /etc/kubernetes/openshift-workload-pinning
                                 user:
                                   name: root
           - path: source-crs/PerformanceProfile-MCP-worker.yaml
             patches:
               - metadata:
                   name: openshift-worker-node-performance-profile
                 spec:
                   cpu: 4
                       isolated: 4-47
                       reserved: 0-3
                   hugepages:
                       defaultHugepagesSize: 1G
                       pages:
                           - count: 32
                             size: 1G
                   realTimeKernel:
                       enabled: true
           - path: source-crs/TunedPerformancePatch-MCP-worker.yaml
             patches:
               - metadata:
                   name: performance-patch-worker
                 spec:
                   profile:
                       - data: |
                         [main]
                         summary=Configuration changes profile inherited from performance created tuned
                         include=openshift-node-performance-openshift-worker-node-performance-profile
                         [bootloader]
                         cmdline_crash=nohz_full=4-47 5
                         [sysctl]
                         kernel.timer_migration=1
                         [scheduler]
                         group.ice-ptp=0:f:10:*:ice-ptp.*
                         [service]
                         service.stalld=start,enable
                         service.chronyd=stop,disable
                         name: performance-patch-worker
                   recommend:
                       - profile: performance-patch-worker

   1

   The policies are applied to all clusters with this label.

   2

   This generic MachineConfig CR is used to configure workload partitioning on the worker node.

   3

   The MCP field must be set to worker.

   4

   The cpu.isolated and cpu.reserved fields must be configured for each particular hardware platform.

   5

   The cmdline_crash CPU set must match the cpu.isolated set in the PerformanceProfile section.

   A generic MachineConfig CR is used to configure workload partitioning on the worker node. You can generate the content of crio and kubelet configuration files.

2. Add the created policy template to the Git repository monitored by the ArgoCD policies application.
3. Add the policy in the kustomization.yaml file.
4. Commit the changes in Git, and then push to the Git repository being monitored by the GitOps ZTP ArgoCD application.

5. To remediate the new policies to your spoke cluster, create a TALM custom resource:

   $ cat <<EOF | oc apply -f -
   apiVersion: ran.openshift.io/v1alpha1
   kind: ClusterGroupUpgrade
   metadata:
     name: example-sno-worker-policies
     namespace: default
   spec:
     backup: false
     clusters:
     - example-sno
     enable: true
     managedPolicies:
     - group-du-sno-config-policy
     - example-sno-workers-config-policy
     - example-sno-config-policy
     preCaching: false
     remediationStrategy:
       maxConcurrency: 1
   EOF

Procedure

1. Create the following PolicyGenTemplate CR:

   apiVersion: ran.openshift.io/v1
   kind: PolicyGenTemplate
   metadata:
     name: "example-sno-workers"
     namespace: "example-sno"
   spec:
     bindingRules:
       sites: "example-sno" 1
     mcp: "worker" 2
     sourceFiles:
       - fileName: MachineConfigGeneric.yaml 3
         policyName: "config-policy"
         metadata:
           labels:
             machineconfiguration.openshift.io/role: worker
           name: enable-workload-partitioning
         spec:
           config:
             storage:
               files:
               - contents:
                   source: data:text/plain;charset=utf-8;base64,W2NyaW8ucnVudGltZS53b3JrbG9hZHMubWFuYWdlbWVudF0KYWN0aXZhdGlvbl9hbm5vdGF0aW9uID0gInRhcmdldC53b3JrbG9hZC5vcGVuc2hpZnQuaW8vbWFuYWdlbWVudCIKYW5ub3RhdGlvbl9wcmVmaXggPSAicmVzb3VyY2VzLndvcmtsb2FkLm9wZW5zaGlmdC5pbyIKcmVzb3VyY2VzID0geyAiY3B1c2hhcmVzIiA9IDAsICJjcHVzZXQiID0gIjAtMyIgfQo=
                 mode: 420
                 overwrite: true
                 path: /etc/crio/crio.conf.d/01-workload-partitioning
                 user:
                   name: root
               - contents:
                   source: data:text/plain;charset=utf-8;base64,ewogICJtYW5hZ2VtZW50IjogewogICAgImNwdXNldCI6ICIwLTMiCiAgfQp9Cg==
                 mode: 420
                 overwrite: true
                 path: /etc/kubernetes/openshift-workload-pinning
                 user:
                   name: root
       - fileName: PerformanceProfile.yaml
         policyName: "config-policy"
         metadata:
           name: openshift-worker-node-performance-profile
         spec:
           cpu: 4
             isolated: "4-47"
             reserved: "0-3"
           hugepages:
             defaultHugepagesSize: 1G
             pages:
               - size: 1G
                 count: 32
           realTimeKernel:
             enabled: true
       - fileName: TunedPerformancePatch.yaml
         policyName: "config-policy"
         metadata:
           name: performance-patch-worker
         spec:
           profile:
             - name: performance-patch-worker
               data: |
                 [main]
                 summary=Configuration changes profile inherited from performance created tuned
                 include=openshift-node-performance-openshift-worker-node-performance-profile
                 [bootloader]
                 cmdline_crash=nohz_full=4-47 5
                 [sysctl]
                 kernel.timer_migration=1
                 [scheduler]
                 group.ice-ptp=0:f:10:*:ice-ptp.*
                 [service]
                 service.stalld=start,enable
                 service.chronyd=stop,disable
           recommend:
           - profile: performance-patch-worker

   1

   The policies are applied to all clusters with this label.

   2

   The MCP field must be set to worker.

   3

   This generic MachineConfig CR is used to configure workload partitioning on the worker node.

   4

   The cpu.isolated and cpu.reserved fields must be configured for each particular hardware platform.

   5

   The cmdline_crash CPU set must match the cpu.isolated set in the PerformanceProfile section.

   A generic MachineConfig CR is used to configure workload partitioning on the worker node. You can generate the content of crio and kubelet configuration files.

2. Add the created policy template to the Git repository monitored by the ArgoCD policies application.
3. Add the policy in the kustomization.yaml file.
4. Commit the changes in Git, and then push to the Git repository being monitored by the GitOps ZTP ArgoCD application.

5. To remediate the new policies to your spoke cluster, create a TALM custom resource:

   $ cat <<EOF | oc apply -f -
   apiVersion: ran.openshift.io/v1alpha1
   kind: ClusterGroupUpgrade
   metadata:
     name: example-sno-worker-policies
     namespace: default
   spec:
     backup: false
     clusters:
     - example-sno
     enable: true
     managedPolicies:
     - group-du-sno-config-policy
     - example-sno-workers-config-policy
     - example-sno-config-policy
     preCaching: false
     remediationStrategy:
       maxConcurrency: 1
   EOF

Procedure

1. If you deployed your cluster by using the example-sno.yaml SiteConfig manifest, add your new worker node to the spec.clusters['example-sno'].nodes list:

   nodes:
   - hostName: "example-node2.example.com"
     role: "worker"
     bmcAddress: "idrac-virtualmedia+https://[1111:2222:3333:4444::bbbb:1]/redfish/v1/Systems/System.Embedded.1"
     bmcCredentialsName:
       name: "example-node2-bmh-secret"
     bootMACAddress: "AA:BB:CC:DD:EE:11"
     bootMode: "UEFI"
     nodeNetwork:
       interfaces:
         - name: eno1
           macAddress: "AA:BB:CC:DD:EE:11"
       config:
         interfaces:
           - name: eno1
             type: ethernet
             state: up
             macAddress: "AA:BB:CC:DD:EE:11"
             ipv4:
               enabled: false
             ipv6:
               enabled: true
               address:
               - ip: 1111:2222:3333:4444::1
                 prefix-length: 64
         dns-resolver:
           config:
             search:
             - example.com
             server:
             - 1111:2222:3333:4444::2
         routes:
           config:
           - destination: ::/0
             next-hop-interface: eno1
             next-hop-address: 1111:2222:3333:4444::1
             table-id: 254

2. Create a BMC authentication secret for the new host, as referenced by the bmcCredentialsName field in the spec.nodes section of your SiteConfig file:

   apiVersion: v1
   data:
     password: "password"
     username: "username"
   kind: Secret
   metadata:
     name: "example-node2-bmh-secret"
     namespace: example-sno
   type: Opaque

3. Commit the changes in Git, and then push to the Git repository that is being monitored by the GitOps ZTP ArgoCD application.

   When the ArgoCD cluster application synchronizes, two new manifests appear on the hub cluster generated by the GitOps ZTP plugin:

   • BareMetalHost

   • NMStateConfig

     Important

     The cpuset field should not be configured for the worker node. Workload partitioning for worker nodes is added through management policies after the node installation is complete.