Chapter 6. Migrating from SiteConfig CRs to ClusterInstance CRs

1. Set up the parallel pipeline by preparing a new Git folder structure in your repository and creating the corresponding Argo CD project and application.

2. To migrate the clusters incrementally, first remove the associated SiteConfig CR from the old pipeline. Then, add a corresponding ClusterInstance CR to the new pipeline.

   Note

   By using the prune=false sync policy in the initial Argo CD application, the resources managed by this pipeline remain intact even after you remove the target cluster from this application. This approach ensures that the existing cluster resources remain operational during the migration process.

   1. Optionally, use the siteconfig-converter tool to automatically convert existing SiteConfig CRs to ClusterInstance CRs.
3. When you complete the cluster migration, delete the original Argo project and application and clean up any related resources.

Procedure

1. Create YAML files for the parallel Argo project and application:

   1. Create a YAML file that defines the AppProject resource:

      Example ztp-app-project-v2.yaml file

      apiVersion: argoproj.io/v1alpha1
      kind: AppProject
      metadata:
        name: ztp-app-project-v2
        namespace: openshift-gitops
      spec:
        clusterResourceWhitelist:
        - group: hive.openshift.io
          kind: ClusterImageSet
        - group: hive.openshift.io
          kind: ClusterImageSet
        - group: cluster.open-cluster-management.io
          kind: ManagedCluster
        - group: ""
          kind: Namespace
        destinations:
        - namespace: '*'
          server: '*'
        namespaceResourceWhitelist:
        - group: ""
          kind: ConfigMap
        - group: ""
          kind: Namespace
        - group: ""
          kind: Secret
        - group: agent-install.openshift.io
          kind: InfraEnv
        - group: agent-install.openshift.io
          kind: NMStateConfig
        - group: extensions.hive.openshift.io
          kind: AgentClusterInstall
        - group: hive.openshift.io
          kind: ClusterDeployment
        - group: metal3.io
          kind: BareMetalHost
        - group: metal3.io
          kind: HostFirmwareSettings
        - group: agent.open-cluster-management.io
          kind: KlusterletAddonConfig
        - group: cluster.open-cluster-management.io
          kind: ManagedCluster
        - group: siteconfig.open-cluster-management.io
          kind: ClusterInstance 

      1

      
        sourceRepos:
        - '*'

      Copy to Clipboard Toggle word wrap

      1

      The ClusterInstance CR manages the siteconfig.open-cluster-management.io object instead of the SiteConfig CR.

   2. Create a YAML file that defines the Application resource:

      Example clusters-v2.yaml file

      apiVersion: argoproj.io/v1alpha1
      kind: Application
      metadata:
        name: clusters-v2
        namespace: openshift-gitops
      spec:
        destination:
          namespace: clusters-sub
          server: https://kubernetes.default.svc
        ignoreDifferences:
        - group: cluster.open-cluster-management.io
          kind: ManagedCluster
          managedFieldsManagers:
          - controller
        project: ztp-app-project-v2 

      1

      
        source:
          path: site-configs-v2 

      2

      
          repoURL: http://infra.5g-deployment.lab:3000/student/ztp-repository.git
          targetRevision: main
        syncPolicy:
          syncOptions:
          - CreateNamespace=true
          - PrunePropagationPolicy=background
          - RespectIgnoreDifferences=true

      Copy to Clipboard Toggle word wrap

      1

      The project field must match the name of the AppProject resource created in the previous step.

      2

      The path field must match the root folder in your Git repository that will contain the ClusterInstance CRs and associated resources.

      Note

      By default, auto-sync is enabled. However, synchronization only occurs when you push configuration data for the cluster to the new configuration folder, or in this example, the site-configs-v2/ folder.

2. Create and commit a root folder in your Git repository that will contain the ClusterInstance CRs and associated resources, for example:

   $ mkdir site-configs-v2
   $ touch site-configs-v2/.gitkeep
   $ git commit -s -m “Creates cluster-instance folder”
   $ git push origin main

   Copy to Clipboard Toggle word wrap

   • The .gitkeep file is a placeholder to ensure that the empty folder is tracked by Git.

     Note

     You only need to create and commit the root site-configs-v2/ folder during pipeline setup. You will mirror the complete site-configs/ folder structure into site-configs-v2/ during the cluster migration procedure.

3. Apply the AppProject and Application resources to the hub cluster by running the following commands:

   $ oc apply -f ztp-app-project-v2.yaml
   $ oc apply -f clusters-v2.yaml

   Copy to Clipboard Toggle word wrap

Verification

1. Verify that the original Argo CD project, ztp-app-project, and the new Argo CD project, ztp-app-project-v2 are present on the hub cluster by running the following command:

   $ oc get appprojects -n openshift-gitops

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                 AGE
   default              46h
   policy-app-project   42h
   ztp-app-project      18h
   ztp-app-project-v2    14s

   Copy to Clipboard Toggle word wrap

2. Verify that the original Argo CD application, clusters, and the new Argo CD application, clusters-v2 are present on the hub cluster by running the following command:

   $ oc get application.argo -n openshift-gitops

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                       SYNC STATUS   HEALTH STATUS
   clusters                   Synced        Healthy
   clusters-v2                Synced        Healthy
   policies                   Synced        Healthy

   Copy to Clipboard Toggle word wrap

Procedure

1. Copy the resources/ folder from the site-configs/ directory into the new site-configs-v2/ directory:

   $ cp -r site-configs/resources site-configs-v2/

   Copy to Clipboard Toggle word wrap

2. Remove the reference to the resources/ folder from the site-configs/kustomization.yaml file. This ensures that the old clusters Argo CD application no longer manages the active-ocp-version resource.

   Example updated site-configs/resources/kustomization.yaml file

   apiVersion: kustomize.config.k8s.io/v1beta1
   kind: Kustomization
   resources:
      - pre-reqs/
      #- resources/
   generators:
      - hub-1/sno1.yaml
      - hub-1/sno2.yaml
      - hub-1/sno3.yaml

   Copy to Clipboard Toggle word wrap

3. Add the resources/ folder to the site-configs-v2/kustomization.yaml file. This step transfers ownership of the ClusterImageSet to the new clusters-v2 application.

   Example updated site-configs-v2/kustomization.yaml file

   apiVersion: kustomize.config.k8s.io/v1beta1
   kind: Kustomization
   resources:
     - resources/

   Copy to Clipboard Toggle word wrap

4. Commit and push the changes to the Git repository.

Verification

1. In Argo CD, verify that the clusters-v2 application is Healthy and Synced.

2. If the active-ocp-version ClusterImageSet resource in the cluster Argo application is out of sync, you can remove the Argo CD application label by running the following command:

   $ oc label clusterimageset active-ocp-version app.kubernetes.io/instance-

   Copy to Clipboard Toggle word wrap

   Example output

   clusterimageset.hive.openshift.io/active-ocp-version unlabeled

   Copy to Clipboard Toggle word wrap

Procedure

1. Mirror the site-configs folder structure to the new site-configs-v2 directory that will contain the ClusterInstance CRs, for example:

   site-configs-v2/
   ├── hub-1/ 

   1

   
   │   └── extra-manifest/
   ├── pre-reqs/
   │   └── sno1/ 

   2

   
   ├── reference-manifest/
   │   └── 4.20/
   └── resources/

   Copy to Clipboard Toggle word wrap

   1

   The hub-1/ folder will contain the ClusterInstance CR for each cluster.

   2

   Mirror the target cluster, in this example sno1, to include the required pre-requisite resources such as the image registry pull secret, the baseboard management controller credentials, and so on.

2. Remove the target cluster from the original Argo CD application by commenting out the resources in the related files in Git:

   1. Comment out the target cluster from the site-configs/kustomization.yaml file, for example:

      $ cat site-configs/kustomization.yaml

      Copy to Clipboard Toggle word wrap

      Example updated site-configs/kustomization.yaml file

      apiVersion: kustomize.config.k8s.io/v1beta1
      kind: Kustomization
      resources:
         - pre-reqs/
         #- resources/
      generators:
         #- hub-1/sno1.yaml
         - hub-1/sno2.yaml
         - hub-1/sno3.yaml

      Copy to Clipboard Toggle word wrap

   2. Comment out the target cluster from the site-configs/pre-reqs/kustomization.yaml file. This removes the site-configs/pre-reqs/sno1 folder, which also requires migration and has resources such as the image registry pull secret, the baseboard management controller credentials, and so on, for example:

      $ cat site-configs/pre-reqs/kustomization.yaml

      Copy to Clipboard Toggle word wrap

      Example updated site-configs/pre-reqs/kustomization.yaml file

      apiVersion: kustomize.config.k8s.io/v1beta1
      kind: Kustomization
      resources:
        #- sno1/
        - sno2/
        - sno3/

      Copy to Clipboard Toggle word wrap

3. Commit the changes to the Git repository.

   Note

   After you commit the changes, the original Argo CD application reports an OutOfSync sync status because the Argo CD application still attempts to monitor the status of the taget cluster’s resources. However, because the sync policy is set to prune=false, the Argo CD application does not delete any resources.

4. To ensure that the original Argo CD application no longer manages the cluster resources, you can remove the Argo CD application label from the resources by running the following command:

   $ for cr in bmh,hfs,clusterdeployment,agentclusterinstall,infraenv,nmstateconfig,configmap,klusterletaddonconfig,secrets; do oc label $cr app.kubernetes.io/instance- --all -n sno1; done && oc label ns sno1 app.kubernetes.io/instance- && oc label managedclusters sno1 app.kubernetes.io/instance-

   Copy to Clipboard Toggle word wrap

   The Argo CD application label is removed from all resources in the sno1 namespace and the sync status returns to Synced.

5. Create the ClusterInstance CR for the target cluster by using the siteconfig-converter tool packaged with the ztp-site-generate container image:

   Note

   The siteconfig-converter tool cannot translate earlier versions of the AgentClusterInstall resource that uses the following deprecated fields in the SiteConfig CR:

   • apiVIP
   • ingressVIP
   • manifestsConfigMapRef

   To solve this issue, you can do one of the following options:

   • Create a custom cluster template that includes these fields. For more information about creating custom templates, see Creating custom templates with the SiteConfig operator
   • Suppress the creation of the AgentClusterInstall resource by adding it to the suppressedManifests list in the ClusterInstance CR, or by using the -s flag in the siteconfig-converter tool. You must remove the resource from the suppressedManifests list when reinstalling the cluster.

   1. Pull the ztp-site-generate container image by running the following command:

      podman pull registry.redhat.io/openshift4/ztp-site-generate-rhel8:4.20

      Copy to Clipboard Toggle word wrap

   2. Run the siteconfig-converter tool interactively through the container by running the following command:

      $ podman run -v "${PWD}":/resources:Z,U -it registry.redhat.io/openshift4/ztp-site-generate-rhel8:{product-version} siteconfig-converter -d /resources/<output_folder> /resources/<path_to_siteconfig_resource>

      Copy to Clipboard Toggle word wrap
      • Replace <output_folder> with the output directory for the generated files.

      • Replace <path_to_siteconfig_resource> with the path to the target SiteConfig CR file.

        Example output

        Successfully read SiteConfig: sno1/sno1
        Converted cluster 1 (sno1) to ClusterInstance: /resources/output/sno1.yaml
        WARNING: extraManifests field is not supported in ClusterInstance and will be ignored. Create one or more configmaps with the exact desired set of CRs for the cluster and include them in the extraManifestsRefs.
        WARNING: Added default extraManifest ConfigMap 'extra-manifests-cm' to extraManifestsRefs. This configmap is created automatically.
        Successfully converted 1 cluster(s) to ClusterInstance files in /resources/output: sno1.yaml
        Generating ConfigMap kustomization files...
        Using ConfigMap name: extra-manifests-cm, namespace: sno1, manifests directory: extra-manifests
        Generating ConfigMap kustomization files with name: extra-manifests-cm, namespace: sno1, manifests directory: extra-manifests
        Generating extraManifests for SiteConfig: /resources/sno1.yaml
        Using absolute path for input file: /resources/sno1.yaml
        Running siteconfig-generator from directory: /resources
        Found extraManifests directory: /resources/output/extra-manifests/sno1
        Moved sno1_containerruntimeconfig_enable-crun-master.yaml to /resources/output/extra-manifests/sno1_containerruntimeconfig_enable-crun-master.yaml
        Moved sno1_containerruntimeconfig_enable-crun-worker.yaml to /resources/output/extra-manifests/sno1_containerruntimeconfig_enable-crun-worker.yaml
        Moved 2 extraManifest files from /resources/output/extra-manifests/sno1 to /resources/output/extra-manifests
        Removed directory: /resources/output/extra-manifests/sno1
        --- Kustomization.yaml Generator ---
        Scanning directory: /resources/output/extra-manifests
        Found and adding: extra-manifests/sno1_containerruntimeconfig_enable-crun-master.yaml
        Found and adding: extra-manifests/sno1_containerruntimeconfig_enable-crun-worker.yaml
        ------------------------------------
        kustomization-configMapGenerator-snippet.yaml generated successfully at: /resources/output/kustomization-configMapGenerator-snippet.yaml
        Content:
        apiVersion: kustomize.config.k8s.io/v1beta1
        kind: Kustomization
        configMapGenerator:
            - files:
                - extra-manifests/sno1_containerruntimeconfig_enable-crun-master.yaml
                - extra-manifests/sno1_containerruntimeconfig_enable-crun-worker.yaml
              name: extra-manifests-cm
              namespace: sno1
        generatorOptions:
            disableNameSuffixHash: true
        
        ------------------------------------

        Copy to Clipboard Toggle word wrap

        Note

        The ClusterInstance CR requires the extra manifests to be defined in a ConfigMap resource.

        To meet this requirement, the siteconfig-converter tool generates a kustomization.yaml snippet. The generated snippet uses Kustomize’s configMapGenerator to automatically package your manifest files into the required ConfigMap resource. You must merge this snippet into your original kustomization.yaml file to ensure that the ConfigMap resource is created and managed alongside your other cluster resources.

6. Configure the new Argo CD application to manage the target cluster by referencing it in the new pipelines Kustomization files, for example:

   $ cat site-configs-v2/kustomization.yaml

   Copy to Clipboard Toggle word wrap

   Example updated site-configs-v2/kustomization.yaml file

   apiVersion: kustomize.config.k8s.io/v1beta1
   kind: Kustomization
   resources:
     - resources/
     - pre-reqs/
     - hub-1/sno1.yaml

   Copy to Clipboard Toggle word wrap

   $ cat  site-configs-v2/pre-reqs/kustomization.yaml

   Copy to Clipboard Toggle word wrap

   Example updated site-configs-v2/pre-reqs/kustomization.yaml file

   apiVersion: kustomize.config.k8s.io/v1beta1
   kind: Kustomization
   resources:
     - sno1/

   Copy to Clipboard Toggle word wrap

7. Commit the changes to the Git repository.

Verification

1. Verify that the ClusterInstance CR is successfully deployed and the provisioning status complete by running the following command:

   $ oc get clusterinstance -A

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                                                         PAUSED   PROVISIONSTATUS   PROVISIONDETAILS         AGE
   clusterinstance.siteconfig.open-cluster-management.io/sno1            Completed         Provisioning completed   27s

   Copy to Clipboard Toggle word wrap

   At this point, the new Argo CD application that uses the ClusterInstance CR is managing the sno1 cluster. You can continue to migrate one or more clusters at a time by repeating these steps until all target clusters are migrated to the new pipeline.

2. Verify the folder structure and files in the site-configs-v2/ directory contain the migrated resources for the sno1 cluster, for example:

   site-configs-v2/
   ├── hub-1/
   │   ├── sno1.yaml 

   1

   
   ├── extra-manifest/
   │   ├── enable-crun-worker.yaml 

   2

   
   │   └── enable-crun-master.yaml
   ├── kustomization.yaml 

   3

   
   ├── pre-reqs/
   │   └── sno1/
   │       ├── bmc-credentials.yaml
   │       ├── namespace.yaml
   │       └── pull-secret.yaml
   ├── kustomization.yaml
   ├── reference-manifest/
   │   └── 4.20/
   └── resources/
       ├── active-ocp-version.yaml
       └── kustomization.yaml

   Copy to Clipboard Toggle word wrap

   1

   This ClusterInstance CR for the sno1 cluster.

   2

   The tool automatically generates the extra manifests referenced by the ClusterInstance CR. After generation, the file names might change. You can rename the files to match the original naming convention in the associated kustomization.yaml file.

   3

   The tool generates a kuztomization.yaml file snippet to create the ConfigMap resources that specifies the extra manifests. You can merge the generated kustomization snippet with your original kuztomization.yaml file.

Procedure

1. Delete the original Argo CD application that managed the SiteConfig CRs:

   $ oc delete application.argo clusters -n openshift-gitops

   Copy to Clipboard Toggle word wrap
   • Replace clusters with the name of your original Argo CD application.

2. Delete the original Argo CD project by running the following command:

   $ oc delete appproject ztp-app-project -n openshift-gitops

   Copy to Clipboard Toggle word wrap
   • Replace ztp-app-project with the name of your original Argo CD project.

Verification

1. Confirm that the original Argo CD application is deleted by running the following command:

   $ oc get appproject -n openshift-gitops

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                 AGE
   default              6d20h
   policy-app-project   2d22h
   ztpv2-app-project    44h

   Copy to Clipboard Toggle word wrap

   • The original Argo CD project in this example, ztp-app-project is not present in the output.

2. Confirm that the original Argo CD project is deleted by running the following command:

   oc get applications.argo -n openshift-gitops

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                       SYNC STATUS   HEALTH STATUS
   clusters-v2                Synced        Healthy
   policies                   Synced        Healthy

   Copy to Clipboard Toggle word wrap

   • The original Argo CD application in this example, clusters is not present in the output.