Chapter 15. Image-based upgrade for single-node OpenShift clusters

1. The persistent storage must be provided by either the LVM Storage or the Local Storage Operator, not both.

Procedure

1. Create the storage.bu file:

   variant: fcos
   version: 1.3.0
   storage:
     disks:
     - device: /dev/disk/by-path/pci-<root_disk> 1
       wipe_table: false
       partitions:
       - label: var-lib-containers
         start_mib: <start_of_partition> 2
         size_mib: <partition_size> 3
     filesystems:
       - path: /var/lib/containers
         device: /dev/disk/by-partlabel/var-lib-containers
         format: xfs
         wipe_filesystem: true
         with_mount_unit: true
         mount_options:
           - defaults
           - prjquota

   1

   Specify the root disk.

   2

   Specify the start of the partition in MiB. If the value is too small, the installation will fail.

   3

   Specify a minimum size for the partition of 500 GB to ensure adequate disk space for precached images. If the value is too small, the deployments after installation will fail.

2. Convert the storage.bu to an Ignition file by running the following command:

   $ butane storage.bu

   Example output

   {"ignition":{"version":"3.2.0"},"storage":{"disks":[{"device":"/dev/disk/by-path/pci-0000:00:17.0-ata-1.0","partitions":[{"label":"var-lib-containers","sizeMiB":0,"startMiB":250000}],"wipeTable":false}],"filesystems":[{"device":"/dev/disk/by-partlabel/var-lib-containers","format":"xfs","mountOptions":["defaults","prjquota"],"path":"/var/lib/containers","wipeFilesystem":true}]},"systemd":{"units":[{"contents":"# Generated by Butane\n[Unit]\nRequires=systemd-fsck@dev-disk-by\\x2dpartlabel-var\\x2dlib\\x2dcontainers.service\nAfter=systemd-fsck@dev-disk-by\\x2dpartlabel-var\\x2dlib\\x2dcontainers.service\n\n[Mount]\nWhere=/var/lib/containers\nWhat=/dev/disk/by-partlabel/var-lib-containers\nType=xfs\nOptions=defaults,prjquota\n\n[Install]\nRequiredBy=local-fs.target","enabled":true,"name":"var-lib-containers.mount"}]}}

3. Copy the output into the .spec.clusters.nodes.ignitionConfigOverride field in the SiteConfig CR:

   [...]
   spec:
     clusters:
       - nodes:
           - hostName: <name>
             ignitionConfigOverride: '{"ignition":{"version":"3.2.0"},"storage":{"disks":[{"device":"/dev/disk/by-path/pci-0000:00:17.0-ata-1.0","partitions":[{"label":"var-lib-containers","sizeMiB":0,"startMiB":250000}],"wipeTable":false}],"filesystems":[{"device":"/dev/disk/by-partlabel/var-lib-containers","format":"xfs","mountOptions":["defaults","prjquota"],"path":"/var/lib/containers","wipeFilesystem":true}]},"systemd":{"units":[{"contents":"# Generated by Butane\n[Unit]\nRequires=systemd-fsck@dev-disk-by\\x2dpartlabel-var\\x2dlib\\x2dcontainers.service\nAfter=systemd-fsck@dev-disk-by\\x2dpartlabel-var\\x2dlib\\x2dcontainers.service\n\n[Mount]\nWhere=/var/lib/containers\nWhat=/dev/disk/by-partlabel/var-lib-containers\nType=xfs\nOptions=defaults,prjquota\n\n[Install]\nRequiredBy=local-fs.target","enabled":true,"name":"var-lib-containers.mount"}]}}'
   [...]

Verification

1. During or after installation, verify on the hub cluster that the BareMetalHost object shows the annotation by running the following command:

   $ oc get bmh -n my-sno-ns my-sno -ojson | jq '.metadata.annotations["bmac.agent-install.openshift.io/ignition-config-overrides"]'

   Example output

   "{\"ignition\":{\"version\":\"3.2.0\"},\"storage\":{\"disks\":[{\"device\":\"/dev/disk/by-path/pci-0000:00:17.0-ata-1.0\",\"partitions\":[{\"label\":\"var-lib-containers\",\"sizeMiB\":0,\"startMiB\":250000}],\"wipeTable\":false}],\"filesystems\":[{\"device\":\"/dev/disk/by-partlabel/var-lib-containers\",\"format\":\"xfs\",\"mountOptions\":[\"defaults\",\"prjquota\"],\"path\":\"/var/lib/containers\",\"wipeFilesystem\":true}]},\"systemd\":{\"units\":[{\"contents\":\"# Generated by Butane\\n[Unit]\\nRequires=systemd-fsck@dev-disk-by\\\\x2dpartlabel-var\\\\x2dlib\\\\x2dcontainers.service\\nAfter=systemd-fsck@dev-disk-by\\\\x2dpartlabel-var\\\\x2dlib\\\\x2dcontainers.service\\n\\n[Mount]\\nWhere=/var/lib/containers\\nWhat=/dev/disk/by-partlabel/var-lib-containers\\nType=xfs\\nOptions=defaults,prjquota\\n\\n[Install]\\nRequiredBy=local-fs.target\",\"enabled\":true,\"name\":\"var-lib-containers.mount\"}]}}"

2. After installation, check the single-node OpenShift disk status by running the following commands:

   # lsblk

   Example output

   NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
   sda      8:0    0 446.6G  0 disk
   ├─sda1   8:1    0     1M  0 part
   ├─sda2   8:2    0   127M  0 part
   ├─sda3   8:3    0   384M  0 part /boot
   ├─sda4   8:4    0 243.6G  0 part /var
   │                                /sysroot/ostree/deploy/rhcos/var
   │                                /usr
   │                                /etc
   │                                /
   │                                /sysroot
   └─sda5   8:5    0 202.5G  0 part /var/lib/containers

   # df -h

   Example output

   Filesystem      Size  Used Avail Use% Mounted on
   devtmpfs        4.0M     0  4.0M   0% /dev
   tmpfs           126G   84K  126G   1% /dev/shm
   tmpfs            51G   93M   51G   1% /run
   /dev/sda4       244G  5.2G  239G   3% /sysroot
   tmpfs           126G  4.0K  126G   1% /tmp
   /dev/sda5       203G  119G   85G  59% /var/lib/containers
   /dev/sda3       350M  110M  218M  34% /boot
   tmpfs            26G     0   26G   0% /run/user/1000

Procedure

1. Create a Namespace object YAML file for the Lifecycle Agent, for example lcao-namespace.yaml:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-lifecycle-agent
     annotations:
       workload.openshift.io/allowed: management

   1. Create the Namespace CR by running the following command:

      $ oc create -f lcao-namespace.yaml

2. Create an OperatorGroup object YAML file for the Lifecycle Agent, for example lcao-operatorgroup.yaml:

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: openshift-lifecycle-agent
     namespace: openshift-lifecycle-agent
   spec:
     targetNamespaces:
     - openshift-lifecycle-agent

   1. Create the OperatorGroup CR by running the following command:

      $ oc create -f lcao-operatorgroup.yaml

3. Create a Subscription CR, for example, lcao-subscription.yaml:

   apiVersion: operators.coreos.com/v1
   kind: Subscription
   metadata:
     name: openshift-lifecycle-agent-subscription
     namespace: openshift-lifecycle-agent
   spec:
     channel: "stable"
     name: lifecycle-agent
     source: redhat-operators
     sourceNamespace: openshift-marketplace

   1. Create the Subscription CR by running the following command:

      $ oc create -f lcao-subscription.yaml

Verification

1. To verify that the installation succeeded, inspect the CSV resource by running the following command:

   $ oc get csv -n openshift-lifecycle-agent

   Example output

   NAME                              DISPLAY                     VERSION               REPLACES                           PHASE
   lifecycle-agent.v4.16.0           Openshift Lifecycle Agent   4.16.0                Succeeded

2. Verify that the Lifecycle Agent is up and running by running the following command:

   $ oc get deploy -n openshift-lifecycle-agent

   Example output

   NAME                                 READY   UP-TO-DATE   AVAILABLE   AGE
   lifecycle-agent-controller-manager   1/1     1            1           14s

Procedure

1. In the OpenShift Container Platform web console, navigate to Operators OperatorHub.
2. Search for the Lifecycle Agent from the list of available Operators, and then click Install.
3. On the Install Operator page, under A specific namespace on the cluster select openshift-lifecycle-agent.
4. Click Install.

Verification

1. To confirm that the installation is successful:

   1. Click Operators Installed Operators.

   2. Ensure that the Lifecycle Agent is listed in the openshift-lifecycle-agent project with a Status of InstallSucceeded.

      Note

      During installation an Operator might display a Failed status. If the installation later succeeds with an InstallSucceeded message, you can ignore the Failed message.

1. Click Operators Installed Operators, and inspect the Operator Subscriptions and Install Plans tabs for any failure or errors under Status.
2. Click Workloads Pods, and check the logs for pods in the openshift-lifecycle-agent project.

Procedure

1. Extract the following CRs from the ztp-site-generate container image and push them to the source-cr directory:

   Example LcaSubscriptionNS.yaml file

   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-lifecycle-agent
     annotations:
       workload.openshift.io/allowed: management
       ran.openshift.io/ztp-deploy-wave: "2"
     labels:
       kubernetes.io/metadata.name: openshift-lifecycle-agent

   Example LcaSubscriptionOperGroup.yaml file

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: lifecycle-agent-operatorgroup
     namespace: openshift-lifecycle-agent
     annotations:
       ran.openshift.io/ztp-deploy-wave: "2"
   spec:
     targetNamespaces:
       - openshift-lifecycle-agent

   Example LcaSubscription.yaml file

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: lifecycle-agent
     namespace: openshift-lifecycle-agent
     annotations:
       ran.openshift.io/ztp-deploy-wave: "2"
   spec:
     channel: "stable"
     name: lifecycle-agent
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     installPlanApproval: Manual
   status:
     state: AtLatestKnown

   Example directory structure

   ├── kustomization.yaml
   ├── sno
   │   ├── example-cnf.yaml
   │   ├── common-ranGen.yaml
   │   ├── group-du-sno-ranGen.yaml
   │   ├── group-du-sno-validator-ranGen.yaml
   │   └── ns.yaml
   ├── source-crs
   │   ├── LcaSubscriptionNS.yaml
   │   ├── LcaSubscriptionOperGroup.yaml
   │   ├── LcaSubscription.yaml

2. Add the CRs to your common PolicyGenTemplate:

   apiVersion: ran.openshift.io/v1
   kind: PolicyGenTemplate
   metadata:
     name: "example-common-latest"
     namespace: "ztp-common"
   spec:
     bindingRules:
       common: "true"
       du-profile: "latest"
     sourceFiles:
       - fileName: LcaSubscriptionNS.yaml
         policyName: "subscriptions-policy"
       - fileName: LcaSubscriptionOperGroup.yaml
         policyName: "subscriptions-policy"
       - fileName: LcaSubscription.yaml
         policyName: "subscriptions-policy"
   [...]

Procedure

1. Extract the following CRs from the ztp-site-generate container image and push them to the source-cr directory:

   Example OadpSubscriptionNS.yaml file

   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-adp
     annotations:
       ran.openshift.io/ztp-deploy-wave: "2"
     labels:
       kubernetes.io/metadata.name: openshift-adp

   Example OadpSubscriptionOperGroup.yaml file

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: redhat-oadp-operator
     namespace: openshift-adp
     annotations:
       ran.openshift.io/ztp-deploy-wave: "2"
   spec:
     targetNamespaces:
     - openshift-adp

   Example OadpSubscription.yaml file

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: redhat-oadp-operator
     namespace: openshift-adp
     annotations:
       ran.openshift.io/ztp-deploy-wave: "2"
   spec:
     channel: stable-1.4
     name: redhat-oadp-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     installPlanApproval: Manual
   status:
     state: AtLatestKnown

   Example OadpOperatorStatus.yaml file

   apiVersion: operators.coreos.com/v1
   kind: Operator
   metadata:
     name: redhat-oadp-operator.openshift-adp
     annotations:
       ran.openshift.io/ztp-deploy-wave: "2"
   status:
     components:
       refs:
       - kind: Subscription
         namespace: openshift-adp
         conditions:
         - type: CatalogSourcesUnhealthy
           status: "False"
       - kind: InstallPlan
         namespace: openshift-adp
         conditions:
         - type: Installed
           status: "True"
       - kind: ClusterServiceVersion
         namespace: openshift-adp
         conditions:
         - type: Succeeded
           status: "True"
           reason: InstallSucceeded

   Example directory structure

   ├── kustomization.yaml
   ├── sno
   │   ├── example-cnf.yaml
   │   ├── common-ranGen.yaml
   │   ├── group-du-sno-ranGen.yaml
   │   ├── group-du-sno-validator-ranGen.yaml
   │   └── ns.yaml
   ├── source-crs
   │   ├── OadpSubscriptionNS.yaml
   │   ├── OadpSubscriptionOperGroup.yaml
   │   ├── OadpSubscription.yaml
   │   ├── OadpOperatorStatus.yaml

2. Add the CRs to your common PolicyGenTemplate:

   apiVersion: ran.openshift.io/v1
   kind: PolicyGenTemplate
   metadata:
     name: "example-common-latest"
     namespace: "ztp-common"
   spec:
     bindingRules:
       common: "true"
       du-profile: "latest"
     sourceFiles:
       - fileName: OadpSubscriptionNS.yaml
         policyName: "subscriptions-policy"
       - fileName: OadpSubscriptionOperGroup.yaml
         policyName: "subscriptions-policy"
       - fileName: OadpSubscription.yaml
         policyName: "subscriptions-policy"
       - fileName: OadpOperatorStatus.yaml
         policyName: "subscriptions-policy"
   [...]

3. Create the DataProtectionApplication CR and the S3 secret only for the target cluster:

   1. Extract the following CRs from the ztp-site-generate container image and push them to the source-cr directory:

      Example DataProtectionApplication.yaml file

      apiVersion: oadp.openshift.io/v1alpha1
      kind: DataProtectionApplication
      metadata:
        name: dataprotectionapplication
        namespace: openshift-adp
        annotations:
          ran.openshift.io/ztp-deploy-wave: "100"
      spec:
        configuration:
          restic:
            enable: false 1
          velero:
            defaultPlugins:
              - aws
              - openshift
            resourceTimeout: 10m
        backupLocations:
          - velero:
              config:
                profile: "default"
                region: minio
                s3Url: $url
                insecureSkipTLSVerify: "true"
                s3ForcePathStyle: "true"
              provider: aws
              default: true
              credential:
                key: cloud
                name: cloud-credentials
              objectStorage:
                bucket: $bucketName 2
                prefix: $prefixName 3
      status:
        conditions:
        - reason: Complete
          status: "True"
          type: Reconciled

      1

      The spec.configuration.restic.enable field must be set to false for an image-based upgrade because persistent volume contents are retained and reused after the upgrade.

      2 3

      The bucket defines the bucket name that is created in S3 backend. The prefix defines the name of the subdirectory that will be automatically created in the bucket. The combination of bucket and prefix must be unique for each target cluster to avoid interference between them. To ensure a unique storage directory for each target cluster, you can use the RHACM hub template function, for example, prefix: {{hub .ManagedClusterName hub}}.

      Example OadpSecret.yaml file

      apiVersion: v1
      kind: Secret
      metadata:
        name: cloud-credentials
        namespace: openshift-adp
        annotations:
          ran.openshift.io/ztp-deploy-wave: "100"
      type: Opaque

      Example OadpBackupStorageLocationStatus.yaml file

      apiVersion: velero.io/v1
      kind: BackupStorageLocation
      metadata:
        namespace: openshift-adp
        annotations:
          ran.openshift.io/ztp-deploy-wave: "100"
      status:
        phase: Available

      The OadpBackupStorageLocationStatus.yaml CR verifies the availability of backup storage locations created by OADP.

   2. Add the CRs to your site PolicyGenTemplate with overrides:

      apiVersion: ran.openshift.io/v1
      kind: PolicyGenTemplate
      metadata:
        name: "example-cnf"
        namespace: "ztp-site"
      spec:
        bindingRules:
          sites: "example-cnf"
          du-profile: "latest"
        mcp: "master"
        sourceFiles:
          ...
          - fileName: OadpSecret.yaml
            policyName: "config-policy"
            data:
              cloud: <your_credentials> 1
          - fileName: DataProtectionApplication.yaml
            policyName: "config-policy"
            spec:
              backupLocations:
                - velero:
                    config:
                      region: minio
                      s3Url: <your_S3_URL> 2
                      profile: "default"
                      insecureSkipTLSVerify: "true"
                      s3ForcePathStyle: "true"
                    provider: aws
                    default: true
                    credential:
                      key: cloud
                      name: cloud-credentials
                    objectStorage:
                      bucket: <your_bucket_name> 3
                      prefix: <cluster_name> 4
          - fileName: OadpBackupStorageLocationStatus.yaml
            policyName: "config-policy"

      1

      Specify your credentials for your S3 storage backend.

      2

      Specify the URL for your S3-compatible bucket.

      3 4

      The bucket defines the bucket name that is created in S3 backend. The prefix defines the name of the subdirectory that will be automatically created in the bucket. The combination of bucket and prefix must be unique for each target cluster to avoid interference between them. To ensure a unique storage directory for each target cluster, you can use the RHACM hub template function, for example, prefix: {{hub .ManagedClusterName hub}}.

1. Dual-stack networking is not supported in this release.
2. If the seed cluster is installed in a disconnected environment, the target clusters must also be installed in a disconnected environment.
3. The proxy configuration does not have to be the same.

Procedure

1. Detach the cluster from the hub to delete any RHACM-specific resources from the seed cluster that must not be in the seed image:

   1. Manually detach the seed cluster by running the following command:

      $ oc delete managedcluster sno-worker-example

      1. Wait until the ManagedCluster CR is removed. After the CR is removed, create the proper SeedGenerator CR. The Lifecycle Agent cleans up the RHACM artifacts.

   2. If you are using GitOps ZTP, detach your cluster by removing the seed cluster’s SiteConfig CR from the kustomization.yaml.

      1. If you have a kustomization.yaml file that references multiple SiteConfig CRs, remove your seed cluster’s SiteConfig CR from the kustomization.yaml:

         apiVersion: kustomize.config.k8s.io/v1beta1
         kind: Kustomization
         
         generators:
         #- example-seed-sno1.yaml
         - example-target-sno2.yaml
         - example-target-sno3.yaml

      2. If you have a kustomization.yaml that references one SiteConfig CR, remove your seed cluster’s SiteConfig CR from the kustomization.yaml and add the generators: {} line:

         apiVersion: kustomize.config.k8s.io/v1beta1
         kind: Kustomization
         
         generators: {}

      3. Commit the kustomization.yaml changes in your Git repository and push the changes to your repository.

         The ArgoCD pipeline detects the changes and removes the managed cluster.

2. Create the Secret object so that you can push the seed image to your registry.

   1. Create the authentication file by running the following commands:

      $ MY_USER=myuserid
      $ AUTHFILE=/tmp/my-auth.json
      $ podman login --authfile ${AUTHFILE} -u ${MY_USER} quay.io/${MY_USER}

      $ base64 -w 0 ${AUTHFILE} ; echo

   2. Copy the output into the seedAuth field in the Secret YAML file named seedgen in the openshift-lifecycle-agent namespace:

      apiVersion: v1
      kind: Secret
      metadata:
        name: seedgen 1
        namespace: openshift-lifecycle-agent
      type: Opaque
      data:
        seedAuth: <encoded_AUTHFILE> 2

      1

      The Secret resource must have the name: seedgen and namespace: openshift-lifecycle-agent fields.

      2

      Specifies a base64-encoded authfile for write-access to the registry for pushing the generated seed images.

   3. Apply the Secret by running the following command:

      $ oc apply -f secretseedgenerator.yaml

3. Create the SeedGenerator CR:

   apiVersion: lca.openshift.io/v1
   kind: SeedGenerator
   metadata:
     name: seedimage 1
   spec:
     seedImage: <seed_container_image> 2

   1

   The SeedGenerator CR must be named seedimage.

   2

   Specify the container image URL, for example, quay.io/example/seed-container-image:<tag>. It is recommended to use the <seed_cluster_name>:<ocp_version> format.

4. Generate the seed image by running the following command:

   $ oc apply -f seedgenerator.yaml

   Important

   The cluster reboots and loses API capabilities while the Lifecycle Agent generates the seed image. Applying the SeedGenerator CR stops the kubelet and the CRI-O operations, then it starts the image generation.

Verification

1. After the cluster recovers and it is available, you can check the status of the SeedGenerator CR by running the following command:

   $ oc get seedgenerator -o yaml

   Example output

   status:
     conditions:
     - lastTransitionTime: "2024-02-13T21:24:26Z"
       message: Seed Generation completed
       observedGeneration: 1
       reason: Completed
       status: "False"
       type: SeedGenInProgress
     - lastTransitionTime: "2024-02-13T21:24:26Z"
       message: Seed Generation completed
       observedGeneration: 1
       reason: Completed
       status: "True"
       type: SeedGenCompleted 1
     observedGeneration: 1

   1

   The seed image generation is complete.

Procedure

1. Create the OADP Backup and Restore CRs for platform artifacts in the same namespace where the OADP Operator is installed, which is openshift-adp.

   1. If the target cluster is managed by RHACM, add the following YAML file for backing up and restoring RHACM artifacts:

      PlatformBackupRestore.yaml for RHACM

      apiVersion: velero.io/v1
      kind: Backup
      metadata:
        name: acm-klusterlet
        annotations:
          lca.openshift.io/apply-label: "apps/v1/deployments/open-cluster-management-agent/klusterlet,v1/secrets/open-cluster-management-agent/bootstrap-hub-kubeconfig,rbac.authorization.k8s.io/v1/clusterroles/klusterlet,v1/serviceaccounts/open-cluster-management-agent/klusterlet,scheduling.k8s.io/v1/priorityclasses/klusterlet-critical,rbac.authorization.k8s.io/v1/clusterroles/open-cluster-management:klusterlet-admin-aggregate-clusterrole,rbac.authorization.k8s.io/v1/clusterrolebindings/klusterlet,operator.open-cluster-management.io/v1/klusterlets/klusterlet,apiextensions.k8s.io/v1/customresourcedefinitions/klusterlets.operator.open-cluster-management.io,v1/secrets/open-cluster-management-agent/open-cluster-management-image-pull-credentials" 1
        labels:
          velero.io/storage-location: default
        namespace: openshift-adp
      spec:
        includedNamespaces:
        - open-cluster-management-agent
        includedClusterScopedResources:
        - klusterlets.operator.open-cluster-management.io
        - clusterroles.rbac.authorization.k8s.io
        - clusterrolebindings.rbac.authorization.k8s.io
        - priorityclasses.scheduling.k8s.io
        includedNamespaceScopedResources:
        - deployments
        - serviceaccounts
        - secrets
        excludedNamespaceScopedResources: []
      ---
      apiVersion: velero.io/v1
      kind: Restore
      metadata:
        name: acm-klusterlet
        namespace: openshift-adp
        labels:
          velero.io/storage-location: default
        annotations:
          lca.openshift.io/apply-wave: "1"
      spec:
        backupName:
          acm-klusterlet

      1

      If your multiclusterHub CR does not have .spec.imagePullSecret defined and the secret does not exist on the open-cluster-management-agent namespace in your hub cluster, remove v1/secrets/open-cluster-management-agent/open-cluster-management-image-pull-credentials.

   2. If you created persistent volumes on your cluster through LVM Storage, add the following YAML file for LVM Storage artifacts:

      PlatformBackupRestoreLvms.yaml for LVM Storage

      apiVersion: velero.io/v1
      kind: Backup
      metadata:
        labels:
          velero.io/storage-location: default
        name: lvmcluster
        namespace: openshift-adp
      spec:
        includedNamespaces:
          - openshift-storage
        includedNamespaceScopedResources:
          - lvmclusters
          - lvmvolumegroups
          - lvmvolumegroupnodestatuses
      ---
      apiVersion: velero.io/v1
      kind: Restore
      metadata:
        name: lvmcluster
        namespace: openshift-adp
        labels:
          velero.io/storage-location: default
        annotations:
          lca.openshift.io/apply-wave: "2" 1
      spec:
        backupName:
          lvmcluster

      1

      The lca.openshift.io/apply-wave value must be lower than the values specified in the application Restore CRs.

2. If you need to restore applications after the upgrade, create the OADP Backup and Restore CRs for your application in the openshift-adp namespace.

   1. Create the OADP CRs for cluster-scoped application artifacts in the openshift-adp namespace.

      Example OADP CRs for cluster-scoped application artifacts for LSO and LVM Storage

      apiVersion: velero.io/v1
      kind: Backup
      metadata:
        annotations:
          lca.openshift.io/apply-label: "apiextensions.k8s.io/v1/customresourcedefinitions/test.example.com,security.openshift.io/v1/securitycontextconstraints/test,rbac.authorization.k8s.io/v1/clusterroles/test-role,rbac.authorization.k8s.io/v1/clusterrolebindings/system:openshift:scc:test" 1
        name: backup-app-cluster-resources
        labels:
          velero.io/storage-location: default
        namespace: openshift-adp
      spec:
        includedClusterScopedResources:
        - customresourcedefinitions
        - securitycontextconstraints
        - clusterrolebindings
        - clusterroles
        excludedClusterScopedResources:
        - Namespace
      ---
      apiVersion: velero.io/v1
      kind: Restore
      metadata:
        name: test-app-cluster-resources
        namespace: openshift-adp
        labels:
          velero.io/storage-location: default
        annotations:
          lca.openshift.io/apply-wave: "3" 2
      spec:
        backupName:
          backup-app-cluster-resources

      1

      Replace the example resource name with your actual resources.

      2

      The lca.openshift.io/apply-wave value must be higher than the value in the platform Restore CRs and lower than the value in the application namespace-scoped Restore CR.

   2. Create the OADP CRs for your namespace-scoped application artifacts.

      Example OADP CRs namespace-scoped application artifacts when LSO is used

      apiVersion: velero.io/v1
      kind: Backup
      metadata:
        labels:
          velero.io/storage-location: default
        name: backup-app
        namespace: openshift-adp
      spec:
        includedNamespaces:
        - test
        includedNamespaceScopedResources:
        - secrets
        - persistentvolumeclaims
        - deployments
        - statefulsets
        - configmaps
        - cronjobs
        - services
        - job
        - poddisruptionbudgets
        - <application_custom_resources> 1
        excludedClusterScopedResources:
        - persistentVolumes
      ---
      apiVersion: velero.io/v1
      kind: Restore
      metadata:
        name: test-app
        namespace: openshift-adp
        labels:
          velero.io/storage-location: default
        annotations:
          lca.openshift.io/apply-wave: "4"
      spec:
        backupName:
          backup-app

      1

      Define custom resources for your application.

      Example OADP CRs namespace-scoped application artifacts when LVM Storage is used

      apiVersion: velero.io/v1
      kind: Backup
      metadata:
        labels:
          velero.io/storage-location: default
        name: backup-app
        namespace: openshift-adp
      spec:
        includedNamespaces:
        - test
        includedNamespaceScopedResources:
        - secrets
        - persistentvolumeclaims
        - deployments
        - statefulsets
        - configmaps
        - cronjobs
        - services
        - job
        - poddisruptionbudgets
        - <application_custom_resources> 1
        includedClusterScopedResources:
        - persistentVolumes 2
        - logicalvolumes.topolvm.io 3
        - volumesnapshotcontents 4
      ---
      apiVersion: velero.io/v1
      kind: Restore
      metadata:
        name: test-app
        namespace: openshift-adp
        labels:
          velero.io/storage-location: default
        annotations:
          lca.openshift.io/apply-wave: "4"
      spec:
        backupName:
          backup-app
        restorePVs: true
        restoreStatus:
          includedResources:
          - logicalvolumes 5

      1

      Define custom resources for your application.

      2

      Required field.

      3

      Required field

      4

      Optional if you use LVM Storage volume snapshots.

      5

      Required field.

      Important

      The same version of the applications must function on both the current and the target release of OpenShift Container Platform.

3. Create the ConfigMap object for your OADP CRs by running the following command:

   $ oc create configmap oadp-cm-example --from-file=example-oadp-resources.yaml=<path_to_oadp_crs> -n openshift-adp

4. Patch the ImageBasedUpgrade CR by running the following command:

   $ oc patch imagebasedupgrades.lca.openshift.io upgrade \
     -p='{"spec": {"oadpContent": [{"name": "oadp-cm-example", "namespace": "openshift-adp"}]}}' \
     --type=merge -n openshift-lifecycle-agent

Procedure

1. Create a YAML file that contains your extra manifests, such as SR-IOV.

   Example SR-IOV resources

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetworkNodePolicy
   metadata:
     name: "example-sriov-node-policy"
     namespace: openshift-sriov-network-operator
   spec:
     deviceType: vfio-pci
     isRdma: false
     nicSelector:
       pfNames: [ens1f0]
     nodeSelector:
       node-role.kubernetes.io/master: ""
     mtu: 1500
     numVfs: 8
     priority: 99
     resourceName: example-sriov-node-policy
   ---
   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetwork
   metadata:
     name: "example-sriov-network"
     namespace: openshift-sriov-network-operator
   spec:
     ipam: |-
       {
       }
     linkState: auto
     networkNamespace: sriov-namespace
     resourceName: example-sriov-node-policy
     spoofChk: "on"
     trust: "off"

2. Create the ConfigMap object by running the following command:

   $ oc create configmap example-extra-manifests-cm --from-file=example-extra-manifests.yaml=<path_to_extramanifest> -n openshift-lifecycle-agent

3. Patch the ImageBasedUpgrade CR by running the following command:

   $ oc patch imagebasedupgrades.lca.openshift.io upgrade \
     -p='{"spec": {"extraManifests": [{"name": "example-extra-manifests-cm", "namespace": "openshift-lifecycle-agent"}]}}' \
     --type=merge -n openshift-lifecycle-agent

Procedure

1. Create a YAML file that contains the CatalogSource CR:

   apiVersion: operators.coreos.com/v1
   kind: CatalogSource
   metadata:
     name: example-catalogsources
     namespace: openshift-marketplace
   spec:
     sourceType: grpc
     displayName: disconnected-redhat-operators
     image: quay.io/example-org/example-catalog:v1

2. Create the ConfigMap object by running the following command:

   $ oc create configmap example-catalogsources-cm --from-file=example-catalogsources.yaml=<path_to_catalogsource_cr> -n openshift-lifecycle-agent

3. Patch the ImageBasedUpgrade CR by running the following command:

   $ oc patch imagebasedupgrades.lca.openshift.io upgrade \
     -p='{"spec": {"extraManifests": [{"name": "example-catalogsources-cm", "namespace": "openshift-lifecycle-agent"}]}}' \
     --type=merge -n openshift-lifecycle-agent

Procedure

1. Ensure that your Git repository that you use with the ArgoCD policies application contains the following directory structure:

   ├── source-crs/
   │   ├── ibu/
   │   │    ├── ImageBasedUpgrade.yaml
   │   │    ├── PlatformBackupRestore.yaml
   │   │    ├── PlatformBackupRestoreLvms.yaml
   ├── ...
   ├── ibu-upgrade-ranGen.yaml
   ├── kustomization.yaml

   Important

   The kustomization.yaml file must be located in the same directory structure as previously shown to reference the ibu-upgrade-ranGen.yaml manifest.

   The source-crs/ibu/PlatformBackupRestore.yaml file is provided in the ZTP container image.

   PlatformBackupRestore.yaml

   apiVersion: velero.io/v1
   kind: Backup
   metadata:
     name: acm-klusterlet
     annotations:
       lca.openshift.io/apply-label: "apps/v1/deployments/open-cluster-management-agent/klusterlet,v1/secrets/open-cluster-management-agent/bootstrap-hub-kubeconfig,rbac.authorization.k8s.io/v1/clusterroles/klusterlet,v1/serviceaccounts/open-cluster-management-agent/klusterlet,scheduling.k8s.io/v1/priorityclasses/klusterlet-critical,rbac.authorization.k8s.io/v1/clusterroles/open-cluster-management:klusterlet-admin-aggregate-clusterrole,rbac.authorization.k8s.io/v1/clusterrolebindings/klusterlet,operator.open-cluster-management.io/v1/klusterlets/klusterlet,apiextensions.k8s.io/v1/customresourcedefinitions/klusterlets.operator.open-cluster-management.io,v1/secrets/open-cluster-management-agent/open-cluster-management-image-pull-credentials" 1
     labels:
       velero.io/storage-location: default
     namespace: openshift-adp
   spec:
     includedNamespaces:
     - open-cluster-management-agent
     includedClusterScopedResources:
     - klusterlets.operator.open-cluster-management.io
     - clusterroles.rbac.authorization.k8s.io
     - clusterrolebindings.rbac.authorization.k8s.io
     - priorityclasses.scheduling.k8s.io
     includedNamespaceScopedResources:
     - deployments
     - serviceaccounts
     - secrets
     excludedNamespaceScopedResources: []
   ---
   apiVersion: velero.io/v1
   kind: Restore
   metadata:
     name: acm-klusterlet
     namespace: openshift-adp
     labels:
       velero.io/storage-location: default
     annotations:
       lca.openshift.io/apply-wave: "1"
   spec:
     backupName:
       acm-klusterlet

   1

   If your multiclusterHub CR does not have .spec.imagePullSecret defined and the secret does not exist on the open-cluster-management-agent namespace in your hub cluster, remove v1/secrets/open-cluster-management-agent/open-cluster-management-image-pull-credentials.

   If you use LVM Storage to create persistent volumes, you can use the source-crs/ibu/PlatformBackupRestoreLvms.yaml provided in the ZTP container image to back up your LVM Storage resources.

   PlatformBackupRestoreLvms.yaml

   apiVersion: velero.io/v1
   kind: Backup
   metadata:
     labels:
       velero.io/storage-location: default
     name: lvmcluster
     namespace: openshift-adp
   spec:
     includedNamespaces:
       - openshift-storage
     includedNamespaceScopedResources:
       - lvmclusters
       - lvmvolumegroups
       - lvmvolumegroupnodestatuses
   ---
   apiVersion: velero.io/v1
   kind: Restore
   metadata:
     name: lvmcluster
     namespace: openshift-adp
     labels:
       velero.io/storage-location: default
     annotations:
       lca.openshift.io/apply-wave: "2" 1
   spec:
     backupName:
       lvmcluster

   1

   The lca.openshift.io/apply-wave value must be lower than the values specified in the application Restore CRs.

2. If you need to restore applications after the upgrade, create the OADP Backup and Restore CRs for your application in the openshift-adp namespace:

   1. Create the OADP CRs for cluster-scoped application artifacts in the openshift-adp namespace:

      Example OADP CRs for cluster-scoped application artifacts for LSO and LVM Storage

      apiVersion: velero.io/v1
      kind: Backup
      metadata:
        annotations:
          lca.openshift.io/apply-label: "apiextensions.k8s.io/v1/customresourcedefinitions/test.example.com,security.openshift.io/v1/securitycontextconstraints/test,rbac.authorization.k8s.io/v1/clusterroles/test-role,rbac.authorization.k8s.io/v1/clusterrolebindings/system:openshift:scc:test" 1
        name: backup-app-cluster-resources
        labels:
          velero.io/storage-location: default
        namespace: openshift-adp
      spec:
        includedClusterScopedResources:
        - customresourcedefinitions
        - securitycontextconstraints
        - clusterrolebindings
        - clusterroles
        excludedClusterScopedResources:
        - Namespace
      ---
      apiVersion: velero.io/v1
      kind: Restore
      metadata:
        name: test-app-cluster-resources
        namespace: openshift-adp
        labels:
          velero.io/storage-location: default
        annotations:
          lca.openshift.io/apply-wave: "3" 2
      spec:
        backupName:
          backup-app-cluster-resources

      1

      Replace the example resource name with your actual resources.

      2

      The lca.openshift.io/apply-wave value must be higher than the value in the platform Restore CRs and lower than the value in the application namespace-scoped Restore CR.

   2. Create the OADP CRs for your namespace-scoped application artifacts in the source-crs/custom-crs directory:

      Example OADP CRs namespace-scoped application artifacts when LSO is used

      apiVersion: velero.io/v1
      kind: Backup
      metadata:
        labels:
          velero.io/storage-location: default
        name: backup-app
        namespace: openshift-adp
      spec:
        includedNamespaces:
        - test
        includedNamespaceScopedResources:
        - secrets
        - persistentvolumeclaims
        - deployments
        - statefulsets
        - configmaps
        - cronjobs
        - services
        - job
        - poddisruptionbudgets
        - <application_custom_resources> 1
        excludedClusterScopedResources:
        - persistentVolumes
      ---
      apiVersion: velero.io/v1
      kind: Restore
      metadata:
        name: test-app
        namespace: openshift-adp
        labels:
          velero.io/storage-location: default
        annotations:
          lca.openshift.io/apply-wave: "4"
      spec:
        backupName:
          backup-app

      1

      Define custom resources for your application.

      Example OADP CRs namespace-scoped application artifacts when LVM Storage is used

      apiVersion: velero.io/v1
      kind: Backup
      metadata:
        labels:
          velero.io/storage-location: default
        name: backup-app
        namespace: openshift-adp
      spec:
        includedNamespaces:
        - test
        includedNamespaceScopedResources:
        - secrets
        - persistentvolumeclaims
        - deployments
        - statefulsets
        - configmaps
        - cronjobs
        - services
        - job
        - poddisruptionbudgets
        - <application_custom_resources> 1
        includedClusterScopedResources:
        - persistentVolumes 2
        - logicalvolumes.topolvm.io 3
        - volumesnapshotcontents 4
      ---
      apiVersion: velero.io/v1
      kind: Restore
      metadata:
        name: test-app
        namespace: openshift-adp
        labels:
          velero.io/storage-location: default
        annotations:
          lca.openshift.io/apply-wave: "4"
      spec:
        backupName:
          backup-app
        restorePVs: true
        restoreStatus:
          includedResources:
          - logicalvolumes 5

      1

      Define custom resources for your application.

      2

      Required field.

      3

      Required field

      4

      Optional if you use LVM Storage volume snapshots.

      5

      Required field.

      Important

      The same version of the applications must function on both the current and the target release of OpenShift Container Platform.

3. Create the oadp-cm ConfigMap object through the oadp-cm-policy in a new PolicyGenTemplate called ibu-upgrade-ranGen.yaml:

   apiVersion: ran.openshift.io/v1
   kind: PolicyGenTemplate
   metadata:
     name: example-group-ibu
     namespace: "ztp-group"
   spec:
     bindingRules:
       group-du-sno: ""
     mcp: "master"
     evaluationInterval:
       compliant: 10s
       noncompliant: 10s
     sourceFiles:
     - fileName: ConfigMapGeneric.yaml
       complianceType: mustonlyhave
       policyName: "oadp-cm-policy"
       metadata:
         name: oadp-cm
         namespace: openshift-adp

4. Create a kustomization.yaml with the following content:

   apiVersion: kustomize.config.k8s.io/v1beta1
   kind: Kustomization
   
   generators: 1
   - ibu-upgrade-ranGen.yaml
   
   configMapGenerator: 2
   - files:
     - source-crs/ibu/PlatformBackupRestore.yaml
     #- source-crs/custom-crs/ApplicationClusterScopedBackupRestore.yaml
     #- source-crs/custom-crs/ApplicationApplicationBackupRestoreLso.yaml
     name: oadp-cm
     namespace: ztp-group
   generatorOptions:
     disableNameSuffixHash: true
   
   
   patches: 3
   - target:
       group: policy.open-cluster-management.io
       version: v1
       kind: Policy
       name: example-group-ibu-oadp-cm-policy
     patch: |-
       - op: replace
         path: /spec/policy-templates/0/objectDefinition/spec/object-templates/0/objectDefinition/data
         value: '{{hub copyConfigMapData "ztp-group" "oadp-cm" hub}}'

   1

   Generates the oadp-cm-policy.

   2

   Creates the oadp-cm ConfigMap object on the hub cluster with Backup and Restore CRs.

   3

   Overrides the data field of ConfigMap added in oadp-cm-policy. A hub template is used to propagate the oadp-cm ConfigMap to all target clusters.

5. Push the changes to your Git repository.

Procedure

1. Label your required extra manifests with the lca.openshift.io/target-ocp-version: <target_version> label in your existing PolicyGenTemplate CR:

   apiVersion: ran.openshift.io/v1
   kind: PolicyGenTemplate
   metadata:
     name: example-sno
   spec:
     bindingRules:
       sites: "example-sno"
       du-profile: "4.15"
     mcp: "master"
     sourceFiles:
       - fileName: SriovNetwork.yaml
         policyName: "config-policy"
         metadata:
           name: "sriov-nw-du-fh"
           labels:
             lca.openshift.io/target-ocp-version: "4.15" 1
         spec:
           resourceName: du_fh
           vlan: 140
       - fileName: SriovNetworkNodePolicy.yaml
         policyName: "config-policy"
         metadata:
           name: "sriov-nnp-du-fh"
           labels:
             lca.openshift.io/target-ocp-version: "4.15"
         spec:
           deviceType: netdevice
           isRdma: false
           nicSelector:
             pfNames: ["ens5f0"]
           numVfs: 8
           priority: 10
           resourceName: du_fh
       - fileName: SriovNetwork.yaml
         policyName: "config-policy"
         metadata:
           name: "sriov-nw-du-mh"
           labels:
             lca.openshift.io/target-ocp-version: "4.15"
         spec:
           resourceName: du_mh
           vlan: 150
       - fileName: SriovNetworkNodePolicy.yaml
         policyName: "config-policy"
         metadata:
           name: "sriov-nnp-du-mh"
           labels:
             lca.openshift.io/target-ocp-version: "4.15"
         spec:
           deviceType: vfio-pci
           isRdma: false
           nicSelector:
             pfNames: ["ens7f0"]
           numVfs: 8
           priority: 10
           resourceName: du_mh
       - fileName: DefaultCatsrc.yaml 2
         policyName: "config-policy"
         metadata:
           name: default-cat-source
           namespace: openshift-marketplace
           labels:
               lca.openshift.io/target-ocp-version: "4.15"
         spec:
             displayName: default-cat-source
             image: quay.io/example-org/example-catalog:v1

   1

   Ensure that the lca.openshift.io/target-ocp-version label matches either the y-stream or the z-stream of the target OpenShift Container Platform version that is specified in the spec.seedImageRef.version field of the ImageBasedUpgrade CR. The Lifecycle Agent only applies the CRs that match the specified version.

   2

   If you do not want to use custom catalog sources, remove this entry.

2. Push the changes to your Git repository.

Procedure

1. Increase the threshold to 65% by running the following command:

   $ oc -n openshift-lifecycle-agent annotate ibu upgrade image-cleanup.lca.openshift.io/disk-usage-threshold-percent='65'

2. (Optional) Remove the threshold override by running the following command:

   $ oc -n  openshift-lifecycle-agent annotate ibu upgrade image-cleanup.lca.openshift.io/disk-usage-threshold-percent-

Procedure

1. Disable the automatic image cleanup by running the following command:

   $ oc -n openshift-lifecycle-agent annotate ibu upgrade image-cleanup.lca.openshift.io/on-prep='Disabled'

2. (Optional) Enable automatic image cleanup again by running the following command:

   $ oc -n  openshift-lifecycle-agent annotate ibu upgrade image-cleanup.lca.openshift.io/on-prep-