Chapter 16. Image-based upgrade for single-node OpenShift clusters

1. The persistent storage must be provided by either the LVM Storage or the Local Storage Operator, not both.

Procedure

1. Create the storage.bu file:

   variant: fcos
   version: 1.3.0
   storage:
     disks:
     - device: /dev/disk/by-path/pci-<root_disk> 

   1

   
       wipe_table: false
       partitions:
       - label: var-lib-containers
         start_mib: <start_of_partition> 

   2

   
         size_mib: <partition_size> 

   3

   
     filesystems:
       - path: /var/lib/containers
         device: /dev/disk/by-partlabel/var-lib-containers
         format: xfs
         wipe_filesystem: true
         with_mount_unit: true
         mount_options:
           - defaults
           - prjquota

   Copy to Clipboard Toggle word wrap

   1

   Specify the root disk.

   2

   Specify the start of the partition in MiB. If the value is too small, the installation will fail.

   3

   Specify a minimum size for the partition of 500 GB to ensure adequate disk space for precached images. If the value is too small, the deployments after installation will fail.

2. Convert the storage.bu to an Ignition file by running the following command:

   $ butane storage.bu

   Copy to Clipboard Toggle word wrap

   Example output

   {"ignition":{"version":"3.2.0"},"storage":{"disks":[{"device":"/dev/disk/by-path/pci-0000:00:17.0-ata-1.0","partitions":[{"label":"var-lib-containers","sizeMiB":0,"startMiB":250000}],"wipeTable":false}],"filesystems":[{"device":"/dev/disk/by-partlabel/var-lib-containers","format":"xfs","mountOptions":["defaults","prjquota"],"path":"/var/lib/containers","wipeFilesystem":true}]},"systemd":{"units":[{"contents":"# Generated by Butane\n[Unit]\nRequires=systemd-fsck@dev-disk-by\\x2dpartlabel-var\\x2dlib\\x2dcontainers.service\nAfter=systemd-fsck@dev-disk-by\\x2dpartlabel-var\\x2dlib\\x2dcontainers.service\n\n[Mount]\nWhere=/var/lib/containers\nWhat=/dev/disk/by-partlabel/var-lib-containers\nType=xfs\nOptions=defaults,prjquota\n\n[Install]\nRequiredBy=local-fs.target","enabled":true,"name":"var-lib-containers.mount"}]}}

   Copy to Clipboard Toggle word wrap

3. Copy the output into the .spec.clusters.nodes.ignitionConfigOverride field in the SiteConfig CR:

   [...]
   spec:
     clusters:
       - nodes:
           - hostName: <name>
             ignitionConfigOverride: '{"ignition":{"version":"3.2.0"},"storage":{"disks":[{"device":"/dev/disk/by-path/pci-0000:00:17.0-ata-1.0","partitions":[{"label":"var-lib-containers","sizeMiB":0,"startMiB":250000}],"wipeTable":false}],"filesystems":[{"device":"/dev/disk/by-partlabel/var-lib-containers","format":"xfs","mountOptions":["defaults","prjquota"],"path":"/var/lib/containers","wipeFilesystem":true}]},"systemd":{"units":[{"contents":"# Generated by Butane\n[Unit]\nRequires=systemd-fsck@dev-disk-by\\x2dpartlabel-var\\x2dlib\\x2dcontainers.service\nAfter=systemd-fsck@dev-disk-by\\x2dpartlabel-var\\x2dlib\\x2dcontainers.service\n\n[Mount]\nWhere=/var/lib/containers\nWhat=/dev/disk/by-partlabel/var-lib-containers\nType=xfs\nOptions=defaults,prjquota\n\n[Install]\nRequiredBy=local-fs.target","enabled":true,"name":"var-lib-containers.mount"}]}}'
   [...]

   Copy to Clipboard Toggle word wrap

Verification

1. During or after installation, verify on the hub cluster that the BareMetalHost object shows the annotation by running the following command:

   $ oc get bmh -n my-sno-ns my-sno -ojson | jq '.metadata.annotations["bmac.agent-install.openshift.io/ignition-config-overrides"]'

   Copy to Clipboard Toggle word wrap

   Example output

   "{\"ignition\":{\"version\":\"3.2.0\"},\"storage\":{\"disks\":[{\"device\":\"/dev/disk/by-path/pci-0000:00:17.0-ata-1.0\",\"partitions\":[{\"label\":\"var-lib-containers\",\"sizeMiB\":0,\"startMiB\":250000}],\"wipeTable\":false}],\"filesystems\":[{\"device\":\"/dev/disk/by-partlabel/var-lib-containers\",\"format\":\"xfs\",\"mountOptions\":[\"defaults\",\"prjquota\"],\"path\":\"/var/lib/containers\",\"wipeFilesystem\":true}]},\"systemd\":{\"units\":[{\"contents\":\"# Generated by Butane\\n[Unit]\\nRequires=systemd-fsck@dev-disk-by\\\\x2dpartlabel-var\\\\x2dlib\\\\x2dcontainers.service\\nAfter=systemd-fsck@dev-disk-by\\\\x2dpartlabel-var\\\\x2dlib\\\\x2dcontainers.service\\n\\n[Mount]\\nWhere=/var/lib/containers\\nWhat=/dev/disk/by-partlabel/var-lib-containers\\nType=xfs\\nOptions=defaults,prjquota\\n\\n[Install]\\nRequiredBy=local-fs.target\",\"enabled\":true,\"name\":\"var-lib-containers.mount\"}]}}"

   Copy to Clipboard Toggle word wrap

2. After installation, check the single-node OpenShift disk status by running the following commands:

   # lsblk

   Copy to Clipboard Toggle word wrap

   Example output

   NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
   sda      8:0    0 446.6G  0 disk
   ├─sda1   8:1    0     1M  0 part
   ├─sda2   8:2    0   127M  0 part
   ├─sda3   8:3    0   384M  0 part /boot
   ├─sda4   8:4    0 243.6G  0 part /var
   │                                /sysroot/ostree/deploy/rhcos/var
   │                                /usr
   │                                /etc
   │                                /
   │                                /sysroot
   └─sda5   8:5    0 202.5G  0 part /var/lib/containers

   Copy to Clipboard Toggle word wrap

   # df -h

   Copy to Clipboard Toggle word wrap

   Example output

   Filesystem      Size  Used Avail Use% Mounted on
   devtmpfs        4.0M     0  4.0M   0% /dev
   tmpfs           126G   84K  126G   1% /dev/shm
   tmpfs            51G   93M   51G   1% /run
   /dev/sda4       244G  5.2G  239G   3% /sysroot
   tmpfs           126G  4.0K  126G   1% /tmp
   /dev/sda5       203G  119G   85G  59% /var/lib/containers
   /dev/sda3       350M  110M  218M  34% /boot
   tmpfs            26G     0   26G   0% /run/user/1000

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a Namespace object YAML file for the Lifecycle Agent:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-lifecycle-agent
     annotations:
       workload.openshift.io/allowed: management

   Copy to Clipboard Toggle word wrap

   1. Create the Namespace CR by running the following command:

      $ oc create -f <namespace_filename>.yaml

      Copy to Clipboard Toggle word wrap

2. Create an OperatorGroup object YAML file for the Lifecycle Agent:

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: openshift-lifecycle-agent
     namespace: openshift-lifecycle-agent
   spec:
     targetNamespaces:
     - openshift-lifecycle-agent

   Copy to Clipboard Toggle word wrap

   1. Create the OperatorGroup CR by running the following command:

      $ oc create -f <operatorgroup_filename>.yaml

      Copy to Clipboard Toggle word wrap

3. Create a Subscription CR for the Lifecycle Agent:

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: openshift-lifecycle-agent-subscription
     namespace: openshift-lifecycle-agent
   spec:
     channel: "stable"
     name: lifecycle-agent
     source: redhat-operators
     sourceNamespace: openshift-marketplace

   Copy to Clipboard Toggle word wrap

   1. Create the Subscription CR by running the following command:

      $ oc create -f <subscription_filename>.yaml

      Copy to Clipboard Toggle word wrap

Verification

1. To verify that the installation succeeded, inspect the CSV resource by running the following command:

   $ oc get csv -n openshift-lifecycle-agent

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                              DISPLAY                     VERSION               REPLACES                           PHASE
   lifecycle-agent.v4.20.0           Openshift Lifecycle Agent   4.20.0                Succeeded

   Copy to Clipboard Toggle word wrap

2. Verify that the Lifecycle Agent is up and running by running the following command:

   $ oc get deploy -n openshift-lifecycle-agent

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                                 READY   UP-TO-DATE   AVAILABLE   AGE
   lifecycle-agent-controller-manager   1/1     1            1           14s

   Copy to Clipboard Toggle word wrap

Procedure

1. In the OpenShift Container Platform web console, navigate to Ecosystem Software Catalog.
2. Search for the Lifecycle Agent from the list of available Operators, and then click Install.
3. On the Install Operator page, under A specific namespace on the cluster select openshift-lifecycle-agent.
4. Click Install.

Verification

1. To confirm that the installation is successful:

   1. Click Ecosystem Installed Operators.

   2. Ensure that the Lifecycle Agent is listed in the openshift-lifecycle-agent project with a Status of InstallSucceeded.

      Note

      During installation an Operator might display a Failed status. If the installation later succeeds with an InstallSucceeded message, you can ignore the Failed message.

1. Click Ecosystem Installed Operators, and inspect the Operator Subscriptions and Install Plans tabs for any failure or errors under Status.
2. Click Workloads Pods, and check the logs for pods in the openshift-lifecycle-agent project.

Procedure

1. Extract the following CRs from the ztp-site-generate container image and push them to the source-cr directory:

   Example LcaSubscriptionNS.yaml file

   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-lifecycle-agent
     annotations:
       workload.openshift.io/allowed: management
       ran.openshift.io/ztp-deploy-wave: "2"
     labels:
       kubernetes.io/metadata.name: openshift-lifecycle-agent

   Copy to Clipboard Toggle word wrap

   Example LcaSubscriptionOperGroup.yaml file

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: lifecycle-agent-operatorgroup
     namespace: openshift-lifecycle-agent
     annotations:
       ran.openshift.io/ztp-deploy-wave: "2"
   spec:
     targetNamespaces:
       - openshift-lifecycle-agent

   Copy to Clipboard Toggle word wrap

   Example LcaSubscription.yaml file

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: lifecycle-agent
     namespace: openshift-lifecycle-agent
     annotations:
       ran.openshift.io/ztp-deploy-wave: "2"
   spec:
     channel: "stable"
     name: lifecycle-agent
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     installPlanApproval: Manual
   status:
     state: AtLatestKnown

   Copy to Clipboard Toggle word wrap

   Example directory structure

   ├── kustomization.yaml
   ├── sno
   │   ├── example-cnf.yaml
   │   ├── common-ranGen.yaml
   │   ├── group-du-sno-ranGen.yaml
   │   ├── group-du-sno-validator-ranGen.yaml
   │   └── ns.yaml
   ├── source-crs
   │   ├── LcaSubscriptionNS.yaml
   │   ├── LcaSubscriptionOperGroup.yaml
   │   ├── LcaSubscription.yaml

   Copy to Clipboard Toggle word wrap

2. Add the CRs to your common PolicyGenerator:

   apiVersion: policy.open-cluster-management.io/v1
   kind: PolicyGenerator
   metadata:
     name: common-latest
   placementBindingDefaults:
     name: common-placement-binding
   policyDefaults:
     namespace: ztp-common
     placement:
       labelSelector:
         common: "true"
         du-profile: "latest"
     remediationAction: inform
     severity: low
     namespaceSelector:
       exclude:
         - kube-*
       include:
         - '*'
     evaluationInterval:
       compliant: 10m
       noncompliant: 10s
   policies:
   - name: common-latest-subscriptions-policy
     policyAnnotations:
       ran.openshift.io/ztp-deploy-wave: "2"
     manifests:
       - path: source-crs/LcaSubscriptionNS.yaml
       - path: source-crs/LcaSubscriptionOperGroup.yaml
       - path: source-crs/LcaSubscription.yaml
   [...]

   Copy to Clipboard Toggle word wrap

Procedure

1. Extract the following CRs from the ztp-site-generate container image and push them to the source-cr directory:

   Example OadpSubscriptionNS.yaml file

   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-adp
     annotations:
       ran.openshift.io/ztp-deploy-wave: "2"
     labels:
       kubernetes.io/metadata.name: openshift-adp

   Copy to Clipboard Toggle word wrap

   Example OadpSubscriptionOperGroup.yaml file

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: redhat-oadp-operator
     namespace: openshift-adp
     annotations:
       ran.openshift.io/ztp-deploy-wave: "2"
   spec:
     targetNamespaces:
     - openshift-adp

   Copy to Clipboard Toggle word wrap

   Example OadpSubscription.yaml file

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: redhat-oadp-operator
     namespace: openshift-adp
     annotations:
       ran.openshift.io/ztp-deploy-wave: "2"
   spec:
     channel: stable-1.4
     name: redhat-oadp-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     installPlanApproval: Manual
   status:
     state: AtLatestKnown

   Copy to Clipboard Toggle word wrap

   Example OadpOperatorStatus.yaml file

   apiVersion: operators.coreos.com/v1
   kind: Operator
   metadata:
     name: redhat-oadp-operator.openshift-adp
     annotations:
       ran.openshift.io/ztp-deploy-wave: "2"
   status:
     components:
       refs:
       - kind: Subscription
         namespace: openshift-adp
         conditions:
         - type: CatalogSourcesUnhealthy
           status: "False"
       - kind: InstallPlan
         namespace: openshift-adp
         conditions:
         - type: Installed
           status: "True"
       - kind: ClusterServiceVersion
         namespace: openshift-adp
         conditions:
         - type: Succeeded
           status: "True"
           reason: InstallSucceeded

   Copy to Clipboard Toggle word wrap

   Example directory structure

   ├── kustomization.yaml
   ├── sno
   │   ├── example-cnf.yaml
   │   ├── common-ranGen.yaml
   │   ├── group-du-sno-ranGen.yaml
   │   ├── group-du-sno-validator-ranGen.yaml
   │   └── ns.yaml
   ├── source-crs
   │   ├── OadpSubscriptionNS.yaml
   │   ├── OadpSubscriptionOperGroup.yaml
   │   ├── OadpSubscription.yaml
   │   ├── OadpOperatorStatus.yaml

   Copy to Clipboard Toggle word wrap

2. Add the CRs to your common PolicyGenTemplate:

   apiVersion: ran.openshift.io/v1
   kind: PolicyGenTemplate
   metadata:
     name: "example-common-latest"
     namespace: "ztp-common"
   spec:
     bindingRules:
       common: "true"
       du-profile: "latest"
     sourceFiles:
       - fileName: OadpSubscriptionNS.yaml
         policyName: "subscriptions-policy"
       - fileName: OadpSubscriptionOperGroup.yaml
         policyName: "subscriptions-policy"
       - fileName: OadpSubscription.yaml
         policyName: "subscriptions-policy"
       - fileName: OadpOperatorStatus.yaml
         policyName: "subscriptions-policy"
   [...]

   Copy to Clipboard Toggle word wrap

3. Create the DataProtectionApplication CR and the S3 secret only for the target cluster:

   1. Extract the following CRs from the ztp-site-generate container image and push them to the source-cr directory:

      Example OadpDataProtectionApplication.yaml file

      apiVersion: oadp.openshift.io/v1alpha1
      kind: DataProtectionApplication
      metadata:
        name: dataprotectionapplication
        namespace: openshift-adp
        annotations:
          ran.openshift.io/ztp-deploy-wave: "100"
      spec:
        configuration:
          restic:
            enable: false 

      1

      
          velero:
            defaultPlugins:
              - aws
              - openshift
            resourceTimeout: 10m
        backupLocations:
          - velero:
              config:
                profile: "default"
                region: minio
                s3Url: $url
                insecureSkipTLSVerify: "true"
                s3ForcePathStyle: "true"
              provider: aws
              default: true
              credential:
                key: cloud
                name: cloud-credentials
              objectStorage:
                bucket: $bucketName 

      2

      
                prefix: $prefixName 

      3

      
      status:
        conditions:
        - reason: Complete
          status: "True"
          type: Reconciled

      Copy to Clipboard Toggle word wrap

      1

      The spec.configuration.restic.enable field must be set to false for an image-based upgrade because persistent volume contents are retained and reused after the upgrade.

      2 3

      The bucket defines the bucket name that is created in S3 backend. The prefix defines the name of the subdirectory that will be automatically created in the bucket. The combination of bucket and prefix must be unique for each target cluster to avoid interference between them. To ensure a unique storage directory for each target cluster, you can use the Red Hat Advanced Cluster Management hub template function, for example, prefix: {{hub .ManagedClusterName hub}}.

      Example OadpSecret.yaml file

      apiVersion: v1
      kind: Secret
      metadata:
        name: cloud-credentials
        namespace: openshift-adp
        annotations:
          ran.openshift.io/ztp-deploy-wave: "100"
      type: Opaque

      Copy to Clipboard Toggle word wrap

      Example OadpBackupStorageLocationStatus.yaml file

      apiVersion: velero.io/v1
      kind: BackupStorageLocation
      metadata:
        name: dataprotectionapplication-1 

      1

      
        namespace: openshift-adp
        annotations:
          ran.openshift.io/ztp-deploy-wave: "100"
      status:
        phase: Available

      Copy to Clipboard Toggle word wrap

      1

      The name value in the BackupStorageLocation resource must follow the <DataProtectionApplication.metadata.name>-<index> pattern. The <index> represents the position of the corresponding backupLocations entry in the spec.backupLocations field in the DataProtectionApplication resource. The position starts from 1. If the metadata.name value of the DataProtectionApplication resource is changed in the OadpDataProtectionApplication.yaml file, update the metadata.name field in the BackupStorageLocation resource accordingly.

      The OadpBackupStorageLocationStatus.yaml CR verifies the availability of backup storage locations created by OADP.

   2. Add the CRs to your site PolicyGenTemplate with overrides:

      apiVersion: ran.openshift.io/v1
      kind: PolicyGenTemplate
      metadata:
        name: "example-cnf"
        namespace: "ztp-site"
      spec:
        bindingRules:
          sites: "example-cnf"
          du-profile: "latest"
        mcp: "master"
        sourceFiles:
          ...
          - fileName: OadpSecret.yaml
            policyName: "config-policy"
            data:
              cloud: <your_credentials> 

      1

      
          - fileName: OadpDataProtectionApplication.yaml 

      2

      
            policyName: "config-policy"
            spec:
              backupLocations:
                - velero:
                    config:
                      region: minio
                      s3Url: <your_S3_URL> 

      3

      
                      profile: "default"
                      insecureSkipTLSVerify: "true"
                      s3ForcePathStyle: "true"
                    provider: aws
                    default: true
                    credential:
                      key: cloud
                      name: cloud-credentials
                    objectStorage:
                      bucket: <your_bucket_name> 

      4

      
                      prefix: <cluster_name> 

      5

      
          - fileName: OadpBackupStorageLocationStatus.yaml
            policyName: "config-policy"

      Copy to Clipboard Toggle word wrap

      1

      Specify your credentials for your S3 storage backend.

      2

      If more than one backupLocations entries are defined in the OadpDataProtectionApplication CR, ensure that each location has a corresponding OadpBackupStorageLocation CR added for status tracking. Ensure that the name of each additional OadpBackupStorageLocation CR is overridden with the correct index as described in the example OadpBackupStorageLocationStatus.yaml file.

      3

      Specify the URL for your S3-compatible bucket.

      4 5

      The bucket defines the bucket name that is created in S3 backend. The prefix defines the name of the subdirectory that will be automatically created in the bucket. The combination of bucket and prefix must be unique for each target cluster to avoid interference between them. To ensure a unique storage directory for each target cluster, you can use the Red Hat Advanced Cluster Management hub template function, for example, prefix: {{hub .ManagedClusterName hub}}.

1. If the seed cluster is installed in a disconnected environment, the target clusters must also be installed in a disconnected environment.
2. The proxy configuration must be either enabled or disabled in both the seed and target clusters. However, the proxy servers configured on the clusters does not have to match.

Procedure

1. Detach the managed cluster from the hub to delete any RHACM-specific resources from the seed cluster that must not be in the seed image:

   1. Manually detach the seed cluster by running the following command:

      $ oc delete managedcluster sno-worker-example

      Copy to Clipboard Toggle word wrap
      1. Wait until the managed cluster is removed. After the cluster is removed, create the proper SeedGenerator CR. The Lifecycle Agent cleans up the RHACM artifacts.

   2. If you are using GitOps ZTP, detach your cluster by removing the seed cluster’s SiteConfig CR from the kustomization.yaml.

      1. If you have a kustomization.yaml file that references multiple SiteConfig CRs, remove your seed cluster’s SiteConfig CR from the kustomization.yaml:

         apiVersion: kustomize.config.k8s.io/v1beta1
         kind: Kustomization
         
         generators:
         #- example-seed-sno1.yaml
         - example-target-sno2.yaml
         - example-target-sno3.yaml

         Copy to Clipboard Toggle word wrap

      2. If you have a kustomization.yaml that references one SiteConfig CR, remove your seed cluster’s SiteConfig CR from the kustomization.yaml and add the generators: {} line:

         apiVersion: kustomize.config.k8s.io/v1beta1
         kind: Kustomization
         
         generators: {}

         Copy to Clipboard Toggle word wrap

      3. Commit the kustomization.yaml changes in your Git repository and push the changes to your repository.

         The ArgoCD pipeline detects the changes and removes the managed cluster.

2. Create the Secret object so that you can push the seed image to your registry.

   1. Create the authentication file by running the following commands:

      $ MY_USER=myuserid
      $ AUTHFILE=/tmp/my-auth.json
      $ podman login --authfile ${AUTHFILE} -u ${MY_USER} quay.io/${MY_USER}

      Copy to Clipboard Toggle word wrap

      $ base64 -w 0 ${AUTHFILE} ; echo

      Copy to Clipboard Toggle word wrap

   2. Copy the output into the seedAuth field in the Secret YAML file named seedgen in the openshift-lifecycle-agent namespace:

      apiVersion: v1
      kind: Secret
      metadata:
        name: seedgen 

      1

      
        namespace: openshift-lifecycle-agent
      type: Opaque
      data:
        seedAuth: <encoded_AUTHFILE> 

      2

      Copy to Clipboard Toggle word wrap

      1

      The Secret resource must have the name: seedgen and namespace: openshift-lifecycle-agent fields.

      2

      Specifies a base64-encoded authfile for write-access to the registry for pushing the generated seed images.

   3. Apply the Secret by running the following command:

      $ oc apply -f secretseedgenerator.yaml

      Copy to Clipboard Toggle word wrap

3. Create the SeedGenerator CR:

   apiVersion: lca.openshift.io/v1
   kind: SeedGenerator
   metadata:
     name: seedimage 

   1

   
   spec:
     seedImage: <seed_container_image> 

   2

   Copy to Clipboard Toggle word wrap

   1

   The SeedGenerator CR must be named seedimage.

   2

   Specify the container image URL, for example, quay.io/example/seed-container-image:<tag>. It is recommended to use the <seed_cluster_name>:<ocp_version> format.

4. Generate the seed image by running the following command:

   $ oc apply -f seedgenerator.yaml

   Copy to Clipboard Toggle word wrap
   Important

   The cluster reboots and loses API capabilities while the Lifecycle Agent generates the seed image. Applying the SeedGenerator CR stops the kubelet and the CRI-O operations, then it starts the image generation.

Procedure

1. Create the OADP Backup and Restore CRs for platform artifacts in the same namespace where the OADP Operator is installed, which is openshift-adp.

   1. If the target cluster is managed by RHACM, add the following YAML file for backing up and restoring RHACM artifacts:

      PlatformBackupRestore.yaml for RHACM

      apiVersion: velero.io/v1
      kind: Backup
      metadata:
        name: acm-klusterlet
        annotations:
          lca.openshift.io/apply-label: "apps/v1/deployments/open-cluster-management-agent/klusterlet,v1/secrets/open-cluster-management-agent/bootstrap-hub-kubeconfig,rbac.authorization.k8s.io/v1/clusterroles/klusterlet,v1/serviceaccounts/open-cluster-management-agent/klusterlet,scheduling.k8s.io/v1/priorityclasses/klusterlet-critical,rbac.authorization.k8s.io/v1/clusterroles/open-cluster-management:klusterlet-admin-aggregate-clusterrole,rbac.authorization.k8s.io/v1/clusterrolebindings/klusterlet,operator.open-cluster-management.io/v1/klusterlets/klusterlet,apiextensions.k8s.io/v1/customresourcedefinitions/klusterlets.operator.open-cluster-management.io,v1/secrets/open-cluster-management-agent/open-cluster-management-image-pull-credentials" 

      1

      
        labels:
          velero.io/storage-location: default
        namespace: openshift-adp
      spec:
        includedNamespaces:
        - open-cluster-management-agent
        includedClusterScopedResources:
        - klusterlets.operator.open-cluster-management.io
        - clusterroles.rbac.authorization.k8s.io
        - clusterrolebindings.rbac.authorization.k8s.io
        - priorityclasses.scheduling.k8s.io
        includedNamespaceScopedResources:
        - deployments
        - serviceaccounts
        - secrets
        excludedNamespaceScopedResources: []
      ---
      apiVersion: velero.io/v1
      kind: Restore
      metadata:
        name: acm-klusterlet
        namespace: openshift-adp
        labels:
          velero.io/storage-location: default
        annotations:
          lca.openshift.io/apply-wave: "1"
      spec:
        backupName:
          acm-klusterlet

      Copy to Clipboard Toggle word wrap

      1

      If your multiclusterHub CR does not have .spec.imagePullSecret defined and the secret does not exist on the open-cluster-management-agent namespace in your hub cluster, remove v1/secrets/open-cluster-management-agent/open-cluster-management-image-pull-credentials.

   2. If you created persistent volumes on your cluster through LVM Storage, add the following YAML file for LVM Storage artifacts:

      PlatformBackupRestoreLvms.yaml for LVM Storage

      apiVersion: velero.io/v1
      kind: Backup
      metadata:
        labels:
          velero.io/storage-location: default
        name: lvmcluster
        namespace: openshift-adp
      spec:
        includedNamespaces:
          - openshift-storage
        includedNamespaceScopedResources:
          - lvmclusters
          - lvmvolumegroups
          - lvmvolumegroupnodestatuses
      ---
      apiVersion: velero.io/v1
      kind: Restore
      metadata:
        name: lvmcluster
        namespace: openshift-adp
        labels:
          velero.io/storage-location: default
        annotations:
          lca.openshift.io/apply-wave: "2" 

      1

      
      spec:
        backupName:
          lvmcluster

      Copy to Clipboard Toggle word wrap

      1

      The lca.openshift.io/apply-wave value must be lower than the values specified in the application Restore CRs.

2. If you need to restore applications after the upgrade, create the OADP Backup and Restore CRs for your application in the openshift-adp namespace.

   1. Create the OADP CRs for cluster-scoped application artifacts in the openshift-adp namespace.

      Example OADP CRs for cluster-scoped application artifacts for LSO and LVM Storage

      apiVersion: velero.io/v1
      kind: Backup
      metadata:
        annotations:
          lca.openshift.io/apply-label: "apiextensions.k8s.io/v1/customresourcedefinitions/test.example.com,security.openshift.io/v1/securitycontextconstraints/test,rbac.authorization.k8s.io/v1/clusterroles/test-role,rbac.authorization.k8s.io/v1/clusterrolebindings/system:openshift:scc:test" 

      1

      
        name: backup-app-cluster-resources
        labels:
          velero.io/storage-location: default
        namespace: openshift-adp
      spec:
        includedClusterScopedResources:
        - customresourcedefinitions
        - securitycontextconstraints
        - clusterrolebindings
        - clusterroles
        excludedClusterScopedResources:
        - Namespace
      ---
      apiVersion: velero.io/v1
      kind: Restore
      metadata:
        name: test-app-cluster-resources
        namespace: openshift-adp
        labels:
          velero.io/storage-location: default
        annotations:
          lca.openshift.io/apply-wave: "3" 

      2

      
      spec:
        backupName:
          backup-app-cluster-resources

      Copy to Clipboard Toggle word wrap

      1

      Replace the example resource name with your actual resources.

      2

      The lca.openshift.io/apply-wave value must be higher than the value in the platform Restore CRs and lower than the value in the application namespace-scoped Restore CR.

   2. Create the OADP CRs for your namespace-scoped application artifacts.

      Example OADP CRs namespace-scoped application artifacts when LSO is used

      apiVersion: velero.io/v1
      kind: Backup
      metadata:
        labels:
          velero.io/storage-location: default
        name: backup-app
        namespace: openshift-adp
      spec:
        includedNamespaces:
        - test
        includedNamespaceScopedResources:
        - secrets
        - persistentvolumeclaims
        - deployments
        - statefulsets
        - configmaps
        - cronjobs
        - services
        - job
        - poddisruptionbudgets
        - <application_custom_resources> 

      1

      
        excludedClusterScopedResources:
        - persistentVolumes
      ---
      apiVersion: velero.io/v1
      kind: Restore
      metadata:
        name: test-app
        namespace: openshift-adp
        labels:
          velero.io/storage-location: default
        annotations:
          lca.openshift.io/apply-wave: "4"
      spec:
        backupName:
          backup-app

      Copy to Clipboard Toggle word wrap

      1

      Define custom resources for your application.

      Example OADP CRs namespace-scoped application artifacts when LVM Storage is used

      apiVersion: velero.io/v1
      kind: Backup
      metadata:
        labels:
          velero.io/storage-location: default
        name: backup-app
        namespace: openshift-adp
      spec:
        includedNamespaces:
        - test
        includedNamespaceScopedResources:
        - secrets
        - persistentvolumeclaims
        - deployments
        - statefulsets
        - configmaps
        - cronjobs
        - services
        - job
        - poddisruptionbudgets
        - <application_custom_resources> 

      1

      
        includedClusterScopedResources:
        - persistentVolumes 

      2

      
        - logicalvolumes.topolvm.io 

      3

      
        - volumesnapshotcontents 

      4

      
      ---
      apiVersion: velero.io/v1
      kind: Restore
      metadata:
        name: test-app
        namespace: openshift-adp
        labels:
          velero.io/storage-location: default
        annotations:
          lca.openshift.io/apply-wave: "4"
      spec:
        backupName:
          backup-app
        restorePVs: true
        restoreStatus:
          includedResources:
          - logicalvolumes 

      5

      Copy to Clipboard Toggle word wrap

      1

      Define custom resources for your application.

      2

      Required field.

      3

      Required field

      4

      Optional if you use LVM Storage volume snapshots.

      5

      Required field.

      Important

      The same version of the applications must function on both the current and the target release of OpenShift Container Platform.

3. Create the ConfigMap object for your OADP CRs by running the following command:

   $ oc create configmap oadp-cm-example --from-file=example-oadp-resources.yaml=<path_to_oadp_crs> -n openshift-adp

   Copy to Clipboard Toggle word wrap

4. Patch the ImageBasedUpgrade CR by running the following command:

   $ oc patch imagebasedupgrades.lca.openshift.io upgrade \
     -p='{"spec": {"oadpContent": [{"name": "oadp-cm-example", "namespace": "openshift-adp"}]}}' \
     --type=merge -n openshift-lifecycle-agent

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a YAML file that contains your extra manifests, such as SR-IOV.

   Example SR-IOV resources

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetworkNodePolicy
   metadata:
     name: "example-sriov-node-policy"
     namespace: openshift-sriov-network-operator
   spec:
     deviceType: vfio-pci
     isRdma: false
     nicSelector:
       pfNames: [ens1f0]
     nodeSelector:
       node-role.kubernetes.io/master: ""
     mtu: 1500
     numVfs: 8
     priority: 99
     resourceName: example-sriov-node-policy
   ---
   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetwork
   metadata:
     name: "example-sriov-network"
     namespace: openshift-sriov-network-operator
   spec:
     ipam: |-
       {
       }
     linkState: auto
     networkNamespace: sriov-namespace
     resourceName: example-sriov-node-policy
     spoofChk: "on"
     trust: "off"

   Copy to Clipboard Toggle word wrap

2. Create the ConfigMap object by running the following command:

   $ oc create configmap example-extra-manifests-cm --from-file=example-extra-manifests.yaml=<path_to_extramanifest> -n openshift-lifecycle-agent

   Copy to Clipboard Toggle word wrap

3. Patch the ImageBasedUpgrade CR by running the following command:

   $ oc patch imagebasedupgrades.lca.openshift.io upgrade \
     -p='{"spec": {"extraManifests": [{"name": "example-extra-manifests-cm", "namespace": "openshift-lifecycle-agent"}]}}' \
     --type=merge -n openshift-lifecycle-agent

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a YAML file that contains the CatalogSource CR:

   apiVersion: operators.coreos.com/v1
   kind: CatalogSource
   metadata:
     name: example-catalogsources
     namespace: openshift-marketplace
   spec:
     sourceType: grpc
     displayName: disconnected-redhat-operators
     image: quay.io/example-org/example-catalog:v1

   Copy to Clipboard Toggle word wrap

2. Create the ConfigMap object by running the following command:

   $ oc create configmap example-catalogsources-cm --from-file=example-catalogsources.yaml=<path_to_catalogsource_cr> -n openshift-lifecycle-agent

   Copy to Clipboard Toggle word wrap

3. Patch the ImageBasedUpgrade CR by running the following command:

   $ oc patch imagebasedupgrades.lca.openshift.io upgrade \
     -p='{"spec": {"extraManifests": [{"name": "example-catalogsources-cm", "namespace": "openshift-lifecycle-agent"}]}}' \
     --type=merge -n openshift-lifecycle-agent

   Copy to Clipboard Toggle word wrap

Procedure

1. Ensure that your Git repository that you use with the ArgoCD policies application contains the following directory structure:

   ├── source-crs/
   │   ├── ibu/
   │   │    ├── ImageBasedUpgrade.yaml
   │   │    ├── PlatformBackupRestore.yaml
   │   │    ├── PlatformBackupRestoreLvms.yaml
   │   │    ├── PlatformBackupRestoreWithIBGU.yaml
   ├── ...
   ├── kustomization.yaml

   Copy to Clipboard Toggle word wrap

   The source-crs/ibu/PlatformBackupRestoreWithIBGU.yaml file is provided in the ZTP container image.

   PlatformBackupRestoreWithIBGU.yaml

   apiVersion: velero.io/v1
   kind: Backup
   metadata:
     name: acm-klusterlet
     annotations:
       lca.openshift.io/apply-label: "apps/v1/deployments/open-cluster-management-agent/klusterlet,v1/secrets/open-cluster-management-agent/bootstrap-hub-kubeconfig,rbac.authorization.k8s.io/v1/clusterroles/klusterlet,v1/serviceaccounts/open-cluster-management-agent/klusterlet,scheduling.k8s.io/v1/priorityclasses/klusterlet-critical,rbac.authorization.k8s.io/v1/clusterroles/open-cluster-management:klusterlet-work:ibu-role,rbac.authorization.k8s.io/v1/clusterroles/open-cluster-management:klusterlet-admin-aggregate-clusterrole,rbac.authorization.k8s.io/v1/clusterrolebindings/klusterlet,operator.open-cluster-management.io/v1/klusterlets/klusterlet,apiextensions.k8s.io/v1/customresourcedefinitions/klusterlets.operator.open-cluster-management.io,v1/secrets/open-cluster-management-agent/open-cluster-management-image-pull-credentials" 

   1

   
     labels:
       velero.io/storage-location: default
     namespace: openshift-adp
   spec:
     includedNamespaces:
     - open-cluster-management-agent
     includedClusterScopedResources:
     - klusterlets.operator.open-cluster-management.io
     - clusterroles.rbac.authorization.k8s.io
     - clusterrolebindings.rbac.authorization.k8s.io
     - priorityclasses.scheduling.k8s.io
     includedNamespaceScopedResources:
     - deployments
     - serviceaccounts
     - secrets
     excludedNamespaceScopedResources: []
   ---
   apiVersion: velero.io/v1
   kind: Restore
   metadata:
     name: acm-klusterlet
     namespace: openshift-adp
     labels:
       velero.io/storage-location: default
     annotations:
       lca.openshift.io/apply-wave: "1"
   spec:
     backupName:
       acm-klusterlet

   Copy to Clipboard Toggle word wrap

   1

   If your multiclusterHub CR does not have .spec.imagePullSecret defined and the secret does not exist on the open-cluster-management-agent namespace in your hub cluster, remove v1/secrets/open-cluster-management-agent/open-cluster-management-image-pull-credentials.

   Note

   If you perform the image-based upgrade directly on managed clusters, use the PlatformBackupRestore.yaml file.

   If you use LVM Storage to create persistent volumes, you can use the source-crs/ibu/PlatformBackupRestoreLvms.yaml provided in the ZTP container image to back up your LVM Storage resources.

   PlatformBackupRestoreLvms.yaml

   apiVersion: velero.io/v1
   kind: Backup
   metadata:
     labels:
       velero.io/storage-location: default
     name: lvmcluster
     namespace: openshift-adp
   spec:
     includedNamespaces:
       - openshift-storage
     includedNamespaceScopedResources:
       - lvmclusters
       - lvmvolumegroups
       - lvmvolumegroupnodestatuses
   ---
   apiVersion: velero.io/v1
   kind: Restore
   metadata:
     name: lvmcluster
     namespace: openshift-adp
     labels:
       velero.io/storage-location: default
     annotations:
       lca.openshift.io/apply-wave: "2" 

   1

   
   spec:
     backupName:
       lvmcluster

   Copy to Clipboard Toggle word wrap

   1

   The lca.openshift.io/apply-wave value must be lower than the values specified in the application Restore CRs.

2. If you need to restore applications after the upgrade, create the OADP Backup and Restore CRs for your application in the openshift-adp namespace:

   1. Create the OADP CRs for cluster-scoped application artifacts in the openshift-adp namespace:

      Example OADP CRs for cluster-scoped application artifacts for LSO and LVM Storage

      apiVersion: velero.io/v1
      kind: Backup
      metadata:
        annotations:
          lca.openshift.io/apply-label: "apiextensions.k8s.io/v1/customresourcedefinitions/test.example.com,security.openshift.io/v1/securitycontextconstraints/test,rbac.authorization.k8s.io/v1/clusterroles/test-role,rbac.authorization.k8s.io/v1/clusterrolebindings/system:openshift:scc:test" 

      1

      
        name: backup-app-cluster-resources
        labels:
          velero.io/storage-location: default
        namespace: openshift-adp
      spec:
        includedClusterScopedResources:
        - customresourcedefinitions
        - securitycontextconstraints
        - clusterrolebindings
        - clusterroles
        excludedClusterScopedResources:
        - Namespace
      ---
      apiVersion: velero.io/v1
      kind: Restore
      metadata:
        name: test-app-cluster-resources
        namespace: openshift-adp
        labels:
          velero.io/storage-location: default
        annotations:
          lca.openshift.io/apply-wave: "3" 

      2

      
      spec:
        backupName:
          backup-app-cluster-resources

      Copy to Clipboard Toggle word wrap

      1

      Replace the example resource name with your actual resources.

      2

      The lca.openshift.io/apply-wave value must be higher than the value in the platform Restore CRs and lower than the value in the application namespace-scoped Restore CR.

   2. Create the OADP CRs for your namespace-scoped application artifacts in the source-crs/custom-crs directory:

      Example OADP CRs namespace-scoped application artifacts when LSO is used

      apiVersion: velero.io/v1
      kind: Backup
      metadata:
        labels:
          velero.io/storage-location: default
        name: backup-app
        namespace: openshift-adp
      spec:
        includedNamespaces:
        - test
        includedNamespaceScopedResources:
        - secrets
        - persistentvolumeclaims
        - deployments
        - statefulsets
        - configmaps
        - cronjobs
        - services
        - job
        - poddisruptionbudgets
        - <application_custom_resources> 

      1

      
        excludedClusterScopedResources:
        - persistentVolumes
      ---
      apiVersion: velero.io/v1
      kind: Restore
      metadata:
        name: test-app
        namespace: openshift-adp
        labels:
          velero.io/storage-location: default
        annotations:
          lca.openshift.io/apply-wave: "4"
      spec:
        backupName:
          backup-app

      Copy to Clipboard Toggle word wrap

      1

      Define custom resources for your application.

      Example OADP CRs namespace-scoped application artifacts when LVM Storage is used

      apiVersion: velero.io/v1
      kind: Backup
      metadata:
        labels:
          velero.io/storage-location: default
        name: backup-app
        namespace: openshift-adp
      spec:
        includedNamespaces:
        - test
        includedNamespaceScopedResources:
        - secrets
        - persistentvolumeclaims
        - deployments
        - statefulsets
        - configmaps
        - cronjobs
        - services
        - job
        - poddisruptionbudgets
        - <application_custom_resources> 

      1

      
        includedClusterScopedResources:
        - persistentVolumes 

      2

      
        - logicalvolumes.topolvm.io 

      3

      
        - volumesnapshotcontents 

      4

      
      ---
      apiVersion: velero.io/v1
      kind: Restore
      metadata:
        name: test-app
        namespace: openshift-adp
        labels:
          velero.io/storage-location: default
        annotations:
          lca.openshift.io/apply-wave: "4"
      spec:
        backupName:
          backup-app
        restorePVs: true
        restoreStatus:
          includedResources:
          - logicalvolumes 

      5

      Copy to Clipboard Toggle word wrap

      1

      Define custom resources for your application.

      2

      Required field.

      3

      Required field

      4

      Optional if you use LVM Storage volume snapshots.

      5

      Required field.

      Important

      The same version of the applications must function on both the current and the target release of OpenShift Container Platform.

3. Create a kustomization.yaml with the following content:

   apiVersion: kustomize.config.k8s.io/v1beta1
   kind: Kustomization
   
   configMapGenerator: 

   1

   
   - files:
     - source-crs/ibu/PlatformBackupRestoreWithIBGU.yaml
     #- source-crs/custom-crs/ApplicationClusterScopedBackupRestore.yaml
     #- source-crs/custom-crs/ApplicationApplicationBackupRestoreLso.yaml
     name: oadp-cm
     namespace: openshift-adp 

   2

   
   generatorOptions:
     disableNameSuffixHash: true

   Copy to Clipboard Toggle word wrap

   1

   Creates the oadp-cm ConfigMap object on the hub cluster with Backup and Restore CRs.

   2

   The namespace must exist on all managed clusters and the hub for the OADP ConfigMap to be generated and copied to the clusters.

4. Push the changes to your Git repository.

Procedure

1. Label your required extra manifests with the lca.openshift.io/target-ocp-version: <target_version> label in your existing site PolicyGenTemplate CR:

   apiVersion: ran.openshift.io/v1
   kind: PolicyGenTemplate
   metadata:
     name: example-sno
   spec:
     bindingRules:
       sites: "example-sno"
       du-profile: "4.15"
     mcp: "master"
     sourceFiles:
       - fileName: SriovNetwork.yaml
         policyName: "config-policy"
         metadata:
           name: "sriov-nw-du-fh"
           labels:
             lca.openshift.io/target-ocp-version: "4.15" 

   1

   
         spec:
           resourceName: du_fh
           vlan: 140
       - fileName: SriovNetworkNodePolicy.yaml
         policyName: "config-policy"
         metadata:
           name: "sriov-nnp-du-fh"
           labels:
             lca.openshift.io/target-ocp-version: "4.15"
         spec:
           deviceType: netdevice
           isRdma: false
           nicSelector:
             pfNames: ["ens5f0"]
           numVfs: 8
           priority: 10
           resourceName: du_fh
       - fileName: SriovNetwork.yaml
         policyName: "config-policy"
         metadata:
           name: "sriov-nw-du-mh"
           labels:
             lca.openshift.io/target-ocp-version: "4.15"
         spec:
           resourceName: du_mh
           vlan: 150
       - fileName: SriovNetworkNodePolicy.yaml
         policyName: "config-policy"
         metadata:
           name: "sriov-nnp-du-mh"
           labels:
             lca.openshift.io/target-ocp-version: "4.15"
         spec:
           deviceType: vfio-pci
           isRdma: false
           nicSelector:
             pfNames: ["ens7f0"]
           numVfs: 8
           priority: 10
           resourceName: du_mh
       - fileName: DefaultCatsrc.yaml 

   2

   
         policyName: "config-policy"
         metadata:
           name: default-cat-source
           namespace: openshift-marketplace
           labels:
               lca.openshift.io/target-ocp-version: "4.15"
         spec:
             displayName: default-cat-source
             image: quay.io/example-org/example-catalog:v1

   Copy to Clipboard Toggle word wrap

   1

   Ensure that the lca.openshift.io/target-ocp-version label matches either the y-stream or the z-stream of the target OpenShift Container Platform version that is specified in the spec.seedImageRef.version field of the ImageBasedUpgrade CR. The Lifecycle Agent only applies the CRs that match the specified version.

   2

   If you do not want to use custom catalog sources, remove this entry.

2. Push the changes to your Git repository.

Procedure

1. Increase the threshold to 65% by running the following command:

   $ oc -n openshift-lifecycle-agent annotate ibu upgrade image-cleanup.lca.openshift.io/disk-usage-threshold-percent='65'

   Copy to Clipboard Toggle word wrap

2. (Optional) Remove the threshold override by running the following command:

   $ oc -n  openshift-lifecycle-agent annotate ibu upgrade image-cleanup.lca.openshift.io/disk-usage-threshold-percent-

   Copy to Clipboard Toggle word wrap

Procedure

1. Disable the automatic image cleanup by running the following command:

   $ oc -n openshift-lifecycle-agent annotate ibu upgrade image-cleanup.lca.openshift.io/on-prep='Disabled'

   Copy to Clipboard Toggle word wrap

2. (Optional) Enable automatic image cleanup again by running the following command:

   $ oc -n  openshift-lifecycle-agent annotate ibu upgrade image-cleanup.lca.openshift.io/on-prep-

   Copy to Clipboard Toggle word wrap