15.2. 为单节点 OpenShift 集群准备基于镜像的升级

流程

1. 创建 storage.bu 文件：

   variant: fcos
   version: 1.3.0
   storage:
     disks:
     - device: /dev/disk/by-id/wwn-<root_disk> 1
       wipe_table: false
       partitions:
       - label: var-lib-containers
         start_mib: <start_of_partition> 2
         size_mib: <partition_size> 3
     filesystems:
       - path: /var/lib/containers
         device: /dev/disk/by-partlabel/var-lib-containers
         format: xfs
         wipe_filesystem: true
         with_mount_unit: true
         mount_options:
           - defaults
           - prjquota

   1

   指定根磁盘。

   2

   以 MiB 为单位指定分区的起始位置。如果该值太小，则安装将失败。

   3

   为 500 GB 分区指定最小大小，以确保为 precached 镜像有足够的磁盘空间。如果值太小，则安装后部署将失败。

2. 将 storage.bu 转换为 Ignition 文件：

   $ butane storage.bu

   输出示例

   {"ignition":{"version":"3.2.0"},"storage":{"disks":[{"device":"/dev/disk/by-id/wwn-0x6b07b250ebb9d0002a33509f24af1f62","partitions":[{"label":"var-lib-containers","sizeMiB":0,"startMiB":250000}],"wipeTable":false}],"filesystems":[{"device":"/dev/disk/by-partlabel/var-lib-containers","format":"xfs","mountOptions":["defaults","prjquota"],"path":"/var/lib/containers","wipeFilesystem":true}]},"systemd":{"units":[{"contents":"# Generated by Butane\n[Unit]\nRequires=systemd-fsck@dev-disk-by\\x2dpartlabel-var\\x2dlib\\x2dcontainers.service\nAfter=systemd-fsck@dev-disk-by\\x2dpartlabel-var\\x2dlib\\x2dcontainers.service\n\n[Mount]\nWhere=/var/lib/containers\nWhat=/dev/disk/by-partlabel/var-lib-containers\nType=xfs\nOptions=defaults,prjquota\n\n[Install]\nRequiredBy=local-fs.target","enabled":true,"name":"var-lib-containers.mount"}]}}

3. 将输出复制到 SiteConfig CR 中的 .spec.clusters.nodes.ignitionConfigOverride 字段中：

   [...]
   spec:
     clusters:
       - nodes:
           - ignitionConfigOverride: '{"ignition":{"version":"3.2.0"},"storage":{"disks":[{"device":"/dev/disk/by-id/wwn-0x6b07b250ebb9d0002a33509f24af1f62","partitions":[{"label":"var-lib-containers","sizeMiB":0,"startMiB":250000}],"wipeTable":false}],"filesystems":[{"device":"/dev/disk/by-partlabel/var-lib-containers","format":"xfs","mountOptions":["defaults","prjquota"],"path":"/var/lib/containers","wipeFilesystem":true}]},"systemd":{"units":[{"contents":"# Generated by Butane\n[Unit]\nRequires=systemd-fsck@dev-disk-by\\x2dpartlabel-var\\x2dlib\\x2dcontainers.service\nAfter=systemd-fsck@dev-disk-by\\x2dpartlabel-var\\x2dlib\\x2dcontainers.service\n\n[Mount]\nWhere=/var/lib/containers\nWhat=/dev/disk/by-partlabel/var-lib-containers\nType=xfs\nOptions=defaults,prjquota\n\n[Install]\nRequiredBy=local-fs.target","enabled":true,"name":"var-lib-containers.mount"}]}}'
   [...]

验证

1. 在安装过程中，在 hub 集群上验证 BareMetalHost 对象显示注解：

   $ oc get bmh -n my-sno-ns my-sno -ojson | jq '.metadata.annotations["bmac.agent-install.openshift.io/ignition-config-overrides"]

   输出示例

   "{\"ignition\":{\"version\":\"3.2.0\"},\"storage\":{\"disks\":[{\"device\":\"/dev/disk/by-id/wwn-0x6b07b250ebb9d0002a33509f24af1f62\",\"partitions\":[{\"label\":\"var-lib-containers\",\"sizeMiB\":0,\"startMiB\":250000}],\"wipeTable\":false}],\"filesystems\":[{\"device\":\"/dev/disk/by-partlabel/var-lib-containers\",\"format\":\"xfs\",\"mountOptions\":[\"defaults\",\"prjquota\"],\"path\":\"/var/lib/containers\",\"wipeFilesystem\":true}]},\"systemd\":{\"units\":[{\"contents\":\"# Generated by Butane\\n[Unit]\\nRequires=systemd-fsck@dev-disk-by\\\\x2dpartlabel-var\\\\x2dlib\\\\x2dcontainers.service\\nAfter=systemd-fsck@dev-disk-by\\\\x2dpartlabel-var\\\\x2dlib\\\\x2dcontainers.service\\n\\n[Mount]\\nWhere=/var/lib/containers\\nWhat=/dev/disk/by-partlabel/var-lib-containers\\nType=xfs\\nOptions=defaults,prjquota\\n\\n[Install]\\nRequiredBy=local-fs.target\",\"enabled\":true,\"name\":\"var-lib-containers.mount\"}]}}"

2. 安装后，检查单节点 OpenShift 磁盘状态：

   # lsblk

   输出示例

   NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
   sda      8:0    0 446.6G  0 disk
   ├─sda1   8:1    0     1M  0 part
   ├─sda2   8:2    0   127M  0 part
   ├─sda3   8:3    0   384M  0 part /boot
   ├─sda4   8:4    0 243.6G  0 part /var
   │                                /sysroot/ostree/deploy/rhcos/var
   │                                /usr
   │                                /etc
   │                                /
   │                                /sysroot
   └─sda5   8:5    0 202.5G  0 part /var/lib/containers

   # df -h

   输出示例

   Filesystem      Size  Used Avail Use% Mounted on
   devtmpfs        4.0M     0  4.0M   0% /dev
   tmpfs           126G   84K  126G   1% /dev/shm
   tmpfs            51G   93M   51G   1% /run
   /dev/sda4       244G  5.2G  239G   3% /sysroot
   tmpfs           126G  4.0K  126G   1% /tmp
   /dev/sda5       203G  119G   85G  59% /var/lib/containers
   /dev/sda3       350M  110M  218M  34% /boot
   tmpfs            26G     0   26G   0% /run/user/1000

流程

1. 为 Lifecycle Agent 创建一个 Namespace 对象 YAML 文件，如 lcao-namespace.yaml ：

   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-lifecycle-agent
     annotations:
       workload.openshift.io/allowed: management

   1. 运行以下命令来创建 Namespace CR：

      $ oc create -f lcao-namespace.yaml

2. 为 Lifecycle Agent 创建 OperatorGroup 对象 YAML 文件，如 lcao-operatorgroup.yaml ：

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: openshift-lifecycle-agent
     namespace: openshift-lifecycle-agent
   spec:
     targetNamespaces:
     - openshift-lifecycle-agent

   1. 运行以下命令来创建 OperatorGroup CR:

      $ oc create -f lcao-operatorgroup.yaml

3. 创建一个 Subscription CR，如 lcao-subscription.yaml ：

   apiVersion: operators.coreos.com/v1
   kind: Subscription
   metadata:
     name: openshift-lifecycle-agent-subscription
     namespace: openshift-lifecycle-agent
   spec:
     channel: "stable"
     name: lifecycle-agent
     source: redhat-operators
     sourceNamespace: openshift-marketplace

   1. 运行以下命令来创建 Subscription CR：

      $ oc create -f lcao-subscription.yaml

验证

1. 要验证安装是否成功，请运行以下命令来检查 CSV 资源：

   $ oc get csv -n openshift-lifecycle-agent

   输出示例

   NAME                              DISPLAY                     VERSION               REPLACES                           PHASE
   lifecycle-agent.v4.16.0           Openshift Lifecycle Agent   4.16.0                Succeeded

2. 运行以下命令验证 Lifecycle Agent 是否正在运行：

   $ oc get deploy -n openshift-lifecycle-agent

   输出示例

   NAME                                 READY   UP-TO-DATE   AVAILABLE   AGE
   lifecycle-agent-controller-manager   1/1     1            1           14s

流程

1. 在 OpenShift Container Platform Web 控制台中导航至 Operators OperatorHub。
2. 从可用的 Operator 列表中选择 Lifecycle Agent，然后单击 Install。
3. 在 Install Operator 页面中，在 A specific namespace on the cluster 下选择 openshift-lifecycle-agent。
4. 点 Install。

验证

1. 确认安装成功：

   1. 点 Operators Installed Operators。

   2. 确保 openshift-lifecycle-agent 项目中列出的 Lifecycle Agent 的 Status 为 InstallSucceeded。

      注意

      在安装过程中，Operator 可能会显示 Failed 状态。如果安装过程结束后有 InstallSucceeded 信息，您可以忽略这个 Failed 信息。

1. 点 Operators Installed Operators，检查 Operator Subscriptions 和 Install Plans 选项卡中的 Status 是否有任何错误。
2. 点 Workloads Pods，在 openshift-lifecycle-agent 项目中检查 pod 的日志。

流程

1. 在 openshift-lifecycle-agent 命名空间中创建以下 CR，并将它们推送到 source-crs/custom-crs 目录：

   LcaSubscriptionNS.yaml 文件示例

   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-lifecycle-agent
     annotations:
       workload.openshift.io/allowed: management
       ran.openshift.io/ztp-deploy-wave: "2"
     labels:
       kubernetes.io/metadata.name: openshift-lifecycle-agent

   LcaSubscriptionOperGroup.yaml 文件示例

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: lifecycle-agent-operatorgroup
     namespace: openshift-lifecycle-agent
     annotations:
       ran.openshift.io/ztp-deploy-wave: "2"
   spec:
     targetNamespaces:
       - openshift-lifecycle-agent

   LcaSubscription.yaml 文件示例

   apiVersion: operators.coreos.com/v1
   kind: Subscription
   metadata:
     name: lifecycle-agent
     namespace: openshift-lifecycle-agent
     annotations:
       ran.openshift.io/ztp-deploy-wave: "2"
   spec:
     channel: "stable"
     name: lifecycle-agent
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     installPlanApproval: Manual
     status:
       state: AtLatestKnown

   目录结构示例

   ├── kustomization.yaml
   ├── sno
   │   ├── example-cnf.yaml
   │   ├── common-ranGen.yaml
   │   ├── group-du-sno-ranGen.yaml
   │   ├── group-du-sno-validator-ranGen.yaml
   │   └── ns.yaml
   ├── source-crs
   │   ├── custom-crs
   │   │   ├── LcaSubscriptionNS.yaml
   │   │   ├── LcaSubscriptionOperGroup.yaml
   │   │   ├── LcaSubscription.yaml

2. 将 CR 添加到通用 PolicyGenTemplate 中：

   apiVersion: ran.openshift.io/v1
   kind: PolicyGenTemplate
   metadata:
     name: "example-common-latest"
     namespace: "ztp-common"
   spec:
     bindingRules:
       common: "true"
       du-profile: "latest"
     sourceFiles:
       - fileName: custom-crs/LcaSubscriptionNS.yaml
         policyName: "subscriptions-policy"
       - fileName: custom-crs/LcaSubscriptionOperGroup.yaml
         policyName: "subscriptions-policy"
       - fileName: custom-crs/LcaSubscription.yaml
         policyName: "subscriptions-policy"
   [...]

流程

1. 在 openshift-adp 命名空间中创建以下 CR，并将其推送到 source-crs/custom-crs 目录：

   OadpSubscriptionNS.yaml 文件示例

   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-adp
     annotations:
       ran.openshift.io/ztp-deploy-wave: "2"
     labels:
       kubernetes.io/metadata.name: openshift-adp

   OadpSubscriptionOperGroup.yaml 文件示例

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: redhat-oadp-operator
     namespace: openshift-adp
     annotations:
       ran.openshift.io/ztp-deploy-wave: "2"
   spec:
     targetNamespaces:
     - openshift-adp

   OadpSubscription.yaml 文件示例

   apiVersion: operators.coreos.com/v1
   kind: Subscription
   metadata:
     name: redhat-oadp-operator
     namespace: openshift-adp
     annotations:
       ran.openshift.io/ztp-deploy-wave: "2"
   spec:
     channel: stable-1.3
     name: redhat-oadp-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace
     installPlanApproval: Manual
   status:
     state: AtLatestKnown

   OadpOperatorStatus.yaml 文件示例

   apiVersion: operators.coreos.com/v1
   kind: Operator
   metadata:
     name: redhat-oadp-operator.openshift-adp
     annotations:
       ran.openshift.io/ztp-deploy-wave: "2"
   status:
     components:
       refs:
       - kind: Subscription
         namespace: openshift-adp
         conditions:
         - type: CatalogSourcesUnhealthy
           status: "False"
       - kind: InstallPlan
         namespace: openshift-adp
         conditions:
         - type: Installed
           status: "True"
       - kind: ClusterServiceVersion
         namespace: openshift-adp
         conditions:
         - type: Succeeded
           status: "True"
           reason: InstallSucceeded

   目录结构示例

   ├── kustomization.yaml
   ├── sno
   │   ├── example-cnf.yaml
   │   ├── common-ranGen.yaml
   │   ├── group-du-sno-ranGen.yaml
   │   ├── group-du-sno-validator-ranGen.yaml
   │   └── ns.yaml
   ├── source-crs
   │   ├── custom-crs
   │   │   ├── OadpSubscriptionNS.yaml
   │   │   ├── OadpSubscriptionOperGroup.yaml
   │   │   ├── OadpSubscription.yaml
   │   │   ├── OadpOperatorStatus.yaml

2. 将 CR 添加到通用 PolicyGenTemplate 中：

   apiVersion: ran.openshift.io/v1
   kind: PolicyGenTemplate
   metadata:
     name: "example-common-latest"
     namespace: "ztp-common"
   spec:
     bindingRules:
       common: "true"
       du-profile: "latest"
     sourceFiles:
       - fileName: custom-crs/OadpSubscriptionNS.yaml
         policyName: "subscriptions-policy"
       - fileName: custom-crs/OadpSubscriptionOperGroup.yaml
         policyName: "subscriptions-policy"
       - fileName: custom-crs/OadpSubscription.yaml
         policyName: "subscriptions-policy"
       - fileName: custom-crs/OadpOperatorStatus.yaml
         policyName: "subscriptions-policy"
   [...]

3. 创建 DataProtectionApplication CR 和 S3 secret：

   1. 在 source-crs/custom-crs 目录中创建以下 CR：

      DataProtectionApplication.yaml 文件示例

      apiVersion: oadp.openshift.io/v1
      kind: DataProtectionApplication
      metadata:
        name: dataprotectionapplication
        namespace: openshift-adp
        annotations:
          ran.openshift.io/ztp-deploy-wave: "100"
      spec:
        configuration:
          restic:
            enable: false 1
          velero:
            defaultPlugins:
              - aws
              - openshift
            resourceTimeout: 10m
        backupLocations:
          - velero:
              config:
                profile: "default"
                region: minio
                s3Url: $url
                insecureSkipTLSVerify: "true"
                s3ForcePathStyle: "true"
              provider: aws
              default: true
              credential:
                key: cloud
                name: cloud-credentials
              objectStorage:
                bucket: $bucketName 2
                prefix: $prefixName 3
      status:
        conditions:
        - reason: Complete
          status: "True"
          type: Reconciled

      1

      对于基于镜像的升级，spec.configuration.restic.enable 字段必须设置为 false，因为升级后持久性卷内容会被保留并重复使用。

      2 3

      bucket 定义在 S3 后端中创建的存储桶名称。前缀定义要在存储桶中自动创建的子目录的名称。bucket 和前缀的组合对于每个目标集群都必须是唯一的，以避免它们间的干扰。要确保每个目标集群的唯一存储目录，您可以使用 RHACM hub 模板功能，例如 prefix: {{hub .ManagedClusterName hub}}。

      OadpSecret.yaml 文件示例

      apiVersion: v1
      kind: Secret
      metadata:
        name: cloud-credentials
        namespace: openshift-adp
        annotations:
          ran.openshift.io/ztp-deploy-wave: "100"
      type: Opaque

      OadpBackupStorageLocationStatus.yaml 文件示例

      apiVersion: velero.io/v1
      kind: BackupStorageLocation
      metadata:
        namespace: openshift-adp
        annotations:
          ran.openshift.io/ztp-deploy-wave: "100"
      status:
        phase: Available

      OadpBackupStorageLocationStatus.yaml CR 验证 OADP 创建的备份存储位置的可用性。

   2. 使用覆盖将 CR 添加到站点 PolicyGenTemplate 中：

      apiVersion: ran.openshift.io/v1
      kind: PolicyGenTemplate
      metadata:
        name: "example-cnf"
        namespace: "ztp-site"
      spec:
        bindingRules:
          sites: "example-cnf"
          du-profile: "latest"
        mcp: "master"
        sourceFiles:
          ...
          - fileName: custom-crs/OadpSecret.yaml
            policyName: "config-policy"
            data:
              cloud: <your_credentials> 1
          - fileName: custom-crs/DataProtectionApplication.yaml
            policyName: "config-policy"
            spec:
              backupLocations:
                - velero:
                    config:
                      region: minio
                      s3Url: <your_S3_URL> 2
                      profile: "default"
                      insecureSkipTLSVerify: "true"
                      s3ForcePathStyle: "true"
                    provider: aws
                    default: true
                    credential:
                      key: cloud
                      name: cloud-credentials
                    objectStorage:
                      bucket: <your_bucket_name> 3
                      prefix: <cluster_name> 4
          - fileName: custom-crs/OadpBackupStorageLocationStatus.yaml
            policyName: "config-policy"

      1

      为您的 S3 存储后端指定您的凭证。

      2

      指定 S3 兼容存储桶的 URL。

      3 4

      bucket 定义在 S3 后端中创建的存储桶名称。前缀 定义存储桶中自动创建的子目录的名称。bucket 和 前缀 的组合对于每个目标集群都必须是唯一的，以避免它们间的干扰。要确保每个目标集群的唯一存储目录，您可以使用 RHACM hub 模板功能，例如 prefix: {{hub .ManagedClusterName hub}}。

1. 这个版本不支持双栈网络。
2. 代理配置不必相同。

流程

1. 从 hub 中分离集群，从 seed 集群中删除任何不能位于 seed 镜像中的特定于集群的资源：

   1. 如果使用 RHACM，请运行以下命令来手动分离 seed 集群：

      $ oc delete managedcluster sno-worker-example

      1. 等待 ManagedCluster CR 被删除。删除 CR 后，创建正确的 SeedGenerator CR。Lifecycle Agent 清理 RHACM 工件。

   2. 如果使用 GitOps ZTP，请通过从 kustomization.yaml 中删除 seed 集群的 SiteConfig CR 来分离集群：

      1. 从 kustomization.yaml 中删除您看到的集群的 SiteConfig CR。

         apiVersion: kustomize.config.k8s.io/v1beta1
         kind: Kustomization
         
         generators:
         #- example-seed-sno1.yaml
         - example-target-sno2.yaml
         - example-target-sno3.yaml

      2. 在 Git 存储库中提交 kustomization.yaml 更改并推送更改。

         ArgoCD 管道检测到更改并删除受管集群。

2. 创建 Secret ：

   1. 运行以下命令来创建身份验证文件：

      $ MY_USER=myuserid
      $ AUTHFILE=/tmp/my-auth.json
      $ podman login --authfile ${AUTHFILE} -u ${MY_USER} quay.io/${MY_USER}

      $ base64 -w 0 ${AUTHFILE} ; echo

   2. 将输出复制到 openshift-lifecycle-agent 命名空间中名为 seedgen 的 Secret YAML 文件中的 seedAuth 字段中：

      apiVersion: v1
      kind: Secret
      metadata:
        name: seedgen 1
        namespace: openshift-lifecycle-agent
      type: Opaque
      data:
        seedAuth: <encoded_AUTHFILE> 2

      1

      Secret 资源必须具有 name: seedgen 和 namespace: openshift-lifecycle-agent 字段。

      2

      指定用于推送生成的 seed 镜像的 write-access 到 registry 的 base64 编码的 authfile。

   3. 运行以下命令来应用 Secret ：

      $ oc apply -f secretseedgenerator.yaml

3. 创建 SeedGenerator CR：

   apiVersion: lca.openshift.io/v1
   kind: SeedGenerator
   metadata:
     name: seedimage 1
   spec:
     seedImage: <seed_container_image> 2

   1

   SeedGenerator CR 必须命名为 seedimage。

   2

   指定容器镜像 URL，如 quay.io/example/seed-container-image:<tag>。建议您使用 < seed_cluster_name>:<ocp_version> 格式。

4. 运行以下命令来生成 seed 镜像：

   $ oc apply -f seedgenerator.yaml

   重要

   当 Lifecycle Agent 生成 seed 镜像时，集群会重启并丢失 API 功能。应用 SeedGenerator CR 会停止 kubelet 和 CRI-O 操作，然后启动镜像生成。

验证

1. 集群恢复并可用后，您可以通过运行以下命令来检查 SeedGenerator CR 的状态：

   $ oc get seedgenerator -o yaml

   输出示例

   status:
     conditions:
     - lastTransitionTime: "2024-02-13T21:24:26Z"
       message: Seed Generation completed
       observedGeneration: 1
       reason: Completed
       status: "False"
       type: SeedGenInProgress
     - lastTransitionTime: "2024-02-13T21:24:26Z"
       message: Seed Generation completed
       observedGeneration: 1
       reason: Completed
       status: "True"
       type: SeedGenCompleted 1
     observedGeneration: 1

   1

   seed 图像生成已完成。

流程

1. 为安装 OADP Operator 的同一命名空间中的平台工件( openshift-adp )创建 OADP Backup 和 Restore CR。

   1. 如果目标集群由 RHACM 管理，请添加以下 YAML 文件来备份和恢复 RHACM 工件：

      用于 RHACM 的 PlatformBackupRestore.yaml

      apiVersion: velero.io/v1
      kind: Backup
      metadata:
        name: acm-klusterlet
        annotations:
          lca.openshift.io/apply-label: "apps/v1/deployments/open-cluster-management-agent/klusterlet,v1/secrets/open-cluster-management-agent/bootstrap-hub-kubeconfig,rbac.authorization.k8s.io/v1/clusterroles/klusterlet,v1/serviceaccounts/open-cluster-management-agent/klusterlet,scheduling.k8s.io/v1/priorityclasses/klusterlet-critical,rbac.authorization.k8s.io/v1/clusterroles/open-cluster-management:klusterlet-admin-aggregate-clusterrole,rbac.authorization.k8s.io/v1/clusterrolebindings/klusterlet,operator.open-cluster-management.io/v1/klusterlets/klusterlet,apiextensions.k8s.io/v1/customresourcedefinitions/klusterlets.operator.open-cluster-management.io,v1/secrets/open-cluster-management-agent/open-cluster-management-image-pull-credentials" 1
        labels:
          velero.io/storage-location: default
        namespace: openshift-adp
      spec:
        includedNamespaces:
        - open-cluster-management-agent
        includedClusterScopedResources:
        - klusterlets.operator.open-cluster-management.io
        - clusterroles.rbac.authorization.k8s.io
        - clusterrolebindings.rbac.authorization.k8s.io
        - priorityclasses.scheduling.k8s.io
        includedNamespaceScopedResources:
        - deployments
        - serviceaccounts
        - secrets
        excludedNamespaceScopedResources: []
      ---
      apiVersion: velero.io/v1
      kind: Restore
      metadata:
        name: acm-klusterlet
        namespace: openshift-adp
        labels:
          velero.io/storage-location: default
        annotations:
          lca.openshift.io/apply-wave: "1"
      spec:
        backupName:
          acm-klusterlet

      1

      如果您的 multiclusterHub CR 没有定义 .spec.imagePullSecret，且 hub 集群中的 open-cluster-management-agent 命名空间中不存在 secret，请删除 v1/secrets/open-cluster-management-agent/open-cluster-management-image-pull-credentials。

   2. 如果您通过 LVM 存储在集群中创建了持久性卷，请为 LVM Storage 工件添加以下 YAML 文件：

      用于 LVM 存储的 PlatformBackupRestoreLvms.yaml

      apiVersion: velero.io/v1
      kind: Backup
      metadata:
        labels:
          velero.io/storage-location: default
        name: lvmcluster
        namespace: openshift-adp
      spec:
        includedNamespaces:
          - openshift-storage
        includedNamespaceScopedResources:
          - lvmclusters
          - lvmvolumegroups
          - lvmvolumegroupnodestatuses
      ---
      apiVersion: velero.io/v1
      kind: Restore
      metadata:
        name: lvmcluster
        namespace: openshift-adp
        labels:
          velero.io/storage-location: default
        annotations:
          lca.openshift.io/apply-wave: "2" 1
      spec:
        backupName:
          lvmcluster

      1

      lca.openshift.io/apply-wave 值必须小于应用程序 Restore CR 中指定的值。

2. （可选）如果您需要升级后恢复应用程序，在 openshift-adp 命名空间中为应用程序创建 OADP 备份和恢复 CR。

   1. 在 openshift-adp 命名空间中为集群范围的应用程序工件创建 OADP CR。

      LSO 和 LVM Storage 的集群范围应用程序工件的 OADP CR 示例

      apiVersion: velero.io/v1
      kind: Backup
      metadata:
        annotations:
          lca.openshift.io/apply-label: "apiextensions.k8s.io/v1/customresourcedefinitions/test.example.com,security.openshift.io/v1/securitycontextconstraints/test,rbac.authorization.k8s.io/v1/clusterroles/test-role,rbac.authorization.k8s.io/v1/clusterrolebindings/system:openshift:scc:test" 1
        name: backup-app-cluster-resources
        labels:
          velero.io/storage-location: default
        namespace: openshift-adp
      spec:
        includedClusterScopedResources:
        - customresourcedefinitions
        - securitycontextconstraints
        - clusterrolebindings
        - clusterroles
        excludedClusterScopedResources:
        - Namespace
      ---
      apiVersion: velero.io/v1
      kind: Restore
      metadata:
        name: test-app-cluster-resources
        namespace: openshift-adp
        labels:
          velero.io/storage-location: default
        annotations:
          lca.openshift.io/apply-wave: "3" 2
      spec:
        backupName:
          backup-app-cluster-resources

      1

      将示例资源名称替换为您的实际资源。

      2

      lca.openshift.io/apply-wave 值必须高于平台 Restore CR 中的值，并低于应用程序命名空间范围 Restore CR 中的值。

   2. 为您的命名空间范围的应用程序工件创建 OADP CR。

      使用 LSO 时 OADP CR 命名空间范围的应用程序工件示例

      apiVersion: velero.io/v1
      kind: Backup
      metadata:
        labels:
          velero.io/storage-location: default
        name: backup-app
        namespace: openshift-adp
      spec:
        includedNamespaces:
        - test
        includedNamespaceScopedResources:
        - secrets
        - persistentvolumeclaims
        - deployments
        - statefulsets
        - configmaps
        - cronjobs
        - services
        - job
        - poddisruptionbudgets
        - <application_custom_resources> 1
        excludedClusterScopedResources:
        - persistentVolumes
      ---
      apiVersion: velero.io/v1
      kind: Restore
      metadata:
        name: test-app
        namespace: openshift-adp
        labels:
          velero.io/storage-location: default
        annotations:
          lca.openshift.io/apply-wave: "4"
      spec:
        backupName:
          backup-app

      1

      为应用程序定义自定义资源。

      使用 LVM 存储时 OADP CR 命名空间范围的应用程序工件示例

      apiVersion: velero.io/v1
      kind: Backup
      metadata:
        labels:
          velero.io/storage-location: default
        name: backup-app
        namespace: openshift-adp
      spec:
        includedNamespaces:
        - test
        includedNamespaceScopedResources:
        - secrets
        - persistentvolumeclaims
        - deployments
        - statefulsets
        - configmaps
        - cronjobs
        - services
        - job
        - poddisruptionbudgets
        - <application_custom_resources> 1
        includedClusterScopedResources:
        - persistentVolumes 2
        - logicalvolumes.topolvm.io 3
        - volumesnapshotcontents 4
      ---
      apiVersion: velero.io/v1
      kind: Restore
      metadata:
        name: test-app
        namespace: openshift-adp
        labels:
          velero.io/storage-location: default
        annotations:
          lca.openshift.io/apply-wave: "4"
      spec:
        backupName:
          backup-app
        restorePVs: true
        restoreStatus:
          includedResources:
          - logicalvolumes 5

      1

      为应用程序定义自定义资源。

      2

      必填字段。

      3

      必填字段

      4

      如果使用 LVM 存储卷快照，则可选。

      5

      必填字段。

      重要

      同一版本的应用程序必须在 OpenShift Container Platform 的当前和目标发行版本中正常工作。

3. 运行以下命令，为 OADP CR 创建 ConfigMap 对象：

   $ oc create configmap oadp-cm-example --from-file=example-oadp-resources.yaml=<path_to_oadp_crs> -n openshift-adp

4. 运行以下命令来修补 ImageBasedUpgrade CR：

   $ oc patch imagebasedupgrades.lca.openshift.io upgrade \
     -p='{"spec": {"oadpContent": [{"name": "oadp-cm-example", "namespace": "openshift-adp"}]}}' \
     --type=merge -n openshift-lifecycle-agent

流程

1. 创建包含额外清单的 YAML 文件，如 SR-IOV。

   SR-IOV 资源示例

   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetworkNodePolicy
   metadata:
     name: "example-sriov-node-policy"
     namespace: openshift-sriov-network-operator
   spec:
     deviceType: vfio-pci
     isRdma: false
     nicSelector:
       pfNames: [ens1f0]
     nodeSelector:
       node-role.kubernetes.io/master: ""
     mtu: 1500
     numVfs: 8
     priority: 99
     resourceName: example-sriov-node-policy
   ---
   apiVersion: sriovnetwork.openshift.io/v1
   kind: SriovNetwork
   metadata:
     name: "example-sriov-network"
     namespace: openshift-sriov-network-operator
   spec:
     ipam: |-
       {
       }
     linkState: auto
     networkNamespace: sriov-namespace
     resourceName: example-sriov-node-policy
     spoofChk: "on"
     trust: "off"

2. 运行以下命令来创建 ConfigMap 对象：

   $ oc create configmap example-extra-manifests-cm --from-file=example-extra-manifests.yaml=<path_to_extramanifest> -n openshift-lifecycle-agent

3. 运行以下命令来修补 ImageBasedUpgrade CR：

   $ oc patch imagebasedupgrades.lca.openshift.io upgrade \
     -p='{"spec": {"extraManifests": [{"name": "example-extra-manifests-cm", "namespace": "openshift-lifecycle-agent"}]}}' \
     --type=merge -n openshift-lifecycle-agent

流程

1. 创建包含 CatalogSource CR 的 YAML 文件：

   apiVersion: operators.coreos.com/v1
   kind: CatalogSource
   metadata:
     name: example-catalogsources
     namespace: openshift-marketplace
   spec:
     sourceType: grpc
     displayName: disconnected-redhat-operators
     image: quay.io/example-org/example-catalog:v1

2. 运行以下命令来创建 ConfigMap 对象：

   $ oc create configmap example-catalogsources-cm --from-file=example-catalogsources.yaml=<path_to_catalogsource_cr> -n openshift-lifecycle-agent

3. 运行以下命令来修补 ImageBasedUpgrade CR：

   $ oc patch imagebasedupgrades.lca.openshift.io upgrade \
     -p='{"spec": {"extraManifests": [{"name": "example-catalogsources-cm", "namespace": "openshift-lifecycle-agent"}]}}' \
     --type=merge -n openshift-lifecycle-agent

流程

1. 确保用于 ArgoCD 策略应用程序的 Git 存储库包含以下目录结构：

   ├── source-crs/
   │   ├── ibu/
   │   │    ├── ImageBasedUpgrade.yaml
   │   │    ├── PlatformBackupRestore.yaml
   │   │    ├── PlatformBackupRestoreLvms.yaml
   ├── ...
   ├── ibu-upgrade-ranGen.yaml
   ├── kustomization.yaml

   重要

   kustomization.yaml 文件必须位于与前面所示相同的目录结构中，以便引用 ibu-upgrade-ranGen.yaml 清单。

   source-crs/ibu/PlatformBackupRestore.yaml 文件在 ZTP 容器镜像中提供。

   PlatformBackupRestore.yaml

   apiVersion: velero.io/v1
   kind: Backup
   metadata:
     name: acm-klusterlet
     annotations:
       lca.openshift.io/apply-label: "apps/v1/deployments/open-cluster-management-agent/klusterlet,v1/secrets/open-cluster-management-agent/bootstrap-hub-kubeconfig,rbac.authorization.k8s.io/v1/clusterroles/klusterlet,v1/serviceaccounts/open-cluster-management-agent/klusterlet,scheduling.k8s.io/v1/priorityclasses/klusterlet-critical,rbac.authorization.k8s.io/v1/clusterroles/open-cluster-management:klusterlet-admin-aggregate-clusterrole,rbac.authorization.k8s.io/v1/clusterrolebindings/klusterlet,operator.open-cluster-management.io/v1/klusterlets/klusterlet,apiextensions.k8s.io/v1/customresourcedefinitions/klusterlets.operator.open-cluster-management.io,v1/secrets/open-cluster-management-agent/open-cluster-management-image-pull-credentials" 1
     labels:
       velero.io/storage-location: default
     namespace: openshift-adp
   spec:
     includedNamespaces:
     - open-cluster-management-agent
     includedClusterScopedResources:
     - klusterlets.operator.open-cluster-management.io
     - clusterroles.rbac.authorization.k8s.io
     - clusterrolebindings.rbac.authorization.k8s.io
     - priorityclasses.scheduling.k8s.io
     includedNamespaceScopedResources:
     - deployments
     - serviceaccounts
     - secrets
     excludedNamespaceScopedResources: []
   ---
   apiVersion: velero.io/v1
   kind: Restore
   metadata:
     name: acm-klusterlet
     namespace: openshift-adp
     labels:
       velero.io/storage-location: default
     annotations:
       lca.openshift.io/apply-wave: "1"
   spec:
     backupName:
       acm-klusterlet

   1

   如果您的 multiclusterHub CR 没有定义 .spec.imagePullSecret，且 hub 集群中的 open-cluster-management-agent 命名空间中不存在 secret，请删除 v1/secrets/open-cluster-management-agent/open-cluster-management-image-pull-credentials。

   如果使用 LVM 存储来创建持久性卷，您可以使用 ZTP 容器镜像中提供的 source-crs/ibu/PlatformBackupRestoreLvms.yaml 来备份 LVM 存储资源。

   PlatformBackupRestoreLvms.yaml

   apiVersion: velero.io/v1
   kind: Backup
   metadata:
     labels:
       velero.io/storage-location: default
     name: lvmcluster
     namespace: openshift-adp
   spec:
     includedNamespaces:
       - openshift-storage
     includedNamespaceScopedResources:
       - lvmclusters
       - lvmvolumegroups
       - lvmvolumegroupnodestatuses
   ---
   apiVersion: velero.io/v1
   kind: Restore
   metadata:
     name: lvmcluster
     namespace: openshift-adp
     labels:
       velero.io/storage-location: default
     annotations:
       lca.openshift.io/apply-wave: "2" 1
   spec:
     backupName:
       lvmcluster

   1

   lca.openshift.io/apply-wave 值必须小于应用程序 Restore CR 中指定的值。

2. 可选： 如果您需要在升级后恢复应用程序，在 openshift-adp 命名空间中为应用程序创建 OADP 备份和恢复 CR：

   1. 在 openshift-adp 命名空间中为集群范围的应用程序工件创建 OADP CR：

      LSO 和 LVM Storage 的集群范围应用程序工件的 OADP CR 示例

      apiVersion: velero.io/v1
      kind: Backup
      metadata:
        annotations:
          lca.openshift.io/apply-label: "apiextensions.k8s.io/v1/customresourcedefinitions/test.example.com,security.openshift.io/v1/securitycontextconstraints/test,rbac.authorization.k8s.io/v1/clusterroles/test-role,rbac.authorization.k8s.io/v1/clusterrolebindings/system:openshift:scc:test" 1
        name: backup-app-cluster-resources
        labels:
          velero.io/storage-location: default
        namespace: openshift-adp
      spec:
        includedClusterScopedResources:
        - customresourcedefinitions
        - securitycontextconstraints
        - clusterrolebindings
        - clusterroles
        excludedClusterScopedResources:
        - Namespace
      ---
      apiVersion: velero.io/v1
      kind: Restore
      metadata:
        name: test-app-cluster-resources
        namespace: openshift-adp
        labels:
          velero.io/storage-location: default
        annotations:
          lca.openshift.io/apply-wave: "3" 2
      spec:
        backupName:
          backup-app-cluster-resources

      1

      将示例资源名称替换为您的实际资源。

      2

      lca.openshift.io/apply-wave 值必须高于平台 Restore CR 中的值，并低于应用程序命名空间范围 Restore CR 中的值。

   2. 在 source-crs/custom-crs 目录中为您的命名空间范围的应用程序工件创建 OADP CR：

      使用 LSO 时 OADP CR 命名空间范围的应用程序工件示例

      apiVersion: velero.io/v1
      kind: Backup
      metadata:
        labels:
          velero.io/storage-location: default
        name: backup-app
        namespace: openshift-adp
      spec:
        includedNamespaces:
        - test
        includedNamespaceScopedResources:
        - secrets
        - persistentvolumeclaims
        - deployments
        - statefulsets
        - configmaps
        - cronjobs
        - services
        - job
        - poddisruptionbudgets
        - <application_custom_resources> 1
        excludedClusterScopedResources:
        - persistentVolumes
      ---
      apiVersion: velero.io/v1
      kind: Restore
      metadata:
        name: test-app
        namespace: openshift-adp
        labels:
          velero.io/storage-location: default
        annotations:
          lca.openshift.io/apply-wave: "4"
      spec:
        backupName:
          backup-app

      1

      为应用程序定义自定义资源。

      使用 LVM 存储时 OADP CR 命名空间范围的应用程序工件示例

      apiVersion: velero.io/v1
      kind: Backup
      metadata:
        labels:
          velero.io/storage-location: default
        name: backup-app
        namespace: openshift-adp
      spec:
        includedNamespaces:
        - test
        includedNamespaceScopedResources:
        - secrets
        - persistentvolumeclaims
        - deployments
        - statefulsets
        - configmaps
        - cronjobs
        - services
        - job
        - poddisruptionbudgets
        - <application_custom_resources> 1
        includedClusterScopedResources:
        - persistentVolumes 2
        - logicalvolumes.topolvm.io 3
        - volumesnapshotcontents 4
      ---
      apiVersion: velero.io/v1
      kind: Restore
      metadata:
        name: test-app
        namespace: openshift-adp
        labels:
          velero.io/storage-location: default
        annotations:
          lca.openshift.io/apply-wave: "4"
      spec:
        backupName:
          backup-app
        restorePVs: true
        restoreStatus:
          includedResources:
          - logicalvolumes 5

      1

      为应用程序定义自定义资源。

      2

      必填字段。

      3

      必填字段

      4

      如果使用 LVM 存储卷快照，则可选。

      5

      必填字段。

      重要

      同一版本的应用程序必须在 OpenShift Container Platform 的当前和目标发行版本中正常工作。

3. 通过名为 ibu-upgrade -ranGen.yaml 的新 PolicyGenTemplate 中的 oadp-cm -policy 来创建 oadp-cm ConfigMap 对象：

   apiVersion: ran.openshift.io/v1
   kind: PolicyGenTemplate
   metadata:
     name: example-group-ibu
     namespace: "ztp-group"
   spec:
     bindingRules:
       group-du-sno: ""
     mcp: "master"
     evaluationInterval:
       compliant: 10s
       noncompliant: 10s
     sourceFiles:
     - fileName: ConfigMapGeneric.yaml
       complianceType: mustonlyhave
       policyName: "oadp-cm-policy"
       metadata:
         name: oadp-cm
         namespace: openshift-adp

4. 使用以下内容创建一个 kustomization.yaml ：

   apiVersion: kustomize.config.k8s.io/v1beta1
   kind: Kustomization
   
   generators: 1
   - ibu-upgrade-ranGen.yaml
   
   configMapGenerator: 2
   - files:
     - source-crs/ibu/PlatformBackupRestore.yaml
     #- source-crs/custom-crs/ApplicationClusterScopedBackupRestore.yaml
     #- source-crs/custom-crs/ApplicationApplicationBackupRestoreLso.yaml
     name: oadp-cm
     namespace: ztp-group
   generatorOptions:
     disableNameSuffixHash: true
   
   
   patches: 3
   - target:
       group: policy.open-cluster-management.io
       version: v1
       kind: Policy
       name: group-ibu-oadp-cm-policy
     patch: |-
       - op: replace
         path: /spec/policy-templates/0/objectDefinition/spec/object-templates/0/objectDefinition/data
         value: '{{hub copyConfigMapData "ztp-group" "oadp-cm" hub}}'

   1

   生成 oadp-cm-policy。

   2

   使用 Backup 和 Restore CR 在 hub 集群中创建 oadp-cm ConfigMap 对象。

   3

   覆盖在 oadp-cm-policy 中添加了 ConfigMap 的 data 字段。hub 模板用于将 oadp-cm ConfigMap 传播到所有目标集群。

5. 将更改推送到您的 Git 存储库。

流程

1. 使用现有 PolicyGenTemplate CR 中的 lca.openshift.io/target-ocp-version: <target_version > 标签标记所需的额外清单：

   apiVersion: ran.openshift.io/v1
   kind: PolicyGenTemplate
   metadata:
     name: example-sno
   spec:
     bindingRules:
       sites: "example-sno"
       du-profile: "4.15"
     mcp: "master"
     sourceFiles:
       - fileName: SriovNetwork.yaml
         policyName: "config-policy"
         metadata:
           name: "sriov-nw-du-fh"
           labels:
             lca.openshift.io/target-ocp-version: "4.15" 1
         spec:
           resourceName: du_fh
           vlan: 140
       - fileName: SriovNetworkNodePolicy.yaml
         policyName: "config-policy"
         metadata:
           name: "sriov-nnp-du-fh"
           labels:
             lca.openshift.io/target-ocp-version: "4.15"
         spec:
           deviceType: netdevice
           isRdma: false
           nicSelector:
             pfNames: ["ens5f0"]
           numVfs: 8
           priority: 10
           resourceName: du_fh
       - fileName: SriovNetwork.yaml
         policyName: "config-policy"
         metadata:
           name: "sriov-nw-du-mh"
           labels:
             lca.openshift.io/target-ocp-version: "4.15"
         spec:
           resourceName: du_mh
           vlan: 150
       - fileName: SriovNetworkNodePolicy.yaml
         policyName: "config-policy"
         metadata:
           name: "sriov-nnp-du-mh"
           labels:
             lca.openshift.io/target-ocp-version: "4.15"
         spec:
           deviceType: vfio-pci
           isRdma: false
           nicSelector:
             pfNames: ["ens7f0"]
           numVfs: 8
           priority: 10
           resourceName: du_mh
       - fileName: DefaultCatsrc.yaml 2
         policyName: "config-policy"
         metadata:
           name: default-cat-source
           namespace: openshift-marketplace
           labels:
               lca.openshift.io/target-ocp-version: "4.15"
         spec:
             displayName: default-cat-source
             image: quay.io/example-org/example-catalog:v1

   1

   确保 lca.openshift.io/target-ocp-version 标签与 ImageBasedUpgrade CR 的 spec.seedImageRef.version 字段中指定的目标 OpenShift Container Platform 版本的 y-stream 或 z-stream 匹配。Lifecycle Agent 仅应用与指定版本匹配的 CR。

   2

   如果您不想使用自定义目录源，请删除此条目。

2. 将更改推送到您的 Git 存储库。