Chapter 5. Persistent storage using local storage

Procedure

1. Edit the previously created local volume to remove any unwanted disks.

   1. Edit the cluster resource:

      $ oc edit localvolume <local_volume_name> -n openshift-local-storage

      Copy to Clipboard Toggle word wrap
   2. Navigate to the lines under devicePaths, and delete any representing unwanted disks.

2. Delete any persistent volumes created.

   $ oc delete pv <pv_name>

   Copy to Clipboard Toggle word wrap

3. Delete directory and included symlinks on the node.

   Warning

   The following step involves accessing a node as the root user. Modifying the state of the node beyond the steps in this procedure could result in cluster instability.

   $ oc debug node/<node_name> -- chroot /host rm -rf /mnt/local-storage/<sc_name> 

   1

   Copy to Clipboard Toggle word wrap

   1

   The name of the storage class used to create the local volumes.

Procedure

1. Delete any local volume resources installed in the project, such as localvolume, localvolumeset, and localvolumediscovery by running the following commands:

   $ oc delete localvolume --all --all-namespaces

   Copy to Clipboard Toggle word wrap

   $ oc delete localvolumeset --all --all-namespaces

   Copy to Clipboard Toggle word wrap

   $ oc delete localvolumediscovery --all --all-namespaces

   Copy to Clipboard Toggle word wrap

2. Uninstall the Local Storage Operator from the web console.

   1. Log in to the OpenShift Container Platform web console.
   2. Navigate to Operators Installed Operators.
   3. Type Local Storage into the filter box to locate the Local Storage Operator.
   4. Click the Options menu at the end of the Local Storage Operator.
   5. Click Uninstall Operator.
   6. Click Remove in the window that appears.

3. The PVs created by the Local Storage Operator will remain in the cluster until deleted. After these volumes are no longer in use, delete them by running the following command:

   $ oc delete pv <pv-name>

   Copy to Clipboard Toggle word wrap

4. Delete the openshift-local-storage project by running the following command:

   $ oc delete project openshift-local-storage

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a YAML file with the configuration for creating a namespace:

   Example YAML configuration for creating a namespace

   apiVersion: v1
   kind: Namespace
   metadata:
     labels:
       openshift.io/cluster-monitoring: "true"
       pod-security.kubernetes.io/enforce: privileged
       pod-security.kubernetes.io/audit: privileged
       pod-security.kubernetes.io/warn: privileged
     name: openshift-storage

   Copy to Clipboard Toggle word wrap

2. Create the namespace by running the following command:

   $ oc create -f <file_name>

   Copy to Clipboard Toggle word wrap

3. Create an OperatorGroup CR YAML file:

   Example OperatorGroup CR

   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: openshift-storage-operatorgroup
     namespace: openshift-storage
   spec:
     targetNamespaces:
     - openshift-storage

   Copy to Clipboard Toggle word wrap

4. Create the OperatorGroup CR by running the following command:

   $ oc create -f <file_name>

   Copy to Clipboard Toggle word wrap

5. Create a Subscription CR YAML file:

   Example Subscription CR

   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: lvms
     namespace: openshift-storage
   spec:
     installPlanApproval: Automatic
     name: lvms-operator
     source: redhat-operators
     sourceNamespace: openshift-marketplace

   Copy to Clipboard Toggle word wrap

6. Create the Subscription CR by running the following command:

   $ oc create -f <file_name>

   Copy to Clipboard Toggle word wrap

Verification

1. To verify that LVM Storage is installed, run the following command:

   $ oc get csv -n openshift-storage -o custom-columns=Name:.metadata.name,Phase:.status.phase

   Copy to Clipboard Toggle word wrap

   Example output

   Name                         Phase
   4.13.0-202301261535          Succeeded

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to the OpenShift Container Platform web console.
2. Click Operators OperatorHub.
3. Click LVM Storage on the OperatorHub page.

4. Set the following options on the Operator Installation page:

   1. Update Channel as stable-4.17.
   2. Installation Mode as A specific namespace on the cluster.
   3. Installed Namespace as Operator recommended namespace openshift-storage. If the openshift-storage namespace does not exist, it is created during the operator installation.

   4. Update approval as Automatic or Manual.

      Note

      If you select Automatic updates, the Operator Lifecycle Manager (OLM) automatically updates the running instance of LVM Storage without any intervention.

      If you select Manual updates, the OLM creates an update request. As a cluster administrator, you must manually approve the update request to update LVM Storage to a newer version.

5. Optional: Select the Enable Operator recommended cluster monitoring on this Namespace checkbox.
6. Click Install.

Procedure

1. Follow the steps in the "Creating the image set configuration" procedure. To create an ImageSetConfiguration custom resource (CR) for LVM Storage, you can use the following example ImageSetConfiguration CR configuration:

   Example ImageSetConfiguration CR for LVM Storage

   kind: ImageSetConfiguration
   apiVersion: mirror.openshift.io/v1alpha2
   archiveSize: 4 

   1

   
   storageConfig: 

   2

   
     registry:
       imageURL: example.com/mirror/oc-mirror-metadata 

   3

   
       skipTLS: false
   mirror:
     platform:
       channels:
       - name: stable-4.17 

   4

   
         type: ocp
       graph: true 

   5

   
     operators:
     - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.17 

   6

   
       packages:
       - name: lvms-operator 

   7

   
         channels:
         - name: stable 

   8

   
     additionalImages:
     - name: registry.redhat.io/ubi9/ubi:latest 

   9

   
     helm: {}

   Copy to Clipboard Toggle word wrap

   1

   Set the maximum size (in GiB) of each file within the image set.

   2

   Specify the location in which you want to save the image set. This location can be a registry or a local directory. You must configure the storageConfig field unless you are using the Technology Preview OCI feature.

   3

   Specify the storage URL for the image stream when using a registry. For more information, see Why use imagestreams.

   4

   Specify the channel from which you want to retrieve the OpenShift Container Platform images.

   5

   Set this field to true to generate the OpenShift Update Service (OSUS) graph image. For more information, see About the OpenShift Update Service.

   6

   Specify the Operator catalog from which you want to retrieve the OpenShift Container Platform images.

   7

   Specify the Operator packages to include in the image set. If this field is empty, all packages in the catalog are retrieved.

   8

   Specify the channels of the Operator packages to include in the image set. You must include the default channel for the Operator package even if you do not use the bundles in that channel. You can find the default channel by running the following command: $ oc mirror list operators --catalog=<catalog_name> --package=<package_name>.

   9

   Specify any additional images to include in the image set.

2. Follow the procedure in the "Mirroring an image set to a mirror registry" section.
3. Follow the procedure in the "Configuring image registry repository mirroring" section.

Procedure

1. Log in to the RHACM CLI using your OpenShift Container Platform credentials.

2. Create a namespace.

   $ oc create ns <namespace>

   Copy to Clipboard Toggle word wrap

3. Create a Policy CR YAML file:

   Example Policy CR to install and configure LVM Storage

   apiVersion: apps.open-cluster-management.io/v1
   kind: PlacementRule
   metadata:
     name: placement-install-lvms
   spec:
     clusterConditions:
     - status: "True"
       type: ManagedClusterConditionAvailable
     clusterSelector: 

   1

   
       matchExpressions:
       - key: mykey
         operator: In
         values:
         - myvalue
   ---
   apiVersion: policy.open-cluster-management.io/v1
   kind: PlacementBinding
   metadata:
     name: binding-install-lvms
   placementRef:
     apiGroup: apps.open-cluster-management.io
     kind: PlacementRule
     name: placement-install-lvms
   subjects:
   - apiGroup: policy.open-cluster-management.io
     kind: Policy
     name: install-lvms
   ---
   apiVersion: policy.open-cluster-management.io/v1
   kind: Policy
   metadata:
     annotations:
       policy.open-cluster-management.io/categories: CM Configuration Management
       policy.open-cluster-management.io/controls: CM-2 Baseline Configuration
       policy.open-cluster-management.io/standards: NIST SP 800-53
     name: install-lvms
   spec:
     disabled: false
     remediationAction: enforce
     policy-templates:
     - objectDefinition:
         apiVersion: policy.open-cluster-management.io/v1
         kind: ConfigurationPolicy
         metadata:
           name: install-lvms
         spec:
           object-templates:
           - complianceType: musthave
             objectDefinition: 

   2

   
               apiVersion: v1
               kind: Namespace
               metadata:
                 labels:
                   openshift.io/cluster-monitoring: "true"
                   pod-security.kubernetes.io/enforce: privileged
                   pod-security.kubernetes.io/audit: privileged
                   pod-security.kubernetes.io/warn: privileged
                 name: openshift-storage
           - complianceType: musthave
             objectDefinition: 

   3

   
               apiVersion: operators.coreos.com/v1
               kind: OperatorGroup
               metadata:
                 name: openshift-storage-operatorgroup
                 namespace: openshift-storage
               spec:
                 targetNamespaces:
                 - openshift-storage
           - complianceType: musthave
             objectDefinition: 

   4

   
               apiVersion: operators.coreos.com/v1alpha1
               kind: Subscription
               metadata:
                 name: lvms
                 namespace: openshift-storage
               spec:
                 installPlanApproval: Automatic
                 name: lvms-operator
                 source: redhat-operators
                 sourceNamespace: openshift-marketplace
           remediationAction: enforce
           severity: low

   Copy to Clipboard Toggle word wrap

   1

   Set the key field and values field in PlacementRule.spec.clusterSelector to match the labels that are configured in the clusters on which you want to install LVM Storage.

   2

   Namespace configuration.

   3

   The OperatorGroup CR configuration.

   4

   The Subscription CR configuration.

4. Create the Policy CR by running the following command:

   $ oc create -f <file_name> -n <namespace>

   Copy to Clipboard Toggle word wrap

   Upon creating the Policy CR, the following custom resources are created on the clusters that match the selection criteria configured in the PlacementRule CR:

   • Namespace
   • OperatorGroup
   • Subscription

Procedure

1. Open the LVMCluster CR YAML file.

2. Configure the LVMCluster CR parameters as described in the following example:

   Example LVMCluster CR YAML file

   apiVersion: lvm.topolvm.io/v1alpha1
   kind: LVMCluster
   metadata:
     name: my-lvmcluster
   spec:
   # ...
     storage:
       deviceClasses:
       - name: vg1  

   1

   
         fstype: ext4 

   2

   
         default: true
         deviceSelector: 

   3

   
   # ...
           forceWipeDevicesAndDestroyAllData: false 

   4

   
         thinPoolConfig: 

   5

   
   # ...
         nodeSelector: 

   6

   
   # ...

   Copy to Clipboard Toggle word wrap

   1

   Set this field to the name of a VG from the previous LVM Storage installation.

   2

   Set this field to ext4 or xfs. By default, this field is set to xfs.

   3

   You can add new devices to the VG that you want to reuse by specifying the new device paths in the deviceSelector field. If you do not want to add new devices to the VG, ensure that the deviceSelector configuration in the current LVM Storage installation is same as that of the previous LVM Storage installation.

   4

   If this field is set to true, LVM Storage wipes all the data on the devices that are added to the VG.

   5

   To retain the thinPoolConfig configuration of the VG that you want to reuse, ensure that the thinPoolConfig configuration in the current LVM Storage installation is same as that of the previous LVM Storage installation. Otherwise, you can configure the thinPoolConfig field as required.

   6

   Configure the requirements to choose the nodes on which you want to create the LVM volume group. If this field is empty, all nodes without no-schedule taints are considered.

3. Save the LVMCluster CR YAML file.

Procedure

1. Create an LVMCluster custom resource (CR) YAML file:

   Example LVMCluster CR YAML file

   apiVersion: lvm.topolvm.io/v1alpha1
   kind: LVMCluster
   metadata:
     name: my-lvmcluster
   spec:
   # ...
     storage:
       deviceClasses: 

   1

   
   # ...
         nodeSelector: 

   2

   
   # ...
         deviceSelector: 

   3

   
   # ...
         thinPoolConfig: 

   4

   
   # ...

   Copy to Clipboard Toggle word wrap

   1

   Contains the configuration to assign the local storage devices to the LVM volume groups.

   2

   Contains the configuration to choose the nodes on which you want to create the LVM volume group. If this field is empty, all nodes without no-schedule taints are considered.

   3

   Contains the configuration to specify the paths to the devices that you want to add to the LVM volume group, and force wipe the devices that are added to the LVM volume group.

   4

   Contains the configuration to create a thin pool in the LVM volume group. If you exclude this field, logical volumes are thick provisioned.

2. Create the LVMCluster CR by running the following command:

   $ oc create -f <file_name>

   Copy to Clipboard Toggle word wrap

   Example output

   lvmcluster/lvmcluster created

   Copy to Clipboard Toggle word wrap

Verification

1. Check that the LVMCluster CR is in the Ready state:

   $ oc get lvmclusters.lvm.topolvm.io -o jsonpath='{.items[*].status}' -n <namespace>

   Copy to Clipboard Toggle word wrap

   Example output

   {"deviceClassStatuses": 

   1

   
   [
     {
       "name": "vg1",
       "nodeStatus": [ 

   2

   
           {
               "devices": [ 

   3

   
                   "/dev/nvme0n1",
                   "/dev/nvme1n1",
                   "/dev/nvme2n1"
               ],
               "node": "kube-node", 

   4

   
               "status": "Ready" 

   5

   
           }
       ]
     }
   ]
   "state":"Ready"} 

   6

   Copy to Clipboard Toggle word wrap

   1

   The status of the device class.

   2

   The status of the LVM volume group on each node.

   3

   The list of devices used to create the LVM volume group.

   4

   The node on which the device class is created.

   5

   The status of the LVM volume group on the node.

   6

   The status of the LVMCluster CR.

   Note

   If the LVMCluster CR is in the Failed state, you can view the reason for failure in the status field.

   Example of status field with the reason for failue:

   status:
     deviceClassStatuses:
       - name: vg1
         nodeStatus:
           - node: my-node-1.example.com
             reason: no available devices found for volume group
             status: Failed
     state: Failed

   Copy to Clipboard Toggle word wrap

2. Optional: To view the storage classes created by LVM Storage for each device class, run the following command:

   $ oc get storageclass

   Copy to Clipboard Toggle word wrap

   Example output

   NAME          PROVISIONER          RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
   lvms-vg1      topolvm.io           Delete          WaitForFirstConsumer   true                   31m

   Copy to Clipboard Toggle word wrap

3. Optional: To view the volume snapshot classes created by LVM Storage for each device class, run the following command:

   $ oc get volumesnapshotclass

   Copy to Clipboard Toggle word wrap

   Example output

   NAME          DRIVER               DELETIONPOLICY   AGE
   lvms-vg1      topolvm.io           Delete           24h

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to the OpenShift Container Platform web console.
2. Click Operators Installed Operators.
3. In the openshift-storage namespace, click LVM Storage.
4. Click Create LVMCluster and select either Form view or YAML view.
5. Configure the required LVMCluster CR parameters.
6. Click Create.

7. Optional: If you want to edit the LVMCLuster CR, perform the following actions:

   1. Click the LVMCluster tab.
   2. From the Actions menu, select Edit LVMCluster.
   3. Click YAML and edit the required LVMCLuster CR parameters.
   4. Click Save.

Verification

1. On the LVMCLuster page, check that the LVMCluster CR is in the Ready state.
2. Optional: To view the available storage classes created by LVM Storage for each device class, click Storage StorageClasses.
3. Optional: To view the available volume snapshot classes created by LVM Storage for each device class, click Storage VolumeSnapshotClasses.

Procedure

1. Log in to the RHACM CLI using your OpenShift Container Platform credentials.

2. Create a ConfigurationPolicy CR YAML file with the configuration to create an LVMCluster CR:

   Example ConfigurationPolicy CR YAML file to create an LVMCluster CR

   apiVersion: policy.open-cluster-management.io/v1
   kind: ConfigurationPolicy
   metadata:
     name: lvms
   spec:
     object-templates:
     - complianceType: musthave
       objectDefinition:
         apiVersion: lvm.topolvm.io/v1alpha1
         kind: LVMCluster
         metadata:
           name: my-lvmcluster
           namespace: openshift-storage
         spec:
           storage:
             deviceClasses: 

   1

   
   # ...
               deviceSelector: 

   2

   
   # ...
               thinPoolConfig: 

   3

   
   # ...
               nodeSelector: 

   4

   
   # ...
     remediationAction: enforce
     severity: low

   Copy to Clipboard Toggle word wrap

   1

   Contains the configuration to assign the local storage devices to the LVM volume groups.

   2

   Contains the configuration to specify the paths to the devices that you want to add to the LVM volume group, and force wipe the devices that are added to the LVM volume group.

   3

   Contains the configuration to create a thin pool in the LVM volume group. If you exclude this field, logical volumes are thick provisioned.

   4

   Contains the configuration to choose the nodes on which you want to create the LVM volume groups. If this field is empty, then all nodes without no-schedule taints are considered.

3. Create the ConfigurationPolicy CR by running the following command:

   $ oc create -f <file_name> -n <cluster_namespace> 

   1

   Copy to Clipboard Toggle word wrap

   1

   Namespace of the OpenShift Container Platform cluster on which LVM Storage is installed.

Procedure

1. Log in to the OpenShift CLI (oc).

2. Delete the LVMCluster CR by running the following command:

   $ oc delete lvmcluster <lvm_cluster_name> -n openshift-storage

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to the OpenShift Container Platform web console.
2. Click Operators Installed Operators to view all the installed Operators.
3. Click LVM Storage in the openshift-storage namespace.
4. Click the LVMCluster tab.
5. From the Actions, select Delete LVMCluster.
6. Click Delete.

Procedure

1. Log in to the RHACM CLI using your OpenShift Container Platform credentials.

2. Delete the ConfigurationPolicy CR YAML file that was created for the LVMCluster CR:

   $ oc delete -f <file_name> -n <cluster_namespace> 

   1

   Copy to Clipboard Toggle word wrap

   1

   Namespace of the OpenShift Container Platform cluster on which LVM Storage is installed.

3. Create a Policy CR YAML file to delete the LVMCluster CR:

   Example Policy CR to delete the LVMCluster CR

   apiVersion: policy.open-cluster-management.io/v1
   kind: Policy
   metadata:
     name: policy-lvmcluster-delete
     annotations:
       policy.open-cluster-management.io/standards: NIST SP 800-53
       policy.open-cluster-management.io/categories: CM Configuration Management
       policy.open-cluster-management.io/controls: CM-2 Baseline Configuration
   spec:
     remediationAction: enforce
     disabled: false
     policy-templates:
       - objectDefinition:
           apiVersion: policy.open-cluster-management.io/v1
           kind: ConfigurationPolicy
           metadata:
             name: policy-lvmcluster-removal
           spec:
             remediationAction: enforce 

   1

   
             severity: low
             object-templates:
               - complianceType: mustnothave
                 objectDefinition:
                   kind: LVMCluster
                   apiVersion: lvm.topolvm.io/v1alpha1
                   metadata:
                     name: my-lvmcluster
                     namespace: openshift-storage 

   2

   
   ---
   apiVersion: policy.open-cluster-management.io/v1
   kind: PlacementBinding
   metadata:
     name: binding-policy-lvmcluster-delete
   placementRef:
     apiGroup: apps.open-cluster-management.io
     kind: PlacementRule
     name: placement-policy-lvmcluster-delete
   subjects:
     - apiGroup: policy.open-cluster-management.io
       kind: Policy
       name: policy-lvmcluster-delete
   ---
   apiVersion: apps.open-cluster-management.io/v1
   kind: PlacementRule
   metadata:
     name: placement-policy-lvmcluster-delete
   spec:
     clusterConditions:
       - status: "True"
         type: ManagedClusterConditionAvailable
     clusterSelector: 

   3

   
       matchExpressions:
         - key: mykey
           operator: In
           values:
             - myvalue

   Copy to Clipboard Toggle word wrap

   1

   The spec.remediationAction in policy-template is overridden by the preceding parameter value for spec.remediationAction.

   2

   This namespace field must have the openshift-storage value.

   3

   Configure the requirements to select the clusters. LVM Storage is uninstalled on the clusters that match the selection criteria.

4. Create the Policy CR by running the following command:

   $ oc create -f <file_name> -n <namespace>

   Copy to Clipboard Toggle word wrap

5. Create a Policy CR YAML file to check if the LVMCluster CR has been deleted:

   Example Policy CR to check if the LVMCluster CR has been deleted

   apiVersion: policy.open-cluster-management.io/v1
   kind: Policy
   metadata:
     name: policy-lvmcluster-inform
     annotations:
       policy.open-cluster-management.io/standards: NIST SP 800-53
       policy.open-cluster-management.io/categories: CM Configuration Management
       policy.open-cluster-management.io/controls: CM-2 Baseline Configuration
   spec:
     remediationAction: inform
     disabled: false
     policy-templates:
       - objectDefinition:
           apiVersion: policy.open-cluster-management.io/v1
           kind: ConfigurationPolicy
           metadata:
             name: policy-lvmcluster-removal-inform
           spec:
             remediationAction: inform 

   1

   
             severity: low
             object-templates:
               - complianceType: mustnothave
                 objectDefinition:
                   kind: LVMCluster
                   apiVersion: lvm.topolvm.io/v1alpha1
                   metadata:
                     name: my-lvmcluster
                     namespace: openshift-storage 

   2

   
   ---
   apiVersion: policy.open-cluster-management.io/v1
   kind: PlacementBinding
   metadata:
     name: binding-policy-lvmcluster-check
   placementRef:
     apiGroup: apps.open-cluster-management.io
     kind: PlacementRule
     name: placement-policy-lvmcluster-check
   subjects:
     - apiGroup: policy.open-cluster-management.io
       kind: Policy
       name: policy-lvmcluster-inform
   ---
   apiVersion: apps.open-cluster-management.io/v1
   kind: PlacementRule
   metadata:
     name: placement-policy-lvmcluster-check
   spec:
     clusterConditions:
       - status: "True"
         type: ManagedClusterConditionAvailable
     clusterSelector:
       matchExpressions:
         - key: mykey
           operator: In
           values:
             - myvalue

   Copy to Clipboard Toggle word wrap

   1

   The policy-template spec.remediationAction is overridden by the preceding parameter value for spec.remediationAction.

   2

   The namespace field must have the openshift-storage value.

6. Create the Policy CR by running the following command:

   $ oc create -f <file_name> -n <namespace>

   Copy to Clipboard Toggle word wrap

Procedure

1. Edit the LVMCluster CR by running the following command:

   $ oc edit <lvmcluster_file_name> -n <namespace>

   Copy to Clipboard Toggle word wrap

2. Add the path to the new device in the deviceSelector field.

   Example LVMCluster CR

   apiVersion: lvm.topolvm.io/v1alpha1
   kind: LVMCluster
   metadata:
     name: my-lvmcluster
   spec:
     storage:
       deviceClasses:
   # ...
         deviceSelector: 

   1

   
           paths: 

   2

   
           - /dev/disk/by-path/pci-0000:87:00.0-nvme-1
           - /dev/disk/by-path/pci-0000:88:00.0-nvme-1
           optionalPaths: 

   3

   
           - /dev/disk/by-path/pci-0000:89:00.0-nvme-1
           - /dev/disk/by-path/pci-0000:90:00.0-nvme-1
   # ...

   Copy to Clipboard Toggle word wrap

   1

   Contains the configuration to specify the paths to the devices that you want to add to the LVM volume group. You can specify the device paths in the paths field, the optionalPaths field, or both. If you do not specify the device paths in both paths and optionalPaths, Logical Volume Manager (LVM) Storage adds the supported unused devices to the LVM volume group. LVM Storage adds the devices to the LVM volume group only if the following conditions are met:

   • The device path exists.
   • The device is supported by LVM Storage. For information about unsupported devices, see "Devices not supported by LVM Storage".

   2

   Specify the device paths. If the device path specified in this field does not exist, or the device is not supported by LVM Storage, the LVMCluster CR moves to the Failed state.

   3

   Specify the optional device paths. If the device path specified in this field does not exist, or the device is not supported by LVM Storage, LVM Storage ignores the device without causing an error.

   Important

   After a device is added to the LVM volume group, it cannot be removed.

3. Save the LVMCluster CR.