Chapter 13. Troubleshooting LVM

Procedure

• Use the following methods to gather different kinds of diagnostic data:

  • Add the -v argument to any LVM command to increase the verbosity level of the command output. Verbosity can be further increased by adding additional v’s. A maximum of four such v’s is allowed, for example, -vvvv.
  • In the log section of the /etc/lvm/lvm.conf configuration file, increase the value of the level option. This causes LVM to provide more details in the system log.

  • If the problem is related to the logical volume activation, enable LVM to log messages during the activation:

    1. Set the activation = 1 option in the log section of the /etc/lvm/lvm.conf configuration file.
    2. Execute the LVM command with the -vvvv option.
    3. Examine the command output.

    4. Reset the activation option to 0.

       If you do not reset the option to 0, the system might become unresponsive during low memory situations.

  • Display an information dump for diagnostic purposes:

    # lvmdump

    Copy to Clipboard Toggle word wrap

  • Display additional system information:

    # lvs -v

    Copy to Clipboard Toggle word wrap

    # pvs --all

    Copy to Clipboard Toggle word wrap

    # dmsetup info --columns

    Copy to Clipboard Toggle word wrap
  • Examine the last backup of the LVM metadata in the /etc/lvm/backup/ directory and archived versions in the /etc/lvm/archive/ directory.

  • Check the current configuration information:

    # lvmconfig

    Copy to Clipboard Toggle word wrap
  • Check the /run/lvm/hints cache file for a record of which devices have physical volumes on them.

Prerequisites

• You have identified the metadata of the missing physical volume. For details, see Finding the metadata of a missing LVM physical volume.

Verification

• Display the active logical volumes:

  # lvs --all --options +devices
  
    LV     VG   Attr   LSize   Origin Snap%  Move Log Copy%  Devices
   mylv myvg   -wi--- 300.00G                               /dev/vdb1 (0),/dev/vdb1(0)
   mylv myvg   -wi--- 300.00G                               /dev/vdb1 (34728),/dev/vdb1(0)

  Copy to Clipboard Toggle word wrap

Verification

• Check the number of extents that the volume group now uses:

  # vgs --options +vg_free_count,vg_extent_count
  
    VG     #PV #LV #SN  Attr   VSize   VFree  Free  #Ext
    myvg   2   1   0   wz--n- 34.30G    0    0     8780

  Copy to Clipboard Toggle word wrap

• Is found in label_header and pv_header structures.
• Is in the second 512-byte sector of the disk. Note that if a non-default location was specified when creating the physical volume (PV), the header can also be in the first or third sector.

• Begins 4096 bytes from the start of the disk.
• Ends 1 MiB from the start of the disk.
• Begins with a 512 byte sector containing the mda_header structure.

Procedure

• Print current metadata text as referenced from valid mda_header:

  # pvck --dump metadata <disk>

  Copy to Clipboard Toggle word wrap

  For example, to print metadata text from valid mda_header:

  # pvck --dump metadata /dev/sdb
    metadata text at 172032 crc Oxc627522f # vgname test segno 59
    ---
    <raw metadata from disk>
    ---

  Copy to Clipboard Toggle word wrap

• Print the locations of all metadata copies found in the metadata area, based on finding a valid mda_header:

  # pvck --dump metadata_all <disk>

  Copy to Clipboard Toggle word wrap

  For example, to print the locations of metadata copies on the /dev/sdb disk:

  # pvck --dump metadata_all /dev/sdb
    metadata at 4608 length 815 crc 29fcd7ab vg test seqno 1 id FaCsSz-1ZZn-mTO4-Xl4i-zb6G-BYat-u53Fxv
    metadata at 5632 length 1144 crc 50ea61c3 vg test seqno 2 id FaCsSz-1ZZn-mTO4-Xl4i-zb6G-BYat-u53Fxv
    metadata at 7168 length 1450 crc 5652ea55 vg test seqno 3 id FaCsSz-1ZZn-mTO4-Xl4i-zb6G-BYat-u53Fxv

  Copy to Clipboard Toggle word wrap

• Search for all copies of metadata in the metadata area without using an mda_header, for example, if headers are missing or damaged:

  # pvck --dump metadata_search <disk>

  Copy to Clipboard Toggle word wrap

  For example, to search for copies of metadata in the metadata area on the /dev/sdb disk without using an mda_header:

  # pvck --dump metadata_search /dev/sdb
    Searching for metadata at offset 4096 size 1044480
    metadata at 4608 length 815 crc 29fcd7ab vg test seqno 1 id FaCsSz-1ZZn-mTO4-Xl4i-zb6G-BYat-u53Fxv
    metadata at 5632 length 1144 crc 50ea61c3 vg test seqno 2 id FaCsSz-1ZZn-mTO4-Xl4i-zb6G-BYat-u53Fxv
    metadata at 7168 length 1450 crc 5652ea55 vg test seqno 3 id FaCsSz-1ZZn-mTO4-Xl4i-zb6G-BYat-u53Fxv

  Copy to Clipboard Toggle word wrap

• Include the -v option in the dump command to show the description from each copy of metadata:

  # pvck --dump metadata -v <disk>

  Copy to Clipboard Toggle word wrap

  For example, to show description from each copy of metadata on the /dev/sdb disk:

  # pvck --dump metadata -v /dev/sdb
    metadata text at 199680 crc 0x628cf243 # vgname my_vg seqno 40
    ---
  my_vg {
  id = "dmEbPi-gsgx-VbvS-Uaia-HczM-iu32-Rb7iOf"
  seqno = 40
  format = "lvm2"
  status = ["RESIZEABLE", "READ", "WRITE"]
  flags = []
  extent_size = 8192
  max_lv = 0
  max_pv = 0
  metadata_copies = 0
  
  physical_volumes {
  
  pv0 {
  id = "8gn0is-Hj8p-njgs-NM19-wuL9-mcB3-kUDiOQ"
  device = "/dev/sda"
  
  device_id_type = "sys_wwid"
  device_id = "naa.6001405e635dbaab125476d88030a196"
  status = ["ALLOCATABLE"]
  flags = []
  dev_size = 125829120
  pe_start = 8192
  pe_count = 15359
  }
  
  pv1 {
  id = "E9qChJ-5ElL-HVEp-rc7d-U5Fg-fHxL-2QLyID"
  device = "/dev/sdb"
  
  device_id_type = "sys_wwid"
  device_id = "naa.6001405f3f9396fddcd4012a50029a90"
  status = ["ALLOCATABLE"]
  flags = []
  dev_size = 125829120
  pe_start = 8192
  pe_count = 15359
  }

  Copy to Clipboard Toggle word wrap

  This file can be used for repair. The first metadata area is used by default for dump metadata. If the disk has a second metadata area at the end of the disk, you can use the --settings "mda_num=2" option to use the second metadata area for dump metadata instead.

Procedure

• If -f <filename> is added to --dump metadata, the raw metadata is written to the named file. You can use this file for repair.
• If -f <filename> is added to --dump metadata_all or --dump metadata_search, then raw metadata from all locations is written to the named file.

• To save one instance of metadata text from --dump metadata_all|metadata_search add --settings "metadata_offset=<offset>" where <offset> is from the listing output "metadata at <offset>":

  # pvck --dump metadata_search --settings metadata_offset=5632 -f meta.txt /dev/sdb
    Searching for metadata at offset 4096 size 1044480
    metadata at 5632 length 1144 crc 50ea61c3 vg test seqno 2 id FaCsSz-1ZZn-mTO4-Xl4i-zb6G-BYat-u53Fxv
  # head -2 meta.txt
  test {
  id = "FaCsSz-1ZZn-mTO4-Xl4i-zb6G-BYat-u53Fxv"

  Copy to Clipboard Toggle word wrap

Prerequisites

• You have identified the metadata of the missing physical volume. For details, see Finding the metadata of a missing LVM physical volume.

Verification

• To verify that the new physical volume is intact and the volume group is functioning correctly, check the output of the following command:

  # vgs

  Copy to Clipboard Toggle word wrap

Procedure

• Repair the headers and metadata on the disk:

  # pvck --repair -f <metadata-file> <disk>

  Copy to Clipboard Toggle word wrap

  where

  • <metadata-file> is a file containing the most recent metadata for the VG. This can be /etc/lvm/backup/vg-name, or it can be a file containing raw metadata text from the pvck --dump metadata_search command output.

  • <disk> is the name of the disk where the PV is supposed to be, for example, /dev/sdb. To prevent data loss, verify that is the correct disk. If you are not certain the disk is correct, contact Red Hat Support.

    Note

    If the metadata file is a backup file, the pvck --repair should be run on each PV that holds metadata in VG. If the metadata file is raw metadata that has been extracted from another PV, the pvck --repair needs to be run only on the damaged PV.

Verification

• To check that the new physical volume is intact and the volume group is functioning correctly, check outputs of the following commands:

  # vgs <vgname>

  Copy to Clipboard Toggle word wrap

  # pvs <pvname>

  Copy to Clipboard Toggle word wrap

  # lvs <lvname>

  Copy to Clipboard Toggle word wrap