Chapter 8. Troubleshooting Ceph placement groups

• Verify your network connection.
• Ensure that Monitors are able to form a quorum.
• Ensure that all healthy OSDs are up and in, and the backfilling and recovery processes are finished.

Prerequisites

• A running Red Hat Ceph Storage cluster.
• Root-level access to the node.

• Inconsistent placement group in a pool
• Inconsistent objects in a placement group
• Inconsistent snapshot sets in a placement group

Prerequisites

• A running Red Hat Ceph Storage cluster in a healthy state.
• Root-level access to the node.

Procedure

• List all the inconsistent placement groups in a pool:

  Syntax

  rados list-inconsistent-pg POOL --format=json-pretty

  Example

  [ceph: root@host01 /]# rados list-inconsistent-pg data --format=json-pretty
  [0.6]

• List inconsistent objects in a placement group with ID:

  Syntax

  rados list-inconsistent-obj PLACEMENT_GROUP_ID

  Example

  [ceph: root@host01 /]# rados list-inconsistent-obj 0.6
  {
      "epoch": 14,
      "inconsistents": [
          {
              "object": {
                  "name": "image1",
                  "nspace": "",
                  "locator": "",
                  "snap": "head",
                  "version": 1
              },
              "errors": [
                  "data_digest_mismatch",
                  "size_mismatch"
              ],
              "union_shard_errors": [
                  "data_digest_mismatch_oi",
                  "size_mismatch_oi"
              ],
              "selected_object_info": "0:602f83fe:::foo:head(16'1 client.4110.0:1 dirty|data_digest|omap_digest s 968 uv 1 dd e978e67f od ffffffff alloc_hint [0 0 0])",
              "shards": [
                  {
                      "osd": 0,
                      "errors": [],
                      "size": 968,
                      "omap_digest": "0xffffffff",
                      "data_digest": "0xe978e67f"
                  },
                  {
                      "osd": 1,
                      "errors": [],
                      "size": 968,
                      "omap_digest": "0xffffffff",
                      "data_digest": "0xe978e67f"
                  },
                  {
                      "osd": 2,
                      "errors": [
                          "data_digest_mismatch_oi",
                          "size_mismatch_oi"
                      ],
                      "size": 0,
                      "omap_digest": "0xffffffff",
                      "data_digest": "0xffffffff"
                  }
              ]
          }
      ]
  }

  The following fields are important to determine what causes the inconsistency:

  • name: The name of the object with inconsistent replicas.
  • nspace: The namespace that is a logical separation of a pool. It’s empty by default.
  • locator: The key that is used as the alternative of the object name for placement.
  • snap: The snapshot ID of the object. The only writable version of the object is called head. If an object is a clone, this field includes its sequential ID.
  • version: The version ID of the object with inconsistent replicas. Each write operation to an object increments it.

  • errors: A list of errors that indicate inconsistencies between shards without determining which shard or shards are incorrect. See the shard array to further investigate the errors.

    • data_digest_mismatch: The digest of the replica read from one OSD is different from the other OSDs.
    • size_mismatch: The size of a clone or the head object does not match the expectation.
    • read_error: This error indicates inconsistencies caused most likely by disk errors.

  • union_shard_error: The union of all errors specific to shards. These errors are connected to a faulty shard. The errors that end with oi indicate that you have to compare the information from a faulty object to information with selected objects. See the shard array to further investigate the errors.

    In the above example, the object replica stored on osd.2 has different digest than the replicas stored on osd.0 and osd.1. Specifically, the digest of the replica is not 0xffffffff as calculated from the shard read from osd.2, but 0xe978e67f. In addition, the size of the replica read from osd.2 is 0, while the size reported by osd.0 and osd.1 is 968.

• List inconsistent sets of snapshots:

  Syntax

  rados list-inconsistent-snapset PLACEMENT_GROUP_ID

  Example

  [ceph: root@host01 /]# rados list-inconsistent-snapset 0.23 --format=json-pretty
  {
      "epoch": 64,
      "inconsistents": [
          {
              "name": "obj5",
              "nspace": "",
              "locator": "",
              "snap": "0x00000001",
              "headless": true
          },
          {
              "name": "obj5",
              "nspace": "",
              "locator": "",
              "snap": "0x00000002",
              "headless": true
          },
          {
              "name": "obj5",
              "nspace": "",
              "locator": "",
              "snap": "head",
              "ss_attr_missing": true,
              "extra_clones": true,
              "extra clones": [
                  2,
                  1
              ]
          }
      ]

  The command returns the following errors:

  • ss_attr_missing: One or more attributes are missing. Attributes are information about snapshots encoded into a snapshot set as a list of key-value pairs.
  • ss_attr_corrupted: One or more attributes fail to decode.
  • clone_missing: A clone is missing.
  • snapset_mismatch: The snapshot set is inconsistent by itself.
  • head_mismatch: The snapshot set indicates that head exists or not, but the scrub results report otherwise.
  • headless: The head of the snapshot set is missing.
  • size_mismatch: The size of a clone or the head object does not match the expectation.

Prerequisites

• Root-level access to the Ceph Monitor node.

Procedure

• Repair the inconsistent placement groups:

  Syntax

  ceph pg repair ID

  Replace ID with the ID of the inconsistent placement group.

Prerequisites

• A running Red Hat Ceph Storage cluster in a healthy state.
• Root-level access to the node.

• See Chapter 3, Troubleshooting networking issues for details.
• See Chapter 4, Troubleshooting Ceph Monitors for details about troubleshooting the most common errors related to Ceph Monitors.
• See Chapter 5, Troubleshooting Ceph OSDs for details about troubleshooting the most common errors related to Ceph OSDs.
• See the Auto-scaling placement groups section in the Red Hat Ceph Storage Storage Strategies Guide for more information on PG autoscaler.