Red Hat Summit Red Hat SummitApoyoConsolaDesarrolladoresIniciar una pruebaContacto

Este contenido no está disponible en el idioma seleccionado.

Chapter 8. Troubleshooting Ceph objects

As a storage administrator, you can use the ceph-objectstore-tool utility to perform high-level or low-level object operations. The ceph-objectstore-tool utility can help you troubleshoot problems related to objects within a particular OSD or placement group.

Important

Manipulating objects can cause unrecoverable data loss. Contact Red Hat support before using the ceph-objectstore-tool utility.

Prerequisites

  • Verify there are no network-related issues.

8.1. Troubleshooting high-level object operations
Copiar enlace

As a storage administrator, you can use the ceph-objectstore-tool utility to perform high-level object operations. The ceph-objectstore-tool utility supports the following high-level object operations:

  • List objects
  • List lost objects
  • Fix lost objects
Important

Manipulating objects can cause unrecoverable data loss. Contact Red Hat support before using the ceph-objectstore-tool utility.

Prerequisites

  • Root-level access to the Ceph OSD nodes.

8.1.1. Listing objects
Copiar enlace

The OSD can contain zero to many placement groups, and zero to many objects within a placement group (PG). The ceph-objectstore-tool utility allows you to list objects stored within an OSD.

Prerequisites

  • Root-level access to the Ceph OSD node.
  • Stopping the ceph-osd daemon.

Procedure

  1. Verify the appropriate OSD is down:

    Syntax

    systemctl status ceph-FSID@osd.OSD_ID
    Copy to Clipboard Toggle word wrap

    Example

    [root@host01 ~]# systemctl status ceph-b404c440-9e4c-11ec-a28a-001a4a0001df@osd.0.service
    Copy to Clipboard Toggle word wrap

  2. Log in to the OSD container:

    Syntax

    cephadm shell --name osd.OSD_ID
    Copy to Clipboard Toggle word wrap

    Example

    [root@host01 ~]# cephadm shell --name osd.0
    Copy to Clipboard Toggle word wrap

  3. Identify all the objects within an OSD, regardless of their placement group:

    Syntax

    ceph-objectstore-tool --data-path PATH_TO_OSD --op list
    Copy to Clipboard Toggle word wrap

    Example

    [ceph: root@host01 /]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --op list
    Copy to Clipboard Toggle word wrap

  4. Identify all the objects within a placement group:

    Syntax

    ceph-objectstore-tool --data-path PATH_TO_OSD --pgid PG_ID --op list
    Copy to Clipboard Toggle word wrap

    Example

    [ceph: root@host01 /]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c --op list
    Copy to Clipboard Toggle word wrap

  5. Identify the PG an object belongs to:

    Syntax

    ceph-objectstore-tool --data-path PATH_TO_OSD --op list OBJECT_ID
    Copy to Clipboard Toggle word wrap

    Example

    [ceph: root@host01 /]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --op list default.region
    Copy to Clipboard Toggle word wrap

8.1.2. Fixing lost objects
Copiar enlace

You can use the ceph-objectstore-tool utility to list and fix lost and unfound objects stored within a Ceph OSD. This procedure applies only to legacy objects.

Prerequisites

  • Root-level access to the Ceph OSD node.
  • Stopping the ceph-osd daemon.

Procedure

  1. Verify the appropriate OSD is down:

    Syntax

    systemctl status ceph-FSID@osd.OSD_ID
    Copy to Clipboard Toggle word wrap

    Example

    [root@host01 ~]# systemctl status ceph-b404c440-9e4c-11ec-a28a-001a4a0001df@osd.0.service
    Copy to Clipboard Toggle word wrap

  2. Log in to the OSD container:

    Syntax

    cephadm shell --name osd.OSD_ID
    Copy to Clipboard Toggle word wrap

    Example

    [root@host01 ~]# cephadm shell --name osd.0
    Copy to Clipboard Toggle word wrap

  3. To list all the lost legacy objects:

    Syntax

    ceph-objectstore-tool --data-path PATH_TO_OSD --op fix-lost --dry-run
    Copy to Clipboard Toggle word wrap

    Example

    [ceph: root@host01 /]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --op fix-lost --dry-run
    Copy to Clipboard Toggle word wrap

  4. Use the ceph-objectstore-tool utility to fix lost and unfound objects. Select the appropriate circumstance:

    1. To fix all lost objects:

      Syntax

      ceph-objectstore-tool --data-path PATH_TO_OSD --op fix-lost
      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@host01 /]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --op fix-lost
      Copy to Clipboard Toggle word wrap

    2. To fix all the lost objects within a placement group:

      Syntax

      ceph-objectstore-tool --data-path PATH_TO_OSD --pgid PG_ID --op fix-lost
      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@host01 /]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c --op fix-lost
      Copy to Clipboard Toggle word wrap

    3. To fix a lost object by its identifier:

      Syntax

      ceph-objectstore-tool --data-path PATH_TO_OSD --op fix-lost OBJECT_ID
      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@host01 /]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --op fix-lost default.region
      Copy to Clipboard Toggle word wrap

8.2. Troubleshooting low-level object operations
Copiar enlace

As a storage administrator, you can use the ceph-objectstore-tool utility to perform low-level object operations. The ceph-objectstore-tool utility supports the following low-level object operations:

  • Manipulate the object’s content
  • Remove an object
  • List the object map (OMAP)
  • Manipulate the OMAP header
  • Manipulate the OMAP key
  • List the object’s attributes
  • Manipulate the object’s attribute key
Important

Manipulating objects can cause unrecoverable data loss. Contact Red Hat support before using the ceph-objectstore-tool utility.

Prerequisites

  • Root-level access to the Ceph OSD nodes.

8.2.1. Manipulating the object’s content
Copiar enlace

With the ceph-objectstore-tool utility, you can get or set bytes on an object.

Important

Setting the bytes on an object can cause unrecoverable data loss. To prevent data loss, make a backup copy of the object.

Prerequisites

  • Root-level access to the Ceph OSD node.
  • Stopping the ceph-osd daemon.

Procedure

  1. Verify the appropriate OSD is down:

    Syntax

    systemctl status ceph-FSID@osd.OSD_ID
    Copy to Clipboard Toggle word wrap

    Example

    [root@host01 ~]# systemctl status ceph-b404c440-9e4c-11ec-a28a-001a4a0001df@osd.0.service
    Copy to Clipboard Toggle word wrap

  2. Log in to the OSD container:

    Syntax

    cephadm shell --name osd.OSD_ID
    Copy to Clipboard Toggle word wrap

    Example

    [root@host01 ~]# cephadm shell --name osd.0
    Copy to Clipboard Toggle word wrap

  3. Find the object by listing the objects of the OSD or placement group (PG).

  4. Before setting the bytes on an object, make a backup and a working copy of the object:

    Syntax

    ceph-objectstore-tool --data-path PATH_TO_OSD --pgid PG_ID \
OBJECT \
get-bytes > OBJECT_FILE_NAME
    Copy to Clipboard Toggle word wrap

    Example

    [ceph: root@host01 /]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c \
'{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}'  \
get-bytes > zone_info.default.backup

[ceph: root@host01 /]#  ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c \
'{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}'  \
get-bytes > zone_info.default.working-copy
    Copy to Clipboard Toggle word wrap

  5. Edit the working copy object file and modify the object contents accordingly.

  6. Set the bytes of the object:

    Syntax

    ceph-objectstore-tool --data-path PATH_TO_OSD --pgid PG_ID \
OBJECT \
set-bytes < OBJECT_FILE_NAME
    Copy to Clipboard Toggle word wrap

    Example

    [ceph: root@host01 /]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c \
'{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}' \
set-bytes < zone_info.default.working-copy
    Copy to Clipboard Toggle word wrap

8.2.2. Removing an object
Copiar enlace

Use the ceph-objectstore-tool utility to remove an object. By removing an object, its contents and references are removed from the placement group (PG).

Important

You cannot recreate an object once it is removed.

Prerequisites

  • Root-level access to the Ceph OSD node.
  • Stopping the ceph-osd daemon.

Procedure

  1. Log in to the OSD container:

    Syntax

    cephadm shell --name osd.OSD_ID
    Copy to Clipboard Toggle word wrap

    Example

    [root@host01 ~]# cephadm shell --name osd.0
    Copy to Clipboard Toggle word wrap

  2. Remove an object:

    Syntax

    ceph-objectstore-tool --data-path PATH_TO_OSD --pgid PG_ID \
OBJECT \
remove
    Copy to Clipboard Toggle word wrap

    Example

    [ceph: root@host01 /]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c \
'{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}' \
remove
    Copy to Clipboard Toggle word wrap

8.2.3. Listing the object map
Copiar enlace

Use the ceph-objectstore-tool utility to list the contents of the object map (OMAP). The output provides you a list of keys.

Prerequisites

  • Root-level access to the Ceph OSD node.
  • Stopping the ceph-osd daemon.

Procedure

  1. Verify the appropriate OSD is down:

    Syntax

    systemctl status ceph-osd@OSD_ID
    Copy to Clipboard Toggle word wrap

    Example

    [root@host01 ~]# systemctl status ceph-b404c440-9e4c-11ec-a28a-001a4a0001df@osd.0.service
    Copy to Clipboard Toggle word wrap

  2. Log in to the OSD container:

    Syntax

    cephadm shell --name osd.OSD_ID
    Copy to Clipboard Toggle word wrap

    Example

    [root@host01 ~]# cephadm shell --name osd.0
    Copy to Clipboard Toggle word wrap

  3. List the object map:

    Syntax

    ceph-objectstore-tool --data-path PATH_TO_OSD --pgid PG_ID \
OBJECT \
list-omap
    Copy to Clipboard Toggle word wrap

    Example

    [ceph: root@host01 /]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c \
'{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}' \
list-omap
    Copy to Clipboard Toggle word wrap

8.2.4. Manipulating the object map header
Copiar enlace

The ceph-objectstore-tool utility outputs the object map (OMAP) header with the values associated with the object’s keys.

Prerequisites

  • Root-level access to the Ceph OSD node.
  • Stopping the ceph-osd daemon.

Procedure

  1. Verify the appropriate OSD is down:

    Syntax

    systemctl status ceph-FSID@osd.OSD_ID
    Copy to Clipboard Toggle word wrap

    Example

    [root@host01 ~]# systemctl status ceph-b404c440-9e4c-11ec-a28a-001a4a0001df@osd.0.service
    Copy to Clipboard Toggle word wrap

  2. Log in to the OSD container:

    Syntax

    cephadm shell --name osd.OSD_ID
    Copy to Clipboard Toggle word wrap

    Example

    [root@host01 ~]# cephadm shell --name osd.0
    Copy to Clipboard Toggle word wrap

  3. Get the object map header:

    Syntax

    ceph-objectstore-tool --data-path PATH_TO_OSD \
--pgid PG_ID OBJECT \
get-omaphdr > OBJECT_MAP_FILE_NAME
    Copy to Clipboard Toggle word wrap

    Example

    [ceph: root@host01 /]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 \
--pgid 0.1c '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}'  \
get-omaphdr > zone_info.default.omaphdr.txt
    Copy to Clipboard Toggle word wrap

  4. Set the object map header:

    Syntax

    ceph-objectstore-tool --data-path PATH_TO_OSD \
--pgid PG_ID OBJECT \
get-omaphdr < OBJECT_MAP_FILE_NAME
    Copy to Clipboard Toggle word wrap

    Example

    [ceph: root@host01 /]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 \
--pgid 0.1c '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}'  \
set-omaphdr < zone_info.default.omaphdr.txt
    Copy to Clipboard Toggle word wrap

8.2.5. Manipulating the object map key
Copiar enlace

Use the ceph-objectstore-tool utility to change the object map (OMAP) key. You need to provide the data path, the placement group identifier (PG ID), the object, and the key in the OMAP.

Prerequisites

  • Root-level access to the Ceph OSD node.
  • Stopping the ceph-osd daemon.

Procedure

  1. Log in to the OSD container:

    Syntax

    cephadm shell --name osd.OSD_ID
    Copy to Clipboard Toggle word wrap

    Example

    [root@host01 ~]# cephadm shell --name osd.0
    Copy to Clipboard Toggle word wrap

  2. Get the object map key:

    Syntax

    ceph-objectstore-tool --data-path PATH_TO_OSD \
--pgid PG_ID OBJECT \
get-omap KEY > OBJECT_MAP_FILE_NAME
    Copy to Clipboard Toggle word wrap

    Example

    [ceph: root@host01 /]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 \
--pgid 0.1c '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}'  \
get-omap "" > zone_info.default.omap.txt
    Copy to Clipboard Toggle word wrap

    • Set the object map key:

      Syntax

      ceph-objectstore-tool --data-path PATH_TO_OSD \
--pgid PG_ID OBJECT \
set-omap KEY < OBJECT_MAP_FILE_NAME
      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@host01 /]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 \
--pgid 0.1c '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}'  \
set-omap "" < zone_info.default.omap.txt
      Copy to Clipboard Toggle word wrap

    • Remove the object map key:

      Syntax

      ceph-objectstore-tool --data-path PATH_TO_OSD \
--pgid PG_ID OBJECT \
rm-omap KEY
      Copy to Clipboard Toggle word wrap

      Example

      [ceph: root@host01 /]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 \
--pgid 0.1c '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}'  \
rm-omap ""
      Copy to Clipboard Toggle word wrap

8.2.6. Listing the object’s attributes
Copiar enlace

Use the ceph-objectstore-tool utility to list an object’s attributes. The output provides you with the object’s keys and values.

Prerequisites

  • Root-level access to the Ceph OSD node.
  • Stopping the ceph-osd daemon.

Procedure

  1. Verify the appropriate OSD is down:

    Syntax

    systemctl status ceph-FSID@osd.OSD_ID
    Copy to Clipboard Toggle word wrap

    Example

    [root@host01 ~]# systemctl status ceph-b404c440-9e4c-11ec-a28a-001a4a0001df@osd.0.service
    Copy to Clipboard Toggle word wrap

  2. Log in to the OSD container:

    Syntax

    cephadm shell --name osd.OSD_ID
    Copy to Clipboard Toggle word wrap

    Example

    [root@host01 ~]# cephadm shell --name osd.0
    Copy to Clipboard Toggle word wrap

  3. List the object’s attributes:

    Syntax

    ceph-objectstore-tool --data-path PATH_TO_OSD \
--pgid PG_ID OBJECT \
list-attrs
    Copy to Clipboard Toggle word wrap

    Example

    [ceph: root@host01 /]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 \
--pgid 0.1c '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}' \
list-attrs
    Copy to Clipboard Toggle word wrap

8.2.7. Manipulating the object attribute key
Copiar enlace

Use the ceph-objectstore-tool utility to change an object’s attributes. To manipulate the object’s attributes you need the data paths, the placement group identifier (PG ID), the object, and the key in the object’s attribute.

Prerequisites

  • Root-level access to the Ceph OSD node.
  • Stop the ceph-osd daemon.

Procedure

  1. Verify the appropriate OSD is down:

    Syntax

    systemctl status ceph-FSID@osd.OSD_ID
    Copy to Clipboard Toggle word wrap

    Example

    [root@host01 ~]# systemctl status ceph-b404c440-9e4c-11ec-a28a-001a4a0001df@osd.0.service
    Copy to Clipboard Toggle word wrap

  2. Log in to the OSD container:

    Syntax

    cephadm shell --name osd.OSD_ID
    Copy to Clipboard Toggle word wrap

    Example

    [root@host01 ~]# cephadm shell --name osd.0
    Copy to Clipboard Toggle word wrap

  3. Get the object’s attributes:

    Syntax

    ceph-objectstore-tool --data-path PATH_TO_OSD \
--pgid PG_ID OBJECT \
get-attr KEY > OBJECT_ATTRS_FILE_NAME
    Copy to Clipboard Toggle word wrap

    Example

    [ceph: root@host01 /]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 \
--pgid 0.1c '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}' \
get-attr "oid" > zone_info.default.attr.txt
    Copy to Clipboard Toggle word wrap

  4. Set an object’s attributes:

    Syntax

    ceph-objectstore-tool --data-path PATH_TO_OSD \
--pgid PG_ID OBJECT \
set-attr KEY < OBJECT_ATTRS_FILE_NAME
    Copy to Clipboard Toggle word wrap

    Example

    [ceph: root@host01 /]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 \
--pgid 0.1c '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}' \
set-attr "oid" < zone_info.default.attr.txt
    Copy to Clipboard Toggle word wrap

  5. Remove an object’s attributes:

    Syntax

    ceph-objectstore-tool --data-path PATH_TO_OSD \
--pgid PG_ID OBJECT  \
rm-attr KEY
    Copy to Clipboard Toggle word wrap

    Example

    [ceph: root@host01 /]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 \
--pgid 0.1c '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}' \
rm-attr "oid"
    Copy to Clipboard Toggle word wrap

Volver arriba
Red Hat logoGithubredditYoutubeTwitter

Aprender

Pruebe, compre y venda

Comunidades

Acerca de la documentación de Red Hat

Ayudamos a los usuarios de Red Hat a innovar y alcanzar sus objetivos con nuestros productos y servicios con contenido en el que pueden confiar. Explore nuestras recientes actualizaciones.

Hacer que el código abierto sea más inclusivo

Red Hat se compromete a reemplazar el lenguaje problemático en nuestro código, documentación y propiedades web. Para más detalles, consulte el Blog de Red Hat.

Acerca de Red Hat

Ofrecemos soluciones reforzadas que facilitan a las empresas trabajar en plataformas y entornos, desde el centro de datos central hasta el perímetro de la red.

Theme

© 2025 Red Hat