Chapter 10. Troubleshooting Ceph objects

Procedure

1. Verify the appropriate OSD is down:

   [root@osd ~]# systemctl status ceph-osd@OSD_NUMBER

   Copy to Clipboard Toggle word wrap

   Example

   [root@osd ~]# systemctl status ceph-osd@1

   Copy to Clipboard Toggle word wrap

2. For containerized deployments, to access the bluestore tool, follow the below steps:

   1. Set noout flag on cluster.

      Example

      [root@mon ~]# ceph osd set noout

      Copy to Clipboard Toggle word wrap

   2. Login to the node hosting the OSD container.

   3. Backup /etc/systemd/system/ceph-osd@.service unit file to /root directory.

      Example

      [root@osd ~]# cp /etc/systemd/system/ceph-osd@.service /root/ceph-osd@.service.backup

      Copy to Clipboard Toggle word wrap

   4. Move /run/ceph-osd@OSD_ID.service-cid file to /root.

      Example

      [root@osd ~]# mv /run/ceph-osd@0.service-cid /root

      Copy to Clipboard Toggle word wrap

   5. Edit /etc/systemd/system/ceph-osd@.service unit file and add -it --entrypoint /bin/bash option to podman command.

      Example

      # Please do not change this file directly since it is managed by Ansible and will be overwritten
      [Unit]
      Description=Ceph OSD
      After=network.target
      
      [Service]
      EnvironmentFile=-/etc/environment
      ExecStartPre=-/usr/bin/rm -f /%t/%n-pid /%t/%n-cid
      ExecStartPre=-/usr/bin/podman rm -f ceph-osd-%i
      ExecStart=/usr/bin/podman run -it --entrypoint /bin/bash \
        -d --conmon-pidfile /%t/%n-pid --cidfile /%t/%n-cid \
        --rm \
        --net=host \
        --privileged=true \
        --pid=host \
        --ipc=host \
        --cpus=2 \
        -v /dev:/dev \
        -v /etc/localtime:/etc/localtime:ro \
        -v /var/lib/ceph:/var/lib/ceph:z \
        -v /etc/ceph:/etc/ceph:z \
        -v /var/run/ceph:/var/run/ceph:z \
        -v /var/run/udev/:/var/run/udev/ \
        -v /var/log/ceph:/var/log/ceph:z \
        -e OSD_BLUESTORE=1 -e OSD_FILESTORE=0 -e OSD_DMCRYPT=0 \
        -e CLUSTER=ceph \
        -v /run/lvm/:/run/lvm/ \
        -e CEPH_DAEMON=OSD_CEPH_VOLUME_ACTIVATE \
        -e CONTAINER_IMAGE=registry.redhat.io/rhceph/rhceph-4-rhel8:latest \
        -e OSD_ID=%i \
        -e DEBUG=stayalive \
        --name=ceph-osd-%i \
         \
        registry.redhat.io/rhceph/rhceph-4-rhel8:latest
      ExecStop=-/usr/bin/sh -c "/usr/bin/podman rm -f `cat /%t/%n-cid`"
      KillMode=none
      Restart=always
      RestartSec=10s
      TimeoutStartSec=120
      TimeoutStopSec=15
      Type=forking
      PIDFile=/%t/%n-pid
      
      [Install]
      WantedBy=multi-user.target

      Copy to Clipboard Toggle word wrap

   6. Reload systemd manager configuration.

      Example

      [root@osd ~]# systemctl daemon-reload

      Copy to Clipboard Toggle word wrap

   7. Restart the OSD service associated with the OSD_ID.

      Syntax

      systemctl restart ceph-osd@OSD_ID.service

      Copy to Clipboard Toggle word wrap

      Replace OSD_ID with the ID of the OSD.

      Example

      [root@osd ~]# systemctl restart ceph-osd@0.service

      Copy to Clipboard Toggle word wrap

   8. Login to the container associated with the OSD_ID.

      Syntax

      podman exec -it ceph-osd-OSD_ID /bin/bash

      Copy to Clipboard Toggle word wrap

      Example

      [root@osd ~]# podman exec -it ceph-osd-0 /bin/bash

      Copy to Clipboard Toggle word wrap

   9. Get osd fsid and activate the OSD to mount OSD’s logical volume (LV).

      Syntax

      ceph-volume lvm list |grep -A15 "osd\.OSD_ID"|grep "osd fsid"
      ceph-volume lvm activate --bluestore OSD_ID OSD_FSID

      Copy to Clipboard Toggle word wrap

      Example

      [root@osd ~]# ceph-volume lvm list |grep -A15 "osd\.0"|grep "osd fsid"
                    osd fsid                  087eee15-6561-40a3-8fe4-9583ba64a4ff
      [root@osd ~]# ceph-volume lvm activate --bluestore 0 087eee15-6561-40a3-8fe4-9583ba64a4ff
      Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-41c69f8f-30e2-4685-9c5c-c605898c5537/osd-data-d073e8b3-0b89-4271-af5b-83045fd000dc --path /var/lib/ceph/osd/ceph-0 --no-mon-config
      Running command: /usr/bin/ln -snf /dev/ceph-41c69f8f-30e2-4685-9c5c-c605898c5537/osd-data-d073e8b3-0b89-4271-af5b-83045fd000dc /var/lib/ceph/osd/ceph-0/block
      Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block
      Running command: /usr/bin/chown -R ceph:ceph /dev/mapper/ceph--41c69f8f--30e2--4685--9c5c--c605898c5537-osd--data--d073e8b3--0b89--4271--af5b--83045fd000dc
      Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/systemctl enable ceph-volume@lvm-0-087eee15-6561-40a3-8fe4-9583ba64a4ff
       stderr: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-0-087eee15-6561-40a3-8fe4-9583ba64a4ff.service  /usr/lib/systemd/system/ceph-volume@.service.
      Running command: /usr/bin/systemctl enable --runtime ceph-osd@0
       stderr: Created symlink /run/systemd/system/ceph-osd.target.wants/ceph osd@0.service  /usr/lib/systemd/system/ceph-osd@.service.
      Running command: /usr/bin/systemctl start ceph-osd@0
       stderr: Running in chroot, ignoring request: start
      --> ceph-volume lvm activate successful for osd ID: 0

      Copy to Clipboard Toggle word wrap

3. Identify all the objects within an OSD, regardless of their placement group:

   [root@osd ~]# ceph-objectstore-tool --data-path PATH_TO_OSD --op list

   Copy to Clipboard Toggle word wrap

   Example

   [root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --op list

   Copy to Clipboard Toggle word wrap

4. Identify all the objects within a placement group:

   [root@osd ~]# ceph-objectstore-tool --data-path PATH_TO_OSD --pgid PG_ID --op list

   Copy to Clipboard Toggle word wrap

   Example

   [root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c --op list

   Copy to Clipboard Toggle word wrap

5. Identify the PG an object belongs to:

   [root@osd ~]# ceph-objectstore-tool --data-path PATH_TO_OSD --op list OBJECT_ID

   Copy to Clipboard Toggle word wrap

   Example

   [root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --op list default.region

   Copy to Clipboard Toggle word wrap

6. For containerized deployments, to revert the changes, follow the below steps:

   1. After exiting the container, copy /etc/systemd/system/ceph-osd@.service unit file from /root directory.

      Example

      [root@osd ~]# cp /etc/systemd/system/ceph-osd@.service /root/ceph-osd@.service.modified
      [root@osd ~]# cp /root/ceph-osd@.service.backup /etc/systemd/system/ceph-osd@.service

      Copy to Clipboard Toggle word wrap

   2. Reload systemd manager configuration.

      Example

      [root@osd ~]# systemctl daemon-reload

      Copy to Clipboard Toggle word wrap

   3. Move /run/ceph-osd@OSD_ID.service-cid file to /tmp.

      Example

      [root@osd ~]# mv /run/ceph-osd@0.service-cid /tmp

      Copy to Clipboard Toggle word wrap

   4. Restart the OSD service associated with the OSD_ID.

      Syntax

      [root@osd ~]# systemctl restart ceph-osd@OSD_ID.service

      Copy to Clipboard Toggle word wrap

      Example

      [root@osd ~]# systemctl restart ceph-osd@0.service

      Copy to Clipboard Toggle word wrap

Procedure

1. Verify the appropriate OSD is down:

   [root@osd ~]# systemctl status ceph-osd@OSD_NUMBER

   Copy to Clipboard Toggle word wrap

   Example

   [root@osd ~]# systemctl status ceph-osd@1

   Copy to Clipboard Toggle word wrap

2. For containerized deployments, to access the bluestore tool, follow the below steps:

   1. Set noout flag on cluster.

      Example

      [root@mon ~]# ceph osd set noout

      Copy to Clipboard Toggle word wrap

   2. Login to the node hosting the OSD container.

   3. Backup /etc/systemd/system/ceph-osd@.service unit file to /root directory.

      Example

      [root@osd ~]# cp /etc/systemd/system/ceph-osd@.service /root/ceph-osd@.service.backup

      Copy to Clipboard Toggle word wrap

   4. Move /run/ceph-osd@OSD_ID.service-cid file to /root.

      Example

      [root@osd ~]# mv /run/ceph-osd@0.service-cid /root

      Copy to Clipboard Toggle word wrap

   5. Edit /etc/systemd/system/ceph-osd@.service unit file and add -it --entrypoint /bin/bash option to podman command.

      Example

      # Please do not change this file directly since it is managed by Ansible and will be overwritten
      [Unit]
      Description=Ceph OSD
      After=network.target
      
      [Service]
      EnvironmentFile=-/etc/environment
      ExecStartPre=-/usr/bin/rm -f /%t/%n-pid /%t/%n-cid
      ExecStartPre=-/usr/bin/podman rm -f ceph-osd-%i
      ExecStart=/usr/bin/podman run -it --entrypoint /bin/bash \
        -d --conmon-pidfile /%t/%n-pid --cidfile /%t/%n-cid \
        --rm \
        --net=host \
        --privileged=true \
        --pid=host \
        --ipc=host \
        --cpus=2 \
        -v /dev:/dev \
        -v /etc/localtime:/etc/localtime:ro \
        -v /var/lib/ceph:/var/lib/ceph:z \
        -v /etc/ceph:/etc/ceph:z \
        -v /var/run/ceph:/var/run/ceph:z \
        -v /var/run/udev/:/var/run/udev/ \
        -v /var/log/ceph:/var/log/ceph:z \
        -e OSD_BLUESTORE=1 -e OSD_FILESTORE=0 -e OSD_DMCRYPT=0 \
        -e CLUSTER=ceph \
        -v /run/lvm/:/run/lvm/ \
        -e CEPH_DAEMON=OSD_CEPH_VOLUME_ACTIVATE \
        -e CONTAINER_IMAGE=registry.redhat.io/rhceph/rhceph-4-rhel8:latest \
        -e OSD_ID=%i \
        -e DEBUG=stayalive \
        --name=ceph-osd-%i \
         \
        registry.redhat.io/rhceph/rhceph-4-rhel8:latest
      ExecStop=-/usr/bin/sh -c "/usr/bin/podman rm -f `cat /%t/%n-cid`"
      KillMode=none
      Restart=always
      RestartSec=10s
      TimeoutStartSec=120
      TimeoutStopSec=15
      Type=forking
      PIDFile=/%t/%n-pid
      
      [Install]
      WantedBy=multi-user.target

      Copy to Clipboard Toggle word wrap

   6. Reload systemd manager configuration.

      Example

      [root@osd ~]# systemctl daemon-reload

      Copy to Clipboard Toggle word wrap

   7. Restart the OSD service associated with the OSD_ID.

      Syntax

      systemctl restart ceph-osd@OSD_ID.service

      Copy to Clipboard Toggle word wrap

      Replace OSD_ID with the ID of the OSD.

      Example

      [root@osd ~]# systemctl restart ceph-osd@0.service

      Copy to Clipboard Toggle word wrap

   8. Login to the container associated with the OSD_ID.

      Syntax

      podman exec -it ceph-osd-OSD_ID /bin/bash

      Copy to Clipboard Toggle word wrap

      Example

      [root@osd ~]# podman exec -it ceph-osd-0 /bin/bash

      Copy to Clipboard Toggle word wrap

   9. Get osd fsid and activate the OSD to mount OSD’s logical volume (LV).

      Syntax

      ceph-volume lvm list |grep -A15 "osd\.OSD_ID"|grep "osd fsid"
      ceph-volume lvm activate --bluestore OSD_ID OSD_FSID

      Copy to Clipboard Toggle word wrap

      Example

      [root@osd ~]# ceph-volume lvm list |grep -A15 "osd\.0"|grep "osd fsid"
                    osd fsid                  087eee15-6561-40a3-8fe4-9583ba64a4ff
      [root@osd ~]# ceph-volume lvm activate --bluestore 0 087eee15-6561-40a3-8fe4-9583ba64a4ff
      Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-41c69f8f-30e2-4685-9c5c-c605898c5537/osd-data-d073e8b3-0b89-4271-af5b-83045fd000dc --path /var/lib/ceph/osd/ceph-0 --no-mon-config
      Running command: /usr/bin/ln -snf /dev/ceph-41c69f8f-30e2-4685-9c5c-c605898c5537/osd-data-d073e8b3-0b89-4271-af5b-83045fd000dc /var/lib/ceph/osd/ceph-0/block
      Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block
      Running command: /usr/bin/chown -R ceph:ceph /dev/mapper/ceph--41c69f8f--30e2--4685--9c5c--c605898c5537-osd--data--d073e8b3--0b89--4271--af5b--83045fd000dc
      Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/systemctl enable ceph-volume@lvm-0-087eee15-6561-40a3-8fe4-9583ba64a4ff
       stderr: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-0-087eee15-6561-40a3-8fe4-9583ba64a4ff.service  /usr/lib/systemd/system/ceph-volume@.service.
      Running command: /usr/bin/systemctl enable --runtime ceph-osd@0
       stderr: Created symlink /run/systemd/system/ceph-osd.target.wants/ceph osd@0.service  /usr/lib/systemd/system/ceph-osd@.service.
      Running command: /usr/bin/systemctl start ceph-osd@0
       stderr: Running in chroot, ignoring request: start
      --> ceph-volume lvm activate successful for osd ID: 0

      Copy to Clipboard Toggle word wrap

3. Use the ceph-objectstore-tool utility to list lost and unfound objects. Select the appropriate circumstance:

   1. To list all the lost objects:

      [root@osd ~]# ceph-objectstore-tool --data-path PATH_TO_OSD --op list-lost

      Copy to Clipboard Toggle word wrap

      Example

      [root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --op list-lost

      Copy to Clipboard Toggle word wrap

   2. To list all the lost objects within a placement group:

      [root@osd ~]# ceph-objectstore-tool --data-path PATH_TO_OSD --pgid PG_ID --op list-lost

      Copy to Clipboard Toggle word wrap

      Example

      [root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c --op list-lost

      Copy to Clipboard Toggle word wrap

   3. To list a lost object by its identifier:

      [root@osd ~]# ceph-objectstore-tool --data-path PATH_TO_OSD --op list-lost OBJECT_ID

      Copy to Clipboard Toggle word wrap

      Example

      [root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --op list-lost default.region

      Copy to Clipboard Toggle word wrap

4. For containerized deployments, to revert the changes, follow the below steps:

   1. After exiting the container, copy /etc/systemd/system/ceph-osd@.service unit file from /root directory.

      Example

      [root@osd ~]# cp /etc/systemd/system/ceph-osd@.service /root/ceph-osd@.service.modified
      [root@osd ~]# cp /root/ceph-osd@.service.backup /etc/systemd/system/ceph-osd@.service

      Copy to Clipboard Toggle word wrap

   2. Reload systemd manager configuration.

      Example

      [root@osd ~]# systemctl daemon-reload

      Copy to Clipboard Toggle word wrap

   3. Move /run/ceph-osd@OSD_ID.service-cid file to /tmp.

      Example

      [root@osd ~]# mv /run/ceph-osd@0.service-cid /tmp

      Copy to Clipboard Toggle word wrap

   4. Restart the OSD service associated with the OSD_ID.

      Syntax

      [root@osd ~]# systemctl restart ceph-osd@OSD_ID.service

      Copy to Clipboard Toggle word wrap

      Example

      [root@osd ~]# systemctl restart ceph-osd@0.service

      Copy to Clipboard Toggle word wrap

Procedure

1. Verify the appropriate OSD is down:

   Syntax

   [root@osd ~]# systemctl status ceph-osd@OSD_NUMBER

   Copy to Clipboard Toggle word wrap

   Example

   [root@osd ~]# systemctl status ceph-osd@1

   Copy to Clipboard Toggle word wrap

2. For containerized deployments, to access the bluestore tool, follow the below steps:

   1. Set noout flag on cluster.

      Example

      [root@mon ~]# ceph osd set noout

      Copy to Clipboard Toggle word wrap

   2. Login to the node hosting the OSD container.

   3. Backup /etc/systemd/system/ceph-osd@.service unit file to /root directory.

      Example

      [root@osd ~]# cp /etc/systemd/system/ceph-osd@.service /root/ceph-osd@.service.backup

      Copy to Clipboard Toggle word wrap

   4. Move /run/ceph-osd@OSD_ID.service-cid file to /root.

      Example

      [root@osd ~]# mv /run/ceph-osd@0.service-cid /root

      Copy to Clipboard Toggle word wrap

   5. Edit /etc/systemd/system/ceph-osd@.service unit file and add -it --entrypoint /bin/bash option to podman command.

      Example

      # Please do not change this file directly since it is managed by Ansible and will be overwritten
      [Unit]
      Description=Ceph OSD
      After=network.target
      
      [Service]
      EnvironmentFile=-/etc/environment
      ExecStartPre=-/usr/bin/rm -f /%t/%n-pid /%t/%n-cid
      ExecStartPre=-/usr/bin/podman rm -f ceph-osd-%i
      ExecStart=/usr/bin/podman run -it --entrypoint /bin/bash \
        -d --conmon-pidfile /%t/%n-pid --cidfile /%t/%n-cid \
        --rm \
        --net=host \
        --privileged=true \
        --pid=host \
        --ipc=host \
        --cpus=2 \
        -v /dev:/dev \
        -v /etc/localtime:/etc/localtime:ro \
        -v /var/lib/ceph:/var/lib/ceph:z \
        -v /etc/ceph:/etc/ceph:z \
        -v /var/run/ceph:/var/run/ceph:z \
        -v /var/run/udev/:/var/run/udev/ \
        -v /var/log/ceph:/var/log/ceph:z \
        -e OSD_BLUESTORE=1 -e OSD_FILESTORE=0 -e OSD_DMCRYPT=0 \
        -e CLUSTER=ceph \
        -v /run/lvm/:/run/lvm/ \
        -e CEPH_DAEMON=OSD_CEPH_VOLUME_ACTIVATE \
        -e CONTAINER_IMAGE=registry.redhat.io/rhceph/rhceph-4-rhel8:latest \
        -e OSD_ID=%i \
        -e DEBUG=stayalive \
        --name=ceph-osd-%i \
         \
        registry.redhat.io/rhceph/rhceph-4-rhel8:latest
      ExecStop=-/usr/bin/sh -c "/usr/bin/podman rm -f `cat /%t/%n-cid`"
      KillMode=none
      Restart=always
      RestartSec=10s
      TimeoutStartSec=120
      TimeoutStopSec=15
      Type=forking
      PIDFile=/%t/%n-pid
      
      [Install]
      WantedBy=multi-user.target

      Copy to Clipboard Toggle word wrap

   6. Reload systemd manager configuration.

      Example

      [root@osd ~]# systemctl daemon-reload

      Copy to Clipboard Toggle word wrap

   7. Restart the OSD service associated with the OSD_ID.

      Syntax

      systemctl restart ceph-osd@OSD_ID.service

      Copy to Clipboard Toggle word wrap

      Replace OSD_ID with the ID of the OSD.

      Example

      [root@osd ~]# systemctl restart ceph-osd@0.service

      Copy to Clipboard Toggle word wrap

   8. Login to the container associated with the OSD_ID.

      Syntax

      podman exec -it ceph-osd-OSD_ID /bin/bash

      Copy to Clipboard Toggle word wrap

      Example

      [root@osd ~]# podman exec -it ceph-osd-0 /bin/bash

      Copy to Clipboard Toggle word wrap

   9. Get osd fsid and activate the OSD to mount OSD’s logical volume (LV).

      Syntax

      ceph-volume lvm list |grep -A15 "osd\.OSD_ID"|grep "osd fsid"
      ceph-volume lvm activate --bluestore OSD_ID OSD_FSID

      Copy to Clipboard Toggle word wrap

      Example

      [root@osd ~]# ceph-volume lvm list |grep -A15 "osd\.0"|grep "osd fsid"
                    osd fsid                  087eee15-6561-40a3-8fe4-9583ba64a4ff
      [root@osd ~]# ceph-volume lvm activate --bluestore 0 087eee15-6561-40a3-8fe4-9583ba64a4ff
      Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-41c69f8f-30e2-4685-9c5c-c605898c5537/osd-data-d073e8b3-0b89-4271-af5b-83045fd000dc --path /var/lib/ceph/osd/ceph-0 --no-mon-config
      Running command: /usr/bin/ln -snf /dev/ceph-41c69f8f-30e2-4685-9c5c-c605898c5537/osd-data-d073e8b3-0b89-4271-af5b-83045fd000dc /var/lib/ceph/osd/ceph-0/block
      Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block
      Running command: /usr/bin/chown -R ceph:ceph /dev/mapper/ceph--41c69f8f--30e2--4685--9c5c--c605898c5537-osd--data--d073e8b3--0b89--4271--af5b--83045fd000dc
      Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/systemctl enable ceph-volume@lvm-0-087eee15-6561-40a3-8fe4-9583ba64a4ff
       stderr: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-0-087eee15-6561-40a3-8fe4-9583ba64a4ff.service  /usr/lib/systemd/system/ceph-volume@.service.
      Running command: /usr/bin/systemctl enable --runtime ceph-osd@0
       stderr: Created symlink /run/systemd/system/ceph-osd.target.wants/ceph osd@0.service  /usr/lib/systemd/system/ceph-osd@.service.
      Running command: /usr/bin/systemctl start ceph-osd@0
       stderr: Running in chroot, ignoring request: start
      --> ceph-volume lvm activate successful for osd ID: 0

      Copy to Clipboard Toggle word wrap

3. To list all the lost legacy objects:

   Syntax

   ceph-objectstore-tool --data-path PATH_TO_OSD --op fix-lost --dry-run

   Copy to Clipboard Toggle word wrap

   Example

   [root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --op fix-lost --dry-run

   Copy to Clipboard Toggle word wrap

4. Use the ceph-objectstore-tool utility to fix lost and unfound objects as a ceph user. Select the appropriate circumstance:

   • To fix all lost objects:

     Syntax

     su - ceph -c 'ceph-objectstore-tool --data-path PATH_TO_OSD --op fix-lost'

     Copy to Clipboard Toggle word wrap

     Example

     [root@osd ~]# su - ceph -c 'ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --op fix-lost'

     Copy to Clipboard Toggle word wrap

   • To fix all the lost objects within a placement group:

     su - ceph -c 'ceph-objectstore-tool --data-path _PATH_TO_OSD_ --pgid _PG_ID_ --op fix-lost'

     Copy to Clipboard Toggle word wrap

     Example

     [root@osd ~]# su - ceph -c 'ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c --op fix-lost'

     Copy to Clipboard Toggle word wrap

   • To fix a lost object by its identifier:

     Syntax

     su - ceph -c 'ceph-objectstore-tool --data-path PATH_TO_OSD --op fix-lost OBJECT_ID'

     Copy to Clipboard Toggle word wrap

     Example

     [root@osd ~]# su - ceph -c 'ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --op fix-lost default.region'

     Copy to Clipboard Toggle word wrap

5. For containerized deployments, to revert the changes, follow the below steps:

   1. After exiting the container, copy /etc/systemd/system/ceph-osd@.service unit file from /root directory.

      Example

      [root@osd ~]# cp /etc/systemd/system/ceph-osd@.service /root/ceph-osd@.service.modified
      [root@osd ~]# cp /root/ceph-osd@.service.backup /etc/systemd/system/ceph-osd@.service

      Copy to Clipboard Toggle word wrap

   2. Reload systemd manager configuration.

      Example

      [root@osd ~]# systemctl daemon-reload

      Copy to Clipboard Toggle word wrap

   3. Move /run/ceph-osd@OSD_ID.service-cid file to /tmp.

      Example

      [root@osd ~]# mv /run/ceph-osd@0.service-cid /tmp

      Copy to Clipboard Toggle word wrap

   4. Restart the OSD service associated with the OSD_ID.

      Syntax

      [root@osd ~]# systemctl restart ceph-osd@OSD_ID.service

      Copy to Clipboard Toggle word wrap

      Example

      [root@osd ~]# systemctl restart ceph-osd@0.service

      Copy to Clipboard Toggle word wrap

Procedure

1. Verify the appropriate OSD is down:

   [root@osd ~]# systemctl status ceph-osd@$OSD_NUMBER

   Copy to Clipboard Toggle word wrap

   Example

   [root@osd ~]# systemctl status ceph-osd@1

   Copy to Clipboard Toggle word wrap

2. For containerized deployments, to access the bluestore tool, follow the below steps:

   1. Set noout flag on cluster.

      Example

      [root@mon ~]# ceph osd set noout

      Copy to Clipboard Toggle word wrap

   2. Login to the node hosting the OSD container.

   3. Backup /etc/systemd/system/ceph-osd@.service unit file to /root directory.

      Example

      [root@osd ~]# cp /etc/systemd/system/ceph-osd@.service /root/ceph-osd@.service.backup

      Copy to Clipboard Toggle word wrap

   4. Move /run/ceph-osd@OSD_ID.service-cid file to /root.

      Example

      [root@osd ~]# mv /run/ceph-osd@0.service-cid /root

      Copy to Clipboard Toggle word wrap

   5. Edit /etc/systemd/system/ceph-osd@.service unit file and add -it --entrypoint /bin/bash option to podman command.

      Example

      # Please do not change this file directly since it is managed by Ansible and will be overwritten
      [Unit]
      Description=Ceph OSD
      After=network.target
      
      [Service]
      EnvironmentFile=-/etc/environment
      ExecStartPre=-/usr/bin/rm -f /%t/%n-pid /%t/%n-cid
      ExecStartPre=-/usr/bin/podman rm -f ceph-osd-%i
      ExecStart=/usr/bin/podman run -it --entrypoint /bin/bash \
        -d --conmon-pidfile /%t/%n-pid --cidfile /%t/%n-cid \
        --rm \
        --net=host \
        --privileged=true \
        --pid=host \
        --ipc=host \
        --cpus=2 \
        -v /dev:/dev \
        -v /etc/localtime:/etc/localtime:ro \
        -v /var/lib/ceph:/var/lib/ceph:z \
        -v /etc/ceph:/etc/ceph:z \
        -v /var/run/ceph:/var/run/ceph:z \
        -v /var/run/udev/:/var/run/udev/ \
        -v /var/log/ceph:/var/log/ceph:z \
        -e OSD_BLUESTORE=1 -e OSD_FILESTORE=0 -e OSD_DMCRYPT=0 \
        -e CLUSTER=ceph \
        -v /run/lvm/:/run/lvm/ \
        -e CEPH_DAEMON=OSD_CEPH_VOLUME_ACTIVATE \
        -e CONTAINER_IMAGE=registry.redhat.io/rhceph/rhceph-4-rhel8:latest \
        -e OSD_ID=%i \
        -e DEBUG=stayalive \
        --name=ceph-osd-%i \
         \
        registry.redhat.io/rhceph/rhceph-4-rhel8:latest
      ExecStop=-/usr/bin/sh -c "/usr/bin/podman rm -f `cat /%t/%n-cid`"
      KillMode=none
      Restart=always
      RestartSec=10s
      TimeoutStartSec=120
      TimeoutStopSec=15
      Type=forking
      PIDFile=/%t/%n-pid
      
      [Install]
      WantedBy=multi-user.target

      Copy to Clipboard Toggle word wrap

   6. Reload systemd manager configuration.

      Example

      [root@osd ~]# systemctl daemon-reload

      Copy to Clipboard Toggle word wrap

   7. Restart the OSD service associated with the OSD_ID.

      Syntax

      systemctl restart ceph-osd@OSD_ID.service

      Copy to Clipboard Toggle word wrap

      Replace OSD_ID with the ID of the OSD.

      Example

      [root@osd ~]# systemctl restart ceph-osd@0.service

      Copy to Clipboard Toggle word wrap

   8. Login to the container associated with the OSD_ID.

      Syntax

      podman exec -it ceph-osd-OSD_ID /bin/bash

      Copy to Clipboard Toggle word wrap

      Example

      [root@osd ~]# podman exec -it ceph-osd-0 /bin/bash

      Copy to Clipboard Toggle word wrap

   9. Get osd fsid and activate the OSD to mount OSD’s logical volume (LV).

      Syntax

      ceph-volume lvm list |grep -A15 "osd\.OSD_ID"|grep "osd fsid"
      ceph-volume lvm activate --bluestore OSD_ID OSD_FSID

      Copy to Clipboard Toggle word wrap

      Example

      [root@osd ~]# ceph-volume lvm list |grep -A15 "osd\.0"|grep "osd fsid"
                    osd fsid                  087eee15-6561-40a3-8fe4-9583ba64a4ff
      [root@osd ~]# ceph-volume lvm activate --bluestore 0 087eee15-6561-40a3-8fe4-9583ba64a4ff
      Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-41c69f8f-30e2-4685-9c5c-c605898c5537/osd-data-d073e8b3-0b89-4271-af5b-83045fd000dc --path /var/lib/ceph/osd/ceph-0 --no-mon-config
      Running command: /usr/bin/ln -snf /dev/ceph-41c69f8f-30e2-4685-9c5c-c605898c5537/osd-data-d073e8b3-0b89-4271-af5b-83045fd000dc /var/lib/ceph/osd/ceph-0/block
      Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block
      Running command: /usr/bin/chown -R ceph:ceph /dev/mapper/ceph--41c69f8f--30e2--4685--9c5c--c605898c5537-osd--data--d073e8b3--0b89--4271--af5b--83045fd000dc
      Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/systemctl enable ceph-volume@lvm-0-087eee15-6561-40a3-8fe4-9583ba64a4ff
       stderr: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-0-087eee15-6561-40a3-8fe4-9583ba64a4ff.service  /usr/lib/systemd/system/ceph-volume@.service.
      Running command: /usr/bin/systemctl enable --runtime ceph-osd@0
       stderr: Created symlink /run/systemd/system/ceph-osd.target.wants/ceph osd@0.service  /usr/lib/systemd/system/ceph-osd@.service.
      Running command: /usr/bin/systemctl start ceph-osd@0
       stderr: Running in chroot, ignoring request: start
      --> ceph-volume lvm activate successful for osd ID: 0

      Copy to Clipboard Toggle word wrap

3. Find the object by listing the objects of the OSD or placement group (PG).

4. Before setting the bytes on an object, make a backup and a working copy of the object:

   [root@osd ~]# ceph-objectstore-tool --data-path PATH_TO_OSD --pgid PG_ID \
   OBJECT \
   get-bytes > OBJECT_FILE_NAME
   
   [root@osd ~]# ceph-objectstore-tool --data-path PATH_TO_OSD --pgid PG_ID \
   OBJECT \
   get-bytes > OBJECT_FILE_NAME

   Copy to Clipboard Toggle word wrap

   Example

   [root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c \
   '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}'  \
   get-bytes > zone_info.default.backup
   
   [root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c \
   '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}'  \
   get-bytes > zone_info.default.working-copy

   Copy to Clipboard Toggle word wrap

5. Edit the working copy object file and modify the object contents accordingly.

6. Set the bytes of the object:

   [root@osd ~]# ceph-objectstore-tool --data-path PATH_TO_OSD --pgid PG_ID \
   OBJECT \
   set-bytes < OBJECT_FILE_NAME

   Copy to Clipboard Toggle word wrap

   Example

   [root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c \
   '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}' \
   set-bytes < zone_info.default.working-copy

   Copy to Clipboard Toggle word wrap

7. For containerized deployments, to revert the changes, follow the below steps:

   1. After exiting the container, copy /etc/systemd/system/ceph-osd@.service unit file from /root directory.

      Example

      [root@osd ~]# cp /etc/systemd/system/ceph-osd@.service /root/ceph-osd@.service.modified
      [root@osd ~]# cp /root/ceph-osd@.service.backup /etc/systemd/system/ceph-osd@.service

      Copy to Clipboard Toggle word wrap

   2. Reload systemd manager configuration.

      Example

      [root@osd ~]# systemctl daemon-reload

      Copy to Clipboard Toggle word wrap

   3. Move /run/ceph-osd@OSD_ID.service-cid file to /tmp.

      Example

      [root@osd ~]# mv /run/ceph-osd@0.service-cid /tmp

      Copy to Clipboard Toggle word wrap

   4. Restart the OSD service associated with the OSD_ID.

      Syntax

      [root@osd ~]# systemctl restart ceph-osd@OSD_ID.service

      Copy to Clipboard Toggle word wrap

      Example

      [root@osd ~]# systemctl restart ceph-osd@0.service

      Copy to Clipboard Toggle word wrap

Procedure

1. Verify the appropriate OSD is down:

   [root@osd ~]# systemctl status ceph-osd@$OSD_NUMBER

   Copy to Clipboard Toggle word wrap

   Example

   [root@osd ~]# systemctl status ceph-osd@1

   Copy to Clipboard Toggle word wrap

2. For containerized deployments, to access the bluestore tool, follow the below steps:

   1. Set noout flag on cluster.

      Example

      [root@mon ~]# ceph osd set noout

      Copy to Clipboard Toggle word wrap

   2. Login to the node hosting the OSD container.

   3. Backup /etc/systemd/system/ceph-osd@.service unit file to /root directory.

      Example

      [root@osd ~]# cp /etc/systemd/system/ceph-osd@.service /root/ceph-osd@.service.backup

      Copy to Clipboard Toggle word wrap

   4. Move /run/ceph-osd@OSD_ID.service-cid file to /root.

      Example

      [root@osd ~]# mv /run/ceph-osd@0.service-cid /root

      Copy to Clipboard Toggle word wrap

   5. Edit /etc/systemd/system/ceph-osd@.service unit file and add -it --entrypoint /bin/bash option to podman command.

      Example

      # Please do not change this file directly since it is managed by Ansible and will be overwritten
      [Unit]
      Description=Ceph OSD
      After=network.target
      
      [Service]
      EnvironmentFile=-/etc/environment
      ExecStartPre=-/usr/bin/rm -f /%t/%n-pid /%t/%n-cid
      ExecStartPre=-/usr/bin/podman rm -f ceph-osd-%i
      ExecStart=/usr/bin/podman run -it --entrypoint /bin/bash \
        -d --conmon-pidfile /%t/%n-pid --cidfile /%t/%n-cid \
        --rm \
        --net=host \
        --privileged=true \
        --pid=host \
        --ipc=host \
        --cpus=2 \
        -v /dev:/dev \
        -v /etc/localtime:/etc/localtime:ro \
        -v /var/lib/ceph:/var/lib/ceph:z \
        -v /etc/ceph:/etc/ceph:z \
        -v /var/run/ceph:/var/run/ceph:z \
        -v /var/run/udev/:/var/run/udev/ \
        -v /var/log/ceph:/var/log/ceph:z \
        -e OSD_BLUESTORE=1 -e OSD_FILESTORE=0 -e OSD_DMCRYPT=0 \
        -e CLUSTER=ceph \
        -v /run/lvm/:/run/lvm/ \
        -e CEPH_DAEMON=OSD_CEPH_VOLUME_ACTIVATE \
        -e CONTAINER_IMAGE=registry.redhat.io/rhceph/rhceph-4-rhel8:latest \
        -e OSD_ID=%i \
        -e DEBUG=stayalive \
        --name=ceph-osd-%i \
         \
        registry.redhat.io/rhceph/rhceph-4-rhel8:latest
      ExecStop=-/usr/bin/sh -c "/usr/bin/podman rm -f `cat /%t/%n-cid`"
      KillMode=none
      Restart=always
      RestartSec=10s
      TimeoutStartSec=120
      TimeoutStopSec=15
      Type=forking
      PIDFile=/%t/%n-pid
      
      [Install]
      WantedBy=multi-user.target

      Copy to Clipboard Toggle word wrap

   6. Reload systemd manager configuration.

      Example

      [root@osd ~]# systemctl daemon-reload

      Copy to Clipboard Toggle word wrap

   7. Restart the OSD service associated with the OSD_ID.

      Syntax

      systemctl restart ceph-osd@OSD_ID.service

      Copy to Clipboard Toggle word wrap

      Replace OSD_ID with the ID of the OSD.

      Example

      [root@osd ~]# systemctl restart ceph-osd@0.service

      Copy to Clipboard Toggle word wrap

   8. Login to the container associated with the OSD_ID.

      Syntax

      podman exec -it ceph-osd-OSD_ID /bin/bash

      Copy to Clipboard Toggle word wrap

      Example

      [root@osd ~]# podman exec -it ceph-osd-0 /bin/bash

      Copy to Clipboard Toggle word wrap

   9. Get osd fsid and activate the OSD to mount OSD’s logical volume (LV).

      Syntax

      ceph-volume lvm list |grep -A15 "osd\.OSD_ID"|grep "osd fsid"
      ceph-volume lvm activate --bluestore OSD_ID OSD_FSID

      Copy to Clipboard Toggle word wrap

      Example

      [root@osd ~]# ceph-volume lvm list |grep -A15 "osd\.0"|grep "osd fsid"
                    osd fsid                  087eee15-6561-40a3-8fe4-9583ba64a4ff
      [root@osd ~]# ceph-volume lvm activate --bluestore 0 087eee15-6561-40a3-8fe4-9583ba64a4ff
      Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-41c69f8f-30e2-4685-9c5c-c605898c5537/osd-data-d073e8b3-0b89-4271-af5b-83045fd000dc --path /var/lib/ceph/osd/ceph-0 --no-mon-config
      Running command: /usr/bin/ln -snf /dev/ceph-41c69f8f-30e2-4685-9c5c-c605898c5537/osd-data-d073e8b3-0b89-4271-af5b-83045fd000dc /var/lib/ceph/osd/ceph-0/block
      Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block
      Running command: /usr/bin/chown -R ceph:ceph /dev/mapper/ceph--41c69f8f--30e2--4685--9c5c--c605898c5537-osd--data--d073e8b3--0b89--4271--af5b--83045fd000dc
      Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/systemctl enable ceph-volume@lvm-0-087eee15-6561-40a3-8fe4-9583ba64a4ff
       stderr: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-0-087eee15-6561-40a3-8fe4-9583ba64a4ff.service  /usr/lib/systemd/system/ceph-volume@.service.
      Running command: /usr/bin/systemctl enable --runtime ceph-osd@0
       stderr: Created symlink /run/systemd/system/ceph-osd.target.wants/ceph osd@0.service  /usr/lib/systemd/system/ceph-osd@.service.
      Running command: /usr/bin/systemctl start ceph-osd@0
       stderr: Running in chroot, ignoring request: start
      --> ceph-volume lvm activate successful for osd ID: 0

      Copy to Clipboard Toggle word wrap

3. Remove an object:

   Syntax

   ceph-objectstore-tool --data-path PATH_TO_OSD --pgid PG_ID \
   OBJECT \
   remove

   Copy to Clipboard Toggle word wrap

   Example

   [root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c \
   '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}' \
   remove

   Copy to Clipboard Toggle word wrap

4. For containerized deployments, to revert the changes, follow the below steps:

   1. After exiting the container, copy /etc/systemd/system/ceph-osd@.service unit file from /root directory.

      Example

      [root@osd ~]# cp /etc/systemd/system/ceph-osd@.service /root/ceph-osd@.service.modified
      [root@osd ~]# cp /root/ceph-osd@.service.backup /etc/systemd/system/ceph-osd@.service

      Copy to Clipboard Toggle word wrap

   2. Reload systemd manager configuration.

      Example

      [root@osd ~]# systemctl daemon-reload

      Copy to Clipboard Toggle word wrap

   3. Move /run/ceph-osd@OSD_ID.service-cid file to /tmp.

      Example

      [root@osd ~]# mv /run/ceph-osd@0.service-cid /tmp

      Copy to Clipboard Toggle word wrap

   4. Restart the OSD service associated with the OSD_ID.

      Syntax

      [root@osd ~]# systemctl restart ceph-osd@OSD_ID.service

      Copy to Clipboard Toggle word wrap

      Example

      [root@osd ~]# systemctl restart ceph-osd@0.service

      Copy to Clipboard Toggle word wrap

Procedure

1. Verify the appropriate OSD is down:

   [root@osd ~]# systemctl status ceph-osd@OSD_NUMBER

   Copy to Clipboard Toggle word wrap

   Example

   [root@osd ~]# systemctl status ceph-osd@1

   Copy to Clipboard Toggle word wrap

2. For containerized deployments, to access the bluestore tool, follow the below steps:

   1. Set noout flag on cluster.

      Example

      [root@mon ~]# ceph osd set noout

      Copy to Clipboard Toggle word wrap

   2. Login to the node hosting the OSD container.

   3. Backup /etc/systemd/system/ceph-osd@.service unit file to /root directory.

      Example

      [root@osd ~]# cp /etc/systemd/system/ceph-osd@.service /root/ceph-osd@.service.backup

      Copy to Clipboard Toggle word wrap

   4. Move /run/ceph-osd@OSD_ID.service-cid file to /root.

      Example

      [root@osd ~]# mv /run/ceph-osd@0.service-cid /root

      Copy to Clipboard Toggle word wrap

   5. Edit /etc/systemd/system/ceph-osd@.service unit file and add -it --entrypoint /bin/bash option to podman command.

      Example

      # Please do not change this file directly since it is managed by Ansible and will be overwritten
      [Unit]
      Description=Ceph OSD
      After=network.target
      
      [Service]
      EnvironmentFile=-/etc/environment
      ExecStartPre=-/usr/bin/rm -f /%t/%n-pid /%t/%n-cid
      ExecStartPre=-/usr/bin/podman rm -f ceph-osd-%i
      ExecStart=/usr/bin/podman run -it --entrypoint /bin/bash \
        -d --conmon-pidfile /%t/%n-pid --cidfile /%t/%n-cid \
        --rm \
        --net=host \
        --privileged=true \
        --pid=host \
        --ipc=host \
        --cpus=2 \
        -v /dev:/dev \
        -v /etc/localtime:/etc/localtime:ro \
        -v /var/lib/ceph:/var/lib/ceph:z \
        -v /etc/ceph:/etc/ceph:z \
        -v /var/run/ceph:/var/run/ceph:z \
        -v /var/run/udev/:/var/run/udev/ \
        -v /var/log/ceph:/var/log/ceph:z \
        -e OSD_BLUESTORE=1 -e OSD_FILESTORE=0 -e OSD_DMCRYPT=0 \
        -e CLUSTER=ceph \
        -v /run/lvm/:/run/lvm/ \
        -e CEPH_DAEMON=OSD_CEPH_VOLUME_ACTIVATE \
        -e CONTAINER_IMAGE=registry.redhat.io/rhceph/rhceph-4-rhel8:latest \
        -e OSD_ID=%i \
        -e DEBUG=stayalive \
        --name=ceph-osd-%i \
         \
        registry.redhat.io/rhceph/rhceph-4-rhel8:latest
      ExecStop=-/usr/bin/sh -c "/usr/bin/podman rm -f `cat /%t/%n-cid`"
      KillMode=none
      Restart=always
      RestartSec=10s
      TimeoutStartSec=120
      TimeoutStopSec=15
      Type=forking
      PIDFile=/%t/%n-pid
      
      [Install]
      WantedBy=multi-user.target

      Copy to Clipboard Toggle word wrap

   6. Reload systemd manager configuration.

      Example

      [root@osd ~]# systemctl daemon-reload

      Copy to Clipboard Toggle word wrap

   7. Restart the OSD service associated with the OSD_ID.

      Syntax

      systemctl restart ceph-osd@OSD_ID.service

      Copy to Clipboard Toggle word wrap

      Replace OSD_ID with the ID of the OSD.

      Example

      [root@osd ~]# systemctl restart ceph-osd@0.service

      Copy to Clipboard Toggle word wrap

   8. Login to the container associated with the OSD_ID.

      Syntax

      podman exec -it ceph-osd-OSD_ID /bin/bash

      Copy to Clipboard Toggle word wrap

      Example

      [root@osd ~]# podman exec -it ceph-osd-0 /bin/bash

      Copy to Clipboard Toggle word wrap

   9. Get osd fsid and activate the OSD to mount OSD’s logical volume (LV).

      Syntax

      ceph-volume lvm list |grep -A15 "osd\.OSD_ID"|grep "osd fsid"
      ceph-volume lvm activate --bluestore OSD_ID OSD_FSID

      Copy to Clipboard Toggle word wrap

      Example

      [root@osd ~]# ceph-volume lvm list |grep -A15 "osd\.0"|grep "osd fsid"
                    osd fsid                  087eee15-6561-40a3-8fe4-9583ba64a4ff
      [root@osd ~]# ceph-volume lvm activate --bluestore 0 087eee15-6561-40a3-8fe4-9583ba64a4ff
      Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-41c69f8f-30e2-4685-9c5c-c605898c5537/osd-data-d073e8b3-0b89-4271-af5b-83045fd000dc --path /var/lib/ceph/osd/ceph-0 --no-mon-config
      Running command: /usr/bin/ln -snf /dev/ceph-41c69f8f-30e2-4685-9c5c-c605898c5537/osd-data-d073e8b3-0b89-4271-af5b-83045fd000dc /var/lib/ceph/osd/ceph-0/block
      Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block
      Running command: /usr/bin/chown -R ceph:ceph /dev/mapper/ceph--41c69f8f--30e2--4685--9c5c--c605898c5537-osd--data--d073e8b3--0b89--4271--af5b--83045fd000dc
      Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/systemctl enable ceph-volume@lvm-0-087eee15-6561-40a3-8fe4-9583ba64a4ff
       stderr: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-0-087eee15-6561-40a3-8fe4-9583ba64a4ff.service  /usr/lib/systemd/system/ceph-volume@.service.
      Running command: /usr/bin/systemctl enable --runtime ceph-osd@0
       stderr: Created symlink /run/systemd/system/ceph-osd.target.wants/ceph osd@0.service  /usr/lib/systemd/system/ceph-osd@.service.
      Running command: /usr/bin/systemctl start ceph-osd@0
       stderr: Running in chroot, ignoring request: start
      --> ceph-volume lvm activate successful for osd ID: 0

      Copy to Clipboard Toggle word wrap

3. List the object map:

   [root@osd ~]# ceph-objectstore-tool --data-path PATH_TO_OSD --pgid PG_ID \
   OBJECT \
   list-omap

   Copy to Clipboard Toggle word wrap

   Example

   [root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --pgid 0.1c \
   '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}' \
   list-omap

   Copy to Clipboard Toggle word wrap

4. For containerized deployments, to revert the changes, follow the below steps:

   1. After exiting the container, copy /etc/systemd/system/ceph-osd@.service unit file from /root directory.

      Example

      [root@osd ~]# cp /etc/systemd/system/ceph-osd@.service /root/ceph-osd@.service.modified
      [root@osd ~]# cp /root/ceph-osd@.service.backup /etc/systemd/system/ceph-osd@.service

      Copy to Clipboard Toggle word wrap

   2. Reload systemd manager configuration.

      Example

      [root@osd ~]# systemctl daemon-reload

      Copy to Clipboard Toggle word wrap

   3. Move /run/ceph-osd@OSD_ID.service-cid file to /tmp.

      Example

      [root@osd ~]# mv /run/ceph-osd@0.service-cid /tmp

      Copy to Clipboard Toggle word wrap

   4. Restart the OSD service associated with the OSD_ID.

      Syntax

      [root@osd ~]# systemctl restart ceph-osd@OSD_ID.service

      Copy to Clipboard Toggle word wrap

      Example

      [root@osd ~]# systemctl restart ceph-osd@0.service

      Copy to Clipboard Toggle word wrap

Procedure

1. For containerized deployments, to access the bluestore tool, follow the below steps:

   1. Set noout flag on cluster.

      Example

      [root@mon ~]# ceph osd set noout

      Copy to Clipboard Toggle word wrap

   2. Login to the node hosting the OSD container.

   3. Backup /etc/systemd/system/ceph-osd@.service unit file to /root directory.

      Example

      [root@osd ~]# cp /etc/systemd/system/ceph-osd@.service /root/ceph-osd@.service.backup

      Copy to Clipboard Toggle word wrap

   4. Move /run/ceph-osd@OSD_ID.service-cid file to /root.

      Example

      [root@osd ~]# mv /run/ceph-osd@0.service-cid /root

      Copy to Clipboard Toggle word wrap

   5. Edit /etc/systemd/system/ceph-osd@.service unit file and add -it --entrypoint /bin/bash option to podman command.

      Example

      # Please do not change this file directly since it is managed by Ansible and will be overwritten
      [Unit]
      Description=Ceph OSD
      After=network.target
      
      [Service]
      EnvironmentFile=-/etc/environment
      ExecStartPre=-/usr/bin/rm -f /%t/%n-pid /%t/%n-cid
      ExecStartPre=-/usr/bin/podman rm -f ceph-osd-%i
      ExecStart=/usr/bin/podman run -it --entrypoint /bin/bash \
        -d --conmon-pidfile /%t/%n-pid --cidfile /%t/%n-cid \
        --rm \
        --net=host \
        --privileged=true \
        --pid=host \
        --ipc=host \
        --cpus=2 \
        -v /dev:/dev \
        -v /etc/localtime:/etc/localtime:ro \
        -v /var/lib/ceph:/var/lib/ceph:z \
        -v /etc/ceph:/etc/ceph:z \
        -v /var/run/ceph:/var/run/ceph:z \
        -v /var/run/udev/:/var/run/udev/ \
        -v /var/log/ceph:/var/log/ceph:z \
        -e OSD_BLUESTORE=1 -e OSD_FILESTORE=0 -e OSD_DMCRYPT=0 \
        -e CLUSTER=ceph \
        -v /run/lvm/:/run/lvm/ \
        -e CEPH_DAEMON=OSD_CEPH_VOLUME_ACTIVATE \
        -e CONTAINER_IMAGE=registry.redhat.io/rhceph/rhceph-4-rhel8:latest \
        -e OSD_ID=%i \
        -e DEBUG=stayalive \
        --name=ceph-osd-%i \
         \
        registry.redhat.io/rhceph/rhceph-4-rhel8:latest
      ExecStop=-/usr/bin/sh -c "/usr/bin/podman rm -f `cat /%t/%n-cid`"
      KillMode=none
      Restart=always
      RestartSec=10s
      TimeoutStartSec=120
      TimeoutStopSec=15
      Type=forking
      PIDFile=/%t/%n-pid
      
      [Install]
      WantedBy=multi-user.target

      Copy to Clipboard Toggle word wrap

   6. Reload systemd manager configuration.

      Example

      [root@osd ~]# systemctl daemon-reload

      Copy to Clipboard Toggle word wrap

   7. Restart the OSD service associated with the OSD_ID.

      Syntax

      systemctl restart ceph-osd@OSD_ID.service

      Copy to Clipboard Toggle word wrap

      Replace OSD_ID with the ID of the OSD.

      Example

      [root@osd ~]# systemctl restart ceph-osd@0.service

      Copy to Clipboard Toggle word wrap

   8. Login to the container associated with the OSD_ID.

      Syntax

      podman exec -it ceph-osd-OSD_ID /bin/bash

      Copy to Clipboard Toggle word wrap

      Example

      [root@osd ~]# podman exec -it ceph-osd-0 /bin/bash

      Copy to Clipboard Toggle word wrap

   9. Get osd fsid and activate the OSD to mount OSD’s logical volume (LV).

      Syntax

      ceph-volume lvm list |grep -A15 "osd\.OSD_ID"|grep "osd fsid"
      ceph-volume lvm activate --bluestore OSD_ID OSD_FSID

      Copy to Clipboard Toggle word wrap

      Example

      [root@osd ~]# ceph-volume lvm list |grep -A15 "osd\.0"|grep "osd fsid"
                    osd fsid                  087eee15-6561-40a3-8fe4-9583ba64a4ff
      [root@osd ~]# ceph-volume lvm activate --bluestore 0 087eee15-6561-40a3-8fe4-9583ba64a4ff
      Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-41c69f8f-30e2-4685-9c5c-c605898c5537/osd-data-d073e8b3-0b89-4271-af5b-83045fd000dc --path /var/lib/ceph/osd/ceph-0 --no-mon-config
      Running command: /usr/bin/ln -snf /dev/ceph-41c69f8f-30e2-4685-9c5c-c605898c5537/osd-data-d073e8b3-0b89-4271-af5b-83045fd000dc /var/lib/ceph/osd/ceph-0/block
      Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block
      Running command: /usr/bin/chown -R ceph:ceph /dev/mapper/ceph--41c69f8f--30e2--4685--9c5c--c605898c5537-osd--data--d073e8b3--0b89--4271--af5b--83045fd000dc
      Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/systemctl enable ceph-volume@lvm-0-087eee15-6561-40a3-8fe4-9583ba64a4ff
       stderr: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-0-087eee15-6561-40a3-8fe4-9583ba64a4ff.service  /usr/lib/systemd/system/ceph-volume@.service.
      Running command: /usr/bin/systemctl enable --runtime ceph-osd@0
       stderr: Created symlink /run/systemd/system/ceph-osd.target.wants/ceph osd@0.service  /usr/lib/systemd/system/ceph-osd@.service.
      Running command: /usr/bin/systemctl start ceph-osd@0
       stderr: Running in chroot, ignoring request: start
      --> ceph-volume lvm activate successful for osd ID: 0

      Copy to Clipboard Toggle word wrap

2. Verify the appropriate OSD is down:

   Syntax

   systemctl status ceph-osd@OSD_NUMBER

   Copy to Clipboard Toggle word wrap

   Example

   [root@osd ~]# systemctl status ceph-osd@1

   Copy to Clipboard Toggle word wrap

3. Get the object map header:

   Syntax

   ceph-objectstore-tool --data-path PATH_TO_OSD \
   --pgid PG_ID OBJECT \
   get-omaphdr > OBJECT_MAP_FILE_NAME

   Copy to Clipboard Toggle word wrap

   Example

   [root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 \
   --pgid 0.1c '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}'  \
   get-omaphdr > zone_info.default.omaphdr.txt

   Copy to Clipboard Toggle word wrap

4. Set the object map header:

   Syntax

   ceph-objectstore-tool --data-path PATH_TO_OSD \
   --pgid PG_ID OBJECT \
   get-omaphdr < OBJECT_MAP_FILE_NAME

   Copy to Clipboard Toggle word wrap

   Example

   [root@osd ~]# su - ceph -c 'ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 \
   --pgid 0.1c '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}'  \
   set-omaphdr < zone_info.default.omaphdr.txt

   Copy to Clipboard Toggle word wrap

5. For containerized deployments, to revert the changes, follow the below steps:

   1. After exiting the container, copy /etc/systemd/system/ceph-osd@.service unit file from /root directory.

      Example

      [root@osd ~]# cp /etc/systemd/system/ceph-osd@.service /root/ceph-osd@.service.modified
      [root@osd ~]# cp /root/ceph-osd@.service.backup /etc/systemd/system/ceph-osd@.service

      Copy to Clipboard Toggle word wrap

   2. Reload systemd manager configuration.

      Example

      [root@osd ~]# systemctl daemon-reload

      Copy to Clipboard Toggle word wrap

   3. Move /run/ceph-osd@OSD_ID.service-cid file to /tmp.

      Example

      [root@osd ~]# mv /run/ceph-osd@0.service-cid /tmp

      Copy to Clipboard Toggle word wrap

   4. Restart the OSD service associated with the OSD_ID.

      Syntax

      [root@osd ~]# systemctl restart ceph-osd@OSD_ID.service

      Copy to Clipboard Toggle word wrap

      Example

      [root@osd ~]# systemctl restart ceph-osd@0.service

      Copy to Clipboard Toggle word wrap

Procedure

1. For containerized deployments, to access the bluestore tool, follow the below steps:

   1. Set noout flag on cluster.

      Example

      [root@mon ~]# ceph osd set noout

      Copy to Clipboard Toggle word wrap

   2. Login to the node hosting the OSD container.

   3. Backup /etc/systemd/system/ceph-osd@.service unit file to /root directory.

      Example

      [root@osd ~]# cp /etc/systemd/system/ceph-osd@.service /root/ceph-osd@.service.backup

      Copy to Clipboard Toggle word wrap

   4. Move /run/ceph-osd@OSD_ID.service-cid file to /root.

      Example

      [root@osd ~]# mv /run/ceph-osd@0.service-cid /root

      Copy to Clipboard Toggle word wrap

   5. Edit /etc/systemd/system/ceph-osd@.service unit file and add -it --entrypoint /bin/bash option to podman command.

      Example

      # Please do not change this file directly since it is managed by Ansible and will be overwritten
      [Unit]
      Description=Ceph OSD
      After=network.target
      
      [Service]
      EnvironmentFile=-/etc/environment
      ExecStartPre=-/usr/bin/rm -f /%t/%n-pid /%t/%n-cid
      ExecStartPre=-/usr/bin/podman rm -f ceph-osd-%i
      ExecStart=/usr/bin/podman run -it --entrypoint /bin/bash \
        -d --conmon-pidfile /%t/%n-pid --cidfile /%t/%n-cid \
        --rm \
        --net=host \
        --privileged=true \
        --pid=host \
        --ipc=host \
        --cpus=2 \
        -v /dev:/dev \
        -v /etc/localtime:/etc/localtime:ro \
        -v /var/lib/ceph:/var/lib/ceph:z \
        -v /etc/ceph:/etc/ceph:z \
        -v /var/run/ceph:/var/run/ceph:z \
        -v /var/run/udev/:/var/run/udev/ \
        -v /var/log/ceph:/var/log/ceph:z \
        -e OSD_BLUESTORE=1 -e OSD_FILESTORE=0 -e OSD_DMCRYPT=0 \
        -e CLUSTER=ceph \
        -v /run/lvm/:/run/lvm/ \
        -e CEPH_DAEMON=OSD_CEPH_VOLUME_ACTIVATE \
        -e CONTAINER_IMAGE=registry.redhat.io/rhceph/rhceph-4-rhel8:latest \
        -e OSD_ID=%i \
        -e DEBUG=stayalive \
        --name=ceph-osd-%i \
         \
        registry.redhat.io/rhceph/rhceph-4-rhel8:latest
      ExecStop=-/usr/bin/sh -c "/usr/bin/podman rm -f `cat /%t/%n-cid`"
      KillMode=none
      Restart=always
      RestartSec=10s
      TimeoutStartSec=120
      TimeoutStopSec=15
      Type=forking
      PIDFile=/%t/%n-pid
      
      [Install]
      WantedBy=multi-user.target

      Copy to Clipboard Toggle word wrap

   6. Reload systemd manager configuration.

      Example

      [root@osd ~]# systemctl daemon-reload

      Copy to Clipboard Toggle word wrap

   7. Restart the OSD service associated with the OSD_ID.

      Syntax

      systemctl restart ceph-osd@OSD_ID.service

      Copy to Clipboard Toggle word wrap

      Replace OSD_ID with the ID of the OSD.

      Example

      [root@osd ~]# systemctl restart ceph-osd@0.service

      Copy to Clipboard Toggle word wrap

   8. Login to the container associated with the OSD_ID.

      Syntax

      podman exec -it ceph-osd-OSD_ID /bin/bash

      Copy to Clipboard Toggle word wrap

      Example

      [root@osd ~]# podman exec -it ceph-osd-0 /bin/bash

      Copy to Clipboard Toggle word wrap

   9. Get osd fsid and activate the OSD to mount OSD’s logical volume (LV).

      Syntax

      ceph-volume lvm list |grep -A15 "osd\.OSD_ID"|grep "osd fsid"
      ceph-volume lvm activate --bluestore OSD_ID OSD_FSID

      Copy to Clipboard Toggle word wrap

      Example

      [root@osd ~]# ceph-volume lvm list |grep -A15 "osd\.0"|grep "osd fsid"
                    osd fsid                  087eee15-6561-40a3-8fe4-9583ba64a4ff
      [root@osd ~]# ceph-volume lvm activate --bluestore 0 087eee15-6561-40a3-8fe4-9583ba64a4ff
      Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-41c69f8f-30e2-4685-9c5c-c605898c5537/osd-data-d073e8b3-0b89-4271-af5b-83045fd000dc --path /var/lib/ceph/osd/ceph-0 --no-mon-config
      Running command: /usr/bin/ln -snf /dev/ceph-41c69f8f-30e2-4685-9c5c-c605898c5537/osd-data-d073e8b3-0b89-4271-af5b-83045fd000dc /var/lib/ceph/osd/ceph-0/block
      Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block
      Running command: /usr/bin/chown -R ceph:ceph /dev/mapper/ceph--41c69f8f--30e2--4685--9c5c--c605898c5537-osd--data--d073e8b3--0b89--4271--af5b--83045fd000dc
      Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/systemctl enable ceph-volume@lvm-0-087eee15-6561-40a3-8fe4-9583ba64a4ff
       stderr: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-0-087eee15-6561-40a3-8fe4-9583ba64a4ff.service  /usr/lib/systemd/system/ceph-volume@.service.
      Running command: /usr/bin/systemctl enable --runtime ceph-osd@0
       stderr: Created symlink /run/systemd/system/ceph-osd.target.wants/ceph osd@0.service  /usr/lib/systemd/system/ceph-osd@.service.
      Running command: /usr/bin/systemctl start ceph-osd@0
       stderr: Running in chroot, ignoring request: start
      --> ceph-volume lvm activate successful for osd ID: 0

      Copy to Clipboard Toggle word wrap

2. Get the object map key:

   Syntax

   ceph-objectstore-tool --data-path PATH_TO_OSD \
   --pgid PG_ID OBJECT \
   get-omap KEY > OBJECT_MAP_FILE_NAME

   Copy to Clipboard Toggle word wrap

   Example

   [root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 \
   --pgid 0.1c '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}'  \
   get-omap "" > zone_info.default.omap.txt

   Copy to Clipboard Toggle word wrap

3. Set the object map key:

   Syntax

   ceph-objectstore-tool --data-path PATH_TO_OSD \
   --pgid PG_ID OBJECT \
   set-omap KEY < OBJECT_MAP_FILE_NAME

   Copy to Clipboard Toggle word wrap

   Example

   [root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 \
   --pgid 0.1c '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}'  \
   set-omap "" < zone_info.default.omap.txt

   Copy to Clipboard Toggle word wrap

   • Remove the object map key:

     Syntax

     ceph-objectstore-tool --data-path PATH_TO_OSD \
     --pgid PG_ID OBJECT \
     rm-omap KEY

     Copy to Clipboard Toggle word wrap

     Example

     [root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 \
     --pgid 0.1c '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}'  \
     rm-omap ""

     Copy to Clipboard Toggle word wrap

4. For containerized deployments, to revert the changes, follow the below steps:

   1. After exiting the container, copy /etc/systemd/system/ceph-osd@.service unit file from /root directory.

      Example

      [root@osd ~]# cp /etc/systemd/system/ceph-osd@.service /root/ceph-osd@.service.modified
      [root@osd ~]# cp /root/ceph-osd@.service.backup /etc/systemd/system/ceph-osd@.service

      Copy to Clipboard Toggle word wrap

   2. Reload systemd manager configuration.

      Example

      [root@osd ~]# systemctl daemon-reload

      Copy to Clipboard Toggle word wrap

   3. Move /run/ceph-osd@OSD_ID.service-cid file to /tmp.

      Example

      [root@osd ~]# mv /run/ceph-osd@0.service-cid /tmp

      Copy to Clipboard Toggle word wrap

   4. Restart the OSD service associated with the OSD_ID.

      Syntax

      [root@osd ~]# systemctl restart ceph-osd@OSD_ID.service

      Copy to Clipboard Toggle word wrap

      Example

      [root@osd ~]# systemctl restart ceph-osd@0.service

      Copy to Clipboard Toggle word wrap

Procedure

1. Verify the appropriate OSD is down:

   [root@osd ~]# systemctl status ceph-osd@OSD_NUMBER

   Copy to Clipboard Toggle word wrap

   Example

   [root@osd ~]# systemctl status ceph-osd@1

   Copy to Clipboard Toggle word wrap

2. For containerized deployments, to access the bluestore tool, follow the below steps:

   1. Set noout flag on cluster.

      Example

      [root@mon ~]# ceph osd set noout

      Copy to Clipboard Toggle word wrap

   2. Login to the node hosting the OSD container.

   3. Backup /etc/systemd/system/ceph-osd@.service unit file to /root directory.

      Example

      [root@osd ~]# cp /etc/systemd/system/ceph-osd@.service /root/ceph-osd@.service.backup

      Copy to Clipboard Toggle word wrap

   4. Move /run/ceph-osd@OSD_ID.service-cid file to /root.

      Example

      [root@osd ~]# mv /run/ceph-osd@0.service-cid /root

      Copy to Clipboard Toggle word wrap

   5. Edit /etc/systemd/system/ceph-osd@.service unit file and add -it --entrypoint /bin/bash option to podman command.

      Example

      # Please do not change this file directly since it is managed by Ansible and will be overwritten
      [Unit]
      Description=Ceph OSD
      After=network.target
      
      [Service]
      EnvironmentFile=-/etc/environment
      ExecStartPre=-/usr/bin/rm -f /%t/%n-pid /%t/%n-cid
      ExecStartPre=-/usr/bin/podman rm -f ceph-osd-%i
      ExecStart=/usr/bin/podman run -it --entrypoint /bin/bash \
        -d --conmon-pidfile /%t/%n-pid --cidfile /%t/%n-cid \
        --rm \
        --net=host \
        --privileged=true \
        --pid=host \
        --ipc=host \
        --cpus=2 \
        -v /dev:/dev \
        -v /etc/localtime:/etc/localtime:ro \
        -v /var/lib/ceph:/var/lib/ceph:z \
        -v /etc/ceph:/etc/ceph:z \
        -v /var/run/ceph:/var/run/ceph:z \
        -v /var/run/udev/:/var/run/udev/ \
        -v /var/log/ceph:/var/log/ceph:z \
        -e OSD_BLUESTORE=1 -e OSD_FILESTORE=0 -e OSD_DMCRYPT=0 \
        -e CLUSTER=ceph \
        -v /run/lvm/:/run/lvm/ \
        -e CEPH_DAEMON=OSD_CEPH_VOLUME_ACTIVATE \
        -e CONTAINER_IMAGE=registry.redhat.io/rhceph/rhceph-4-rhel8:latest \
        -e OSD_ID=%i \
        -e DEBUG=stayalive \
        --name=ceph-osd-%i \
         \
        registry.redhat.io/rhceph/rhceph-4-rhel8:latest
      ExecStop=-/usr/bin/sh -c "/usr/bin/podman rm -f `cat /%t/%n-cid`"
      KillMode=none
      Restart=always
      RestartSec=10s
      TimeoutStartSec=120
      TimeoutStopSec=15
      Type=forking
      PIDFile=/%t/%n-pid
      
      [Install]
      WantedBy=multi-user.target

      Copy to Clipboard Toggle word wrap

   6. Reload systemd manager configuration.

      Example

      [root@osd ~]# systemctl daemon-reload

      Copy to Clipboard Toggle word wrap

   7. Restart the OSD service associated with the OSD_ID.

      Syntax

      systemctl restart ceph-osd@OSD_ID.service

      Copy to Clipboard Toggle word wrap

      Replace OSD_ID with the ID of the OSD.

      Example

      [root@osd ~]# systemctl restart ceph-osd@0.service

      Copy to Clipboard Toggle word wrap

   8. Login to the container associated with the OSD_ID.

      Syntax

      podman exec -it ceph-osd-OSD_ID /bin/bash

      Copy to Clipboard Toggle word wrap

      Example

      [root@osd ~]# podman exec -it ceph-osd-0 /bin/bash

      Copy to Clipboard Toggle word wrap

   9. Get osd fsid and activate the OSD to mount OSD’s logical volume (LV).

      Syntax

      ceph-volume lvm list |grep -A15 "osd\.OSD_ID"|grep "osd fsid"
      ceph-volume lvm activate --bluestore OSD_ID OSD_FSID

      Copy to Clipboard Toggle word wrap

      Example

      [root@osd ~]# ceph-volume lvm list |grep -A15 "osd\.0"|grep "osd fsid"
                    osd fsid                  087eee15-6561-40a3-8fe4-9583ba64a4ff
      [root@osd ~]# ceph-volume lvm activate --bluestore 0 087eee15-6561-40a3-8fe4-9583ba64a4ff
      Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-41c69f8f-30e2-4685-9c5c-c605898c5537/osd-data-d073e8b3-0b89-4271-af5b-83045fd000dc --path /var/lib/ceph/osd/ceph-0 --no-mon-config
      Running command: /usr/bin/ln -snf /dev/ceph-41c69f8f-30e2-4685-9c5c-c605898c5537/osd-data-d073e8b3-0b89-4271-af5b-83045fd000dc /var/lib/ceph/osd/ceph-0/block
      Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block
      Running command: /usr/bin/chown -R ceph:ceph /dev/mapper/ceph--41c69f8f--30e2--4685--9c5c--c605898c5537-osd--data--d073e8b3--0b89--4271--af5b--83045fd000dc
      Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
      Running command: /usr/bin/systemctl enable ceph-volume@lvm-0-087eee15-6561-40a3-8fe4-9583ba64a4ff
       stderr: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-0-087eee15-6561-40a3-8fe4-9583ba64a4ff.service  /usr/lib/systemd/system/ceph-volume@.service.
      Running command: /usr/bin/systemctl enable --runtime ceph-osd@0
       stderr: Created symlink /run/systemd/system/ceph-osd.target.wants/ceph osd@0.service  /usr/lib/systemd/system/ceph-osd@.service.
      Running command: /usr/bin/systemctl start ceph-osd@0
       stderr: Running in chroot, ignoring request: start
      --> ceph-volume lvm activate successful for osd ID: 0

      Copy to Clipboard Toggle word wrap

3. List the object’s attributes:

   ceph-objectstore-tool --data-path PATH_TO_OSD \
   --pgid PG_ID OBJECT \
   list-attrs

   Copy to Clipboard Toggle word wrap

   Example

   [root@osd ~]# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 \
   --pgid 0.1c '{"oid":"zone_info.default","key":"","snapid":-2,"hash":235010478,"max":0,"pool":11,"namespace":""}' \
   list-attrs

   Copy to Clipboard Toggle word wrap

4. For containerized deployments, to revert the changes, follow the below steps:

   1. After exiting the container, copy /etc/systemd/system/ceph-osd@.service unit file from /root directory.

      Example

      [root@osd ~]# cp /etc/systemd/system/ceph-osd@.service /root/ceph-osd@.service.modified
      [root@osd ~]# cp /root/ceph-osd@.service.backup /etc/systemd/system/ceph-osd@.service

      Copy to Clipboard Toggle word wrap

   2. Reload systemd manager configuration.

      Example

      [root@osd ~]# systemctl daemon-reload

      Copy to Clipboard Toggle word wrap

   3. Move /run/ceph-osd@OSD_ID.service-cid file to /tmp.

      Example

      [root@osd ~]# mv /run/ceph-osd@0.service-cid /tmp

      Copy to Clipboard Toggle word wrap

   4. Restart the OSD service associated with the OSD_ID.

      Syntax

      [root@osd ~]# systemctl restart ceph-osd@OSD_ID.service

      Copy to Clipboard Toggle word wrap

      Example

      [root@osd ~]# systemctl restart ceph-osd@0.service

      Copy to Clipboard Toggle word wrap