Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 7. Management of ceph-immutable-object-cache daemons

As a storage administrator, use the ceph-immutable-object-cache daemons to cache the parent image content on the local disk. This cache is in the local caching directory. Future reads on that data use the local cache.

Figure 7.1. Ceph immutable cache daemon

Ceph immutable cache daemon
View larger image
Ceph immutable cache daemon

Cloned Block device images usually modify only a small fraction of the parent image. For example, in a virtual desktop interface (VDI), the virtual machines are cloned from the same base image and initially differ only by the hostname and the IP address. During the bootup, if you use a local cache of the parent image, this speeds up reads on the caching host. This change reduces the client to cluster network traffic.

Reasons to use ceph-immutable-object-cache daemons

The ceph-immutable-object-cache daemon is a part of Red Hat Ceph Storage. It is a scalable, open-source, and distributed storage system. It connects to local clusters with the RADOS protocol, relying on default search paths to find ceph.conf files, monitor addresses and authentication information for them such as /etc/ceph/CLUSTER.conf, /etc/ceph/CLUSTER.keyring, and /etc/ceph/CLUSTER.NAME.keyring, where CLUSTER is the human-friendly name of the cluster, and NAME is the RADOS user to connect as an example, client.ceph-immutable-object-cache.

Key components of the daemon

The ceph-immutable-object-cache daemon has the following parts:

  • Domain socket based inter-process communication (IPC): The daemon listens on a local domain socket on start-up and waits for connections from librbd clients.
  • Least recently used (LRU) based promotion or demotion policy: The daemon maintains in-memory statistics of cache-hits on each cache file. It demotes the cold cache if capacity reaches to the configured threshold.
  • File-based caching store: The daemon maintains a simple file based cache store. On promotion the RADOS objects are fetched from RADOS cluster and stored in the local caching directory.

When you open each cloned RBD image, librbd tries to connect to the cache daemon through its Unix domain socket. Once successfully connected, librbd coordinates with the daemon on the subsequent reads. If there is a read that is not cached, the daemon promotes the RADOS object to the local caching directory, so the next read on that object is serviced from cache. The daemon also maintains simple LRU statistics so that under capacity pressure it evicts cold cache files as needed.

Note

For better performance, use SSDs as the underlying storage.

The ceph-immutable-object-cache is a daemon for object cache of RADOS objects among Ceph clusters.

Important

To use the ceph-immutable-object-cache daemon, you must be able to connect RADOS clusters.

The daemon promotes the objects to a local directory. These cache objects service the future reads. You can configure the daemon by installing the ceph-immutable-object-cache package.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • At least one SSD for the cache.

Procedure

  1. Enable the RBD shared read only parent image cache. Add the following parameters under [client] in the /etc/ceph/ceph.conf file:

    Example

    [root@ceph-host01 ~]# vi /etc/ceph/ceph.conf

[client]
rbd parent cache enabled = true
rbd plugins = parent_cache
    Copy to Clipboard Toggle word wrap

    Restart the cluster.

  2. Install the ceph-immutable-object-cache package:

    Example

    [root@ceph-host1 ~]# dnf install ceph-immutable-object-cache
    Copy to Clipboard Toggle word wrap

  3. Create a unique Ceph user ID, the keyring:

    Syntax

    ceph auth get-or-create client.ceph-immutable-object-cache.USER_NAME mon 'profile rbd' osd 'profile rbd-read-only'
    Copy to Clipboard Toggle word wrap

    Example

    [root@ceph-host1 ~]# ceph auth get-or-create client.ceph-immutable-object-cache.user mon 'profile rbd' osd 'profile rbd-read-only'

[client.ceph-immutable-object-cache.user]
	key = AQCVPH1gFgHRAhAAp8ExRIsoxQK4QSYSRoVJLw==
    Copy to Clipboard Toggle word wrap

    Copy this keyring.

  4. In the /etc/ceph directory, create a file and paste the keyring:

    Example

    [root@ceph-host1 ]# vi /etc/ceph/ceph.client.ceph-immutable-object-cache.user.keyring

[client.ceph-immutable-object-cache.user]
	key = AQCVPH1gFgHRAhAAp8ExRIsoxQK4QSYSRoVJLw
    Copy to Clipboard Toggle word wrap

  5. Enable the daemon:

    Syntax

    systemctl enable ceph-immutable-object-cache@ceph-immutable-object-cache.USER_NAME
    Copy to Clipboard Toggle word wrap

    Specify the USER_NAME as the daemon instance.

    Example

    [root@ceph-host1 ~]# systemctl enable ceph-immutable-object-cache@ceph-immutable-object-cache.user

Created symlink /etc/systemd/system/ceph-immutable-object-cache.target.wants/ceph-immutable-object-cache@ceph-immutable-object-cache.user.service  /usr/lib/systemd/system/ceph-immutable-object-cache@.service.
    Copy to Clipboard Toggle word wrap

  6. Start the ceph-immutable-object-cache daemon:

    Syntax

    systemctl start ceph-immutable-object-cache@ceph-immutable-object-cache.USER_NAME
    Copy to Clipboard Toggle word wrap

    Example

    [root@ceph-host1 ~]# systemctl start ceph-immutable-object-cache@ceph-immutable-object-cache.user
    Copy to Clipboard Toggle word wrap

Verification

  • Check the status of the configuration:

    Syntax

    systemctl status ceph-immutable-object-cache@ceph-immutable-object-cache.USER_NAME
    Copy to Clipboard Toggle word wrap

    Example

    [root@ceph-host1 ~]# systemctl status ceph-immutable-object-cache@ceph-immutable-object-cache.user

● ceph-immutable-object-cache@ceph-immutable-object-cache.user>
  Loaded: loaded (/usr/lib/systemd/system/ceph-immutable-objec>
  Active: active (running) since Mon 2021-04-19 13:49:06 IST; >
Main PID: 85020 (ceph-immutable-)
   Tasks: 15 (limit: 49451)
  Memory: 8.3M
  CGroup: /system.slice/system-ceph\x2dimmutable\x2dobject\x2d>
          └─85020 /usr/bin/ceph-immutable-object-cache -f --cl>
    Copy to Clipboard Toggle word wrap

A few important generic settings of ceph-immutable-object-cache daemons are listed.

immutable_object_cache_sock
Description
The path to the domain socket used for communication between librbd clients and the ceph-immutable-object-cache daemon.
Type
String
Default
/var/run/ceph/immutable_object_cache_sock
immutable_object_cache_path
Description
The immutable object cache data directory.
Type
String
Default
/tmp/ceph_immutable_object_cache
immutable_object_cache_max_size
Description
The maximum size for immutable cache.
Type
Size
Default
1G
immutable_object_cache_watermark
Description
The high-water mark for the cache. The value is between zero and one. If the cache size reaches this threshold the daemon starts to delete cold cache based on LRU statistics.
Type
Float
Default
0.9

The ceph-immutable-object-cache daemons supports throttling which supports the settings described.

immutable_object_cache_qos_schedule_tick_min
Description
Minimum schedule tick for immutable object cache.
Type
Milliseconds
Default
50
immutable_object_cache_qos_iops_limit
Description
User-defined immutable object cache IO operations limit per second.
Type
Integer
Default
0
immutable_object_cache_qos_iops_burst
Description
User-defined burst limit of immutable object cache IO operations.
Type
Integer
Default
0
immutable_object_cache_qos_iops_burst_seconds
Description
User-defined burst duration in seconds of immutable object cache IO operations.
Type
Seconds
Default
1
immutable_object_cache_qos_bps_limit
Description
User-defined immutable object cache IO bytes limit per second.
Type
Integer
Default
0
immutable_object_cache_qos_bps_burst
Description
User-defined burst limit of immutable object cache IO bytes.
Type
Integer
Default
0
immutable_object_cache_qos_bps_burst_seconds
Description
The desired burst limit of read operations.
Type
Seconds
Default
1
Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat