Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 13. Add an OSD

When you have a cluster up and running, you may add OSDs or remove OSDs from the cluster at runtime.

A Ceph OSD generally consists of one ceph-osd daemon for one storage drive (and its associated journal) within a host machine. If your host has multiple storage drives, you may map one ceph-osd daemon for each drive.

Generally, it’s a good idea to check the capacity of your cluster to see if you are reaching the upper end of its capacity. As your cluster reaches its near full ratio, you should add one or more OSDs to expand your cluster’s capacity.

Important

Do not let your cluster reach its full ratio before adding an OSD. OSD failures that occur after the cluster reaches its near full ratio may cause the cluster to exceed its full ratio. Ceph will block write access to protect your data until you resolve storage capacity issues. DO NOT REMOVE OSDs without considering the impact on the full ratio first.

OSDs and their supporting hardware should be similarly configured as a storage strategy for the pool(s) that will use the OSDs. Ceph prefers uniform hardware across pools for a consistent performance profile. For best performance, consider a CRUSH hierarchy with drives of the same type/size. View the Storage Strategies guide for details.

If you are adding drives of dissimilar size, you can adjust their weights. When you add the OSD to the CRUSH map, consider the weight you give to the new OSD. Hard drive capacity grows approximately 40% per year, so newer OSD hosts may have larger hard drives than older hosts in the cluster (i.e., they may have greater weight).

13.1. ceph-deploy
Copy link

Before creating OSDs, consider the following:

  • We recommend using the XFS filesystem (default).
  • We recommend using SSDs for journals. It is common to partition SSDs to serve multiple OSDs. Ensure that the number of SSD partitions does not exceed your SSD’s sequential write limits. Also, ensure that SSD partitions are properly aligned, or their write performance will suffer.
  • We recommend using ceph-deploy disk zap on a Ceph OSD drive before executing ceph-deploy osd create. For example: 
ceph-deploy disk zap <ceph-node>:<data-drive>
Copy to Clipboard Toggle word wrap

From your admin node, use ceph-deploy to prepare the OSDs. 

ceph-deploy osd prepare <ceph-node>:<data-drive> <journal-partition> [<ceph-node>:<data-drive> <journal-partition>]
Copy to Clipboard Toggle word wrap

For example: 

ceph-deploy osd prepare node2:sdb ssdb node3:sdd ssdb node4:sdd ssdb
Copy to Clipboard Toggle word wrap

In the foregoing example, sdb is a spinning hard drive. Ceph will use the entire drive for OSD data. ssdb is a partition on an SSD drive, which Ceph will use to store the journal for the OSD.

Once you prepare OSDs, use ceph-deploy to activate the OSDs. 

ceph-deploy osd activate <ceph-node>:<data-drive> <journal-partition> [<ceph-node>:<data-drive> <journal-partition>]
Copy to Clipboard Toggle word wrap

For example: 

ceph-deploy osd activate node2:sdb ssdb node3:sdd ssdb node4:sdd ssdb
Copy to Clipboard Toggle word wrap

To achieve an active + clean state, you must add as many OSDs as the value of osd pool default size = <n> from your Ceph configuration file.

13.2. manual
Copy link

This procedure sets up a ceph-osd daemon, configures it to use one drive, and configures the cluster to distribute data to the OSD. If your host has multiple drives, you may add an OSD for each drive by repeating this procedure.

To add an OSD, create a data directory for it, mount a drive to that directory, add the OSD to the cluster, and then add it to the CRUSH map.

  1. Create the OSD. If no UUID is given, it will be set automatically when the OSD starts up. The following command will output the OSD number, which you will need for subsequent steps. : 

    ceph osd create [{uuid}]
    Copy to Clipboard Toggle word wrap

  2. Create the default directory on your new OSD. : 

    ssh {new-osd-host}
sudo mkdir /var/lib/ceph/osd/ceph-{osd-number}
    Copy to Clipboard Toggle word wrap

  3. If the OSD is for a drive other than the OS drive, prepare it for use with Ceph, and mount it to the directory you just created: 

    ssh {new-osd-host}
sudo mkfs -t {fstype} /dev/{drive}
sudo mount -o user_xattr /dev/{hdd} /var/lib/ceph/osd/ceph-{osd-number}
    Copy to Clipboard Toggle word wrap

  4. Initialize the OSD data directory. : 

    ssh {new-osd-host}
ceph-osd -i {osd-num} --mkfs --mkkey
    Copy to Clipboard Toggle word wrap

    The directory must be empty before you can run ceph-osd.

  5. Register the OSD authentication key. The value of ceph for ceph-{osd-num} in the path is the $cluster-$id. If your cluster name differs from ceph, use your cluster name instead.: 

    ceph auth add osd.{osd-num} osd 'allow *' mon 'allow rwx' -i /var/lib/ceph/osd/ceph-{osd-num}/keyring
    Copy to Clipboard Toggle word wrap

  6. Add the OSD to the CRUSH map so that the OSD can begin receiving data. The ceph osd crush add command allows you to add OSDs to the CRUSH hierarchy wherever you wish. If you specify at least one bucket, the command will place the OSD into the most specific bucket you specify, and it will move that bucket underneath any other buckets you specify. Important: If you specify only the root bucket, the command will attach the OSD directly to the root, but CRUSH rules expect OSDs to be inside of hosts.

    Execute the following: 

    ceph osd crush add {id-or-name} {weight}  [{bucket-type}={bucket-name} ...]
    Copy to Clipboard Toggle word wrap

    You may also decompile the CRUSH map, add the OSD to the device list, add the host as a bucket (if it’s not already in the CRUSH map), add the device as an item in the host, assign it a weight, recompile it and set it.

13.2.1. Starting the OSD
Copy link

After you add an OSD to Ceph, the OSD is in your configuration. However, it is not yet running. The OSD is down and in. You must start your new OSD before it can begin receiving data. For sysvinit, execute: 

sudo /etc/init.d/ceph start osd.{osd-num}
Copy to Clipboard Toggle word wrap

Once you start your OSD, it is up and in.

13.2.2. Observe the Data Migration
Copy link

Once you have added your new OSD to the CRUSH map, Ceph will begin rebalancing the server by migrating placement groups to your new OSD. You can observe this process with the ceph tool. : 

ceph -w
Copy to Clipboard Toggle word wrap

You should see the placement group states change from active+clean to active, some degraded objects, and finally active+clean when migration completes. (Control-c to exit.)

Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat