SupportConsoleDevelopersStart a trialContact

Appendix B. Ceph network configuration options

These are the common network configuration options for Ceph.

public_network
Description
The IP address and netmask of the public (front-side) network (for example, 192.168.0.0/24). Set in [global]. You can specify comma-delimited subnets.
Type
<ip-address>/<netmask> [, <ip-address>/<netmask>]
Required
No
Default
N/A
public_addr
Description
The IP address for the public (front-side) network. Set for each daemon.
Type
IP Address
Required
No
Default
N/A
cluster_network
Description
The IP address and netmask of the cluster network (for example, 10.0.0.0/24). Set in [global]. You can specify comma-delimited subnets.
Type
<ip-address>/<netmask> [, <ip-address>/<netmask>]
Required
No
Default
N/A
cluster_addr
Description
The IP address for the cluster network. Set for each daemon.
Type
Address
Required
No
Default
N/A
ms_type
Description
The messenger type for the network transport layer. Red Hat supports the simple and the async messenger type using posix semantics.
Type
String.
Required
No.
Default
async+posix
ms_public_type
Description
The messenger type for the network transport layer of the public network. It operates identically to ms_type, but is applicable only to the public or front-side network. This setting enables Ceph to use a different messenger type for the public or front-side and cluster or back-side networks.
Type
String.
Required
No.
Default
None.
ms_cluster_type
Description
The messenger type for the network transport layer of the cluster network. It operates identically to ms_type, but is applicable only to the cluster or back-side network. This setting enables Ceph to use a different messenger type for the public or front-side and cluster or back-side networks.
Type
String.
Required
No.
Default
None.

Host options

You must declare at least one Ceph Monitor in the Ceph configuration file, with a mon addr setting under each declared monitor. Ceph expects a host setting under each declared monitor, metadata server and OSD in the Ceph configuration file.

Important

Do not use localhost. Use the short name of the node, not the fully-qualified domain name (FQDN). Do not specify any value for host when using a third party deployment system that retrieves the node name for you.

mon_addr
Description
A list of <hostname>:<port> entries that clients can use to connect to a Ceph monitor. If not set, Ceph searches [mon.*] sections.
Type
String
Required
No
Default
N/A
host
Description
The host name. Use this setting for specific daemon instances (for example, [osd.0]).
Type
String
Required
Yes, for daemon instances.
Default
localhost

TCP options

Ceph disables TCP buffering by default.

ms_tcp_nodelay
Description
Ceph enables ms_tcp_nodelay so that each request is sent immediately (no buffering). Disabling Nagle’s algorithm increases network traffic, which can introduce congestion. If you experience large numbers of small packets, you may try disabling ms_tcp_nodelay, but be aware that disabling it will generally increase latency.
Type
Boolean
Required
No
Default
true
ms_tcp_rcvbuf
Description
The size of the socket buffer on the receiving end of a network connection. Disabled by default.
Type
32-bit Integer
Required
No
Default
0
ms_tcp_read_timeout
Description
If a client or daemon makes a request to another Ceph daemon and does not drop an unused connection, the tcp read timeout defines the connection as idle after the specified number of seconds.
Type
Unsigned 64-bit Integer
Required
No
Default
900 15 minutes.

Bind options

The bind options configure the default port ranges for the Ceph OSD daemons. The default range is 6800:7100. You can also enable Ceph daemons to bind to IPv6 addresses.

Important

Verify that the firewall configuration allows you to use the configured port range.

ms_bind_port_min
Description
The minimum port number to which an OSD daemon will bind.
Type
32-bit Integer
Default
6800
Required
No
ms_bind_port_max
Description
The maximum port number to which an OSD daemon will bind.
Type
32-bit Integer
Default
7300
Required
No.
ms_bind_ipv6
Description
Enables Ceph daemons to bind to IPv6 addresses.
Type
Boolean
Default
false
Required
No

Asynchronous messenger options

These Ceph messenger options configure the behavior of AsyncMessenger.

ms_async_transport_type
Description
Transport type used by the AsyncMessenger. Red Hat supports the posix setting, but does not support the dpdk or rdma settings at this time. POSIX uses standard TCP/IP networking and is the default value. Other transport types are experimental and are NOT supported.
Type
String
Required
No
Default
posix
ms_async_op_threads
Description
Initial number of worker threads used by each AsyncMessenger instance. This configuration setting SHOULD equal the number of replicas or erasure code chunks, but it may be set lower if the CPU core count is low or the number of OSDs on a single server is high.
Type
64-bit Unsigned Integer
Required
No
Default
3
ms_async_max_op_threads
Description
The maximum number of worker threads used by each AsyncMessenger instance. Set to lower values if the OSD host has limited CPU count, and increase if Ceph is underutilizing CPUs are underutilized.
Type
64-bit Unsigned Integer
Required
No
Default
5
ms_async_set_affinity
Description
Set to true to bind AsyncMessenger workers to particular CPU cores.
Type
Boolean
Required
No
Default
true
ms_async_affinity_cores
Description
When ms_async_set_affinity is true, this string specifies how AsyncMessenger workers are bound to CPU cores. For example, 0,2 will bind workers #1 and #2 to CPU cores #0 and #2, respectively. NOTE: When manually setting affinity, make sure to not assign workers to virtual CPUs created as an effect of hyper threading or similar technology, because they are slower than physical CPU cores.
Type
String
Required
No
Default
(empty)
ms_async_send_inline
Description
Send messages directly from the thread that generated them instead of queuing and sending from the AsyncMessenger thread. This option is known to decrease performance on systems with a lot of CPU cores, so it’s disabled by default.
Type
Boolean
Required
No
Default
false

Connection mode configuration options

For most connections, there are options that control the modes that are used for encryption and compression.

ms_cluster_mode
Description
Connection mode used for intra-cluster communication between Ceph daemons. If multiple modes are listed, the modes listed first are preferred.
Type
String
Default
crc secure
ms_service_mode
Description
A list of permitted modes for clients to use when connecting to the storage cluster.
Type
String
Default
crc secure
ms_client_mode
Description
A list of connection modes, in order of preference, for clients to use when interacting with a Ceph cluster.
Type
String
Default
crc secure
ms_mon_cluster_mode
Description
The connection mode to use between Ceph monitors.
Type
String
Default
secure crc
ms_mon_service_mode
Description
A list of permitted modes for clients or other Ceph daemons to use when connecting to monitors.
Type
String
Default
secure crc
ms_mon_client_mode
Description
A list of connection modes, in order of preference, for clients or non-monitor daemons to use when connecting to Ceph monitors.
Type
String
Default
secure crc

Compression mode configuration options

With the messenger v2 protocol, you can use the configuration options for the compression modes.

ms_compress_secure
Description
Combining encryption with compression reduces the level of security of messages between peers. In case, both the encryption and compression are enabled, compression setting is ignored and message is not compressed. Override this setting with option. Send messages directly from the thread that generated them instead of queuing and sending from the AsyncMessenger thread. This option is known to decrease performance on systems with a lot of CPU cores, so it’s disabled by default.
Type
Boolean
Default
false
ms_osd_compress_mode
Description
Compression policy to use in messenger for communication with Ceph OSDs.
Type
String
Default
none
Valid choices
none or force
ms_osd_compress_min_size
Description
Minimal message size eligible for on-wire compression.
Type
Integer
Default
1 Ki
ms_osd_compression_algorithm
Description
Compression algorithm for connections with OSD in order of preference
Type
String
Default
snappy
Valid choices
snappy, zstd, zlib, or lz4
Red Hat logoGithubRedditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

© 2024 Red Hat, Inc.