Chapter 13. Configuring time synchronization
Accurate timekeeping in an IT environment is important. A consistent time across all network devices improves the traceability of log files and certain protocols rely on synchronized clocks. For example, Kerberos uses time stamps to prevent replay attacks.
13.1. Using the Chrony suite to configure NTP
Accurate timekeeping is important for several reasons in IT. In networking for example, accurate time stamps in packets and logs are required. In Linux systems, the NTP
protocol is implemented by a daemon running in user space.
The user space daemon updates the system clock running in the kernel. The system clock can keep time by using various clock sources. Usually, the Time Stamp Counter (TSC) is used. The TSC is a CPU register which counts the number of cycles since it was last reset. It is very fast, has a high resolution, and there are no interruptions.
Starting with Red Hat Enterprise Linux 8, the NTP
protocol is implemented by the chronyd
daemon, available from the repositories in the chrony
package.
The following sections describe how to use the chrony suite to configure NTP.
13.1.1. Introduction to chrony suite
chrony is an implementation of the Network Time Protocol (NTP)
. You can use chrony:
-
To synchronize the system clock with
NTP
servers - To synchronize the system clock with a reference clock, for example a GPS receiver
- To synchronize the system clock with a manual time input
-
As an
NTPv4(RFC 5905)
server or peer to provide a time service to other computers in the network
chrony performs well in a wide range of conditions, including intermittent network connections, heavily congested networks, changing temperatures (ordinary computer clocks are sensitive to temperature), and systems that do not run continuously, or run on a virtual machine.
Typical accuracy between two machines synchronized over the Internet is within a few milliseconds, and for machines on a LAN within tens of microseconds. Hardware timestamping or a hardware reference clock may improve accuracy between two machines synchronized to a sub-microsecond level.
chrony consists of chronyd
, a daemon that runs in user space, and chronyc, a command line program which can be used to monitor the performance of chronyd
and to change various operating parameters when it is running.
The chrony daemon, chronyd
, can be monitored and controlled by the command line utility chronyc. This utility provides a command prompt which allows entering a number of commands to query the current state of chronyd
and make changes to its configuration. By default, chronyd
accepts only commands from a local instance of chronyc, but it can be configured to accept monitoring commands also from remote hosts. The remote access should be restricted.
13.1.2. Using chronyc to control chronyd
You can control chronyd
by using the chronyc command line utility.
Procedure
To make changes to the local instance of
chronyd
using the command line utility chronyc in interactive mode, enter the following command asroot
:# chronyc
chronyc must run as
root
if some of the restricted commands are to be used.The chronyc command prompt will be displayed as follows:
chronyc>
-
To list all of the commands, type
help
. Alternatively, the utility can also be invoked in non-interactive command mode if called together with a command as follows:
chronyc command
Changes made using chronyc are not permanent, they will be lost after a chronyd
restart. For permanent changes, modify /etc/chrony.conf
.
13.2. Using Chrony
The following sections describe how to install, start, and stop chronyd
, and how to check if chrony
is synchronized. Sections also describe how to manually adjust System Clock.
13.2.1. Managing chrony
The following procedure describes how to install, start, stop, and check the status of chronyd
.
Procedure
The chrony suite is installed by default on Red Hat Enterprise Linux. To ensure that it is, run the following command as
root
:# dnf install chrony
The default location for the chrony daemon is
/usr/sbin/chronyd
. The command line utility will be installed to/usr/bin/chronyc
.To check the status of
chronyd
, issue the following command:$ systemctl status chronyd chronyd.service - NTP client/server Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled) Active: active (running) since Wed 2013-06-12 22:23:16 CEST; 11h ago
To start
chronyd
, issue the following command asroot
:# systemctl start chronyd
To ensure
chronyd
starts automatically at system start, issue the following command asroot
:# systemctl enable chronyd
To stop
chronyd
, issue the following command asroot
:# systemctl stop chronyd
To prevent
chronyd
from starting automatically at system start, issue the following command asroot
:# systemctl disable chronyd
13.2.2. Checking if chrony is synchronized
The following procedure describes how to check if chrony is synchronized with the use of the tracking
, sources
, and sourcestats
commands.
Procedure
To check chrony tracking, issue the following command:
$ chronyc tracking Reference ID : CB00710F (ntp-server.example.net) Stratum : 3 Ref time (UTC) : Fri Jan 27 09:49:17 2017 System time : 0.000006523 seconds slow of NTP time Last offset : -0.000006747 seconds RMS offset : 0.000035822 seconds Frequency : 3.225 ppm slow Residual freq : 0.000 ppm Skew : 0.129 ppm Root delay : 0.013639022 seconds Root dispersion : 0.001100737 seconds Update interval : 64.2 seconds Leap status : Normal
The sources command displays information about the current time sources that
chronyd
is accessing. To check chrony sources, issue the following command:$ chronyc sources 210 Number of sources = 3 MS Name/IP address Stratum Poll Reach LastRx Last sample =============================================================================== #* GPS0 0 4 377 11 -479ns[ -621ns] /- 134ns ^? a.b.c 2 6 377 23 -923us[ -924us] +/- 43ms ^ d.e.f 1 6 377 21 -2629us[-2619us] +/- 86ms
You can specify the optional
-v
argument to print more verbose information. In this case, extra caption lines are shown as a reminder of the meanings of the columns.The
sourcestats
command displays information about the drift rate and offset estimation process for each of the sources currently being examined bychronyd
. To check chrony source statistics, issue the following command:$ chronyc sourcestats 210 Number of sources = 1 Name/IP Address NP NR Span Frequency Freq Skew Offset Std Dev =============================================================================== abc.def.ghi 11 5 46m -0.001 0.045 1us 25us
The optional argument
-v
can be specified, meaning verbose. In this case, extra caption lines are shown as a reminder of the meanings of the columns.
Additional resources
-
chronyc(1)
man page
13.2.3. Manually adjusting the System Clock
The following procedure describes how to manually adjust the System Clock.
Procedure
To step the system clock immediately, bypassing any adjustments in progress by slewing, issue the following command as
root
:# chronyc makestep
If the rtcfile
directive is used, the real-time clock should not be manually adjusted. Random adjustments would interfere with chrony's need to measure the rate at which the real-time clock drifts.
13.2.4. Disabling a chrony dispatcher script
The chrony
dispatcher script manages the online and offline state of the NTP servers. As a system administrator, you can disable the dispatcher script to keep chronyd
polling the servers constantly.
If you enable NetworkManager on your system to manage networking configuration, the NetworkManager executes the chrony
dispatcher script during interface reconfiguration, stop or start operations. However, if you configure certain interfaces or routes outside of NetworkManager, you can encounter the following situation:
- The dispatcher script might run when no route to the NTP servers exists, causing the NTP servers to switch to the offline state.
- If you establish the route later, the script does not run again by default, and the NTP servers remain in the offline state.
To ensure that chronyd
can synchronize with your NTP servers, which have separately managed interfaces, disable the dispatcher script.
Prerequisites
- You installed NetworkManager on your system and enabled it.
- Root access
Procedure
To disable the
chrony
dispatcher script, create a symlink to/dev/null
:# ln -s /dev/null /etc/NetworkManager/dispatcher.d/20-chrony-onoffline
NoteAfter this change, the NetworkManager cannot execute the dispatcher script, and the NTP servers remain in the online state at all times.
13.2.5. Setting up chrony for a system in an isolated network
For a network that is never connected to the Internet, one computer is selected to be the primary timeserver. The other computers are either direct clients of the server, or clients of clients. On the server, the drift file must be manually set with the average rate of drift of the system clock. If the server is rebooted, it will obtain the time from surrounding systems and calculate an average to set its system clock. Thereafter it resumes applying adjustments based on the drift file. The drift file will be updated automatically when the settime
command is used.
The following procedure describes how to set up chrony for a system in an isolated network.
Procedure
On the system selected to be the server, using a text editor running as
root
, edit/etc/chrony.conf
as follows:driftfile /var/lib/chrony/drift commandkey 1 keyfile /etc/chrony.keys initstepslew 10 client1 client3 client6 local stratum 8 manual allow 192.0.2.0/24
Where
192.0.2.0/24
is the network or subnet address from which the clients are allowed to connect. For more details, seechrony.conf(7)
man pageOn the systems selected to be direct clients of the server, using a text editor running as
root
, edit the/etc/chrony.conf
as follows:server ntp1.example.net driftfile /var/lib/chrony/drift logdir /var/log/chrony log measurements statistics tracking keyfile /etc/chrony.keys commandkey 24 local stratum 10 initstepslew 20 ntp1.example.net allow 192.0.2.123
Where
192.0.2.123
is the address of the server, andntp1.example.net
is the host name of the server. Clients with this configuration will resynchronize with the server if it restarts.
On the client systems which are not to be direct clients of the server, the /etc/chrony.conf
file should be the same except that the local
and allow
directives should be omitted.
In an isolated network, you can also use the local
directive that enables a local reference mode, which allows chronyd
operating as an NTP
server to appear synchronized to real time, even when it was never synchronized or the last update of the clock happened a long time ago.
To allow multiple servers in the network to use the same local configuration and to be synchronized to one another, without confusing clients that poll more than one server, use the orphan
option of the local
directive which enables the orphan mode. Each server needs to be configured to poll all other servers with local
. This ensures that only the server with the smallest reference ID has the local reference active and other servers are synchronized to it. When the server fails, another one will take over.
13.2.6. Configuring remote monitoring access
chronyc can access chronyd
in two ways:
- Internet Protocol, IPv4 or IPv6.
-
Unix domain socket, which is accessible locally by the
root
orchrony
user.
By default, chronyc connects to the Unix domain socket. The default path is /var/run/chrony/chronyd.sock
. If this connection fails, which can happen for example when chronyc is running under a non-root user, chronyc tries to connect to 127.0.0.1 and then ::1.
Only the following monitoring commands, which do not affect the behavior of chronyd
, are allowed from the network:
- activity
- manual list
- rtcdata
- smoothing
- sources
- sourcestats
- tracking
- waitsync
The set of hosts from which chronyd
accepts these commands can be configured with the cmdallow
directive in the configuration file of chronyd
, or the cmdallow
command in chronyc. By default, the commands are accepted only from localhost (127.0.0.1 or ::1).
All other commands are allowed only through the Unix domain socket. When sent over the network, chronyd
responds with a Not authorised
error, even if it is from localhost.
The following procedure describes how to access chronyd remotely with chronyc.
Procedure
Allow access from both IPv4 and IPv6 addresses by adding the following to the
/etc/chrony.conf
file:bindcmdaddress 0.0.0.0
or
bindcmdaddress ::
Allow commands from the remote IP address, network, or subnet by using the
cmdallow
directive.Add the following content to the
/etc/chrony.conf
file:cmdallow 192.168.1.0/24
Open port 323 in the firewall to connect from a remote system:
# firewall-cmd --zone=public --add-port=323/udp
Optionally, you can open port 323 permanently using the
--permanent
option:# firewall-cmd --permanent --zone=public --add-port=323/udp
If you opened port 323 permanently, reload the firewall configuration:
# firewall-cmd --reload
Additional resources
-
chrony.conf(5)
man page
13.2.7. Managing time synchronization using RHEL system roles
You can manage time synchronization on multiple target machines using the timesync
role. The timesync
role installs and configures an NTP or PTP implementation to operate as an NTP or PTP client to synchronize the system clock.
The timesync
role replaces the configuration of the given or detected provider service on the managed host. Previous settings are lost, even if they are not specified in the role variables. The only preserved setting is the choice of provider if the timesync_ntp_provider
variable is not defined.
The following example shows how to apply the timesync
role in a situation with just one pool of servers.
Example 13.1. An example playbook applying the timesync role for a single pool of servers
--- - hosts: timesync-test vars: timesync_ntp_servers: - hostname: 2.rhel.pool.ntp.org pool: yes iburst: yes roles: - rhel-system-roles.timesync
For a detailed reference on timesync
role variables, install the rhel-system-roles
package, and see the README.md
or README.html
files in the /usr/share/doc/rhel-system-roles/timesync
directory.
Additional resources
13.2.8. Additional resources
-
chronyc(1)
man page -
chronyd(8)
man page - Frequently Asked Questions
13.3. Chrony with HW timestamping
Hardware timestamping is a feature supported in some Network Interface Controller (NICs) which provides accurate timestamping of incoming and outgoing packets. NTP
timestamps are usually created by the kernel and chronyd with the use of the system clock. However, when HW timestamping is enabled, the NIC uses its own clock to generate the timestamps when packets are entering or leaving the link layer or the physical layer. When used with NTP
, hardware timestamping can significantly improve the accuracy of synchronization. For best accuracy, both NTP
servers and NTP
clients need to use hardware timestamping. Under ideal conditions, a sub-microsecond accuracy may be possible.
Another protocol for time synchronization that uses hardware timestamping is PTP
.
Unlike NTP
, PTP
relies on assistance in network switches and routers. If you want to reach the best accuracy of synchronization, use PTP
on networks that have switches and routers with PTP
support, and prefer NTP
on networks that do not have such switches and routers.
The following sections describe how to:
- Verify support for hardware timestamping
- Enable hardware timestamping
- Configure client polling interval
- Enable interleaved mode
- Configure server for large number of clients
- Verify hardware timestamping
- Configure PTP-NTP bridge
13.3.1. Verifying support for hardware timestamping
To verify that hardware timestamping with NTP
is supported by an interface, use the ethtool -T
command. An interface can be used for hardware timestamping with NTP
if ethtool
lists the SOF_TIMESTAMPING_TX_HARDWARE
and SOF_TIMESTAMPING_TX_SOFTWARE
capabilities and also the HWTSTAMP_FILTER_ALL
filter mode.
Example 13.2. Verifying support for hardware timestamping on a specific interface
# ethtool -T eth0
Output:
Timestamping parameters for eth0: Capabilities: hardware-transmit (SOF_TIMESTAMPING_TX_HARDWARE) software-transmit (SOF_TIMESTAMPING_TX_SOFTWARE) hardware-receive (SOF_TIMESTAMPING_RX_HARDWARE) software-receive (SOF_TIMESTAMPING_RX_SOFTWARE) software-system-clock (SOF_TIMESTAMPING_SOFTWARE) hardware-raw-clock (SOF_TIMESTAMPING_RAW_HARDWARE) PTP Hardware Clock: 0 Hardware Transmit Timestamp Modes: off (HWTSTAMP_TX_OFF) on (HWTSTAMP_TX_ON) Hardware Receive Filter Modes: none (HWTSTAMP_FILTER_NONE) all (HWTSTAMP_FILTER_ALL) ptpv1-l4-sync (HWTSTAMP_FILTER_PTP_V1_L4_SYNC) ptpv1-l4-delay-req (HWTSTAMP_FILTER_PTP_V1_L4_DELAY_REQ) ptpv2-l4-sync (HWTSTAMP_FILTER_PTP_V2_L4_SYNC) ptpv2-l4-delay-req (HWTSTAMP_FILTER_PTP_V2_L4_DELAY_REQ) ptpv2-l2-sync (HWTSTAMP_FILTER_PTP_V2_L2_SYNC) ptpv2-l2-delay-req (HWTSTAMP_FILTER_PTP_V2_L2_DELAY_REQ) ptpv2-event (HWTSTAMP_FILTER_PTP_V2_EVENT) ptpv2-sync (HWTSTAMP_FILTER_PTP_V2_SYNC) ptpv2-delay-req (HWTSTAMP_FILTER_PTP_V2_DELAY_REQ)
13.3.2. Enabling hardware timestamping
To enable hardware timestamping, use the hwtimestamp
directive in the /etc/chrony.conf
file. The directive can either specify a single interface, or a wildcard character can be used to enable hardware timestamping on all interfaces that support it. Use the wildcard specification in case that no other application, like ptp4l from the linuxptp
package, is using hardware timestamping on an interface. Multiple hwtimestamp
directives are allowed in the chrony configuration file.
Example 13.3. Enabling hardware timestamping by using the hwtimestamp directive
hwtimestamp eth0 hwtimestamp eth1 hwtimestamp *
13.3.3. Configuring client polling interval
The default range of a polling interval (64-1024 seconds) is recommended for servers on the Internet. For local servers and hardware timestamping, a shorter polling interval needs to be configured in order to minimize offset of the system clock.
The following directive in /etc/chrony.conf
specifies a local NTP
server using one second polling interval:
server ntp.local minpoll 0 maxpoll 0
13.3.4. Enabling interleaved mode
NTP
servers that are not hardware NTP
appliances, but rather general purpose computers running a software NTP
implementation, like chrony, will get a hardware transmit timestamp only after sending a packet. This behavior prevents the server from saving the timestamp in the packet to which it corresponds. In order to enable NTP
clients receiving transmit timestamps that were generated after the transmission, configure the clients to use the NTP
interleaved mode by adding the xleave
option to the server directive in /etc/chrony.conf
:
server ntp.local minpoll 0 maxpoll 0 xleave
13.3.5. Configuring server for large number of clients
The default server configuration allows a few thousands of clients at most to use the interleaved mode concurrently. To configure the server for a larger number of clients, increase the clientloglimit
directive in /etc/chrony.conf
. This directive specifies the maximum size of memory allocated for logging of clients' access on the server:
clientloglimit 100000000
13.3.6. Verifying hardware timestamping
To verify that the interface has successfully enabled hardware timestamping, check the system log. The log should contain a message from chronyd
for each interface with successfully enabled hardware timestamping.
Example 13.4. Log messages for interfaces with enabled hardware timestamping
chronyd[4081]: Enabled HW timestamping on eth0 chronyd[4081]: Enabled HW timestamping on eth1
When chronyd
is configured as an NTP
client or peer, you can have the transmit and receive timestamping modes and the interleaved mode reported for each NTP
source by the chronyc ntpdata
command:
Example 13.5. Reporting the transmit, receive timestamping and interleaved mode for each NTP source
# chronyc ntpdata
Output:
Remote address : 203.0.113.15 (CB00710F) Remote port : 123 Local address : 203.0.113.74 (CB00714A) Leap status : Normal Version : 4 Mode : Server Stratum : 1 Poll interval : 0 (1 seconds) Precision : -24 (0.000000060 seconds) Root delay : 0.000015 seconds Root dispersion : 0.000015 seconds Reference ID : 47505300 (GPS) Reference time : Wed May 03 13:47:45 2017 Offset : -0.000000134 seconds Peer delay : 0.000005396 seconds Peer dispersion : 0.000002329 seconds Response time : 0.000152073 seconds Jitter asymmetry: +0.00 NTP tests : 111 111 1111 Interleaved : Yes Authenticated : No TX timestamping : Hardware RX timestamping : Hardware Total TX : 27 Total RX : 27 Total valid RX : 27
Example 13.6. Reporting the stability of NTP measurements
# chronyc sourcestats
With hardware timestamping enabled, stability of NTP
measurements should be in tens or hundreds of nanoseconds, under normal load. This stability is reported in the Std Dev
column of the output of the chronyc sourcestats
command:
Output:
210 Number of sources = 1 Name/IP Address NP NR Span Frequency Freq Skew Offset Std Dev ntp.local 12 7 11 +0.000 0.019 +0ns 49ns
13.3.7. Configuring PTP-NTP bridge
If a highly accurate Precision Time Protocol (PTP
) primary timeserver is available in a network that does not have switches or routers with PTP
support, a computer may be dedicated to operate as a PTP
client and a stratum-1 NTP
server. Such a computer needs to have two or more network interfaces, and be close to the primary timeserver or have a direct connection to it. This will ensure highly accurate synchronization in the network.
Configure the ptp4l and phc2sys programs from the linuxptp
packages to use one interface to synchronize the system clock using PTP
.
Configure chronyd
to provide the system time using the other interface:
Example 13.7. Configuring chronyd to provide the system time using the other interface
bindaddress 203.0.113.74 hwtimestamp eth1 local stratum 1
13.4. Overview of Network Time Security (NTS) in chrony
Network Time Security (NTS) is an authentication mechanism for Network Time Protocol (NTP), designed to scale substantial clients. It verifies that the packets received from the server machines are unaltered while moving to the client machine. Network Time Security (NTS) includes a Key Establishment (NTS-KE) protocol that automatically creates the encryption keys used between the server and its clients.
NTS is not compatible with the FIPS and OSPP profile. When you enable the FIPS and OSPP profile, chronyd
that is configured with NTS can abort with a fatal message. You can disable the OSPP profile and FIPS mode for chronyd
service by adding the GNUTLS_FORCE_FIPS_MODE=0
to the /etc/sysconfig/chronyd
file.
13.4.1. Enabling Network Time Security (NTS) in the client configuration file
By default, Network Time Security (NTS) is not enabled. You can enable NTS in the /etc/chrony.conf
. For that, perform the following steps:
Prerequisites
- Server with the NTS support
Procedure
In the client configuration file:
Specify the server with the
nts
option in addition to the recommendediburst
option.For example: server time.example.com iburst nts server nts.netnod.se iburst nts server ptbtime1.ptb.de iburst nts
To avoid repeating the Network Time Security-Key Establishment (NTS-KE) session during system boot, add the following line to
chrony.conf
, if it is not present:ntsdumpdir /var/lib/chrony
To disable synchronization with Network Time Protocol (NTP) servers provided by
DHCP
, comment out or remove the following line inchrony.conf
, if it is present:sourcedir /run/chrony-dhcp
- Save your changes.
Restart the
chronyd
service:systemctl restart chronyd
Verification
Verify if the
NTS
keys were successfully established:# chronyc -N authdata Name/IP address Mode KeyID Type KLen Last Atmp NAK Cook CLen ================================================================ time.example.com NTS 1 15 256 33m 0 0 8 100 nts.sth1.ntp.se NTS 1 15 256 33m 0 0 8 100 nts.sth2.ntp.se NTS 1 15 256 33m 0 0 8 100
The
KeyID
,Type
, andKLen
should have non-zero values. If the value is zero, check the system log for error messages fromchronyd
.Verify the client is making NTP measurements:
# chronyc -N sources MS Name/IP address Stratum Poll Reach LastRx Last sample ========================================================= time.example.com 3 6 377 45 +355us[ +375us] +/- 11ms nts.sth1.ntp.se 1 6 377 44 +237us[ +237us] +/- 23ms nts.sth2.ntp.se 1 6 377 44 -170us[ -170us] +/- 22ms
The
Reach
column should have a non-zero value; ideally 377. If the value rarely gets 377 or never gets to 377, it indicates that NTP requests or responses are getting lost in the network.
Additional resources
-
chrony.conf(5)
man page
13.4.2. Enabling Network Time Security (NTS) on the server
If you run your own Network Time Protocol (NTP) server, you can enable the server Network Time Security (NTS) support to facilitate its clients to synchronize securely.
If the NTP server is a client of other servers, that is, it is not a Stratum 1 server, it should use NTS or symmetric key for its synchronization.
Prerequisites
-
Server private key in
PEM
format -
Server certificate with required intermediate certificates in
PEM
format
Procedure
Specify the private key and the certificate file in
chrony.conf
. For example:ntsserverkey /etc/pki/tls/private/<ntp-server.example.net>.key ntsservercert /etc/pki/tls/certs/<ntp-server.example.net>.crt
Ensure that both the key and certificate files are readable by the chrony system user, by setting the group ownership. For example:
# chown :chrony /etc/pki/tls//<ntp-server.example.net>.
-
Ensure the
ntsdumpdir /var/lib/chrony
directive is present in thechrony.conf
. Restart the
chronyd
service:# systemctl restart chronyd
ImportantIf the server has a firewall, it needs to allow both the
UDP 123
andTCP 4460
ports for NTP and Network Time Security-Key Establishment (NTS-KE).
Verification
Perform a quick test from a client machine with the following command:
$ chronyd -Q -t 3 'server ntp-server.example.net iburst nts maxsamples 1' 2021-09-15T13:45:26Z chronyd version 4.1 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS +NTS +SECHASH +IPV6 +DEBUG) 2021-09-15T13:45:26Z Disabled control of system clock 2021-09-15T13:45:28Z System clock wrong by 0.002205 seconds (ignored) 2021-09-15T13:45:28Z chronyd exiting
The
System clock wrong
message indicates the NTP server is accepting NTS-KE connections and responding with NTS-protected NTP messages.Verify the NTS-KE connections and authenticated NTP packets observed on the server:
# chronyc serverstats NTP packets received : 7 NTP packets dropped : 0 Command packets received : 22 Command packets dropped : 0 Client log records dropped : 0 NTS-KE connections accepted: 1 NTS-KE connections dropped : 0 Authenticated NTP packets: 7
If the value of the
NTS-KE connections accepted
andAuthenticated NTP packets
field is a non-zero value, it means that at least one client was able to connect to the NTS-KE port and send an authenticated NTP request.