Time Synchronization

NTP is a TCP/IP protocol for synchronizing time over a network. Basically a client requests the current time from a server, and uses it to set its own clock.

Behind this simple description, there is a lot of complexity - there are tiers of NTP servers, with the tier one NTP servers connected to atomic clocks, and tier two and three servers spreading the load of actually handling requests across the Internet. Also the client software is a lot more complex than you might think - it has to factor out communication delays, and adjust the time in a way that does not upset all the other processes that run on the server. But luckily all that complexity is hidden from you!

Ubuntu by default uses timedatectl / timesyncd to synchronize time and users can optionally use chrony to serve the Network Time Protocol.

Synchronizing your systems time

Since Ubuntu 16.04 timedatectl / timesyncd (which are part of systemd) replace most of ntpdate / ntp.

timesyncd is available by default and replaces not only ntpdate, but also the client portion of chrony (or formerly ntpd). So on top of the one-shot action that ntpdate provided on boot and network activation, now timesyncd by default regularly checks and keeps your local time in sync. It also stores time updates locally, so that after reboots monotonically advances if applicable.

If chrony is installed timedatectl steps back to let chrony do the time keeping. That shall ensure that no two time syncing services are fighting. While no more recommended to be used, this still also applies to ntpd being installed to retain any kind of old behavior/config that you had through an upgrade. But it also implies that on an upgrade from a former release ntp/ntpdate might still be installed and therefore renders the new systemd based services disabled.

ntpdate is considered deprecated in favor of timedatectl (or chrony) and thereby no more installed by default. timesyncd will generally do the right thing keeping your time in sync, and chrony will help with more complex cases. But if you had one of a few known special ntpdate use cases, consider the following:

  • If you require a one-shot sync use: chronyd -q

  • If you require a one-shot time check, without setting the time use: chronyd -Q

Configuring timedatectl and timesyncd

The current status of time and time configuration via timedatectl and timesyncd can be checked with timedatectl status.

$ timedatectl status
                       Local time: Fr 2018-02-23 08:47:13 UTC
                   Universal time: Fr 2018-02-23 08:47:13 UTC
                         RTC time: Fr 2018-02-23 08:47:13
                        Time zone: Etc/UTC (UTC, +0000)
        System clock synchronized: yes
 systemd-timesyncd.service active: yes
                  RTC in local TZ: no

If chrony is running it will automatically switch to:

[...]
 systemd-timesyncd.service active: no 

Via timedatectl an admin can control the timezone, how the system clock should relate to the hwclock and if permanent synronization should be enabled or not. See man timedatectl for more details.

timesyncd itself is still a normal service, so you can check its status also more in detail via.

$ systemctl status systemd-timesyncd
  systemd-timesyncd.service - Network Time Synchronization
   Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2018-02-23 08:55:46 UTC; 10s ago
     Docs: man:systemd-timesyncd.service(8)
 Main PID: 3744 (systemd-timesyn)
   Status: "Synchronized to time server 91.189.89.198:123 (ntp.ubuntu.com)."
    Tasks: 2 (limit: 4915)
   CGroup: /system.slice/systemd-timesyncd.service
           |-3744 /lib/systemd/systemd-timesyncd

Feb 23 08:55:46 bionic-test systemd[1]: Starting Network Time Synchronization...
Feb 23 08:55:46 bionic-test systemd[1]: Started Network Time Synchronization.
Feb 23 08:55:46 bionic-test systemd-timesyncd[3744]: Synchronized to time server 91.189.89.198:123 (ntp.ubuntu.com).

The nameserver to fetch time for timedatectl and timesyncd from can be specified in /etc/systemd/timesyncd.conf and additional config files can be stored in /etc/systemd/timesyncd.conf.d/. The entries for NTP= and FallbackNTP= are space separated lists. See man timesyncd.conf for more.

Serve the Network Time Protocol

If in addition to synchronizing your system you also want to serve NTP information you need an NTP server. There are several options with chrony, ntpd and open-ntp. The recommended solution is chrony.

chrony(d)

The NTP daemon chronyd calculates the drift and offset of your system clock and continuously adjusts it, so there are no large corrections that could lead to inconsistent logs for instance. The cost is a little processing power and memory, but for a modern server this is usually negligible.

Installation

To install chrony, from a terminal prompt enter:

sudo apt install chrony

This will provide two binaries:

  • chronyd - the actual daemon to sync and serve via the NTP protocol

  • chronyc - command-line interface for chrony daemon

Chronyd Configuration

Edit /etc/chrony/chrony.conf to add/remove server lines. By default these servers are configured:

# Use servers from the NTP Pool Project. Approved by Ubuntu Technical Board
# on 2011-02-08 (LP: #104525). See http://www.pool.ntp.org/join.html for
# more information.
pool 0.ubuntu.pool.ntp.org iburst
pool 1.ubuntu.pool.ntp.org iburst
pool 2.ubuntu.pool.ntp.org iburst
pool 3.ubuntu.pool.ntp.org iburst

See man chrony.conf for more details on the configuration options. After changing the any of the config file you have to restart chrony:

sudo systemctl restart chrony.service

Of the pool 2.ubuntu.pool.ntp.org as well as ntp.ubuntu.com also support ipv6 if needed. If one needs to force ipv6 there also is ipv6.ntp.ubuntu.com which is not configured by default.

View status

Use chronyc to see query the status of the chrony daemon. For example to get an overview of the currently available and selected time sources.

chronyc sources

MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^+ gamma.rueckgr.at              2   8   377   135  -1048us[-1048us] +/-   29ms
^- 2b.ncomputers.org             2   8   377   204  -1141us[-1124us] +/-   50ms
^+ www.kashra.com                2   8   377   139  +3483us[+3483us] +/-   18ms
^+ stratum2-4.NTP.TechFak.U>     2   8   377   143  -2090us[-2073us] +/-   19ms
^- zepto.mcl.gg                  2   7   377     9   -774us[ -774us] +/-   29ms
^- mirrorhost.pw                 2   7   377    78   -660us[ -660us] +/-   53ms
^- atto.mcl.gg                   2   7   377     8   -823us[ -823us] +/-   50ms
^- static.140.107.46.78.cli>     2   8   377     9  -1503us[-1503us] +/-   45ms
^- 4.53.160.75                   2   8   377   137    -11ms[  -11ms] +/-  117ms
^- 37.44.185.42                  3   7   377    10  -3274us[-3274us] +/-   70ms
^- bagnikita.com                 2   7   377    74  +3131us[+3131us] +/-   71ms
^- europa.ellipse.net            2   8   377   204   -790us[ -773us] +/-   97ms
^- tethys.hot-chilli.net         2   8   377   141   -797us[ -797us] +/-   59ms
^- 66-232-97-8.static.hvvc.>     2   7   377   206  +1669us[+1686us] +/-  133ms
^+ 85.199.214.102                1   8   377   205   +175us[ +192us] +/-   12ms
^* 46-243-26-34.tangos.nl        1   8   377   141   -123us[ -106us] +/-   10ms
^- pugot.canonical.com           2   8   377    21    -95us[  -95us] +/-   57ms
^- alphyn.canonical.com          2   6   377    23  -1569us[-1569us] +/-   79ms
^- golem.canonical.com           2   7   377    92  -1018us[-1018us] +/-   31ms
^- chilipepper.canonical.com     2   8   377    21  -1106us[-1106us] +/-   27ms

chronyc sourcestats

210 Number of sources = 20
Name/IP Address            NP  NR  Span  Frequency  Freq Skew  Offset  Std Dev
==============================================================================
gamma.rueckgr.at           25  15   32m     -0.007      0.142   -878us   106us
2b.ncomputers.org          26  16   35m     -0.132      0.283  -1169us   256us
www.kashra.com             25  15   32m     -0.092      0.259  +3426us   195us
stratum2-4.NTP.TechFak.U>  25  14   32m     -0.018      0.130  -2056us    96us
zepto.mcl.gg               13  11   21m     +0.148      0.196   -683us    66us
mirrorhost.pw               6   5   645     +0.117      0.445   -591us    19us
atto.mcl.gg                21  13   25m     -0.069      0.199   -904us   103us
static.140.107.46.78.cli>  25  18   34m     -0.005      0.094  -1526us    78us
4.53.160.75                25  10   32m     +0.412      0.110    -11ms    84us
37.44.185.42               24  12   30m     -0.983      0.173  -3718us   122us
bagnikita.com              17   7   31m     -0.132      0.217  +3527us   139us
europa.ellipse.net         26  15   35m     +0.038      0.553   -473us   424us
tethys.hot-chilli.net      25  11   32m     -0.094      0.110   -864us    88us
66-232-97-8.static.hvvc.>  20  11   35m     -0.116      0.165  +1561us   109us
85.199.214.102             26  11   35m     -0.054      0.390   +129us   343us
46-243-26-34.tangos.nl     25  16   32m     +0.129      0.297   -307us   198us
pugot.canonical.com        25  14   34m     -0.271      0.176   -143us   135us
alphyn.canonical.com       17  11  1100     -0.087      0.360  -1749us   114us
golem.canonical.com        23  12   30m     +0.057      0.370   -988us   229us
chilipepper.canonical.com  25  18   34m     -0.084      0.224  -1116us   169us

Certain chronyc commands are privileged and can not be run via the network without explicitly allowing them. See section Command and monitoring access in man chrony.conf for more details. A local admin can use sudo as usually as this will grant him access to the local admin socket /var/run/chrony/chronyd.sock.

PPS Support

Chrony supports various PPS types natively. It can use kernel PPS API as well as PTP hardware clock. Most general GPS receivers can be leveraged via GPSD. The latter (and potentially more) can be accessed via SHM or via a socket (recommended). All of the above can be used to augment chrony with additional high quality time sources for better accuracy, jitter, drift, longer-or-short term accuracy (Usually each kind of clock type is good at one of those, but non-perfect at the others). For more details on configuration see some of the external PPS/GPSD resource listed below.

Note: at the release of 20.04 there was a bug which until fixed you might want to add this content to your /etc/apparmor.d/local/usr.sbin.gpsd.

Example configuration for GPSD to feed Chrony

For the setup you need
$ sudo apt install gpsd chrony

But you will want to test/debug your setup and especially the GPS reception, therefore also install
$ sudo apt install pps-tools gpsd-clients

GPS devices usually will communicate via serial interfaces, yet the most common type these days are USB GPS devices which have a serial converter behind USB. If you want to use this for PPS please be aware that the majority does not signal PPS via USB. Check the GPSD hardware list for details. The examples below were run with a Navisys GR701-W.

When plugging in such a device (or at boot time) dmesg should report serial connection of some sorts, example:

[   52.442199] usb 1-1.1: new full-speed USB device number 3 using xhci_hcd
[   52.546639] usb 1-1.1: New USB device found, idVendor=067b, idProduct=2303, bcdDevice= 4.00
[   52.546654] usb 1-1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[   52.546665] usb 1-1.1: Product: USB-Serial Controller D
[   52.546675] usb 1-1.1: Manufacturer: Prolific Technology Inc. 
[   52.602103] usbcore: registered new interface driver usbserial_generic
[   52.602244] usbserial: USB Serial support registered for generic
[   52.609471] usbcore: registered new interface driver pl2303
[   52.609503] usbserial: USB Serial support registered for pl2303
[   52.609564] pl2303 1-1.1:1.0: pl2303 converter detected
[   52.618366] usb 1-1.1: pl2303 converter now attached to ttyUSB0

We see above that it appeared as ttyUSB0. To later on have chrony accept being feeded time information by that we have to set it up in /etc/chrony/chrony.conf (Please replace USB0 to whatever applies to your setup):

refclock SHM 0 refid GPS precision 1e-1 offset 0.9999 delay 0.2
refclock SOCK /var/run/chrony.ttyUSB0.sock refid PPS

Then restart chrony to make the socket available and waiting.
sudo systemctl restart chrony

Next one needs to tell gpsd which device to manager, therefore in /etc/default/gpsd we set:
DEVICES="/dev/ttyUSB0"

Furthermore the default use-case of gpsd is, well for gps position tracking. Therefore it will normally not consume any cpu since it is not running the service but waiting on a socket for clients. Furthermore the client will then tell gpsd what it requests and gpsd will only do that.
For the use case of gpsd as PPS-providing-daemon you want to set the option to:

  • immediately start even without a client connected, this can be set in GPSD_OPTIONS of /etc/default/gpsd:
    • GPSD_OPTIONS="-n"
  • enable the service itself and not wait for a client to reach the socket in the future:
    • sudo systemctl enable /lib/systemd/system/gpsd.service

Restarting gpsd will now initialize the PPS from GPS and in dmesg you will see

 pps_ldisc: PPS line discipline registered
 pps pps0: new PPS source usbserial0
 pps pps0: source "/dev/ttyUSB0" added

In case you have multiple PPS the tool ppsfind might be useful to identify which PPS belongs to which GPS. You could check that with:

$ sudo ppsfind /dev/ttyUSB0 
pps0: name=usbserial0 path=/dev/ttyUSB0

To get any further you need your GPS to get a lock.
Tools like cgps or gpsmon need to report a 3D Fix for the data being any good.
Then you’d want to check to really have PPS data reported on that with ppstest

$ cgps
...
│ Status:         3D FIX (7 secs) ...

Next one might want to ensure that the pps device really submits PPS data, to do so run ppstest:

$ sudo ppstest /dev/pps0 
trying PPS source "/dev/pps0"
found PPS source "/dev/pps0"
ok, found 1 source(s), now start fetching data...
source 0 - assert 1588140739.099526246, sequence: 69 - clear  1588140739.999663721, sequence: 70
source 0 - assert 1588140740.099661485, sequence: 70 - clear  1588140739.999663721, sequence: 70
source 0 - assert 1588140740.099661485, sequence: 70 - clear  1588140740.999786664, sequence: 71
source 0 - assert 1588140741.099792447, sequence: 71 - clear  1588140740.999786664, sequence: 71

Ok, gpsd is now running, the GPS reception has found a fix and this is fed into chrony.
Now lets check that from the point of view of chrony.

Initially (e.g. before gpsd has started or before it has a lock) those will be new and “untrusted” sources marked with an “?”. If your devices stay in the “?” state and won’t leave it even after quite some time then gpsd seems not to feed any data to chrony and you’d need to debug why.

chronyc> sources
210 Number of sources = 10
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
#? GPS                           0   4     0     -     +0ns[   +0ns] +/-    0ns
#? PPS                           0   4     0     -     +0ns[   +0ns] +/-    0ns

Over time chrony will classify them as good or bad.
In the example case the raw GPS had too much deviation but PPS is good.

chronyc> sources
210 Number of sources = 10
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
#x GPS 0 4 177 24 -876ms[ -876ms] +/- 200ms
#- PPS 0 4 177 21 +916us[ +916us] +/- 63us
^- chilipepper.canonical.com 2 6 37 53 +33us[ +33us] +/- 33ms

And finally after a while it used the hardware PPS input as it was better:

chronyc> sources
210 Number of sources = 10
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
#x GPS                           0   4   377    20   -884ms[ -884ms] +/-  200ms
#* PPS                           0   4   377    18  +6677ns[  +52us] +/-   58us
^- alphyn.canonical.com          2   6   377    20  -1303us[-1258us] +/-  114ms

The PPS might also be ok but used in a combined way with e.g. the selected server.
See man chronyc for more details about how all these combinations will look like.

chronyc> sources
210 Number of sources = 11
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
#? GPS                           0   4     0     -     +0ns[   +0ns] +/-    0ns
#+ PPS                           0   4   377    22   +154us[ +154us] +/- 8561us
^* chilipepper.canonical.com     2   6   377    50   -353us[ -300us] +/-   44ms

And if you wonder if your shm based GPS data is any good, you can check for that as well.
First of all chrony will not only tell you if it classified it as good or bad. Via sourcestats you can also check the details

chronyc> sourcestats
210 Number of sources = 10
Name/IP Address            NP  NR  Span  Frequency  Freq Skew  Offset  Std Dev
==============================================================================
GPS                        20   9   302     +1.993     11.501   -868ms  1208us
PPS                         6   3    78     +0.324      5.009  +3365ns    41us
golem.canonical.com        15  10   783     +0.859      0.509   -750us   108us

You can also track the raw data that gpsd or other ntpd compliant refclocks are sending via shared memory by using ntpshmmon:

$ sudo ntpshmmon -o  
ntpshmmon: version 3.20
#      Name          Offset       Clock                 Real                 L Prc
sample NTP1          0.000223854  1588265805.000223854  1588265805.000000000 0 -10
sample NTP0          0.125691783  1588265805.125999851  1588265805.000308068 0 -20
sample NTP1          0.000349341  1588265806.000349341  1588265806.000000000 0 -10
sample NTP0          0.130326636  1588265806.130634945  1588265806.000308309 0 -20
sample NTP1          0.000485216  1588265807.000485216  1588265807.000000000 0 -10

References

Last updated a month ago. Help improve this document in the forum.