“A man with a watch knows what time it is. A man with two watches is never sure.” –Segal’s Law
Time is a fundamental requirement for computer systems and networks. Many things require time as an input, and if your clock is out of sync you can have a lot of problems ranging from annoyances to outright failures. The beign types of issues include having the wrong time on emails that you send, moderate failures might include build problems with files on shared storage, and significant issues might include authentication and authorization errors locking you and your processes out.
To avoid these aforementioned calamities, Ubuntu comes with several different tools to keep your servers time in sync. In this example we’re setting up the venerable classic ntpd. Before we begin let’s see if you have something on your system syncing time yet.
sudo netstat -panu |grep ":123"
This command runs the command netstat as the root user. It asks that the output of netstat include the process pid and name (-p), it also outputs all sockets (-a), it skips DNS lookups (-n), and show us UDP information (-u). The grep simply looks for connections on port 123 which is the port that NTP uses for communication.
Don’t be confused between NTP the protocol and ntpd the program that we’re using to run the NTP protocol. There are other programs that implement the NTP protocol therefore looking for connections on UDP port 123 is a way to see if any program on your system is running the NTP protocol. If you see a bunch of connections ending to IP addresses ending in “:123” then we know something on your system is communicating via NTP and you will want to chase down that program and fix it, but if you don’t see traffic on port 123 then you’re clear to continue.
sudo apt install ntpdate
sudo ntpdate 0.ubuntu.pool.ntp.org
sudo apt install ntpd
sudo systemctl enable ntpd
sudo ntpq -p
In the above set of commands we first installed ntpdate and updated the system clock against a timeserver from the ubuntu pool. We needed to do this because there is a limit to how much ntpd will change the clock. If the clock is off by a lot ntpd will give up and exit. In a computer you have a choice between stepping the clock to set it and slewing the clock to set it. If you step the clock then your computer system sees big jumps in time and it could cause programs to encounter issues if they are very time sensitive. The ntpd software slews the clock which you can think of as simply slowing or speeding up the clock to get it in sync rather than just jumping it instantaneously to the correct time.
After getting the clock set to about the right time, we installed ntpd and made sure that it’s set to start on boot using the systemctl command. Enabling ntpd should also make sure that it’s running. Finally we ask ntpq to print a list of time peers. It can take a little while for ntpd to determine and sync with its peers, sometimes as much as several hours. You may need to rerun the “ntpq -p” command to see the list of peers after waiting for things to sync up.
Once ntpd is connected and healthy, the peers list should look something like this:
# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
0.ubuntu.pool.n .POOL. 16 p - 64 0 0.000 0.000 0.000
1.ubuntu.pool.n .POOL. 16 p - 64 0 0.000 0.000 0.000
2.ubuntu.pool.n .POOL. 16 p - 64 0 0.000 0.000 0.000
3.ubuntu.pool.n .POOL. 16 p - 64 0 0.000 0.000 0.000
ntp.ubuntu.com .POOL. 16 p - 64 0 0.000 0.000 0.000
*ntp.your.org .CDMA. 1 u 138 256 377 42.557 0.504 1.729
-time-c-g.nist.g .NIST. 1 u 191 256 377 67.130 -2.896 2.265
+lithium.constan 18.26.4.105 2 u 131 256 377 56.666 0.363 4.925
+hadb2.smatwebde 209.51.161.238 2 u 115 256 377 15.316 -1.296 2.818
+four0.fairy.mat 128.227.205.3 2 u 121 256 377 35.536 0.496 3.428
+y.ns.gin.ntt.ne 249.224.99.213 2 u 128 256 377 13.005 -0.474 3.970
-tock.no-such-ag 129.6.15.29 2 u 87 256 377 54.728 -1.931 4.265
In this example output we’ve found several time peers. What we are really looking for is for one of the peers to start it’s line with a “*” and most of the peers should have 377 in the reach field. This indicates that we’re in sync with one peer and have good communication with the others.
Now that you have a list of time sources, the reasonable question is, “Why do I need so many?” I opened with a quote from Segal’s law, so it seems reasonable to conclude with a explanation of how ntpd figures out what time it really is. You’re welcome to delve into the depths of RFC 5905 to look at exactly how the time sync works, but I’ll give a high level layman’s explanation here knowing that it isn’t completely accurate, but it should be good enough for the average Sys Admin’s needs.
NTP needs at least 4 time servers to peer with. You can have more, but you shouldn’t have less. When looking at the time provided by those servers we can start to see which ones are the most consistent. Some peers will consistently provide a stream of time data is a reliable metrone fashion, while another might have fits and starts in it’s transmissions due to network congestion or faulty hardware. Remember that IP networks and particularly UDP does not guarantee the order or consistency of packet arrivals so the network quality between peers can become an issue.
If a peer is shown to be inconsistent it can be excluded from the time calculations. Also we can exclude peers that are significantly different in time. If a peer is the only one that’s 5 minutes faster than the others we can assume it’s off and that the majority of time servers are correct. It’s less important that we know exactly what time it really is, than it is that we all agree on a time. If all the computers in the world agreed that the time was 1 minute off from what it really is, then that’s just fine as long as everything on those networks and systems truly agrees to the time.
After excluding the inconsistent and obviously wrong time servers then we can use the remaining servers to see what time they cluster around and find the most accurate peers. We can then set our clock to sync with that coordinated time, and monitor our synchronization with them.