Timekeeping on Linux-servers: Chrony/NTP configuration, RTC drift and virtualization pitfalls

“The time is wrong” sounds trivial, but in operation it is often the start of a chain reaction: TLS certificates are judged invalid, Kerberos logins fail, cluster quorums collapse, backups fall into incorrect retention windows, and logs can no longer be reliably correlated. Timekeeping on Linux servers is therefore less a matter of convenience than a fundamental operational discipline. At the same time, time is a topic that in modern environments derails more quickly than before due to virtualization, cloud images, snapshot/RESTore workflows and heterogeneous hardware.

This article combines practical knowledge on Chrony/NTP configuration, the role of the RTC (Real-Time Clock, i.e. the hardware clock) and typical virtualization pitfalls. The goal is a setup that is stable in normal operation, detects deviations early and reacts in a controlled way during incidents (e.g. after resume, snapshot-RESTore or host migration) — without carrying “time jumps” as a hidden risk into the platform.

Timekeeping on Linux servers: Why correct time is critical in operation

Many components rely on a monotonic, plausible system time:

• TLS and certificate validation: If the clock is too far in the future or past, handshakes fail (client and server evaluate “not before/not after”).
• Kerberos/SSO: Ticket validity tolerates only small deviations. Time drift behaves like an authentication failure.
• Logs, SIEM, incident response: Timestamps are the backbone of correlation. Drift makes root-cause analysis unnecessarily difficult.
• Databases and replication: Timeouts, TTLs, audit trails and scheduled jobs depend on consistent time.
• Schedulers, cron, systemd-timers: Jumps can cause jobs to run twice or to be skipped.

It is important to distinguish between “setting the clock” (wall clock) and “measuring time” (monotonic time). Linux uses monotonic time sources for many measurements, but protocols, certificates and logs depend on the wall clock. This is precisely where jumps and drift have an effect.

Fundamentals: Clearly distinguish system clock, RTC and time sources

On classic servers there are at least two clocks:

• System clock (kernel clock): runs in RAM and is initialized at boot. This clock is continuously corrected by NTP/Chrony.
• RTC (Real-Time Clock): a battery-backed hardware clock on the motherboard. It is the first reference at boot before network time is available.

The RTC is notorious for drift (deviation per time) because it is often a low-cost crystal, temperature and aging play a role, and servers are not always operated in ideal conditions. Drift is normal — what matters is how you deal with it.

Another term is stratum: this is the “distance” to the reference time. Stratum-1 servers are directly attached to a reference (e.g. GNSS, radio clock, PTP/PHC), Stratum-2 servers obtain time from Stratum-1, and so on. In enterprise operations Stratum-2/3 is common and entirely sufficient, provided redundancy and monitoring are in place.

Chrony vs. classic NTP and systemd-timesyncd: What fits in server operation?

In the Linux environment you typically encounter three variants:

• chronyd (Chrony): a modern NTP client/server, robust with changing connectivity, quick to discipline the clock and practical in VMs. Chrony is the default in many distributions.
• ntpd (classic NTP daemon): solid, but increasingly replaced by Chrony in many environments. Continues to work, but is often less flexible under „non-ideal“ conditions.
• systemd-timesyncd: simple SNTP client (simplified NTP). Suitable for desktops or very simple server setups, but in professional operation it often lacks diagnostic and server functions.

For administrators, Chrony is usually the pragmatic choice: good status and debug commands, controlled „slewing“ (gradual correction) and clear configuration options for virtualization scenarios.

Assessment: Is time synchronization active and stable?

Before you reconfigure: record the current state so you can compare later.

1) Basic check with timedatectl

timedatectl status

Pay particular attention to:

• System clock synchronized: indicates whether the time service reports synchronization.
• NTP service: which service is active (chronyd, systemd-timesyncd or other).
• RTC in local TZ: should typically be no in server environments (RTC in UTC).

2) Chrony status: sources and offset

chronyc tracking
chronyc sources -v
chronyc sourcestats -v

Interpretation (concise and practical):

• System time / Last offset: current deviation relative to the selected source.
• RMS offset: the „typical“ deviation, useful for assessing trends.
• Frequency: how much the local clock needs to be corrected (ppm). Noticeably high values can indicate an unstable clock source or virtualization issues.
• sources: shows reachable servers, stratum, delay/jitter. Redundancy is important: multiple sources should be reliably reachable.

3) Logs: finding jumps and resync events

journalctl -u chronyd --since "-7 days" | tail -n 200

Look for indications such as „stepped“, „makestep“, „reset“, „time jump“. Recurring jumps are a warning sign because they can affect timers, databases and security mechanisms.

Configuring Chrony: stable sources, clear correction strategy

A resilient setup starts with a clear source strategy. In enterprise networks typical options are:

• internal NTP servers (recommended): centrally controllable, firewall-friendly, consistent policy.
• public pools (only if permitted): e.g. vendor- or pool-based sources, but subject to policy and compliance review.
• local reference (fallback): e.g. a „local“ source as the last resort — use extreme caution so you do not accidentally distribute „wrong time“ across the network.

Example: Chrony as a client with internal time sources

Path and details vary by distribution (commonly /etc/chrony/chrony.conf or /etc/chrony.conf). The following example shows a typical pattern: multiple sources, controlled step behavior at boot, stable drift statistics.

# /etc/chrony/chrony.conf (Beispiel)

# Mehrere, voneinander unabhängige NTP-Server (idealerweise unterschiedliche Hosts/Standorte)
server ntp1.example.net iburst
server ntp2.example.net iburst
server ntp3.example.net iburst

# Drift-Datei: Chrony lernt die Frequenzabweichung Ihrer Hardware/VM
# (Pfad je nach Distribution)
driftfile /var/lib/chrony/chrony.drift

# Schnelles Einregeln nach Boot: bei großen Abweichungen wenige Male „step“ erlauben,
# danach nur noch slewing (schrittweise) korrigieren.
makestep 1.0 3

# RTC beim Stop/Start aktualisieren, damit Boot-Zeit plausibler ist
rtcsync

# Optional: Wenn der Host nur selten Netz hat, kann eine Offline-Strategie sinnvoll sein
# (hier nicht aktiviert)
# offline

# Logging sparsam, aber ausreichend für Troubleshooting
logdir /var/log/chrony
# log measurements statistics tracking

Why these parameters? „iburst“ speeds up the initial measurements after startup. „makestep“ allows a few hard steps at the beginning to quickly reach a plausible range. After that you avoid steps because they cause side effects during normal operation. „rtcsync“ ensures the RTC regularly receives the corrected time, which stabilizes boot phases — without idealizing the RTC as the primary source of truth.

RESTarting and validating Chrony

systemctl enable --now chronyd
systemctl RESTart chronyd
chronyc tracking
chronyc sources -v

After switching, synchronization should stabilize within a short time. If it does not, the usual culprits are network paths (UDP/123), DNS, firewall rules or problems with the time sources.

Understanding and managing RTC drift: UTC, hwclock and typical failure patterns

Many time issues do not originate in NTP but at boot: if the RTC is significantly off, the system starts with the wrong time. Services that run very early (e.g. logging, early security checks, some agents) then write „incorrect“ timestamps before NTP corrects them.

Operate the RTC in UTC (standard for server operation)

The clean approach is: RTC in UTC, timezone in Linux via tzdata/Zoneinfo. „RTC in local TZ“ creates unnecessary edge cases, especially with DST (summer/winter time) and dual-boot scenarios.

timedatectl set-timezone Europe/Berlin
timedatectl set-local-rtc 0

The timezone is a presentation layer. For protocols, clusters and distributed systems, UTC is the common baseline.

Check RTC state and make drift visible

hwclock --verbose
hwclock --show
date -u

If RTC and system time diverge significantly, first clarify: Is NTP/Chrony configured correctly? Is the RTC persistent at all (battery/UEFI settings)? In VMs the „RTC“ is often virtual and depends on the hypervisor timing.

When „rtcsync“ helps — and when it does not

„rtcsync“ (in Chrony) synchronizes the RTC periodically from the system time. That helps when the RTC only drifts moderately. But if the RTC is extremely unstable (dead battery, defective hardware, aggressive virtualization timing), then with „rtcsync“ you will write ‚correct‘ values, but you do not resolve the underlying cause. Monitoring and, if necessary, hardware/host fixes are required.

Virtualization pitfalls: time drift, TSC issues, snapshot/RESTore and live migration

In VMs there’s an additional layer: the guest sees a virtual clock that is influenced by the hypervisor. Typical causes for drift and jumps:

• Host overcommit and CPU steal: If vCPUs do not run regularly, the VM „misses“ time intervals.
• Pause/Resume, suspend: The VM was paused; the wall clock jumps on resume.
• Snapshots/RESTore: A RESTore can reset the VM into the „past“ — timestamps, certificates and logs then appear inconsistent.
• Live migration: Depending on the platform, clock source and timer emulation can change.

An important term is TSC (Time Stamp Counter): a CPU counter that can serve as a time base. In virtualization TSC is either emulated or passed through as „stable.“ If the TSC is not stable or changes during migration, the guest experiences time „stutter“.

Golden rule: Only one instance may set time ‚hard‘

Many hypervisors offer „guest tools“ that actively correct time inside the guest (sometimes using hard steps). At the same time Chrony/NTP runs in the guest. If both intervene, you get a hard-to-reproduce mixed behavior: Chrony observes jumps, marks sources as unreliable, or constantly fights the corrections.

Practical recommendation:

• In VMs: use Chrony as the primary time service in the guest.
• Enable hypervisor time sync only when there is a clear policy for it (e.g. only initially at boot) and it does not compete with Chrony.

Exactly how you configure this depends on VMware, Hyper-V, KVM/QEMU, Proxmox or cloud hypervisors. The decisive principle is: no competing ‚time setters‘.

After snapshot/RESTore: verify in a controlled manner before starting applications

A RESTore from a snapshot is effectively a „time jump.“ If the VM jumps into the past, the following effects can occur:

• Certificates appear „not yet valid“ or already expired (depending on the direction).
• Journald/log rotation produces gaps or duplicate time ranges.
• Token-based authentication (OIDC, Kerberos) breaks.

Having a fixed checkpoint in the runbook is operationally helpful: after a RESTore, stabilize the time service first, then start applications. If you use systemd, you can model dependencies via „network-online.target“ and service start order. The important point is the concept, not necessarily a specific unit design.

Troubleshooting playbook: When time drifts or jumps

For daily operations a reproducible check sequence is more important than „the perfect“ theory. The following playbook is structured so you can quickly distinguish between network, configuration, hypervisor and hardware.

Step 1: Is a time service active at all?

timedatectl status
systemctl list-unit-files | grep -E "chrony|ntp|timesync"
systemctl status chronyd --no-pager

If multiple services are active (e.g. chronyd and systemd-timesyncd), choose one deliberately and disable the other to prevent double correction.

Step 2: Can you reach the NTP sources?

NTP typically uses UDP port 123. Firewalls and security groups are a common reason for „unsynchronized“.

chronyc sources -v
chronyc activity
ss -anu | grep ":123" || true

If sources appear as unreachable: check DNS, check routing/firewall, and ensure the source really speaks NTP (not SNTP-only behind a load balancer without UDP stability).

Step 3: Large offset after boot or RESTore?

If systems boot with a large offset, services can write incorrect timestamps in the first minutes. Check the RTC and the Chrony policy („makestep“). For a one-time correction a step can be useful, but not as an ongoing state.

chronyc tracking
hwclock --show
journalctl -u chronyd --since "-24 hours" | grep -Ei "step|jump|makestep|reset"

Step 4: VM-specific indicators

If the drift occurs mainly under load, CPU steal/overcommit is a candidate. You will often then see „spikes“ in offsets, not just a constant drift.

uptime
vmstat 1 5
chronyc sourcestats -v

Interpretation: high run queue, fluctuating values and noisy jitter/delay patterns indicate scheduling problems. In such cases „more NTP“ does not solve the problem; you need resource or host fixes.

Best practices for cloud and hybrid environments

In cloud and hybrid setups additional boundary conditions apply: images are cloned, instances go to sleep, network paths change, and there are often provider-specific time services. Practical rules:

• Multiple time sources: at least three, ideally from different failure domains (e.g. two internal, one external or a provider-near service – depending on policy).
• Monitoring instead of gut feeling: offsets, jitter, reachability and „unsynchronized“ states belong in your monitoring.
• No step during normal operation: steps only controlled (boot/initialization) or in incident cases according to the runbook.
• Consider immutable images: drift files and persistence paths (e.g. /var/lib/chrony) must be accounted for in image strategies, otherwise Chrony never „learns“ and behaves after each reboot as if freshly installed.

Monitoring checklist: What you should actually alert on

Not every deviation is an incident. Meaningful alerting criteria are more about states and trends:

• chronyd not active or “System clock synchronized: no” for an extended period.
• No reachable sources (reachability drops).
• Offset above threshold for X minutes (threshold depends on the application, e.g. stricter for Kerberos).
• Frequent steps/jumps in logs (indication of competing time providers or VM issues).

For internal links in your magazine, a link to monitoring runbooks (metrics, alerts, escalation) is typically appropriate here, so that time is not only “configured” but also “operated”.

Fallback strategy: What to do when time is “broken”?

The most critical situation is a massively incorrect time in production systems. An uncontrolled jump can cause more damage than a slow correction. A practical fallback strategy comprises three elements: containment, correction, validation.

1) Containment: limit impact

• If possible: remove the affected instance from the load balancer or put services into “read-only”/maintenance mode.
• Prevent the instance from distributing incorrect time to others (if it is itself an NTP server: disable NTP serving temporarily).

2) Correction: controlled, not frantic

If the deviation is small to medium, prefer slewing (gradual correction). For very large deviations, a step may sometimes be unavoidable — but then plan it: consider applications, reevaluate tokens/SSO, account for log consistency.

A common approach is to let Chrony do the work and only temporarily allow step behavior for the incident. Do not change policies permanently “on suspicion”.

3) Validation: What must be checked after a time change?

• Auth: Kerberos/SSO, API tokens, service accounts.
• TLS: test critical endpoints with certificates, especially mTLS paths.
• Scheduler: check cron/systemd-timers for duplicate executions, apply job runbooks if necessary.
• Logs: mark the time window so Incident Response can correctly classify the anomaly.

Common pitfalls from daily operations (and how to avoid them)

Pitfall 1: “RTC in local TZ” and DST chaos

If the RTC is based on local time, timestamps shift at DST changes and boot order becomes hard to explain. Solution: set RTC to UTC, use the time zone only for display.

Pitfall 2: A single time source

A single NTP server is a single point of failure. Even if it appears stable: maintenance, DNS issues, or routing changes are enough to cause drift. Solution: at least three sources.

Pitfall 3: Competing time synchronization in VMs

Hypervisor tools provide time, Chrony provides time, and both intervene with different strategies. Solution: a clear owner decision, documented in the platform standard.

Pitfall 4: Misunderstanding snapshots as a “safe state”

Snapshots are not a consistent time anchor. After a RESTore, verifying time is mandatory. Solution: a RESTore runbook with time checks before application start.

Conclusion: Time is infrastructure — and requires operational discipline

Timekeeping on Linux servers is robust when you bring together three things: clean sources (redundant and reachable), a controlled correction strategy (steps only applied deliberately) and a realistic view of the RTC and virtualization (drift is normal, jumps are risks). Chrony is a practical tool for this, but the operating model is decisive: monitoring, runbooks for incidents and clear rules for VM time mechanisms.

If you anchor these standards in build templates and operational documentation, time issues will not disappear “magically” — but they will be rare, quickly diagnosable and, above all, manageable.

For this topic, Ntp servers Linux and time synchronization Linux are also important. This article puts these aspects into a clear context and shows what matters in everyday operations.