Log integrity and tamper evidence: How to build signed, append-only logs with offsite-verifiable archiving

Anyone who regards logs only as „debugging“ underestimates their value in a critical situation. As soon as an incident, an insider event or an audit occurs, a log becomes part of the chain of evidence. This is where Log integrity comes in: the ability to demonstrate that log data is complete and has not been altered unnoticed after the fact. In practice it is less about perfect immutability (which is rare in complex systems) than about Tamper-Evidence, i.e. detectability of manipulation: every modification leaves traces that can be verified outside the compromised system.

This article shows how to build signed, append-only logs and an offsite-verifiable archiving so that they work in operation: with clear prerequisites, typical pitfalls, verification steps, an actionable architectural pattern and a fallback strategy. The focus is not a single tool, but a robust mechanism you can implement with existing components (Syslog/journald, log forwarder, object storage, backup repository, SIEM).

What „Log integrity“ really means in operation

Integrity is often confused in everyday use with „read-only“. For logs that is too narrow. An attacker does not necessarily have to edit files; often it suffices to stop log sources, selectively filter them or tamper with timestamps. Log integrity is therefore a bundle of three objectives:

• Untamperedness: Individual entries must not be changed without detection (Tamper-Evidence).
• Completeness: Gaps must be detectable (e.g. via sequences or hash chains).
• Verifiability: The proof must be possible outside the compromised environment (offsite verification).

Important: „Append-only“ (append-only) is a write model, not a security proof. If a host is compromised, an attacker can often still bypass „append-only“ by operating elsewhere (reconfiguring forwarders, deleting queues, abusing storage credentials). Therefore you additionally need cryptographic chaining and an Offsite-Anchor.

Threat model: What signed, append-only logs protect against — and what they do not?

A robust design starts with the threat model. Tamper-Evidence typically targets the following classes:

• Post-compromise cover-up: After a breach, traces are deleted or altered to conceal persistence and data exfiltration.
• Insider manipulation: An admin or contractor with privileges alters audit trails.
• Compliance pressure: You must be able to demonstrably show that logs were not „tampered with“.

What it does not automatically protect against:

• False content: If an application itself produces false log lines, your system will sign those as well. Integrity does not validate the truthfulness of content.
• Missing capture: If the source does not log or logging is disabled, no signature can retroactively recover that. You can only make gaps visible.
• Complete key loss: If signing keys and verification anchors are compromised, the proof becomes worthless. Key architecture is therefore a core component.

Architectural pattern: Append-only + hash chain + signature + Offsite-Anchor

In practice, a multi-stage pattern has proven effective that works independently of specific products:

1. Collection: Logs originate at sources (Linux journald/syslog, Windows Event Logs, appliances, SaaS audit APIs).
2. Transport with buffering: A forwarder (agent or relay) transports reliably, ideally with queue/backpressure (so that data are not simply lost during network issues).
3. Append-only storage: A write-once–like path, e.g. object storage with Object Lock (WORM) or storage with an immutability feature.
4. Integrity proof: Hash chain (each line/batch contains the hash of the predecessor) plus digital signatures (asymmetric, e.g. Ed25519/ECDSA/RSA) per time window/bucket.
5. Offsite anchor: An external, independently secured location where you periodically store a „fingerprint“ (root hash/manifest signature), so that a compromised logging platform cannot be retroactively rewritten unnoticed.

Terms briefly classified: A hash chain cryptographically links data blocks; any modification to a block breaks the chain. A digital signature uses a private key for signing and a public key for verification; this allows any verifier to confirm authenticity without write privileges. An offsite anchor is an independent verification point (different account, different provider, offline medium, separate security tenant).

Why „append-only“ alone is not enough (and where it still makes sense)

Append-only is still valuable because it increases the effort required for tampering and reduces accidental modification. Typical implementations:

• Filesystem append-only flags (e.g. under Linux) – useful against „accidental deletion“, but not robust in the event of root compromise.
• Object storage with immutability (WORM/Object Lock) – significantly stronger, because even admins in the same account often cannot delete while retention is active.
• Write-once media (e.g. tape, offline exports) – very strong as a final recovery anchor, but slower for search/indexing.

The core weakness: If an attacker gains access early enough, they can attempt to prevent data before retention takes effect (stop forwarding) or redirect write paths. Therefore you must consider completeness (gaps) and offsite verification.

Signed logs in practice: What exactly is signed

Many teams don’t fail because of cryptography, but because of the question: „Do we sign every line?“ That is rarely necessary and often operationally expensive (CPU, overhead, key operations). A batch approach has proven effective:

• Logs are cut into short time windows (e.g. 1 minute or 5 minutes) or size windows (e.g. 50–200 MB).
• For each window a Manifest is created (list of hashes of the files/chunks, plus metadata such as time range, source, sequence numbers).
• This Manifest is digitally signed.
• Additionally a Root-Hash (e.g. Merkle-Root or hash of the Manifest) is anchored offsite.

This achieves: tampering with a single file is detectable (hash does not match), and tampering with the entire history is also detectable (offsite anchor no longer matches). At the same time the log pipeline remains performant.

Pitfall: time synchronization as a hidden integrity breach

Integrity strongly depends on time. If systems have different clocks, apparent gaps or incorrect ordering occur. For administrators this is often the first error found in audits. The minimum is consistent NTP/Chrony/PTP (time services), plus monitoring for drift. For highly critical proofs some teams additionally use a timestamping service (Trusted Timestamping): it cryptographically confirms that a hash existed at a given time. This is especially useful when you place offsite anchors in an external system and later need to prove they were not „created after the fact“.

Offsite-verifiable archiving: separate write path, read path and auditor

„Offsite“ does not only mean „another data center.“ For the purpose of tamper detection it is about administrative independence:

• Separate account/tenant: Logging writes to a storage whose retention is managed by a separate security or compliance account.
• Separate keys: Signature keys do not reside on the log servers; verification keys are widely distributed (Read-only).
• Read-only auditor role: Verification must work without modification rights, ideally also from an isolated system (Jump Host/Forensics-VM).

The goal is an asymmetric power model: production can deliver but cannot retroactively „clean up“ records. Auditors can verify, but not manipulate.

Concrete implementation scheme (tool-agnostic) with file structure and manifest

A practical scheme that can be applied to many environments works with a clear, predictable structure. Example of a daily directory tree per source:

/archive/logs/<source>/YYYY/MM/DD/HH/part-000123.log.gz
/archive/logs/<source>/YYYY/MM/DD/HH/manifest-YYYYMMDDHH.json
/archive/logs/<source>/YYYY/MM/DD/HH/manifest-YYYYMMDDHH.sig
/archive/anchors/YYYY/MM/DD/anchor-YYYYMMDDHH.txt

The Manifest (JSON) contains hashes of the log chunks and the chain information. Example structure:

{
  "source": "vpn-gateway-01",
  "window": {
    "start": "2026-08-09T10:00:00Z",
    "end":   "2026-08-09T11:00:00Z"
  },
  "sequence": {
    "first": 12001,
    "last":  12067
  },
  "prev_manifest_hash": "b5f6...",
  "chunks": [
    {"file": "part-00012001.log.gz", "sha256": "0c1a..."},
    {"file": "part-00012002.log.gz", "sha256": "9f3e..."}
  ],
  "manifest_sha256": "(optional self-hash)"
}

Why this works: the sequence makes gaps visible, the prev_manifest_hash forms a chain across time windows, and the chunk hashes secure the files. Even if someone replaces individual files, the Manifest no longer matches; if they replace Manifest and files together, they must forge the offsite anchor chain.

Key management: the most common reason why signatures are worthless in audits

Signed logs depend on the key model. A clear separation is essential:

• Signing key (private key): Only the signing service may use it. It should not reside on the same system as the log collection. Ideal: HSM (Hardware Security Module) or cloud KMS with a signing API; alternatively an isolated signing host with strict access controls.
• Verification key (public key): May be widely distributed (auditors, forensics, SIEM validators), since it only verifies.
• Rotation: Plan key rotation (e.g. annually or semi-annually) and keep old public keys to verify historical data.

Pitfall: If the same admin team can export the private key and re-sign at any time, the evidentiary proof can be undermined. Therefore check organizationally: who has access to the private key, who manages retention, and who operates the verification instance?

How-to: Integrity check (Runbook) with OpenSSL and hash lists

You need at least two verifiable pieces of evidence: (1) signature verification of the manifest and (2) hash verification of the chunks. The following procedure is generic and works for many signature formats (here as an example: the manifest hash was signed with a private key, verification with the public key).

1) Verify signature

# Dateien
MANIFEST="manifest-2026080910.json"
SIG="manifest-2026080910.sig"
PUBKEY="log-signing-public.pem"

# Prüfen (Beispiel RSA/ECDSA über SHA256)
openssl dgst -sha256 -verify "$PUBKEY" -signature "$SIG" "$MANIFEST"

Expectation: „Verified OK“. If not, either the manifest has been modified, the wrong public-key version is in use, or the signature algorithm does not match the key. Document these errors in operations (Key-ID, algorithm, validity periods).

2) Verify chunk hashes

Extract the hash list from the manifest (e.g. with jq) and verify the files. Example:

# Hashliste aus Manifest erzeugen: <sha256>  <filename>
jq -r '.chunks[] | "(.sha256)  (.file)"' "$MANIFEST" > checksums.sha256

# Prüfen
sha256sum -c checksums.sha256

If individual files are missing or hashes do not match, integrity is broken or the archive structure is no longer correct. You must then additionally check whether it was a transport/retention problem (e.g. queue loss) or manipulation (e.g. selective deletion).

Troubleshooting: Typical causes of gaps and integrity errors

1) Queue/Backpressure not properly handled

Many log arrows look fine in diagrams but break under peak load: the forwarder drops events or blocks applications. Pay attention to true persistence queues (disk-backed) and explicit limits. Warning signs are “dropped messages”, “queue full”, “retry storm”.

Diagnostic steps:

• Forwarder metrics: drop count, retry count, queue fill level.
• Storage latency: if object storage/indexing is slow, the forwarder must be able to buffer.
• Capacity: log volume per day, growth, retention.

2) Rotation/compression collides with signature windows

If you sign files first and then later alter them via rotation/compression (e.g. gzip afterward), signature verification will predictably fail. Rule: sign the final format, not intermediate states. Either: normalize/compress first, then hash and sign. Or: sign the raw stream, but archive that exact raw stream unchanged.

3) Time drift and DST/timezone mixing

If parts of the landscape use local time (CET/CEST) and others use UTC, you get “missing hours” or duplicated windows. Operationally, UTC for archive paths is the most robust. Keep timezone conversion out of the log pipeline: store in UTC, present/correlate in tools as needed.

4) Permissions: the logging component can delete even though it shouldn’t

A common design mistake is giving the log writer delete rights because it’s “easier”. For tamper-evidence that is poison. Better: write-only (Put), optional List for diagnosis, but no Delete. Retention should be controlled via a separate admin path.

Checklist: Minimum requirements for tamper-evident log archiving

• Source coverage: Which systems provide audit logs (AD, VPN, firewall, IAM, cloud control plane, databases, business software, admin portals)?
• Time consistency: NTP/Chrony active, drift monitoring, UTC in the archive.
• Transport: TLS-protected, persistent queue, defined retry strategy.
• Append-only storage: Object Lock/WORM or equivalent immutability, retention documented.
• Integrity layer: hash chain + signed manifest per time window/bucket.
• Offsite anchor: root-hash/manifest signature in a separate tenant/account or offline.
• Key model: private key isolated, rotation planned, retain old public keys.
• Verification: repeatable verification steps (runbook), automated spot checks.
• Alerting: gaps, signature errors, queue overflow, retention changes.

Implementation in stages: How to reach the goal without a Big-Bang

In existing environments a Big-Bang is rarely sensible. A staged approach reduces risk and helps gain acceptance:

Stage 1: Offsite archive + Immutability

Ensure logs are reliably moved out of the production domain, including retention. This is the basis against „server gone, logs gone“ and against ransomware that encrypts local log servers. Initially use the existing log pipeline, but harden storage and permissions.

Stage 2: Manifest + Hashes

Generate a manifest with hashes for each time window. This is not yet a signature, but it enables consistency checks and forces you to define file boundaries and windows.

Stage 3: Signatures and Offsite Anchor

Add signatures and anchor hashes offsite. From this point it becomes audit-proof, provided key management and roles are properly separated.

Stage 4: Automated Verification and Incident Integration

Automate spot checks (e.g. 10 random windows daily) and treat verification failures as an incident class (runbook, ownership, SLAs). Tamper-evidence is only valuable if indicators of manipulation are handled operationally.

Fallback strategy: What to do when signature verification fails?

A verification failure is not automatically an „attack“. Often it is an operational error. Nevertheless, you should proceed as with a security event, but with pragmatic escalation:

1. Determine scope: Does it affect one window, a single source, or all? Correlate with deployments/changes (forwarder update, storage policy).
2. Separate cause: Is a file missing (transport), does a hash mismatch (change/bit rot/process error), or is only the signature invalid (key/algorithm/format)?
3. Check offsite anchor: Do the externally anchored hashes match? If yes, the offsite system is likely intact; if not, escalate more strongly.
4. Check source status: Did the sources continue logging? Are there drop metrics, queue overflows, disk-full?
5. Forensic preservation: Preserve the affected artifacts (manifest, signature, affected chunks, metadata) read-only before you „repair“.
6. Recovery: If it was an operational error, reconstruct from original sources or secondary log paths where possible (e.g. SIEM ingest, syslog relays).

Important: Avoid „re-signing to make it green again“. That’s exactly the pattern that makes auditors suspicious. If a correction is necessary, it must remain visible as a correction (new window, new signature, documented reason).

Best practices for WordPress and admin portal operations: Where tamper-evidence is especially helpful

In WordPress-adjacent environments (admin portals, editorial systems, APIs for login/SSO, plugins, reverse proxies) the logging landscape is often heterogeneous. Typical sources you should focus on for integrity:

• Webserver/Proxy: access, errors, WAF decisions (important for brute-force, exploit attempts, unusual paths).
• Auth/SSO: IdP logs (login, MFA, token issuance), especially for privileged users.
• WordPress-Audit-Events: admin actions (plugin install, theme changes, user roles, API keys), if available.
• Database: admin logins, privileged queries (depending on DB and policy setup).
• System: sudo/SSH, package installations, service RESTarts, cron/systemd timers.

Practical tip: Adopt two perspectives in these areas: (1) application/web logs and (2) infrastructure/identity logs. Attackers can more easily influence one perspective than both. Tamper-evidence then helps you show which perspective remained consistent.

Offsite verification: How to regularly test whether your evidence is truly independent

Offsite verification is only credible if you test it. A simple, repeatable drill (monthly or quarterly) looks like this:

1. Select a time period (e.g., one hour) and a critical source.
2. Retrieve the archive files read-only from the offsite storage.
3. Verify the manifest signature and chunk hashes as described above.
4. Check the offsite anchor (e.g., root hash) against your independent repository.
5. Document the result, Key ID, tools used and hashes in a verification record.

If that feels too manual: automate the steps, but retain at least one manual drill in which an operator without specialist knowledge strictly executes the runbook. That is invaluable in a real incident.

Conclusion: Tamper evidence is an operational process, not a feature

Signed, append-only logs with offsite-verifiable archival are not a luxury but a robust answer to a real operational question: „Can we still trust our own logs when something goes wrong?“ The key is a layered design: append-only storage, hash chains and signed manifests for tamper detection, plus an offsite anchor that remains independent from the production domain. If you combine that with queue stability, time consistency, clean key management and a fallback strategy, you gain not only better auditability but also increased security in incident response and forensics.

In day-to-day operations this pays off especially when you operationalize integrity checks: sampling, alerts on gaps, clear ownership and a runbook that works at 03:00. Then log integrity is not just a promise, but a verifiable state.