Debugging systemd service dependencies: journalctl, systemctl and coredump analysis

Anyone who needs to debug systemd service dependencies in production requires a structured sequence of checks instead of frantic RESTarts. This runbook is aimed at administrators, system engineers and operators: it combines systemctl (status and dependency graph), journalctl (context-rich logs) and coredumpctl (crash analysis) into a practical troubleshooting sequence with fault patterns, risks, verification steps and clear fallback strategies.

Why dependencies are often the real cause

In systemd every unit (e.g. .service, .socket, .mount, .target) represents a desired state. Dependencies determine which units are started or in which order. The distinction between ordering (After=/Before=) and binding (Wants=/Requires=) is crucial: ordering only specifies who starts after whom; binding determines whether one unit enforces the start of another or is stopped when the other fails. Incorrectly modeled dependencies lead to boot delays, race conditions, RESTart loops or services that are „active“ but provide no functionality.

Initial assessment: determine the symptom class

At the start clarify: does the unit not start, is it stuck in the activating state, is it running but non-functional, or does it crash? This classification determines whether you should primarily investigate dependencies, resources (mounts, ports) or process faults.

Service does not start or remains „activating“

Main causes: blocking ExecStartPre steps, missing mounts, timeouts or jobs waiting on other units. First determine whether systemd is waiting for a prerequisite or whether a start command is failing.

Service runs but lacks functionality

Often only the ordering (After=) is defined, but not the start guarantee (Wants=/Requires=), or the service wrote before a mount. Check whether the required resources actually exist and are reachable.

Service crashes or RESTarts endlessly

Logs and core dumps are central here. Dependencies can still be involved if the process crashes due to a missing resource (e.g. a database). Distinguish between a crash caused by a signal (e.g. SIGSEGV) and an OOM kill by the kernel.

Step 1: systemctl status — quick, but targeted

systemctl status provides exit codes, RESTart counters, trigger information and recent log lines. It’s the first glance, but does not replace the remainder of the chain.

systemctl status --no-pager --full meinservice.service

Pay attention to:

• Active: For expected daemons, active (running) is correct; active (exited) is correct for Type=oneshot, but suspicious for daemons.
• ExecStart / ExecStartPre: Exit status != 0 indicates configuration or resource errors.
• TriggeredBy: Indications of socket or path activation.
• StartLimit: Rate limits prevent endless RESTarts.

Step 2: make dependencies visible

Distinguish which units are intended to be started (Wants/Requires) and which are only ordered (After/Before). That shows whether systemd ensures a prerequisite or merely waits.

systemctl list-dependencies --all meinservice.service
systemctl list-dependencies --all --reverse meinservice.service
systemctl show meinservice.service -p Wants -p Requires -p After -p Before -p ExecStart -p Type -p RESTart

Interpretation: Wenn in After= etwas steht, aber nicht in Wants/Requires=, existiert nur eine Reihenfolge, keine Startgarantie. Wenn Requires= gesetzt ist, schlägt der Start bei einem Fehler der Abhängigkeit fehl — bewusst einsetzen, wenn ohne Voraussetzung Datenverlust droht.

Step 3: Jobs, blockers and the critical chain

Jobs are scheduled actions; hanging jobs block chains. Use list-jobs and systemd-analyze to identify bottlenecks.

systemctl list-jobs
systemd-analyze critical-chain
systemd-analyze blame

systemd-analyze critical-chain shows which Units delay the boot path — useful for boot blockages caused by wait-online services or mounts.

Step 4: Use journalctl selectively

The journal combines timestamps, PID and unit context. Filter deliberately, otherwise you will get lost in the data. Work along the chain: logs of the affected service, then of the blocking Units.

journalctl -u meinservice.service -b --no-pager -n 200
journalctl -u meinservice.service --since "30 min ago" --no-pager
journalctl -u abhaengige-unit.service -b --no-pager -n 200

Use priority filters deliberately, but also check info messages, since many services log errors only as informational.

Journal configuration, limits and common pitfalls

journald has configuration limits: limited storage, rate limiting and optional forwarding to syslog or an external log server. If logs are missing, check storage and rate-limit events.

# Prüfen, ob persistentes Journal aktiv ist
ls -ld /var/log/journal || echo "kein persistent journal"
# Zeige journald-Konfiguration
cat /etc/systemd/journald.conf
# Journald-Usage ansehen
journalctl --disk-usage
# Alte Logs platzsparend entfernen
journalctl --vacuum-size=500M
journalctl --vacuum-time=7d

Note: rate limiting can suppress many recurring error messages. In an incident, do not disable it by default; first identify the cause and only lift rate limits temporarily for testing.

Common pitfalls and quick checks

network.target vs. network-online.target

network.target only signals that the network stack has been initialized, not necessarily that routing or DNS work. network-online.target requires a wait-online service; if that is misconfigured, it will block startup chains.

systemctl status --no-pager NetworkManager-wait-online.service
systemctl status --no-pager systemd-networkd-wait-online.service

Mounts: path exists, storage missing

Services sometimes write into local directories when a NAS/block device is missing — later the “data” disappears when the mount arrives afterwards. Use RequiresMountsFor= to ensure mounts.

findmnt -T /var/lib/meinservice
systemctl status --no-pager var-lib-meinservice.mount

Cyclic dependencies

Cyclic dependencies occur when A depends on B and B indirectly depends on A. systemd reports such cycles; resolve them by decoupling (Wants instead of Requires) or by using a dedicated Target as coordinator.

Socket activation

With .socket Units the service only starts on connection. That is efficient, but can confuse monitoring: the service can be inactive even though everything is functioning correctly.

systemctl status --no-pager meinservice.socket
systemctl status --no-pager meinservice.service
ss -lntp

If a service crashes: use coredumpctl safely

Core dumps provide detailed crash information but can contain sensitive data. Check retention policies and permissions before generating or distributing dumps. Systemd manages dumps via systemd-coredump; the configuration is in /etc/systemd/coredump.conf.

# Prüfe coredump-Konfiguration
cat /etc/systemd/coredump.conf 2>/dev/null || true
# Liste verfügbare Dumps für eine Unit
coredumpctl --no-pager list meinservice.service
# Detailansicht eines bestimmten Dumps
coredumpctl info PID

For the actual analysis you can extract the core or open it directly with gdb. Production systems often lack debug symbols; without symbols stack traces are limited but often still useful.

# Direkt mit gdb starten (nur in gesicherter Umgebung)
coredumpctl debug PID
# Alternativ: Core und Executable exportieren
coredumpctl dump PID --output=core.pid
file /usr/bin/meinservice
gdb /usr/bin/meinservice core.pid

Debug symbols: install package-specific -dbg/-debuginfo packages on an „analysis VM“ or use a symbol server. Avoid broad debug installations on production nodes for memory and security reasons.

Security restrictions and namespace effects

Unit settings such as ProtectSystem=, PrivateTmp=, NoNewPrivileges= or Capability-Revoke can deny services access even though dependencies are satisfied. Such hardening options are important for security, but can cause unexpected malfunctions.

# Prüfen, ob Härtung gesetzt ist
systemctl show meinservice.service -p ProtectSystem -p PrivateTmp -p NoNewPrivileges -p CapabilityBoundingSet
# SELinux-Denies prüfen (falls aktiv)
ausearch -m AVC -ts recent || true
# AppArmor-Denies prüfen
dmesg | grep apparmor || true

If an audit daemon reports access problems, read the audit logs before relaxing hardening options.

Wants, Requires, BindsTo: rules for sensible models

When to choose which behavior? Briefly:

• Wants=: soft binding, attempts to start, tolerates failures of the dependency.
• Requires=: hard binding, fails or stops on dependency failures.
• BindsTo=: like Requires, but additionally the unit is stopped if the dependency disappears (e.g. device removed).

Example: If your service depends on a database, choose Wants=database.service + After=database.service for soft behavior with controlled timeouts; choose Requires= if starting without the DB causes critical failures.

# Drop-in: /etc/systemd/system/meinservice.service.d/override.conf
[Unit]
Wants=postgresql.service
After=postgresql.service
# Alternativ für harte Abhängigkeit
# Requires=postgresql.service
# After=postgresql.service

Canary deployment and rollback strategy for unit changes

Systemd overrides are powerful; treat them like code:

1. Version control drop-ins in Git and document the change.
2. Apply changes first to a single instance (Canary).
3. Automated tests: start check, health check (port, endpoint), smoke tests.

# Canary: Änderung anwenden und prüfen
systemctl edit meinservice.service   # erstellt Drop-in
systemctl daemon-reload
systemctl RESTart meinservice.service
# Smoke-Test
ss -lnt | grep 12345
curl -f http://127.0.0.1:12345/health || echo "Healthcheck failed"
# Rollback
rm /etc/systemd/system/meinservice.service.d/override.conf
systemctl daemon-reload
systemctl RESTart meinservice.service

Operational checklist for hosting operators

• Collect initial status, list-jobs, dependency graph and relevant journal excerpts.
• Check mounts, DNS/backend reachability and ports before making unit changes.
• Limit debug installations to analysis boxes, not production.
• Version drop-ins and test rollbacks regularly in staging.
• Set up centralized log forwarding so boot and kernel logs are not lost.

Runbook: step sequence for an incident

1. Capture state: collect status, jobs, logs.
2. Determine error class: hung, running but non-functional, crashing.
3. Identify dependencies: list-dependencies + critical-chain.
4. Check blockers: journal of the blocking unit, mount/network/socket status.
5. Inspect crash path: coredumpctl info, kernel logs for OOM.
6. Check configuration: systemctl cat, verify drop-ins.
7. Controlled RESTart to test hypotheses; then collect and save logs.

# Schnell-Check-Skript (kompakt)
systemctl status --no-pager --full meinservice.service
systemctl list-dependencies --all meinservice.service
systemd-analyze critical-chain
journalctl -u meinservice.service -b --no-pager -n 200
coredumpctl --no-pager list meinservice.service || true
systemctl cat meinservice.service

Rollback, Rescue and worst-case scenarios

If an override worsens the situation, remove the drop-in and reload systemd. Test rollback steps regularly. In the worst case, boot into rescue mode, mount the system partition and remove interfering drop-ins manually.

# Drop-in entfernen und Rollback
rm -f /etc/systemd/system/meinservice.service.d/override.conf
systemctl daemon-reload
systemctl RESTart meinservice.service
systemctl status --no-pager --full meinservice.service

Conclusion

Debugging systemd service dependencies means reading chains instead of just RESTarting services. With a fixed verification sequence — check status, analyze dependency graph, identify jobs/blockers, evaluate logs and core dumps — you find the root cause faster and apply changes safely and in a rollback-capable manner. For hosting environments this approach reduces incident times and minimizes side effects such as data loss due to incorrect mounts or boot blockages. Maintain versioning for drop-ins, documented rollbacks and a structured canary strategy so changes can be rolled out in a controlled and safe way.

Debugging systemd service dependencies: operational architecture, fleet and integration aspects

Beyond single-node debugging, there are a number of architectural and operational aspects that can prevent incidents or cause unexpected dependencies during changes. Decision-makers and IT leadership should anchor these points in deployment and observability paths so that debugging occurs at the infrastructure level rather than at the symptom-treatment level.

Key aspects:

• Unit source and package management: systemd loads unit files from /lib/systemd/system (vendor) and /etc/systemd/system (admin). Use systemd-delta to detect overridden entries and avoid surprises after package updates.
• Enable/Mask strategy: Set non-production units deliberately to masked to prevent automatic starts. In CI, verify whether deployments leave units enabled/disabled.
• Transient units and simulations: For test runs use systemd-run or transient units in an isolated test environment before rolling changes out across the fleet.
• Configuration drift and automation: Version unit changes in the configuration repo and validate unit syntax automatically (systemd-analyze verify) as part of the release pipelines. This prevents differing dependency models between staging and production.
• Observability and alerting: Model health checks that verify not only process liveness but also application health (e.g. DB connection). Create alerts for StartLimitHit, repeated RESTart loops and increased journald disk usage so issues are detected early.
• Data protection and core dumps: Core dumps can contain sensitive data. Define policies for where dumps are stored and who has access; use an analysis VM with a symbol server instead of widespread debug installations on production systems.

Risks when changing: package updates can overwrite units, masking/unintentional enables can lead to start cycles, and inconsistent network-online implementations produce intermittent boot stalls. Therefore run unit checks as part of the release process and perform canary rollouts using targeted inventory tags. This ties systemd fault diagnosis to sustainable operational control and prevents recurring dependency incidents in your hosting environment.

systemd dependencies are also important for this topic. The article explains these aspects clearly and shows what matters in day-to-day operations.