Automated runbook: server remediation via AI recommendation and semi-automated execution over SSH

When a production server starts acting “strange”, daily operations often have only a narrow corridor: quickly stabilize, document cleanly, and avoid introducing new risks. This is where an Automated Runbook comes in: it translates recurring incident patterns into traceable verification steps and actions. New is that an AI module (LLM, Large Language Model – a language model that summarizes texts and generates suggestions) produces an AI suggestion from logs, metrics and context. Execution is, however, semi-automated: an operator confirms the steps, and only then are they applied in a controlled manner over SSH (Secure Shell – encrypted remote login).

This article presents a practical architecture and an operational model that works in admin teams: with clear security guardrails, checklists, common pitfalls, verification steps, implementation patterns and a robust fallback strategy. The emphasis is not “AI can do everything”, but: How do we use AI sensibly without hollowing out operations?

Automated Runbook: Why semi-automated — and not fully automatic?

Fully automatic remediation sounds tempting, but in practice it often fails for two reasons: missing context and hard-to-predict side effects. An LLM can formulate plausible steps, but it has no true “truth”; it generates text based on patterns. In server remediation, however, side effects are concrete: restarts, configuration changes, package updates, database rebuilds or firewall rules can cause collateral damage.

Semi-automation is therefore a stable compromise:

• Fast diagnostic path: AI proposes structured checks (e.g. “Disk full?”, “OOM killer?”, “DNS latency?”), including expected outputs.
• Operator as gate: A person assesses risk, timing, dependencies and only approves what fits the change window and the service criticality.
• Deterministic execution: The Runbook-Engine executes predefined, versioned steps — not “any” AI-generated shell text.

The goal is less heroic admin, more repeatable operations: same symptoms, same checks, same logs, same audit trails.

Reference architecture: AI suggestion, Runbook-Engine and SSH execution

For most environments, an architecture with a clear separation between “proposal” and “execution” proves effective:

• Signal sources: Monitoring (metrics), log platform, tracing, CMDB/asset data, ticketing/ITSM. A unique host/service identifier is important.
• Context collector: A job gathers relevant snippets (e.g. the last 10 minutes of logs, current alerts, last deployments, known maintenances). This determines whether the AI makes “good” suggestions.
• AI assistant (proposal module): Generates a diagnosis and action plan, including a risk assessment. Output preferably as structured JSON (not just free text).
• Runbook catalog: Versioned runbooks (Git), with approved actions, parameters, preconditions and rollback definitions.
• Runbook executor: Executes actions over SSH, writes logs, enforces timeouts, collects outputs, sets exit codes and stops on deviations.
• Gate & Audit: Four-eyes principle, change ID, approval, logging (who approved what and when?).

Important is the role distribution: the LLM recommends, the executor acts. This prevents „creative“ text outputs from being executed directly as commands.

SSH operating model: Bastion Host, keys, privileges

SSH is technically simple, but operationally full of detail. A robust model uses a Bastion Host (jump server as a controlled entry point), short-lived credentials (e.g. time-limited keys/certificates) and Least Privilege (only the rights the runbook requires). Practically this means:

• The executor connects only to the bastion, and from there to target hosts (network segmentation, central audit point).
• On target hosts there is a dedicated „runbook“ user with restricted sudo rights (only defined commands).
• Each action is assigned to a ticket/incident (change and audit trail).

Which incident types are suitable for server remediation via runbook?

Not everything is runbook-suitable. Good candidates are recurring and observable problems with clear checkpoints:

• Disk full: log rotation/journal, temp directories, crash dumps, old artifacts.
• Service hung: health check fails, process alive but unresponsive (e.g. deadlocks, thread exhaustion).
• OOM/memory pressure: out-of-memory killer, swap thrashing, leak indicators.
• DNS/network errors: resolver problems, faulty route, MTU/fragmentation.
• Expired certificates: chain issues, wrong truststore, expiring client certificates.

Poor candidates are „one-off“ special cases, changes with a high blast radius (e.g. kernel upgrade during an incident) or unclear symptoms without reliable telemetry.

Prerequisites: telemetry, identities, runbook design

For the AI proposal to be more than guesswork, solid foundations are required:

1) Telemetry with correlation

Logs, metrics and alerts must come together. „Correlation“ in operations means: hostname, instance ID, service name, deployment version and time window are consistent. Without that the AI will, in doubt, suggest „restart“, because it cannot recognize a differentiated cause.

2) Deterministic runbook actions

A runbook is more than a wiki text. For semi-automated execution you need idempotent steps (runnable multiple times without harm) and preconditions that prevent a step running in the wrong context. Example: „only start if free space < 5% and /var is the cause“.

3) Change and approval rules

Even during an incident: changes must be traceable. Minimum standard: change ID, approval (at least one operator), and a log that contains input, output, exit code and timestamp for each step.

Embedding the AI proposal correctly: output as a plan, not as shell

When an LLM generates free-form shell commands, you face two risks: unvalidated syntax and unvalidated intent. Better: the LLM provides a plan that your runbook engine validates against a catalog of allowed actions.

A practical format is JSON, which the engine strictly validates (schema validation):

{
  "incident_id": "INC-2026-071",
  "target": {
    "hostname": "app-17",
    "environment": "prod"
  },
  "hypotheses": [
    {
      "name": "disk_pressure_var",
      "evidence": ["/var usage high", "journald size increased"],
      "confidence": 0.72
    }
  ],
  "proposed_runbook": {
    "id": "Linux-disk-remediation",
    "steps": [
      {"action": "collect_disk_state", "params": {"paths": ["/", "/var"]}},
      {"action": "journald_vacuum", "params": {"retain": "1G"}},
      {"action": "logrotate_force", "params": {"dry_run": true}}
    ]
  },
  "risk_notes": [
    "Vacuum kann Debug-Logs entfernen; vorher Incident-Logs sichern.",
    "logrotate nur nach Review ohne dry_run ausführen."
  ]
}

Important: the engine accepts only runbook IDs and actions that exist in the catalog. Everything else is discarded. That way AI remains an assistance system, not a remote root.

Implementation: Runbook executor over SSH with review, logging and stop rules

An executor does not need to be complex, but it must be consistent. Three properties are critical in operation:

• Auditability: Each step writes standardized logs (start/end, target, command, output hash, exit code).
• Security boundaries: timeouts, allowed commands, blocked targets (e.g. domain controllers, storage controllers), rate limits.
• Stop rules: On deviations, do not ‚keep trying‘, stop and escalate.

Example: SSH execution via bastion with restricted sudo privileges

In the following example the executor uses a dedicated user and enforces non-interactive execution. This is not a complete product, but a tangible pattern for your own runbooks.

#!/usr/bin/env bash
set -euo pipefail

BASTION="bastion01"
TARGET="$1"               # z.B. app-17
RUNBOOK_ID="$2"           # z.B. Linux-disk-remediation
INCIDENT_ID="$3"          # z.B. INC-2026-071

SSH_OPTS=(
  -o BatchMode=yes
  -o StrictHostKeyChecking=yes
  -o ConnectTimeout=8
  -o ServerAliveInterval=10
  -o ServerAliveCountMax=3
  -J "runbook@${BASTION}"
)

log(){
  printf '%s %s %s\n' "$(date -Is)" "${INCIDENT_ID}" "$*"
}

run(){
  local cmd="$1"
  log "STEP cmd=${cmd}"
  ssh "${SSH_OPTS[@]}" "runbook@${TARGET}" -- "${cmd}"
  log "STEP exit=$?"
}

log "START runbook=${RUNBOOK_ID} target=${TARGET}"

# Beispiel-Schritte (in der Praxis aus einem signierten Katalog geladen)
run "sudo -n /usr/local/sbin/collect_disk_state"
run "sudo -n /usr/local/sbin/journald_vacuum --retain=1G"

log "DONE runbook=${RUNBOOK_ID} target=${TARGET}"

Why these details matter: BatchMode prevents password prompts, StrictHostKeyChecking reduces MitM risks (Man-in-the-Middle), and -J (Jump) enforces the bastion as the entry point. Stop rules arise here via set -e: as soon as a step fails, the script exits in a controlled way.

Runbook design in practice: Preconditions, Dry-Run, Idempotence

For admin teams, three principles make the difference between „automation helps“ and „automation causes trouble“:

Preconditions (preconditions) enforce context

Before you delete, stop, or restart, check the state. Example: perform disk remediation only when a filesystem is truly tight and not, for example, when an NFS mount is hung (otherwise you’ll worsen the situation through timeouts).

#!/usr/bin/env bash
set -euo pipefail

THRESHOLD_PERCENT=95

# Check: Which mounts are critical?
df -P | awk 'NR>1 {print $5 " " $6}' | while read -r use mount; do
  pct=${use%%%}
  if [ "${pct}" -ge "${THRESHOLD_PERCENT}" ]; then
    echo "CRITICAL ${mount} ${pct}%"
  fi
done

# Check: journald size (can fill /var)
if command -v journalctl >/dev/null 2>&1; then
  journalctl --disk-usage || true
fi

The goal is not „pretty output“, but an objective signal: Runbook may only continue when preconditions are met.

Dry-Run as standard

For potentially destructive steps, the first run should be „dry“ (only show what would happen). This fits perfectly with semi-automatic execution: the operator sees the effect and then confirms the real step.

#!/usr/bin/env bash
set -euo pipefail

# Example: test logrotate first, then execute
logrotate -d /etc/logrotate.conf
# Only after approval:
# logrotate -f /etc/logrotate.conf

Idempotence: same action, same effect

Idempotent means: if a step runs twice, no additional harm occurs. Example: „start service“ is idempotent, „patch configuration multiple times“ often is not. Therefore prefer „replace/ensure“ patterns over „append“.

Risks and typical pitfalls (from an operations perspective)

AI-driven runbooks rarely fail because of SSH – they fail because of boundary conditions. The most common pitfalls:

1) Wrong host or wrong environment

A classic: an alert comes from „prod“, but the context collector accesses „stage“ logs (identical names, incorrect labels). Countermeasure: strict checks on environment and asset ID, plus „deny lists“ for particularly critical systems.

2) Incomplete data

When logs are missing (rotation, Forwarding-Backpressure) or metrics are not up to date, the AI proposal becomes uncertain. Treat „no data“ as its own signal. In the runbook: first repair telemetry (e.g., check the Logforwarder-Queue), then remediate.

3) Side effects from „helpful“ standard measures

RESTart as the default is risky when the system is stuck in a recovery loop, a database replication is lagging, or a storage is currently degraded. Runbooks therefore need stop rules and „do-not-do“ lists, e.g. no package updates during an incident without a separate change gate.

4) Permissions too broad or too RESTrictive

Too broad: the runbook user has blanket sudo, making every mistake a potential incident trigger. Too RESTrictive: the runbook aborts and admins bypass the process. A sudoers whitelist with explicit command paths has proven effective.

Example: sudoers whitelist for runbook actions

# /etc/sudoers.d/runbook
Defaults:runbook !requiretty
runbook ALL=(root) NOPASSWD: 
  /usr/local/sbin/collect_disk_state, 
  /usr/local/sbin/journald_vacuum, 
  /usr/local/sbin/service_healthcheck, 
  /bin/systemctl RESTart myservice

Important: only absolute paths, no shell wildcards, and after changes always validate with visudo (syntax check) before rolling out.

Pre-execution checks: Operator checklist

Before you enable semi-automatic execution via SSH, a short, disciplined checklist helps. It is deliberately formulated to be ‚operations-oriented‘:

• Scope: Affected hosts/services clear? Correct environment (prod/test)?
• Impact: What is the worst-case risk of the proposed steps (RESTart, data loss, log loss)?
• Dependencies: Do other services depend on the host (e.g. shared DB, proxy, queue)?
• Timebox: How long may the remediation take? Is there a maintenance window or SLA limits?
• Observability: Which metric/checks indicate success? (e.g. error rate decreases, Disk < 90%, Healthcheck green)
• Rollback: Is there a defined fallback strategy per step?
• Approval/Audit: Ticket/incident ID present, approval documented.

If any of these questions remains „unclear“, that is a signal: collect data first, then act.

Rollback and fallback strategy: What to do if remediation fails?

A fallback strategy is not optional. It is part of the runbook. In practice, three levels have proven useful:

1) Stepwise rollback (where possible)

Configuration changes should be performed as „backup & replace“: back up the previous version, activate the new version, validate, and revert on errors.

#!/usr/bin/env bash
set -euo pipefail

CFG="/etc/myservice/myservice.conf"
BK="${CFG}.$(date +%Y%m%d%H%M%S).bak"

cp -a "${CFG}" "${BK}"

# Beispiel: neue Konfiguration aus gerendertem Artefakt einspielen
cp -a /var/lib/runbook/rendered/myservice.conf "${CFG}"

systemctl reload myservice

# Validierung: Service muss aktiv sein
systemctl is-active --quiet myservice

echo "OK: config applied; backup at ${BK}"

2) Safe Stop: Automation halts, human takes over

If preconditions break, exit codes are unexpected, or validation fails, the system must stop. Important: do not continue automating, but freeze the state (secure logs, save current outputs) and escalate to 2nd/3rd-Level.

3) „Known Good“ path

For critical services, a prepared fallback is worth having: the last known good state (e.g. previous package, previous config, previous container image). Even without CI/CD this can be modeled via an artifact repository and defined versions. Crucially, the path must be tested in advance.

Security and Compliance: Audit, Prompt Data, Secrets

When it comes to AI suggestions, the data question is central: which logs go where? Who may see them? And what happens to secrets? Some established guardrails:

• Prompt hygiene: Secrets (tokens, private keys, passwords) are masked before the LLM call. Masking means: remove or replace known patterns (e.g. „Authorization: Bearer …“).
• Data minimization: Send only the relevant log lines and time windows, not “everything”.
• On-Prem/Private LLM where necessary: If compliance requires it, keep the LLM in your controlled environment.
• Audit logging: Every decision (AI suggestion, operator approval, executed steps) is recorded in an immutable, auditable log.

Also important: the Runbook engine is an administrative access point. It belongs in your threat modeling: network segmentation, hardening, patch management, MFA/SSO at the approval gate and clear emergency processes.

Practical blueprint: a runbook flow that works in day-to-day operations

An operationally useful flow is short enough for an incident but strict enough for security. A proven pattern:

1. Trigger: Alert or ticket creates an incident ID and target(s).
2. Context collect: Defined queries (logs/metrics/events) are collected and stored.
3. AI suggestion: LLM provides hypotheses + proposed runbook + risks as JSON.
4. Mapping: Engine checks: Is there a matching, approved runbook? Are actions allowed?
5. Operator review: Checklist + approval of individual steps (e.g. diagnose first, then remediate).
6. Execute via SSH: Stepwise execution with timeouts, stop rules, outputs.
7. Validate: Success is checked against defined SLO/health signals.
8. Close & learn: Improve the runbook: missing preconditions, new pitfalls, better data collection.

If you already use Ansible or a runbook platform, many of these elements can be integrated. The core remains: AI generates suggestions, execution stays controlled, versioned and auditable.

Troubleshooting: when SSH remediation itself causes problems

The runbook system itself can be a source of failure. Typical causes and quick checks:

SSH cannot connect

• Network path: Is the bastion reachable? Is the target host reachable? Are routing/ACLs correct?
• Host keys: StrictHostKeyChecking blocks after rebuild (expected). Process: define host-key rotation instead of ’simply disabling‘.
• Auth: Short-lived credentials expired? Time drift (NTP) on bastion/target?

A minimal connection check that can run as a pre-step in the runbook:

#!/usr/bin/env bash
set -euo pipefail

BASTION="bastion01"
TARGET="$1"

ssh -o BatchMode=yes -o ConnectTimeout=5 "runbook@${BASTION}" -- "echo BASTION_OK"
ssh -o BatchMode=yes -o ConnectTimeout=5 -J "runbook@${BASTION}" "runbook@${TARGET}" -- "echo TARGET_OK"

Commands fail with sudo errors

Then the sudoers whitelist is usually wrong (path is incorrect, requiretty active, or the command internally invokes a shell). Check whether the runbook really uses only the permitted absolute paths, and whether „sudo -n“ (non-interactive) is set.

Runbook does „nothing“ but reports success

This is a design problem: missing validation. Each remediation step needs a metric or a state that changes. Example: After Log-Vacuum, „df“ must be below the threshold again; otherwise the step is considered unsuccessful.

Conclusion: AI is the accelerator, the Runbook remains the brake

Server remediation based on AI suggestions and semi-automatic execution over SSH works reliably when you clearly separate roles: AI provides hypotheses and structured plans, your Runbook-Engine executes only approved actions, and an operator keeps a hand on the gate. With preconditions, Dry-Run, idempotence, Audit-Logs and a tested fallback strategy you avoid the most common risks: wrong targets, unclear data, and „creative“ changes during an incident.

If you want to explore the topic further, the next step is to consistently standardize your remote access architecture (bastion, logging, permissions) and your Runbook-Governance (versioning, reviews, change integration). That turns quick assistance into a robust operational process.

Runbook automation is also important for this topic. The article situates these aspects clearly and shows what matters in day-to-day operations.