Runbook Automation with Guardrails: Turning Read-Only Probes into Supervised Self-Healing

By | November 18, 2026

Runbook Automation with Guardrails: Turning Read-Only Probes into Supervised Self-Healing

In Self-Compacting & Bi-Temporal Agent Memory and Zero-Trust Signal Gateways for Local AI Agents, we built an autonomous agent runtime that verifies live infrastructure state (~/scripts/antigravity_verify.py) and communicates exclusively through a hardened Signal “Note to Self” gateway (~/scripts/signal_bot.py).

Until now, antigravity_verify.py was strictly read-only. It told you what was broken—a stalled NUT UPS daemon, an unmounted NFS share, or a dropped Tailscale policy route—but left you to SSH in and fix it manually.

Turning a diagnostic probe into an automated remediation engine is a classic engineering trap: unconstrained self-healing is just automated outage amplification. If a disk is full or hardware is failing, a naive auto-restarter will thrash systemd units in an infinite loop.

Here is how we upgraded our read-only probe into a Supervised Self-Healing Sentinel with strict guardrails, circuit breakers, and one-tap human approval over Signal.


Architecture Overview

Autonomous Self-Healing Sentinel Architecture

The pipeline separates detection, gating, approval, and execution into four decoupled stages:

  1. Functional Probes (systemd --user timer every 10 minutes): Checks not just TCP ports, but actual daemon behavior.
  2. Guardrail Gate (Meta/states/sentinel-proposals.json): Enforces cooldowns, TTLs, and a 24-hour circuit breaker before creating a short 4-character proposal ID.
  3. Lock-Free Signal Outbox (Meta/states/sentinel-outbox.jsonl): Avoids signal-cli lock contention by handing alerts to the running JSON-RPC daemon.
  4. Idempotent Execution (--approve <id>): Re-verifies the failure before running the runbook, executes the bounded command, and runs a post-check probe.

1. Functional Probes Over Shallow TCP Checks

An open TCP port does not mean a service is healthy. A NUT UPS server can listen on port 3493 while its USB driver is wedged, and an NFS mountpoint can exist in /etc/mtab while returning stale handle hangs.

We extended antigravity_verify.py with three functional probes alongside standard TCP socket checks:

def check_nut_telemetry() -> dict:
    """Verify NUT daemon actually serves UPS telemetry, not just an open port."""
    proc = subprocess.run(
        ["upsc", "volt_ups@nas-local", "ups.status"],
        capture_output=True, text=True, timeout=3.0,
    )
    val = proc.stdout.strip()
    if proc.returncode == 0 and val:
        return {"ok": True, "detail": f"status={val}"}
    return {"ok": False, "detail": (proc.stderr or "").strip() or f"rc={proc.returncode}"}


def check_nfs_mount() -> dict:
    """Verify NFS mount is both mounted and responsive (no stale handle)."""
    mp = subprocess.run(["mountpoint", "-q", "/mnt/nas_storage"], timeout=2.0)
    if mp.returncode != 0:
        return {"ok": False, "detail": "not mounted"}
    st = subprocess.run(["stat", "-t", "/mnt/nas_storage"], capture_output=True, timeout=2.5)
    return {"ok": st.returncode == 0, "detail": "mounted & responsive" if st.returncode == 0 else "stale mount"}

Similarly, the Tailscale probe inspects tailscale status --json (BackendState == "Running") and verifies that policy routing table 52 contains active routes (ip route show table 52).


2. Declarative Runbooks & Non-Negotiable Guardrails

Every remediable failure maps to an explicit entry in RUNBOOKS, tagged with its blast radius (low or medium), an idempotent command, and a post-verification function:

RUNBOOKS = {
    "nut_ups_nas": {
        "title": "Restart NUT UPS Daemon",
        "blast_radius": "low",
        "command": [
            "ssh", "-o", "ConnectTimeout=5", "-o", "BatchMode=yes",
            "user@nas-local",
            "sudo -n systemctl restart nut-driver-enumerator.service nut-server.service",
        ],
        "verify_fn": lambda: check_nut_telemetry()["ok"],
    },
    "nfs_local_mount": {
        "title": "Remount Primary NFS Share Locally",
        "blast_radius": "low",
        "command": ["bash", "~/scripts/mount_nas_nfs.sh", "mount"],
        "verify_fn": lambda: check_nfs_mount()["ok"],
    },
}

Four Guardrails That Prevent Runaway Loops

  1. 60-Minute Proposal TTL: Approvals are time-bounded. If you tap approve a1b2 three hours later when network topology has changed, the proposal is already marked expired.
  2. 15-Minute Cooldown: Rejected or recently triggered runbooks enter a cooldown window so a flaky sensor cannot spam your phone every 10 minutes.
  3. 24-Hour Circuit Breaker (MAX_ATTEMPTS_PER_DAY = 3): If a runbook runs 3 times in 24 hours and the service keeps failing, the Sentinel halts proposals for that runbook and escalates with a ⛔ Circuit Breaker Active alert.
  4. Self-Protected Daemon Boundary: A child subprocess spawned by signal-bot.service must never restart signal-bot.service itself—doing so severs the parent process mid-reply. Self-protected units are flagged for external/manual intervention only.

3. Idempotent Pre-Check & Post-Verification

When an operator replies approve <id> over Signal, the Sentinel does not blindly execute the shell command. It runs the probe again first:

# Idempotent pre-check: did the issue already self-resolve?
if rb["verify_fn"]():
    p["status"] = "executed_ok"
    return f"✅ [{p['id']}] {p['title']}: Already healthy (self-resolved). No action needed."

If the failure is still present, the Sentinel executes the runbook, waits 1.5 seconds for daemon stabilization, runs rb["verify_fn"]() again, and logs the final outcome to Meta/agent-log.md.


4. Solving signal-cli Lock Contention with an Atomic Outbox

Because signal_bot.py holds a persistent signal-cli jsonRpc child process open, running signal-cli send from a separate systemd timer fails on the SQLite/config lock.

Instead of spawning a second client, antigravity-sentinel.timer writes JSON lines to Meta/states/sentinel-outbox.jsonl. Inside signal_bot.py, a lightweight background thread polls the outbox every 5 seconds, atomically renames it to .processing via os.replace(), and dispatches each alert through the existing send_signal_reply(proc, msg) pipe:

def sentinel_outbox_watcher():
    outbox = Path("~/Vault/Meta/states/sentinel-outbox.jsonl").expanduser()
    while proc.poll() is None:
        if outbox.exists() and outbox.stat().st_size > 0:
            processing = outbox.with_suffix(".processing")
            outbox.replace(processing)
            for raw in processing.read_text(encoding="utf-8").splitlines():
                if raw.strip():
                    msg_text = json.loads(raw).get("message", "").strip()
                    if msg_text:
                        send_signal_reply(proc, msg_text)
            processing.unlink(missing_ok=True)
        time.sleep(5)

From your phone, the entire workflow takes two words:

  • heal — Triggers an on-demand Sentinel scan and lists any pending proposal IDs.
  • approve <id> / reject <id> — Executes (with pre/post verification) or rejects a proposal in seconds.

Key Takeaways

  • Read-only probes are the foundation, not the ceiling—reusing the exact same probe functions as pre-checks and post-checks guarantees runbook idempotency.
  • Short-lived tokens + circuit breakers make remote mobile approvals safe enough for real infrastructure.
  • Decoupled outbox files keep single-writer daemons like signal-cli lock-free without adding a message broker.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.