{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "kernelspec": { "display_name": "bash", "language": "bash", "name": "bash" }, "language_info": { "name": "bash", "version": "1.0.0" }, "blog_metadata": { "topic": "Self-Healing Ceph: Building an OSD Latency Watchdog for a Proxmox Home Lab", "slug": "self-healing-ceph-building-an-osd-latency-watchdog-for-a-pro", "generated_by": "LinkedIn Post Generator + Azure OpenAI", "generated_at": "2026-07-09T14:45:18.620Z" } }, "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Self-Healing Ceph: Building an OSD Latency Watchdog for a Proxmox Home Lab\n", "\n", "A single slow OSD can create intermittent VM freezes, sticky Proxmox UI behavior, and confusing storage symptoms that are easy to misdiagnose. This notebook turns the blog workflow into hands-on validation steps: inspect live Ceph OSD latency, simulate watchdog decisions safely, generate the watchdog and systemd units, and validate guardrails before enabling automation." ] }, { "cell_type": "code", "metadata": {}, "source": [ "set -euo pipefail\n", "if command -v apt-get >/dev/null 2>&1; then\n", " export DEBIAN_FRONTEND=noninteractive\n", " apt-get update\n", " apt-get install -y jq python3 python3-pip smartmontools sysstat util-linux\n", "fi\n", "python3 -m pip install --upgrade pip || true" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": {}, "source": [ "import json\n", "import os\n", "import pathlib\n", "import shutil\n", "import subprocess\n", "import sys\n", "import tempfile\n", "import textwrap\n", "import time" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Baseline the symptom with live Ceph OSD latency\n", "\n", "This helper inspects `ceph osd perf -f json` and prints the current worst OSD by combined apply and commit latency. Run it several times over a few minutes; if the same OSD keeps appearing, that is a stronger signal than a single noisy sample." ] }, { "cell_type": "code", "metadata": {}, "source": [ "#!/usr/bin/env bash\n", "set -euo pipefail\n", "cat >/tmp/ceph-worst-osd.sh <<'BASH'\n", "#!/usr/bin/env bash\n", "set -euo pipefail\n", "ceph osd perf -f json | jq -r '\n", " sort_by(-(.apply_latency_ms + .commit_latency_ms)) |\n", " .[0] as $o |\n", " \"worst=osd.\\($o.id) apply=\\($o.apply_latency_ms)ms commit=\\($o.commit_latency_ms)ms\"\n", "'\n", "BASH\n", "chmod +x /tmp/ceph-worst-osd.sh\n", "printf 'Created %s\\n' /tmp/ceph-worst-osd.sh\n", "printf 'Dry-run example (requires ceph CLI access):\\n'\n", "sed -n '1,120p' /tmp/ceph-worst-osd.sh\n", "if command -v ceph >/dev/null 2>&1 && command -v jq >/dev/null 2>&1; then\n", " /tmp/ceph-worst-osd.sh || true\n", "else\n", " echo 'Skipping execution: ceph and/or jq not available in this environment.'\n", "fi" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Required variables and assumptions for the watchdog\n", "\n", "The watchdog uses these tunables:\n", "\n", "- `THRESH_MS`: latency threshold in milliseconds\n", "- `REQUIRED_HITS`: consecutive bad samples required before action\n", "- `COOLDOWN`: minimum seconds between restarts\n", "- `BUDGET`: maximum restarts per hour\n", "- `MIN_UP_OSDS`: minimum number of up OSDs required before acting\n", "\n", "Assumptions:\n", "\n", "- `ceph` CLI works on the node\n", "- `jq` is installed\n", "- root privileges are available for `systemctl restart`\n", "- state is stored under `/var/lib/osd-watchdog/state.json`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Build the primary Bash watchdog\n", "\n", "This is the main watchdog implementation from the post, with consecutive-hit tracking, single-offender enforcement, recovery/backfill suppression, cooldowns, and an hourly restart budget. It is intentionally narrow: if conditions are ambiguous, it logs and does nothing." ] }, { "cell_type": "code", "metadata": {}, "source": [ "#!/usr/bin/env bash\n", "set -euo pipefail\n", "cat >/tmp/osd-latency-watchdog.sh <<'BASH'\n", "#!/usr/bin/env bash\n", "set -euo pipefail\n", "\n", "STATE=/var/lib/osd-watchdog/state.json\n", "LOGTAG=osd-watchdog\n", "\n", "THRESH_MS=\"${THRESH_MS:-80}\"\n", "COOLDOWN=\"${COOLDOWN:-1800}\"\n", "BUDGET=\"${BUDGET:-2}\"\n", "REQUIRED_HITS=\"${REQUIRED_HITS:-3}\"\n", "MIN_UP_OSDS=\"${MIN_UP_OSDS:-2}\"\n", "\n", "perf=\"$(ceph osd perf -f json)\"\n", "status=\"$(ceph osd stat -f json)\"\n", "health=\"$(ceph health detail -f json 2>/dev/null || ceph health -f json)\"\n", "\n", "mkdir -p \"$(dirname \"$STATE\")\"\n", "[[ -f \"$STATE\" ]] || echo '{\"last_restart\":0,\"events\":[],\"hits\":{}}' > \"$STATE\"\n", "\n", "bad_ids=\"$(jq -r --argjson t \"$THRESH_MS\" '\n", " map(select((.apply_latency_ms // 0) > $t or (.commit_latency_ms // 0) > $t))\n", " | sort_by(-(.apply_latency_ms + .commit_latency_ms))\n", " | .[].id\n", "' <<<\"$perf\")\"\n", "\n", "bad_count=\"$(wc -w <<<\"$bad_ids\" | tr -d ' ')\"\n", "if [[ \"$bad_count\" -eq 0 ]]; then\n", " tmp=\"$(mktemp)\"\n", " jq '.hits = {}' \"$STATE\" > \"$tmp\" && mv \"$tmp\" \"$STATE\"\n", " logger -t \"$LOGTAG\" \"healthy: no OSD above ${THRESH_MS}ms\"\n", " exit 0\n", "fi\n", "\n", "if [[ \"$bad_count\" -gt 1 ]]; then\n", " logger -t \"$LOGTAG\" \"skip: multiple OSDs above threshold (${bad_ids//$'\\n'/,})\"\n", " exit 0\n", "fi\n", "\n", "target=\"$(head -n1 <<<\"$bad_ids\")\"\n", "\n", "up_osds=\"$(jq -r '.num_up_osds // 0' <<<\"$status\")\"\n", "[[ \"$up_osds\" -gt \"$MIN_UP_OSDS\" ]] || {\n", " logger -t \"$LOGTAG\" \"skip: only $up_osds OSDs up\"\n", " exit 0\n", "}\n", "\n", "recovery_active=\"$(jq -r '\n", " [\n", " (.checks // {})[]?.summary?.message?,\n", " (.checks // {})[]?.detail[]?.message?\n", " ]\n", " | map(select(type==\"string\"))\n", " | any(test(\"recovery|backfill|degraded|peering|undersized\"; \"i\"))\n", "' <<<\"$health\")\"\n", "\n", "[[ \"$recovery_active\" == \"false\" ]] || {\n", " logger -t \"$LOGTAG\" \"skip: cluster health indicates recovery/backfill/degraded activity\"\n", " exit 0\n", "}\n", "\n", "tmp=\"$(mktemp)\"\n", "jq --arg id \"$target\" '\n", " .hits = (.hits // {}) |\n", " .hits[$id] = ((.hits[$id] // 0) + 1)\n", "' \"$STATE\" > \"$tmp\" && mv \"$tmp\" \"$STATE\"\n", "\n", "hits=\"$(jq -r --arg id \"$target\" '.hits[$id] // 0' \"$STATE\")\"\n", "if (( hits < REQUIRED_HITS )); then\n", " logger -t \"$LOGTAG\" \"defer: osd.$target above threshold, consecutive hits ${hits}/${REQUIRED_HITS}\"\n", " exit 0\n", "fi\n", "\n", "now=\"$(date +%s)\"\n", "last=\"$(jq -r '.last_restart // 0' \"$STATE\")\"\n", "(( now - last >= COOLDOWN )) || {\n", " logger -t \"$LOGTAG\" \"skip: cooldown active for osd.$target\"\n", " exit 0\n", "}\n", "\n", "hour_ago=$((now - 3600))\n", "used=\"$(jq -r --argjson h \"$hour_ago\" '[.events[] | select(.ts >= $h)] | length' \"$STATE\")\"\n", "(( used < BUDGET )) || {\n", " logger -t \"$LOGTAG\" \"skip: hourly restart budget exhausted\"\n", " exit 0\n", "}\n", "\n", "systemctl restart \"ceph-osd@${target}.service\"\n", "\n", "tmp=\"$(mktemp)\"\n", "jq --argjson now \"$now\" --arg id \"$target\" '\n", " .last_restart = $now\n", " | .events += [{\"ts\":$now,\"osd\":($id|tonumber)}]\n", " | .hits[$id] = 0\n", "' \"$STATE\" > \"$tmp\" && mv \"$tmp\" \"$STATE\"\n", "\n", "logger -t \"$LOGTAG\" \"restarted ceph-osd@${target}.service after ${REQUIRED_HITS} consecutive bad samples over ${THRESH_MS}ms\"\n", "BASH\n", "chmod +x /tmp/osd-latency-watchdog.sh\n", "printf 'Created %s\\n' /tmp/osd-latency-watchdog.sh\n", "sed -n '1,260p' /tmp/osd-latency-watchdog.sh" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Validate watchdog decision logic safely with mocked Ceph output\n", "\n", "Before touching a real cluster, validate the control flow with mocked `ceph`, `systemctl`, and `logger` commands. This harness runs the watchdog against synthetic scenarios so you can confirm healthy exits, multi-OSD suppression, recovery suppression, debounce behavior, cooldowns, and budget enforcement." ] }, { "cell_type": "code", "metadata": {}, "source": [ "#!/usr/bin/env bash\n", "set -euo pipefail\n", "WORKDIR=\"$(mktemp -d)\"\n", "MOCKBIN=\"$WORKDIR/bin\"\n", "STATE_DIR=\"$WORKDIR/state\"\n", "mkdir -p \"$MOCKBIN\" \"$STATE_DIR\"\n", "cp /tmp/osd-latency-watchdog.sh \"$WORKDIR/osd-latency-watchdog.sh\"\n", "sed -i \"s|^STATE=.*$|STATE=$STATE_DIR/state.json|\" \"$WORKDIR/osd-latency-watchdog.sh\"\n", "chmod +x \"$WORKDIR/osd-latency-watchdog.sh\"\n", "\n", "cat >\"$MOCKBIN/logger\" <<'BASH'\n", "#!/usr/bin/env bash\n", "set -euo pipefail\n", "echo \"LOGGER: $*\"\n", "BASH\n", "\n", "cat >\"$MOCKBIN/systemctl\" <<'BASH'\n", "#!/usr/bin/env bash\n", "set -euo pipefail\n", "echo \"SYSTEMCTL: $*\"\n", "BASH\n", "\n", "cat >\"$MOCKBIN/ceph\" <<'BASH'\n", "#!/usr/bin/env bash\n", "set -euo pipefail\n", "scenario=\"${MOCK_SCENARIO:-healthy}\"\n", "cmd=\"$*\"\n", "case \"$scenario:$cmd\" in\n", " healthy:osd\\ perf\\ -f\\ json)\n", " cat <<'JSON'\n", "[\n", " {\"id\":0,\"apply_latency_ms\":2,\"commit_latency_ms\":3},\n", " {\"id\":1,\"apply_latency_ms\":4,\"commit_latency_ms\":5},\n", " {\"id\":2,\"apply_latency_ms\":3,\"commit_latency_ms\":4}\n", "]\n", "JSON\n", " ;;\n", " healthy:osd\\ stat\\ -f\\ json)\n", " echo '{\"num_up_osds\":3}'\n", " ;;\n", " healthy:health\\ detail\\ -f\\ json|healthy:health\\ -f\\ json)\n", " echo '{\"checks\":{}}'\n", " ;;\n", "\n", " onebad:osd\\ perf\\ -f\\ json)\n", " cat <<'JSON'\n", "[\n", " {\"id\":0,\"apply_latency_ms\":2,\"commit_latency_ms\":3},\n", " {\"id\":1,\"apply_latency_ms\":120,\"commit_latency_ms\":95},\n", " {\"id\":2,\"apply_latency_ms\":3,\"commit_latency_ms\":4}\n", "]\n", "JSON\n", " ;;\n", " onebad:osd\\ stat\\ -f\\ json)\n", " echo '{\"num_up_osds\":3}'\n", " ;;\n", " onebad:health\\ detail\\ -f\\ json|onebad:health\\ -f\\ json)\n", " echo '{\"checks\":{}}'\n", " ;;\n", "\n", " multibad:osd\\ perf\\ -f\\ json)\n", " cat <<'JSON'\n", "[\n", " {\"id\":0,\"apply_latency_ms\":120,\"commit_latency_ms\":95},\n", " {\"id\":1,\"apply_latency_ms\":140,\"commit_latency_ms\":90},\n", " {\"id\":2,\"apply_latency_ms\":3,\"commit_latency_ms\":4}\n", "]\n", "JSON\n", " ;;\n", " multibad:osd\\ stat\\ -f\\ json)\n", " echo '{\"num_up_osds\":3}'\n", " ;;\n", " multibad:health\\ detail\\ -f\\ json|multibad:health\\ -f\\ json)\n", " echo '{\"checks\":{}}'\n", " ;;\n", "\n", " recovery:osd\\ perf\\ -f\\ json)\n", " cat <<'JSON'\n", "[\n", " {\"id\":1,\"apply_latency_ms\":120,\"commit_latency_ms\":95}\n", "]\n", "JSON\n", " ;;\n", " recovery:osd\\ stat\\ -f\\ json)\n", " echo '{\"num_up_osds\":3}'\n", " ;;\n", " recovery:health\\ detail\\ -f\\ json|recovery:health\\ -f\\ json)\n", " cat <<'JSON'\n", "{\"checks\":{\"PG_DEGRADED\":{\"summary\":{\"message\":\"Degraded data redundancy\"},\"detail\":[{\"message\":\"recovery in progress\"}]}}}\n", "JSON\n", " ;;\n", "\n", " lowup:osd\\ perf\\ -f\\ json)\n", " cat <<'JSON'\n", "[\n", " {\"id\":1,\"apply_latency_ms\":120,\"commit_latency_ms\":95}\n", "]\n", "JSON\n", " ;;\n", " lowup:osd\\ stat\\ -f\\ json)\n", " echo '{\"num_up_osds\":2}'\n", " ;;\n", " lowup:health\\ detail\\ -f\\ json|lowup:health\\ -f\\ json)\n", " echo '{\"checks\":{}}'\n", " ;;\n", "\n", " *)\n", " echo \"Unhandled mock ceph invocation for scenario=$scenario cmd=$cmd\" >&2\n", " exit 1\n", " ;;\n", "esac\n", "BASH\n", "chmod +x \"$MOCKBIN/logger\" \"$MOCKBIN/systemctl\" \"$MOCKBIN/ceph\"\n", "\n", "export PATH=\"$MOCKBIN:$PATH\"\n", "export THRESH_MS=80 REQUIRED_HITS=3 COOLDOWN=1800 BUDGET=2 MIN_UP_OSDS=2\n", "\n", "run_case() {\n", " local scenario=\"$1\"\n", " echo\n", " echo \"===== scenario: $scenario =====\"\n", " MOCK_SCENARIO=\"$scenario\" \"$WORKDIR/osd-latency-watchdog.sh\" || true\n", " echo \"state:\"\n", " cat \"$STATE_DIR/state.json\" 2>/dev/null || echo '{}'\n", "}\n", "\n", "run_case healthy\n", "run_case multibad\n", "run_case recovery\n", "run_case lowup\n", "run_case onebad\n", "run_case onebad\n", "run_case onebad\n", "run_case onebad\n", "\n", "echo\n", "echo \"Artifacts in $WORKDIR\"" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Generate the systemd oneshot service unit\n", "\n", "A oneshot service keeps each watchdog run isolated and easy to audit in the journal. This is preferable to a long-running shell loop for lab automation." ] }, { "cell_type": "code", "metadata": {}, "source": [ "#!/usr/bin/env bash\n", "set -euo pipefail\n", "cat >/tmp/osd-latency-watchdog.service <<'INI'\n", "[Unit]\n", "Description=Ceph OSD latency watchdog\n", "After=network-online.target\n", "Wants=network-online.target\n", "\n", "[Service]\n", "Type=oneshot\n", "ExecStart=/usr/local/sbin/osd-latency-watchdog.sh\n", "User=root\n", "Group=root\n", "Nice=10\n", "IOSchedulingClass=best-effort\n", "INI\n", "printf 'Created %s\\n' /tmp/osd-latency-watchdog.service\n", "cat /tmp/osd-latency-watchdog.service" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Generate the systemd timer unit\n", "\n", "The timer runs the watchdog on a fixed cadence. `Persistent=true` helps the node catch up cleanly after downtime instead of silently missing scheduled runs." ] }, { "cell_type": "code", "metadata": {}, "source": [ "#!/usr/bin/env bash\n", "set -euo pipefail\n", "cat >/tmp/osd-latency-watchdog.timer <<'INI'\n", "[Unit]\n", "Description=Run Ceph OSD latency watchdog on a fixed cadence\n", "\n", "[Timer]\n", "OnBootSec=2min\n", "OnUnitActiveSec=5min\n", "Persistent=true\n", "Unit=osd-latency-watchdog.service\n", "\n", "[Install]\n", "WantedBy=timers.target\n", "INI\n", "printf 'Created %s\\n' /tmp/osd-latency-watchdog.timer\n", "cat /tmp/osd-latency-watchdog.timer" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Install helper for a Proxmox/Ceph node\n", "\n", "This helper copies the watchdog and units into place, reloads systemd, enables the timer, and prints recent journal output. Review the files before running it on a real node." ] }, { "cell_type": "code", "metadata": {}, "source": [ "#!/usr/bin/env bash\n", "set -euo pipefail\n", "cat >/tmp/install-osd-watchdog.sh <<'BASH'\n", "#!/usr/bin/env bash\n", "set -euo pipefail\n", "install -Dm755 ./osd-latency-watchdog.sh /usr/local/sbin/osd-latency-watchdog.sh\n", "install -Dm644 ./osd-latency-watchdog.service /etc/systemd/system/osd-latency-watchdog.service\n", "install -Dm644 ./osd-latency-watchdog.timer /etc/systemd/system/osd-latency-watchdog.timer\n", "systemctl daemon-reload\n", "systemctl enable --now osd-latency-watchdog.timer\n", "systemctl list-timers --all | grep osd-latency-watchdog\n", "journalctl -u osd-latency-watchdog.service -n 20 --no-pager || true\n", "BASH\n", "chmod +x /tmp/install-osd-watchdog.sh\n", "printf 'Created %s\\n' /tmp/install-osd-watchdog.sh\n", "sed -n '1,200p' /tmp/install-osd-watchdog.sh" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Optional simplified reference implementation in Bash\n", "\n", "The post also included a shorter watchdog variant with fewer guardrails. It is useful as a compact reference, but the fuller version above is the better starting point for hands-on validation." ] }, { "cell_type": "code", "metadata": {}, "source": [ "#!/usr/bin/env bash\n", "set -euo pipefail\n", "cat >/tmp/osd-watchdog-simple.sh <<'BASH'\n", "#!/usr/bin/env bash\n", "set -euo pipefail\n", "STATE=/var/lib/osd-watchdog/state.json; LOGTAG=osd-watchdog\n", "THRESH_MS=\"${THRESH_MS:-80}\"; COOLDOWN=\"${COOLDOWN:-1800}\"; BUDGET=\"${BUDGET:-2}\"; MIN_UP_OSDS=\"${MIN_UP_OSDS:-2}\"\n", "perf=\"$(ceph osd perf -f json)\"; status=\"$(ceph osd stat -f json)\"\n", "target=\"$(jq -r --argjson t \"$THRESH_MS\" 'map(select((.apply_latency_ms // 0) > $t or (.commit_latency_ms // 0) > $t)) | sort_by(-(.apply_latency_ms + .commit_latency_ms)) | .[0].id // empty' <<<\"$perf\")\"\n", "[[ -n \"${target:-}\" ]] || { logger -t \"$LOGTAG\" \"healthy: no OSD above ${THRESH_MS}ms\"; exit 0; }\n", "up_osds=\"$(jq -r '.num_up_osds // 0' <<<\"$status\")\"; [[ \"$up_osds\" -gt \"$MIN_UP_OSDS\" ]] || { logger -t \"$LOGTAG\" \"skip: only $up_osds OSDs up\"; exit 0; }\n", "ceph osd safe-to-destroy \"$target\" >/dev/null 2>&1 && { logger -t \"$LOGTAG\" \"skip: osd.$target appears out/safe-to-destroy semantics unexpected\"; exit 0; } || true\n", "mkdir -p \"$(dirname \"$STATE\")\"; [[ -f \"$STATE\" ]] || echo '{\"last_restart\":0,\"events\":[]}' > \"$STATE\"\n", "now=\"$(date +%s)\"; last=\"$(jq -r '.last_restart // 0' \"$STATE\")\"; (( now - last >= COOLDOWN )) || { logger -t \"$LOGTAG\" \"skip: cooldown active for osd.$target\"; exit 0; }\n", "hour_ago=$((now - 3600)); used=\"$(jq -r --argjson h \"$hour_ago\" '[.events[] | select(.ts >= $h)] | length' \"$STATE\")\"\n", "(( used < BUDGET )) || { logger -t \"$LOGTAG\" \"skip: hourly restart budget exhausted\"; exit 0; }\n", "systemctl restart \"ceph-osd@${target}.service\"\n", "tmp=\"$(mktemp)\"; jq --argjson now \"$now\" --arg id \"$target\" '.last_restart=$now | .events += [{\"ts\":$now,\"osd\":($id|tonumber)}]' \"$STATE\" > \"$tmp\" && mv \"$tmp\" \"$STATE\"\n", "logger -t \"$LOGTAG\" \"restarted ceph-osd@${target}.service due to latency > ${THRESH_MS}ms\"\n", "BASH\n", "chmod +x /tmp/osd-watchdog-simple.sh\n", "printf 'Created %s\\n' /tmp/osd-watchdog-simple.sh\n", "sed -n '1,220p' /tmp/osd-watchdog-simple.sh" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Python reference translated into executable Bash generation\n", "\n", "The original post included a Python sample for easier testing and extension. This cell writes that reference script to disk so you can compare logic paths or adapt it later, while keeping the notebook's executable cells in Bash." ] }, { "cell_type": "code", "metadata": {}, "source": [ "#!/usr/bin/env bash\n", "set -euo pipefail\n", "cat >/tmp/osd-watchdog.py <<'PY'\n", "#!/usr/bin/env python3\n", "import json, subprocess, time, pathlib\n", "THRESH_MS, COOLDOWN, BUDGET = 80, 1800, 2\n", "state_path = pathlib.Path(\"/var/lib/osd-watchdog/state.json\")\n", "perf = json.loads(subprocess.check_output([\"ceph\", \"osd\", \"perf\", \"-f\", \"json\"]))\n", "bad = sorted([o for o in perf if o.get(\"apply_latency_ms\", 0) > THRESH_MS or o.get(\"commit_latency_ms\", 0) > THRESH_MS],\n", " key=lambda o: o.get(\"apply_latency_ms\", 0) + o.get(\"commit_latency_ms\", 0), reverse=True)\n", "if not bad: raise SystemExit(\"healthy\")\n", "state = json.loads(state_path.read_text()) if state_path.exists() else {\"last_restart\": 0, \"events\": []}\n", "now = int(time.time())\n", "recent = [e for e in state[\"events\"] if e[\"ts\"] >= now - 3600]\n", "if now - state[\"last_restart\"] < COOLDOWN or len(recent) >= BUDGET: raise SystemExit(\"rate-limited\")\n", "osd_id = bad[0][\"id\"]\n", "subprocess.check_call([\"systemctl\", \"restart\", f\"ceph-osd@{osd_id}.service\"])\n", "state[\"last_restart\"] = now; state[\"events\"] = recent + [{\"ts\": now, \"osd\": osd_id}]\n", "state_path.parent.mkdir(parents=True, exist_ok=True); state_path.write_text(json.dumps(state))\n", "print(f\"restarted osd.{osd_id}\")\n", "PY\n", "chmod +x /tmp/osd-watchdog.py\n", "printf 'Created %s\\n' /tmp/osd-watchdog.py\n", "sed -n '1,220p' /tmp/osd-watchdog.py" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Safe manual validation checklist\n", "\n", "Use this sequence before enabling live restarts:\n", "\n", "- confirm `ceph osd perf` identifies the expected offender\n", "- run the watchdog manually first\n", "- test healthy, one-bad, multi-bad, and recovery scenarios\n", "- verify consecutive-hit debounce works\n", "- verify cooldown and hourly budget work\n", "- only then install the service and timer on a real node\n", "\n", "Remember: repeated incidents are evidence of bad storage, not a reason to keep restarting forever." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Summary\n", "\n", "This notebook converted the blog workflow into a practical validation path: inspect live OSD latency, generate a guarded Bash watchdog, test it with mocked Ceph responses, and prepare systemd units for controlled deployment. The key principle is fail-closed automation: restart one OSD only when a single offender is sustained, the cluster is otherwise stable, and rate limits still allow action.\n", "\n", "## Next Steps\n", "\n", "- baseline `ceph osd perf` on your own cluster under normal and burst load\n", "- tune `THRESH_MS`, `REQUIRED_HITS`, `COOLDOWN`, and `BUDGET`\n", "- add SMART correlation and repeat-offender cutoffs before trusting live automation\n", "- deploy in dry-run mode first and review journal output\n", "- replace suspect consumer SSDs if the watchdog fires repeatedly" ] } ] }