DAIM-OS Self-Healing Artifact: Event-Driven Link-Failure Recovery for a DAIM-OS Client Using OVSDB Notifications and Reactive Path Recomputation

Reproducibility artifact for "Event-Driven Link-Failure Recovery for a DAIM-OS Client Using OVSDB Notifications and Reactive Path Recomputation" (Paper 3 in the DAIM six-paper programme, prototype-stage, not yet submitted). Contains an event-driven repair agent (daim_link_agent.py) that subscribes to real OVSDB Interface.link_state notifications, recomputes an alternate path with breadth-first search over a declared topology graph, installs it through the existing DAIM-OS OVS adapter, and suppresses repeated repair processing on a flapping link via an IDLE/ACTIVE/HELD-DOWN hold-down state machine, keyed by physical edge rather than interface name since v0.3.0. An edge is only treated as recovered once every interface observing it independently confirms `up` (v0.4.0) -- a persistent per-interface state, not a single last-report-wins value per edge. A fresh agent process also correctly reads OVSDB's initial table snapshot at startup (v0.5.0), rather than only matching later "new"-actioned rows, so an interface already down when the agent starts is detected and routed around instead of silently assumed up; that snapshot is now also validated to be exactly "up" or "down" rather than treating any unrecognised value as implicitly up, and every fatal startup path cleans up the OVSDB monitor subprocess rather than leaking it (v0.6.0). Four real defects are documented in the history: a stale-state bug (v0.2.0, found via external manuscript review); a cross-interface hold-down gap (v0.3.0, found by the first live-network run of the flapping-link protocol against a real Mininet/OVS link -- OVSDB reports a link's two interfaces' transitions independently, and the pre-fix hold-down only covered the interface that triggered the repair); a last-report-wins edge-recovery gap (v0.4.0, found by code review); and a startup-state synchronization gap (v0.5.0, found by code review and confirmed against a live ovsdb-server -- the pre-fix agent produced 100% packet loss when a link was already down at startup, 0% after the fix). A fifth, separate bug in this artifact's own verification harness (stage3_startup_already_down.py's packet-loss check, a substring match that misclassified a 100%-loss run as loss-free) is also documented and fixed (v0.6.0), with its own regression test -- the underlying 100%/0% result was unaffected, since the figure-generation script parsed the numeric loss value independently, but the harness's own boolean field was wrong until this fix. v0.7.0 extends link_state validation to the ongoing event stream, not just the startup snapshot (decide_link_event() now returns a distinct invalid_link_state action instead of silently falling through to a no-op), and fixes a mislabelling bug in the flapping-link comparison figure: its "BFS/recompute calls" row previously counted every non-suppressed action including "recovered", which never calls BFS, so it reported 4-to-2 when the real count (repair/noop/repair_failed only) is 1-to-1 for this specific live schedule -- the BFS-call-reduction claim is Figure 5's logic-level result, not this live figure's. v0.8.0 adds multi-OVS connection multiplexing and per-hop flow routing (REMOTE_ENDPOINTS, load_topology_config(), select()-based multiplexing of multiple ovsdb-client monitor connections in monitor_link_rows()), measured against a real two-VM testbed joined by a GRE tunnel: a remote-edge failure -- on switches the agent shares no local OVSDB connection with -- was correctly detected and repaired across five repetitions. v0.9.0 fixes two issues found by external review of that same result: apply_flow() bypassed the DAIM-OS OVS adapter (daim_ovs_flow) for remote bridges and called ovs-ofctl directly, inconsistent with this paper's own adapter claim -- fixed by routing both local and remote bridges through the same adapter binary, confirmed empirically to already support a remote target with no C-code change; and install_path()/withdraw_path() discarded apply_flow()'s per-call success/failure, so a partially-failed installation could be logged as repair_installed -- fixed by a new execute_repair() that only reports success, and only advances current_path, when every flow-mod call actually succeeded, otherwise reporting a distinct repair_incomplete event. The n=5 measurement was re-run against the fixed code: still 5/5 clean, mean repair-action time 191.48 ms (up from 185.75 ms pre-fix, the honest cost of the extra process-spawn hop the adapter path adds), independently bounded by the packet-loss gap in the concurrent probe traffic (an independent process in the agent's own VM, not a different host -- corrected from an earlier, imprecise description). v0.10.0 fixes a second review pass's two further findings, neither requiring new live-network data: execute_repair()'s own failure path was itself unsound, since withdraw_path(current_path) runs before install_path(), so a withdraw-succeeded/install-failed outcome means neither the old nor the new path is actually installed -- reporting the old path as still current (v0.9.0's behaviour) was its own false claim; current_path now becomes None on any partial failure, with withdraw_path(None) and a subsequent repair attempt from None both handled safely without crashing. Separately, bfs_path()/path_to_flows() hardcoded the literal host names "h1"/"h2" instead of the declared SOURCE/DEST/HOST_ATTACHMENT globals load_topology_config() already allows overriding; harmless for both topologies measured here (both use h1/h2) but would have raised KeyError for a differently-named-host deployment, so this is a prerequisite for future multi-topology work, fixed now. This same pass also corrected a real math error in the v0.9.0 report's own cross-check, which claimed the agent-reported repair-action time fell inside the independent ping-outage bound in every repetition -- false for repetition 5 by 4.69 ms, now stated correctly as 4-of-5 with an honest explanation (repair_end_ns marks control-plane completion, not necessarily the same instant as data-plane restoration). v0.11.0 fixes a third review pass's three further findings: execute_repair()'s current_path=None (v0.10.0) was not itself a guarantee of retry, since decide_link_event() never re-triggers a repair for an edge already in down_edges -- fixed with maybe_retry_repair(), called from main()'s periodic tick, retrying a degraded state until the underlying failure clears with no new OVSDB event required; the startup path had the identical false-success gap, fixed by execute_startup_install() mirroring execute_repair()'s contract; and withdraw_path() matched flow deletion far too broadly (a bare in_port match, confirmed empirically to delete an unrelated flow sharing only that port) -- fixed with OpenFlow cookie scoping (AGENT_COOKIE), entirely within daim_link_agent.py, no adapter C-code change. The cookie-scoping fix changes the wire-level flow-mod strings sent on every call, so the n=5 multi-OVS measurement was re-run again: still 5/5 clean, mean repair-action time 203.54 ms (up from 191.48 ms pre-cookie-scoping). v0.12.0 closes four further evidence-gate items from a fifth review pass. AGENT_COOKIE, a single fixed constant correct only for a single-agent deployment, is replaced by _agent_cookie(), derived deterministically per (SOURCE, DEST) pair, so two agents protecting different pairs on a shared switch no longer collide. The repair-action time metric definition (manuscript Section 6.3) is corrected -- the timer starts inside execute_repair(), after decide_link_event()'s own BFS has already run, not from the start of BFS recomputation as previously stated; a wording fix, no numbers changed. monitor_link_rows() now attempts one reconnect on a dead ovsdb-client monitor stream, reconciling state via resync_from_reconnect() and triggering a real repair if the resync reveals the installed path no longer avoids every down edge -- verified live by killing a real remote monitor connection and confirming both a clean reconnect and continued correct detection/repair afterward. install_path() gained a forwarding-consistency pre-check (_conflicting_flow_cookie(), a read-only ovs-ofctl dump-flows query outside the adapter): a foreign-cookie flow already occupying the exact match a new install would use is now rejected rather than silently overwritten, verified live against a real conflicting flow. Most substantially, execute_repair() now implements a two-phase prepare/commit protocol -- the new path is staged before the old path is withdrawn, and a staging failure rolls back to the untouched old path rather than degrading to current_path=None. Implementing this the naive way (deleting by OpenFlow match at commit and rollback) turned out to be unsafe: the switch attached to SOURCE and the switch attached to DEST each have one flow whose match is identical across every alternate path, so a naive delete removed the live, just-staged entry outright rather than leaving or restoring it -- confirmed live as a real blackhole in an otherwise cleanly-reported repair, fixed by _withdraw_stale_path() and _rollback_staged_path(), which skip or restore that colliding match instead of deleting it, re-verified against the same live fault injection that found the bug. The hold-down window's start moved out of decide_link_event() (which set it at decision time, before any I/O ran) into main(), gated on execute_repair() reporting an actual commit, re-verified live by observing correct suppression of the same physical link's other interface reporting the identical fault moments later. v0.13.0 closes three further gaps in this same rollback protocol, found by a sixth review pass reading the code again after it was already tested and live-verified: rollback's own result was previously discarded, so a rollback that itself failed partway through was reported as a confirmed-intact old path anyway -- fixed with a distinct repair_rollback_incomplete event and current_path=None whenever rollback does not fully succeed. Rollback recomputed the intended new path from scratch instead of acting on what install_path() actually confirmed staged, meaning a flow the forwarding-consistency check correctly rejected could still be "restored" at a colliding boundary match -- silently overwriting the very foreign flow that check had just protected -- fixed by having install_path() return the real staged list and _rollback_staged_flows() (renamed from _rollback_staged_path()) act only on it. And staging installed flows in natural SOURCE-to-DEST order rather than deferring boundary-hop collisions to last, so a live switch could repoint onto the still-incomplete new path before the rest of it was ready -- fixed with old_path-aware staging order, non-colliding flows first. All three re-verified against the same live fault injection that found v0.12.0's original blackhole. Also this round: _agent_cookie() widened from a truncated 32-bit value to the full 64-bit OpenFlow cookie-field width; and the forwarding-consistency check's documentation corrected to state precisely what it guarantees (a conflict present at check time is detected and rejected, not an atomic race-free guarantee against a concurrent writer -- consistent with this paper's declared one-process-per-pair deployment model). Contains pure-logic unit tests for the path computation and all hold-down/startup behaviours (suppression, stale-state reconciliation, cross-interface coverage, edge-confirmation, startup synchronization, link_state validation at both startup and runtime, multi-OVS target routing, adapter unification, partial-failure honesty, endpoint genericity, cookie-scoped deletion, startup-install honesty, repair-retry liveness, pair-scoped-cookie determinism, monitor-subprocess reconnect, forwarding-consistency rejection, two-phase rollback with boundary-hop-collision handling, the rollback protocol's own result-honesty/staged-subset/staging-order correctness, and ambiguous-timeout read-back resolution (v0.14.0: a seventh review pass found apply_flow() treated a subprocess timeout as a confirmed failure identically to a non-zero exit, though a timeout only means the client gave up waiting, not that the switch never applied the operation -- fixed by reading the switch's actual state back directly and resolving True/False only once confirmed, raising rather than guessing if even the read-back cannot tell; live-verified only as a regression check on the ordinary path, since no measured repetition has ever actually timed out; and service-restoration decomposition instrumentation (v0.15.0, the first Layer-2 evaluation step, deliberately landed before further live experiments so the final timing dataset is collected once against fully instrumented code): bfs_start_ns/bfs_end_ns bracketing every bfs_path() call site, stage_start_ns/stage_end_ns and commit_start_ns/commit_end_ns in execute_repair(), all copied into the logged event without changing repair_start_ns/repair_end_ns's pre-existing meaning -- live-verified against the multi-OVS testbed, confirming an internally consistent BFS/stage/commit breakdown on a real fault injection), all runnable without Mininet/OVS; live-network flapping-link, startup-already-down, and multi-OVS-deployment scripts and their raw data and event logs across every pre-fix and post-fix run, retained side by side; the network-level Stage 3 raw data (five autonomous link-failure repetitions, plus scripted non-autonomous switch/controller-failure conditions) and reports; and the figure-generation script and resulting figures used in the manuscript.