Skip to content

Resume in-flight scripts by ticket instead of re-dispatching

Placeholder ppxd requested to merge feat/deployment-resume-by-ticket into main

Summary

  • Closes the double-execution window when a server crash interrupts a deployment mid-script: on resume, the Halibut strategy re-attaches to the still-running agent script via its ScriptTicket instead of launching a duplicate. Matters most for non-idempotent installs on Halibut-backed targets (KubernetesAgent, Tentacle polling/listening).
  • No schema change — reuses the previously-unused DeploymentExecutionCheckpoint.InFlightScriptsJson column ({machineId: ticket}). The strategy records the ticket right after StartScript and clears it once observed; IInFlightScriptStore serialises the per-task read-modify-write with an in-process lock (one Hangfire worker owns a task).
  • Non-breaking by construction — re-attach happens only on a clear "agent still holds it" signal; the sole fall-back to a fresh dispatch is the Tentacle's Complete + UnknownResult (-1) "unknown ticket" reply, which is exactly today's behaviour. The store dependency is optional (null → no-op), so non-Halibut strategies and existing constructions are unaffected.
  • Batch-boundary checkpoint saves no longer overwrite the ledger (clobber-fix), and the checkpoint row is created up-front (EnsureExistsAsync, insert-only) so first-batch tickets are durable; a fresh -1 row re-runs batch 0 exactly as before.

Test plan

  • Unit — InFlightScriptMap (11): add/remove/lookup round-trip, overwrite, blank/malformed JSON treated as empty
  • Unit — AgentHasUsableScript decision matrix (7): Running/Pending/Complete-real → re-attach; Pending re-attaches to avoid the queued-script double-exec; Complete + UnknownResult / null → fresh dispatch
  • Integration vs Postgres — InFlightScriptStore (6): record/clear round-trip, machines independent, batch save does not clobber in-flight, idempotent EnsureExists, no-row fail-safe, concurrent record under the per-task lock
  • Full unit suite green (5650), existing integration checkpoint tests green (14), full solution builds clean
  • Follow-up: full crash → resume → re-attach E2E (real Halibut agent + simulated server restart)

Implements roadmap item #1 (closed) from the post-1.8.3 deployment-architecture review.

Merge request reports