Skip to content

Defer tentacle upgrade when a deployment script is in flight on the machine

Placeholder ppxd requested to merge fix/p3-upgrade-defers-to-active-deploy into main

Summary

  • A tentacle upgrade restarts the agent process. If a deployment currently has a script in flight on that agent (recorded before the StartScript RPC, per the in-flight checkpoint), the restart kills the running script. The upgrade now consults the deployment checkpoint's in-flight state and defers (returns Failed with a retry-later detail) when the machine is busy, instead of dispatching.
  • Asymmetric by design: only the upgrade — rare, operator-triggered, already holding the per-machine upgrade lock — checks for an active deployment. The deployment path is unchanged: no new lock, no self-contention on parallel same-machine StartWithPrevious steps, no new dependency on the deploy hot path. (The earlier machine-exclusive-mutex approach was abandoned precisely because a symmetric lock over-serialized coexisting deploys.)
  • New IInFlightScriptStore.IsMachineBusyAsync(machineId) scans only checkpoints that still carry in-flight entries — a checkpoint exists only for an active/paused deployment and is deleted on success — so the scan set stays small. Backed by a pure InFlightScriptMap.ContainsMachine helper.

Placement & residual window

The check sits inside the existing per-machine upgrade Redis lock, immediately before dispatch and before the H4 metadata write — so a deferred upgrade leaves no metadata stragglers, and the cheap up-to-date / downgrade short-circuits still run first (a no-op upgrade never scans). The TOCTOU window between "machine is free" and "agent restarts" is minimal and accepted: a deploy that starts a script in that sub-second window is itself resilient — a killed in-flight script is classified transient and pauses/re-attaches on resume (#434), rather than failing the deployment.

Failed (not a new enum value) matches the established H4 contention convention for "couldn't dispatch right now, retry later" — non-breaking.

Test plan

  • Unit — InFlightScriptMap.ContainsMachine: matches any slot for the machine regardless of step/action; other-machine-only → false; empty/malformed/legacy JSON → false (never falsely blocks).
  • Unit — MachineUpgradeService: defers without dispatching when the store reports busy (asserts the strategy is never invoked + the operator-facing detail); proceeds to dispatch when free.
  • Integration (real Postgres) — InFlightScriptStore.IsMachineBusyAsync: in-flight script → busy (unrelated machine not); cleared slot → free; scan spans tasks (two tasks' in-flight machines both reported); untouched machine → free.
  • Full unit suite green (6035) + InFlightScriptStore integration suite green (13).

Merge request reports