Skip to content

Pause (resumable) on a transient infra failure instead of failing

Placeholder ppxd requested to merge fix/p1b-transient-observe-pause into main

Summary

  • A Halibut RPC drop that outlives the library's own retries, or an agent that goes unreachable mid-script, used to fail the deployment terminally AND — via the strategy's finally that cleared the in-flight pointer on any exit — destroy the reattach pointer, so a resume re-dispatched a duplicate of the still-running script (the P1 idempotency gap, second half).
  • Classify a transient infrastructure failure (base/transport HalibutClientException, AgentUnreachableException, CircuitOpenException, or an AggregateException of only those) as a PAUSE: the task transitions to Paused with its checkpoint AND in-flight pointer preserved, and a resume re-attaches to the still-running script. Reuses the #428 pause/resume machinery.
  • Unconditional (no env-var opt-out): failing fast on a transient blip would only discard already-completed progress and risk a duplicate run — there is no legitimate use case. (The wall-clock-timeout toggle SQUID_DEPLOYMENT_TIMEOUT_RESUMABLE from #428 is separate and unaffected.)
  • The transient predicate (TransientFailureClassifier, the single source of truth) excludes the permanent Halibut protocol/invocation subtypes (ServiceInvocation / NoMatchingServiceOrMethod / MethodNotFound / ServiceNotFound / AmbiguousMethodMatch) — classifying those as transient would pause-loop forever.
  • A transient propagates out of ExecuteStepsPhase for both required and non-required steps (non-required was previously swallowed into FailureEncountered → OnFailureAsync → checkpoint+pointer deleted). The re-attach probe preserves the pointer on a transient probe failure (was: cleared it → duplicate-dispatch risk). The runner's transient catch is guarded by !timeoutCts && !registryCts && !ct so cancel/timeout win.

Test plan

  • Unit (6012/6012): TransientFailureClassifier matrix (permanent subtypes excluded; CircuitOpen/AgentUnreachable/base included; aggregate all/mixed/empty); DeploymentPipelineRunnerTransientPauseTests (transient→pause, genuine-failure→fail+rethrow, aggregate-of-transients, transient-during-user-cancel does NOT pause); TargetCatchClassifierTests transient cases
  • Unit — real ExecuteStepsPhase: DeploymentExecutionLoggingTests.TransientStrategyFailure_PropagatesOutOfPhase_RegardlessOfRequired (a throwing strategy proves a transient propagates for BOTH required + non-required, FailureEncountered stays false); contrast pinned by StepFailed_DoesNotLogCompleted (non-required GENUINE failure still swallowed)
  • Integration (real Postgres): HalibutResumeReattachTests — observe-throws-transient preserves the in-flight pointer; reattach-probe transient failure preserves the pointer (no fresh dispatch); full reattach + timeout-resume suites green
  • Full solution builds; non-breaking
  • Adversarial multi-agent review (concurrency / resume / misclassification / peers) — its 5 HIGH findings (non-required swallow, over-broad predicate, probe-clears-pointer, cancel-guard, CircuitOpen) all fixed + regression-tested

Merge request reports