-
Mars.P authored
* Pause (resumable) on a transient infra failure instead of failing A Halibut RPC drop that outlives the library's own retries, or an agent that goes unreachable mid-script, used to fail the deployment terminally AND — via the strategy's finally that cleared the in-flight pointer on any exit — destroy the reattach pointer, so a resume re-dispatched a duplicate of the still-running script. Classify a transient infrastructure failure (HalibutClientException / AgentUnreachableException, or an AggregateException of only those) as a PAUSE: the task transitions to Paused with its checkpoint AND in-flight pointer preserved, and a resume re-attaches to the still-running script. Reuses the #428 pause/resume machinery; env-gated by SQUID_DEPLOYMENT_TRANSIENT_RESUMABLE (default true) so operators can opt back into fail-fast. - HalibutMachineExecutionStrategy: clear the in-flight pointer ONLY on a definitive observation (return), preserve it on a throw. - TargetCatchClassifier: a transient infra failure leaves the target in-flight (not terminal) and does NOT fail-fast healthy peers — they finish, then the deployment pauses. Owns the shared IsTransientInfraFailure. - DeploymentPipelineRunner: a transient failure routes to OnTransientPauseAsync (Paused + checkpoint preserved), not OnFailureAsync; does not rethrow. - DeploymentCompletionHandler: OnTransientPauseAsync (mirrors OnTimedOutAsync). A genuine (non-transient) exception, and a transient drop racing a real cancel/timeout, still fail terminally. Tests: - Unit: TargetCatchClassifier transient cases + IsTransientInfraFailure matrix; DeploymentPipelineRunner transient->pause vs fail-fast vs genuine failure, aggregate-of-transients, env-var pin + parse matrix - Integration: HalibutResumeReattach observe-throws-transient preserves the in-flight pointer for resume * Harden transient-pause classification (adversarial review fixes) Adversarial review of the transient-pause change found five reachable paths that defeated the resumable-pause guarantee. Fixes: - Centralise the transient definition in TransientFailureClassifier (Squid.Core.Halibut.Resilience) so the pipeline runner, per-target classifier, per-action catch, and the re-attach probe all agree. - NARROW the predicate: a permanent Halibut protocol/invocation failure (ServiceInvocation / NoMatchingServiceOrMethod / MethodNotFound / ServiceNotFound / AmbiguousMethodMatch) is NOT transient — classifying it so would pause-loop forever. Base/transport HalibutClientException, AgentUnreachableException, and CircuitOpenException ARE transient. - Non-required step: a transient now propagates (rethrow before the per-target/terminal failure recording) regardless of step.IsRequired — previously it was swallowed into FailureEncountered -> OnFailureAsync, which deleted the checkpoint AND the preserved in-flight pointer, orphaning the still-running script. - Re-attach probe: a TRANSIENT probe failure on resume now preserves the pointer and propagates (was: type-blind catch cleared it and dispatched fresh -> duplicate-execution risk when the agent recovers). - Runner transient catch is guarded by !timeoutCts && !registryCts && !ct so a user-cancel / wall-clock timeout racing a transient RPC drop is not reclassified as a pause (cancel/timeout win). - CircuitOpenException is treated as transient (its own contract) so a tripped breaker on resume pauses + preserves the pointer instead of forcing a terminal Failed. Tests: - Unit: TransientFailureClassifier matrix (permanent subtypes excluded, CircuitOpen/AgentUnreachable/base included, aggregate all/mixed/empty); runner transient-during-user-cancel does NOT pause - Integration: reattach-probe transient failure preserves the pointer and does not dispatch fresh * Make transient-pause unconditional (drop env-var toggle) + real-phase tests Remove the SQUID_DEPLOYMENT_TRANSIENT_RESUMABLE escape hatch added earlier in this PR: pausing-resumable on a transient infra blip is the only correct behaviour (failing fast would discard already-completed progress and risk a duplicate run — no legitimate use case), so it needs no opt-out. The env var never shipped, so removing it is non-breaking. The wall-clock-timeout toggle (SQUID_DEPLOYMENT_TIMEOUT_RESUMABLE, shipped in 1.8.18) is unaffected. The runner's transient catch keeps its cancel/timeout guard (!timeoutCts && !registryCts && !ct) so cancel/timeout still win. Tests: - Drop the env-var pin / parse-matrix / fail-fast tests (no longer apply) - Add real-ExecuteStepsPhase propagation tests (DeploymentExecutionLoggingTests.TransientStrategyFailure_PropagatesOutOfPhase_RegardlessOfRequired): a throwing strategy on a step proves a transient PROPAGATES out of the phase for BOTH required AND non-required steps (closing the review's non-required-swallow gap end-to-end), and FailureEncountered stays false; the existing StepFailed_DoesNotLogCompleted pins the contrast that a non-required GENUINE failure is still swallowed * Exclude CircuitOpenException from transient (keep fail-fast-on-open-breaker) The K8s E2E HalibutCircuitBreakerE2ETests.BreakerForcedOpen_* asserts an open breaker fails the deployment fast (Failed) — a deliberate production-safety invariant. Classifying CircuitOpenException as transient (from the review's HIGH #5 suggestion) flipped that to Paused and broke it. Excluding it is the more correct design, by the same principle as the permanent-Halibut-subtype exclusion: the per-machine breaker only opens after 3+ consecutive failures — a SUSTAINED agent problem, not a one-off blip — and is raised BEFORE any script is dispatched (fail-fast), so there is no in-flight script to re-attach to. Pausing on it would loop on a dead agent. The narrow resume-with-open-breaker orphan #5 described is better addressed by probe-before-breaker (a separate focused change), not by this blunt reclassification. The other four review HIGH fixes (non-required propagation, narrowed predicate, reattach-probe preserve, cancel/timeout guard) are unchanged. Test flipped: CircuitOpen_IsNotTransient.
Mars.P authored* Pause (resumable) on a transient infra failure instead of failing A Halibut RPC drop that outlives the library's own retries, or an agent that goes unreachable mid-script, used to fail the deployment terminally AND — via the strategy's finally that cleared the in-flight pointer on any exit — destroy the reattach pointer, so a resume re-dispatched a duplicate of the still-running script. Classify a transient infrastructure failure (HalibutClientException / AgentUnreachableException, or an AggregateException of only those) as a PAUSE: the task transitions to Paused with its checkpoint AND in-flight pointer preserved, and a resume re-attaches to the still-running script. Reuses the #428 pause/resume machinery; env-gated by SQUID_DEPLOYMENT_TRANSIENT_RESUMABLE (default true) so operators can opt back into fail-fast. - HalibutMachineExecutionStrategy: clear the in-flight pointer ONLY on a definitive observation (return), preserve it on a throw. - TargetCatchClassifier: a transient infra failure leaves the target in-flight (not terminal) and does NOT fail-fast healthy peers — they finish, then the deployment pauses. Owns the shared IsTransientInfraFailure. - DeploymentPipelineRunner: a transient failure routes to OnTransientPauseAsync (Paused + checkpoint preserved), not OnFailureAsync; does not rethrow. - DeploymentCompletionHandler: OnTransientPauseAsync (mirrors OnTimedOutAsync). A genuine (non-transient) exception, and a transient drop racing a real cancel/timeout, still fail terminally. Tests: - Unit: TargetCatchClassifier transient cases + IsTransientInfraFailure matrix; DeploymentPipelineRunner transient->pause vs fail-fast vs genuine failure, aggregate-of-transients, env-var pin + parse matrix - Integration: HalibutResumeReattach observe-throws-transient preserves the in-flight pointer for resume * Harden transient-pause classification (adversarial review fixes) Adversarial review of the transient-pause change found five reachable paths that defeated the resumable-pause guarantee. Fixes: - Centralise the transient definition in TransientFailureClassifier (Squid.Core.Halibut.Resilience) so the pipeline runner, per-target classifier, per-action catch, and the re-attach probe all agree. - NARROW the predicate: a permanent Halibut protocol/invocation failure (ServiceInvocation / NoMatchingServiceOrMethod / MethodNotFound / ServiceNotFound / AmbiguousMethodMatch) is NOT transient — classifying it so would pause-loop forever. Base/transport HalibutClientException, AgentUnreachableException, and CircuitOpenException ARE transient. - Non-required step: a transient now propagates (rethrow before the per-target/terminal failure recording) regardless of step.IsRequired — previously it was swallowed into FailureEncountered -> OnFailureAsync, which deleted the checkpoint AND the preserved in-flight pointer, orphaning the still-running script. - Re-attach probe: a TRANSIENT probe failure on resume now preserves the pointer and propagates (was: type-blind catch cleared it and dispatched fresh -> duplicate-execution risk when the agent recovers). - Runner transient catch is guarded by !timeoutCts && !registryCts && !ct so a user-cancel / wall-clock timeout racing a transient RPC drop is not reclassified as a pause (cancel/timeout win). - CircuitOpenException is treated as transient (its own contract) so a tripped breaker on resume pauses + preserves the pointer instead of forcing a terminal Failed. Tests: - Unit: TransientFailureClassifier matrix (permanent subtypes excluded, CircuitOpen/AgentUnreachable/base included, aggregate all/mixed/empty); runner transient-during-user-cancel does NOT pause - Integration: reattach-probe transient failure preserves the pointer and does not dispatch fresh * Make transient-pause unconditional (drop env-var toggle) + real-phase tests Remove the SQUID_DEPLOYMENT_TRANSIENT_RESUMABLE escape hatch added earlier in this PR: pausing-resumable on a transient infra blip is the only correct behaviour (failing fast would discard already-completed progress and risk a duplicate run — no legitimate use case), so it needs no opt-out. The env var never shipped, so removing it is non-breaking. The wall-clock-timeout toggle (SQUID_DEPLOYMENT_TIMEOUT_RESUMABLE, shipped in 1.8.18) is unaffected. The runner's transient catch keeps its cancel/timeout guard (!timeoutCts && !registryCts && !ct) so cancel/timeout still win. Tests: - Drop the env-var pin / parse-matrix / fail-fast tests (no longer apply) - Add real-ExecuteStepsPhase propagation tests (DeploymentExecutionLoggingTests.TransientStrategyFailure_PropagatesOutOfPhase_RegardlessOfRequired): a throwing strategy on a step proves a transient PROPAGATES out of the phase for BOTH required AND non-required steps (closing the review's non-required-swallow gap end-to-end), and FailureEncountered stays false; the existing StepFailed_DoesNotLogCompleted pins the contrast that a non-required GENUINE failure is still swallowed * Exclude CircuitOpenException from transient (keep fail-fast-on-open-breaker) The K8s E2E HalibutCircuitBreakerE2ETests.BreakerForcedOpen_* asserts an open breaker fails the deployment fast (Failed) — a deliberate production-safety invariant. Classifying CircuitOpenException as transient (from the review's HIGH #5 suggestion) flipped that to Paused and broke it. Excluding it is the more correct design, by the same principle as the permanent-Halibut-subtype exclusion: the per-machine breaker only opens after 3+ consecutive failures — a SUSTAINED agent problem, not a one-off blip — and is raised BEFORE any script is dispatched (fail-fast), so there is no in-flight script to re-attach to. Pausing on it would loop on a dead agent. The narrow resume-with-open-breaker orphan #5 described is better addressed by probe-before-breaker (a separate focused change), not by this blunt reclassification. The other four review HIGH fixes (non-required propagation, narrowed predicate, reattach-probe preserve, cancel/timeout guard) are unchanged. Test flipped: CircuitOpen_IsNotTransient.
Loading