Skip to content

Phase 12.J.E.7 — stale-lock recovery + healthcheck-fatal opt-in

Placeholder ppxd requested to merge phase12.J.E.7-rollback-edge-cases into main

Summary

Two narrow production hardenings, both real-world recovery paths:

1. Stale-lock recovery (E11.u2)

Pre-this-PR: a crashed dispatch (host reboot mid-upgrade, OOM kill, any reason a .ps1 died without removing its lock) left upgrade.lock with a dead PID. Every subsequent upgrade dispatch on that machine failed with exit 13 forever until manual intervention. For a fleet operator, a single mid-upgrade crash permanently blocked all future upgrades on that host.

Now: .ps1 reads recorded PID, probes via Get-Process, takes one of three actions:

  • Live PID → exit 13 (real concurrent dispatch correctly rejected)
  • Dead PID → log warning + Remove-Item + proceed (auto-recovery)
  • Non-numeric content → treated as stale (corrupt lock file is safe to break)

2. Healthcheck-fatal opt-in

Previously: post-Start healthcheck timeout = "warning + proceed". For operators where /healthz IS the canonical liveness contract, leaving a Stopped+swapped service is worse than rollback — but flipping the default would break every deployment with a slow-starting agent.

Now: opt-in via SQUID_TARGET_WINDOWS_TENTACLE_HEALTHCHECK_FATAL=true. Strict mode → Invoke-Rollback (reuses J.E.6 infra). Default stays permissive.

E2E tests (2 new)

Test Scenario Mechanism
E11u2_StaleLockWithDeadPid_BrokenAndDispatchProceeds Crashed-dispatch recovery Spawn-and-die: cmd.exe /c exit 0 → WaitForExit → use the now-dead PID as the stale lock content. Realistic crashed-process simulation, not a fixed magic number.
HealthcheckFatalMode_TimeoutTriggersAutoRollback Strict mode timeout Render with healthcheckFatal=true + unreachable healthz URL → expect exit 9 + ROLLED_BACK + marker back at v1

Both tests reverse-assert: stale-lock recovery is invisible to operator final status (they see SUCCESS), and FATAL mode is genuinely opt-in (default still produces SUCCESS — proven by E1.h continuing to pass).

Unit tests (8 new)

  • HealthcheckFatalEnvVar_ConstantNamePinned (Rule 8)
  • ResolveHealthcheckFatal_NoEnvVar_ReturnsFalse (default permissive)
  • ResolveHealthcheckFatal_RecognisedValue_ParsesCorrectly (Theory, 12 cases)
  • ResolveHealthcheckFatal_UnrecognisedValue_FallsBackToFalseWithWarning (Theory, 5 cases — operator typos)
  • RenderInnerScript_HealthcheckFatalPlaceholder_SubstitutedAsPowerShellBoolean (pins $true literal — NOT 'true' string which would always be truthy)
  • RenderInnerScript_HealthcheckFatal_DefaultsToFalseLiteralInRender
  • RenderInnerScript_StaleLockBreak_PinnedStructurally (4 structural ops)

Plus drift detector updates (placeholder set extended to include HEALTHCHECK_FATAL).

Local verification

  • 249/249 unit tests pass (was 227 + 22 new)
  • 87/87 cross-platform E2E pass (was 85 + 2 new)

Deferred

  • E6.u1 (Phase B Move-Item failure) — fault injection unreliable; revisit if production hits this
  • E4.h (already-up-to-date) — needs version-stamped binary; J.E.8

🤖 Generated with Claude Code

Merge request reports