Skip to content

Phase 12.J.E.6 — auto-rollback on Phase B failure (E7.u1)

Placeholder ppxd requested to merge phase12.J.E.6-rollback-on-failure into main

Summary

🛡️ Ship-blocking production gap closed. Pre-this-PR, if a new tentacle binary's OnStart threw post-swap (DI graph crash, broken config, etc.), the operator's machine was left in a broken state with the bad binary in INSTALL_DIR + service Stopped + last-upgrade.json reporting generic 'FAILED'. Manual SSH-and-restore was required across every machine.

This PR adds an Invoke-Rollback path: detect Start-Service failure → archive broken binary at .failed → restore .bak → restart old binary → emit ROLLED_BACK status. The operator sees ROLLED_BACK in the UI and the agent auto-recovers to its previous version.

Production changes (upgrade-windows-tentacle.ps1)

  • New Invoke-Rollback function (~50 lines): Stop new svc → archive broken to .failed → Move .bak → INSTALL_DIR → Start old → WaitForStatus(Running) → write status. Three status enum exits: ROLLED_BACK (clean), ROLLBACK_NEEDED (no .bak), ROLLBACK_CRITICAL_FAILED (old won't start either).
  • Start-Service wrapped in try/catch: adds WaitForStatus('Running', '00:00:30') so OnStart exception arrives synchronously (without it, the catch can't see the asynchronous SCM failure). On exception → Invoke-Rollback with exit 8.
  • New documented exit code 8: "Start-Service post-swap failed → rollback fired"

Test infrastructure

  • TestUpgradeService.CrashOnStartMarkerFileName sentinel (same binary, configuration via filesystem state): if crash-on-start.marker exists in exeDir, OnStart throws → SCM rejects start → rollback fires. Clean teardown via existing fixture — no separate crashing-binary project to maintain.
  • UpgradeLifecycleContext.BuildV2BundleZip(crashOnStart=false): opt-in parameter that includes the sentinel in the bundle.

E2E test (1 new)

E7u1_NewBinaryOnStartCrashes_TriggersAutoRollbackToV1:

  • Stage v1 service → build v2 bundle WITH crash sentinel → run upgrade
  • Assertions:
    • Exit code 8
    • last-upgrade.json status = ROLLED_BACK with detail naming Start-Service failure
    • Marker file = 1.0.0 (proves v1 is running again — only way is v1 successfully started post-rollback)
    • .failed directory exists (broken v2 binary preserved for post-mortem)

Unit test (1 new — 7 structural pins)

RenderInnerScript_RollbackContract_PinnedStructurally:

  1. Invoke-Rollback function exists
  2. Start-Service wrapped in try/catch calling Invoke-Rollback
  3. ROLLED_BACK status emit
  4. ROLLBACK_CRITICAL_FAILED enum
  5. Exit code 8
  6. .failed archive path for post-mortem
  7. WaitForStatus('Running') for synchronous OnStart-exception capture

Each pin has an actionable customMessage explaining the regression scenario + fix pattern.

Local verification

  • 227/227 unit tests pass
  • 85/85 cross-platform E2E tests pass

Deferred to J.E.7+

  • E6.u1 (Phase B Move-Item failure) — requires reliable filesystem-level fault injection
  • Healthcheck-fatal env var (currently warning + proceed; opt-in rollback on healthcheck timeout)
  • E4.h (already-up-to-date) — requires version-stamped test binary

🤖 Generated with Claude Code

Merge request reports