Skip to content

fix(production): Windows upgrade .ps1 inner script silent parse failure

Summary

🐛 Three production / test issues caught by Phase 12.J.E.3's high-fidelity E2E on first Windows runner pass:

  1. Production P0: {{INSTALL_METHODS}} mentioned by name inside a #-prefixed comment got rewritten by String.Replace alongside the real placeholder, splicing multi-line PowerShell into a comment line → inner script parse failure → silent upgrade fail.
  2. Production P0: Get-FileHash cmdlet auto-loader on the windows-latest runner image throws CommandNotFoundException even when Invoke-WebRequest (same Microsoft.PowerShell.Utility module) loads fine moments earlier — likely a partial module-cache state under $ErrorActionPreference = 'Stop' + Set-StrictMode -Version Latest.
  3. Test bug: J.E.3's full-bundle upgrade test passed its pre-built zip via LocalReleaseMirror.StageBinary, which auto-wraps content in a fresh zip — leading to the bundle becoming a single inner entry inside a wrapper zip instead of the expected multi-entry shape.

Pre-J.E.3, NO test in the suite ran the rendered .ps1 end-to-end:

  • ShaVerify E2E only exercised the FETCH path, not the local hash compute
  • WrapperE2E base64-decoded the inner but didn't execute it
  • TentacleUpgradeE2E only verified the wrapper exited 0 (its work was done before the inner ran)

Together, the three production-flow steps (placeholder substitution → SHA verify → extract+swap) had 100% silent-failure regression risk on Windows. PR #196's high-fidelity investment paid off immediately.

Operator impact (pre-fix)

Every Windows tentacle upgrade since the affected commits was silently failing:

  • Wrapper exit code stayed at 0 (its work was schedule-the-task; that worked)
  • Strategy mapped to MachineUpgradeStatus.Initiated
  • Operator UI showed "Initiated" forever
  • last-upgrade.json was never written (parse error / SHA cmdlet error → outer catch wrote FAILED with a different exit code, but the rapid-polling burst couldn't read the new agent version because Phase B's binary swap never happened)
  • After rapid-polling expires the server fell back to version-comparison and inferred "failed"

Fixes (3 commits)

1. upgrade-windows-tentacle.ps1 — placeholder-in-comment fix

  • Rewrote line 190's comment to refer to "the placeholder below" instead of {{INSTALL_METHODS}} verbatim
  • Added IMPORTANT note documenting the discipline + cross-referencing the new pin

2. upgrade-windows-tentacle.ps1 — Get-FileHash → direct .NET SHA256

  • Replaced (Get-FileHash -Path $archivePath -Algorithm SHA256).Hash.ToLower() with [System.Security.Cryptography.SHA256]::Create().ComputeHash([System.IO.File]::ReadAllBytes($archivePath)) (try/finally Dispose). Bypasses the auto-loader entirely + ~5x faster on ~10MB archives.
  • Comment uses indirect phrasing ("the cmdlet form" / "the SHA cmdlet") to avoid mentioning Get-FileHash verbatim — same string-pin discipline as fix #1 (closed)

3. LocalReleaseMirror — new StagePreBuiltArchive(byte[]) API

  • Test was using StageBinary(filename, content) which auto-wraps content into a fresh zip — fine for install-script E2E (single binary file) but wrong for upgrade-lifecycle E2E (full multi-entry bundle)
  • New API stages pre-built archive bytes the mirror serves verbatim. SHA256 companion still works (hashes the pre-built bytes; same per-URL cache so .zip and .sha256 stay in sync byte-for-byte)
  • E1.h test updated with wire-shape sanity assertion that catches the same bug class if a future setup regresses to StageBinary OR a future BuildV2 change drops the canonical exe placeholder

Pin against regression (3 unit tests)

  1. RenderInnerScript_PlaceholderTokens_AppearExactlyOnceInTemplate — every {{TOKEN}} in the template appears EXACTLY ONCE. Catches future comment-line mentions reintroducing fix #1 (closed)'s bug class. Bonus: broadens existing catch-all regex [A-Z_]+ → [A-Z0-9_]+ (was silently missing EXPECTED_SHA256).
  2. RenderInnerScript_ShaVerifyUsesDirectDotNetApi_NotGetFileHashCmdlet — positive pins on [System.Security.Cryptography.SHA256] + ComputeHash + [System.IO.File]::ReadAllBytes; negative pin asserts Get-FileHash does NOT appear.
  3. ShaVerify drift detector updated to pin direct .NET API instead of Get-FileHash.

Verification

Local (all platforms)

  • 111/111 unit tests including 8 RenderInnerScript pins
  • 81/81 cross-platform E2E

Windows runner (run id 25468374459) ✅

26/26 tests passed across 5 categories:

  • TentacleUpgradeLifecycleE2E (J.E.3): 6/6 ✅
    • E1h_FullLifecycle_HappyPath_WritesSuccessStatusAndSwapsBinary — 2m2s (real Phase A download + SHA verify + extract + Phase B Stop/swap/Start + healthcheck + last-upgrade.json writeback)
    • E1u1_DownloadVersionNotFound_ExitsTwoAndWritesFailedStatusWithDownloadDetail — 470ms
    • E12u1_Sha256Mismatch_ExitsSevenAndWritesFailedStatusWithChecksumDetail — 1s
    • E8h_LastUpgradeJson_AfterSuccess_RoundTripsViaCapabilitiesProbe — 2m6s
    • E8u1_CorruptLastUpgradeJson_ParseReturnsNullWithoutThrow — 6ms
    • UpgradeScript_PlaceholderSet_PinnedToProductionContract — 2ms
  • WindowsUpgradeWrapperE2E: 4/4 ✅
  • WindowsUpgradeServiceE2E: 3/3 ✅
  • WindowsUpgradeShaVerifyE2E: 7/7 ✅
  • TentacleUpgradeE2E: 3/3 ✅
  • WindowsServiceFixtureSmokeE2E: 2/2 ✅
  • StubSquidServerSmokeE2E: 1/1 ✅

Total run time: 8m46s; J.E.3 lifecycle tests dominated at ~4m (real .ps1 with real service + 30s healthcheck poll per test).

🤖 Generated with Claude Code

Merge request reports

Loading