Skip to content

Fix Windows Tentacle deploy crashing on AV-locked Halibut temp files

Summary

Real production failure during IIS deploy on Windows Tentacle:

``` IOException: The process cannot access the file 'C:\Windows\TEMP\{guid}_1' because it is being used by another process. at Halibut.Transport.Protocol.TemporaryFileStream.SaveToAsync(string filePath, ...) at Squid.Tentacle.ScriptExecution.LocalScriptService.WriteAdditionalFiles(...) ```

The locked file is Halibut's INTERNAL backing temp where the inbound DataStream was buffered. Right after Halibut closes its write handle, Windows Defender's real-time protection opens the file exclusively to scan it. If `WriteAdditionalFiles` tries to read the source during that scan window (typically <300ms, but can spike to 1-2s on slow disks / older Defender engines), `File.Open` throws and the entire StartScript RPC crashes — turning a transient AV-scan timing issue into a hard deploy failure.

Fix

Wrap the `SaveToAsync` call in a retry-with-backoff loop. 5 attempts × 100/200/400/800/1000ms exponential backoff (~2.5s total). IOException-only retry; non-IO exceptions propagate immediately. Cancellation token observed before each attempt AND during inter-attempt delay so operator-initiated CancelScript unwinds in <1s.

After all attempts fail, throws a wrapper `IOException` naming the file + attempt count + env-var hint; `InnerException` preserves the original raw exception so root-cause stays visible.

Refactored into two methods for testability:

  • `SaveDataStreamWithRetry(ScriptFile, tempPath, ct)` — production call site
  • `RetryOnTransientIO(Action operation, contextName, ct)` — generic, unit-testable

Air-gapped escape hatch (Rule 8): `SQUID_TENTACLE_SAVE_DATASTREAM_MAX_ATTEMPTS` env var raises the budget for operators with aggressive AV.

Test plan

  • `dotnet build Squid.sln` → 0 errors
  • `dotnet test --filter LocalScriptServiceRetryTests` → 16/16 pass (15s)
  • Defect-detection note: the test suite caught a real bug in my initial implementation — `catch (IOException ex) when (attempt < maxAttempts)` skipped the catch on the FINAL attempt, propagating the raw IOException instead of the wrapper. Fixed before commit.
  • CI: full pipeline
  • Manual smoke (post-merge): retry IIS deploy on the Windows Tentacle that hit the original error; should succeed (or fail with the new wrapper message naming the env-var escape hatch)

Local Tentacle Tests sweep shows 37 unrelated environmental failures (`DiskSpaceChecker` requires 10% free on the temp drive; my dev mac is below threshold). These are unrelated to this change and won't reproduce on CI runners.

Backward compatibility

  • No public API removed; `SaveDataStreamWithRetry` is internal and only called by `WriteAdditionalFiles`
  • Behaviour is strictly more lenient — calls that previously failed with raw `IOException` now either succeed after retry, or fail with a richer wrapper that surfaces remediation steps
  • Cancellation semantics preserved (token observed at every step)

Coverage matrix (LocalScriptServiceRetryTests, 16 cases)

Case What it pins
`SuccessOnFirstAttempt_NoRetry` No retry overhead when operation succeeds first
`TransientIOException_RetriesUntilSuccess` (×4) Recovers from 1, 2, 3, 4 transient failures
`AllAttemptsFail_ThrowsWrappedIOExceptionWithAttemptCount` Wrapper message includes filename + attempt count + env var; InnerException preserves original
`NonIOException_ThrowsImmediatelyWithoutRetry` Non-IO exceptions aren't masked by retry sleep
`CancellationBeforeFirstAttempt_ThrowsImmediately` Pre-cancelled token short-circuits
`CancellationDuringBackoff_AbortsRetryLoop` Mid-retry cancel unwinds promptly
`SaveDataStreamMaxAttemptsEnvVar_LiteralPinned` Env var name pinned (Rule 8)
`EnvVarRaisesAttemptCount` Operator override works
`EnvVarInvalidValue_FallsBackToDefault` (×4) Garbage values don't crash, fall back gracefully
`AppliesBackoff_BetweenAttempts` Real sleep happens; timing budget exists

Merge request reports

Loading