Fix Windows Tentacle deploy crashing on AV-locked Halibut temp files
Summary
Real production failure during IIS deploy on Windows Tentacle:
``` IOException: The process cannot access the file 'C:\Windows\TEMP\{guid}_1' because it is being used by another process. at Halibut.Transport.Protocol.TemporaryFileStream.SaveToAsync(string filePath, ...) at Squid.Tentacle.ScriptExecution.LocalScriptService.WriteAdditionalFiles(...) ```
The locked file is Halibut's INTERNAL backing temp where the inbound DataStream was buffered. Right after Halibut closes its write handle, Windows Defender's real-time protection opens the file exclusively to scan it. If `WriteAdditionalFiles` tries to read the source during that scan window (typically <300ms, but can spike to 1-2s on slow disks / older Defender engines), `File.Open` throws and the entire StartScript RPC crashes — turning a transient AV-scan timing issue into a hard deploy failure.
Fix
Wrap the `SaveToAsync` call in a retry-with-backoff loop. 5 attempts × 100/200/400/800/1000ms exponential backoff (~2.5s total). IOException-only retry; non-IO exceptions propagate immediately. Cancellation token observed before each attempt AND during inter-attempt delay so operator-initiated CancelScript unwinds in <1s.
After all attempts fail, throws a wrapper `IOException` naming the file + attempt count + env-var hint; `InnerException` preserves the original raw exception so root-cause stays visible.
Refactored into two methods for testability:
- `SaveDataStreamWithRetry(ScriptFile, tempPath, ct)` — production call site
- `RetryOnTransientIO(Action operation, contextName, ct)` — generic, unit-testable
Air-gapped escape hatch (Rule 8): `SQUID_TENTACLE_SAVE_DATASTREAM_MAX_ATTEMPTS` env var raises the budget for operators with aggressive AV.
Test plan
-
`dotnet build Squid.sln` → 0 errors -
`dotnet test --filter LocalScriptServiceRetryTests` → 16/16 pass (15s) -
Defect-detection note: the test suite caught a real bug in my initial implementation — `catch (IOException ex) when (attempt < maxAttempts)` skipped the catch on the FINAL attempt, propagating the raw IOException instead of the wrapper. Fixed before commit. -
CI: full pipeline -
Manual smoke (post-merge): retry IIS deploy on the Windows Tentacle that hit the original error; should succeed (or fail with the new wrapper message naming the env-var escape hatch)
Local Tentacle Tests sweep shows 37 unrelated environmental failures (`DiskSpaceChecker` requires 10% free on the temp drive; my dev mac is below threshold). These are unrelated to this change and won't reproduce on CI runners.
Backward compatibility
- No public API removed; `SaveDataStreamWithRetry` is internal and only called by `WriteAdditionalFiles`
- Behaviour is strictly more lenient — calls that previously failed with raw `IOException` now either succeed after retry, or fail with a richer wrapper that surfaces remediation steps
- Cancellation semantics preserved (token observed at every step)
Coverage matrix (LocalScriptServiceRetryTests, 16 cases)
| Case | What it pins |
|---|---|
| `SuccessOnFirstAttempt_NoRetry` | No retry overhead when operation succeeds first |
| `TransientIOException_RetriesUntilSuccess` (×4) | Recovers from 1, 2, 3, 4 transient failures |
| `AllAttemptsFail_ThrowsWrappedIOExceptionWithAttemptCount` | Wrapper message includes filename + attempt count + env var; InnerException preserves original |
| `NonIOException_ThrowsImmediatelyWithoutRetry` | Non-IO exceptions aren't masked by retry sleep |
| `CancellationBeforeFirstAttempt_ThrowsImmediately` | Pre-cancelled token short-circuits |
| `CancellationDuringBackoff_AbortsRetryLoop` | Mid-retry cancel unwinds promptly |
| `SaveDataStreamMaxAttemptsEnvVar_LiteralPinned` | Env var name pinned (Rule 8) |
| `EnvVarRaisesAttemptCount` | Operator override works |
| `EnvVarInvalidValue_FallsBackToDefault` (×4) | Garbage values don't crash, fall back gracefully |
| `AppliesBackoff_BetweenAttempts` | Real sleep happens; timing budget exists |