Skip to content
  • Mars.P's avatar
    4f09b624
    Wire disk-pressure self-heal into the Tentacle host lifecycle (#435) · 4f09b624
    Mars.P authored
    * Wire disk-pressure self-heal into the Tentacle host lifecycle
    
    The self-heal scaffolding (SelfHealController, DiskPressureHealAction,
    DefaultWorkspaceCleanupPolicy, and their tests) was built but never
    scheduled — TentacleFlavor returned BackgroundTasks = [], so a disk-full
    agent failed every deployment via LocalScriptService's admission gate
    with no auto-reclaim of completed-script workspaces.
    
    Wire it:
    - WorkspaceProbe enumerates {tempRoot}/squid-tentacle-{ticketId} dirs and
      classifies each from its persisted ScriptState — Complete+0 => Succeeded,
      Complete+nonzero => Failed, started-not-complete => Active, no state =>
      Unknown — plus recursive size + last-write, so the cleanup policy can pick
      oldest-evictable workspaces while protecting a retention window.
    - SelfHealBackgroundTask adapts SelfHealController to the host's
      ITentacleBackgroundTask lifecycle (Start on launch, drain on shutdown);
      ForLocalWorkspaces builds the disk-pressure action over the temp root with
      the running script backend as the live-workspace veto.
    - TentacleFlavor now schedules it (BackgroundTasks = [selfHeal]).
    
    The running-script reporter veto means an in-flight deployment's workspace
    is never swept; only completed/orphaned workspaces are reclaimed, and only
    under disk pressure.
    
    Tests:
    - WorkspaceProbeTests: real temp dirs + real ScriptState files — status
      classification matrix, prefix filtering, size, missing-root
    - SelfHealBackgroundTaskTests: RunAsync starts the heal loop + drains on cancel
    - TentacleFlavorSelfHealWiringTests: the flavor schedules a SelfHeal task
      (the dead-code-is-now-live regression guard)
    
    * Harden disk self-heal: retention TTL floor, symlink-safe sizing, corrupt-state reclaim, backoff
    
    The wiring that takes the disk self-heal live (the 30s DiskPressureHealAction over
    {tempRoot}/squid-tentacle-*) is a second sweeper over the same directories the
    24h orphan sweep owns, and the probe/policy had several robustness gaps. Address
    them before the sweep runs in production:
    
    - Honour the operator's orphan-workspace TTL as a retention floor: under mild
      (non-critical) pressure the new sweep no longer evicts inside the post-mortem
      window the existing age-based orphan sweep promises; under critical pressure the
      floor is bypassed so reclaiming disk still wins.
    - Protect a just-created workspace via a 60s fresh-grace floor (even under
      critical pressure), closing the StartScript TOCTOU gap between CreateDirectory
      and the first state Save / running-script registration.
    - Classify a present-but-corrupt state file as Unknown instead of skipping the
      workspace forever — otherwise a dead-but-corrupt workspace is permanently
      un-reclaimable, exactly when disk pressure most needs the space.
    - Skip reparse points in the recursive size walk so a directory symlink (extracted
      package or runtime ln -s) cannot loop the enumeration or mis-attribute a
      symlink target's size to the workspace.
    - Warn once + back off exponentially when a sweep reclaims everything it is
      allowed to but the disk stays under pressure (non-workspace usage), instead of
      churning every 30s.
    - Expose the retention counts + low-disk trigger behind env vars with pinned
      const names (SelfHealOptions), mirroring the orphan-TTL env-var pattern.
    
    All floors default off in DefaultWorkspaceCleanupPolicy/DiskPressure so the
    parameterless constructors behave exactly as before; only the live
    ForLocalWorkspaces wiring injects them — non-breaking.
    
    Tests: SelfHealOptions env-var/default pins; corrupt-state, recursive nested-size
    and symlink-safety probe cases; non-critical-TTL + critical-bypass + fresh-grace
    policy cases; backoff growth/cap/reset; and a high-fidelity wired-chain E2E
    (real probe + policy + ScriptStateStore + Directory.Delete) proving real reclaim
    while a Running workspace and the in-TTL window survive.
    
    * Pin SelfHealOptions bounds + parse branches, invariant-culture percentage parse
    
    Make MinKeepCount/MaxKeepCount public (mirroring LocalScriptService's orphan-TTL
    bounds) and extract the env-value parsing into internal testable helpers so the
    reject/fallback branches are directly pinned (Rule 9 branch coverage). Parse the
    free-percentage with invariant culture so an operator's '0.20' reads the same on a
    comma-decimal locale server.
    4f09b624
    Wire disk-pressure self-heal into the Tentacle host lifecycle (#435)
    Mars.P authored
    * Wire disk-pressure self-heal into the Tentacle host lifecycle
    
    The self-heal scaffolding (SelfHealController, DiskPressureHealAction,
    DefaultWorkspaceCleanupPolicy, and their tests) was built but never
    scheduled — TentacleFlavor returned BackgroundTasks = [], so a disk-full
    agent failed every deployment via LocalScriptService's admission gate
    with no auto-reclaim of completed-script workspaces.
    
    Wire it:
    - WorkspaceProbe enumerates {tempRoot}/squid-tentacle-{ticketId} dirs and
      classifies each from its persisted ScriptState — Complete+0 => Succeeded,
      Complete+nonzero => Failed, started-not-complete => Active, no state =>
      Unknown — plus recursive size + last-write, so the cleanup policy can pick
      oldest-evictable workspaces while protecting a retention window.
    - SelfHealBackgroundTask adapts SelfHealController to the host's
      ITentacleBackgroundTask lifecycle (Start on launch, drain on shutdown);
      ForLocalWorkspaces builds the disk-pressure action over the temp root with
      the running script backend as the live-workspace veto.
    - TentacleFlavor now schedules it (BackgroundTasks = [selfHeal]).
    
    The running-script reporter veto means an in-flight deployment's workspace
    is never swept; only completed/orphaned workspaces are reclaimed, and only
    under disk pressure.
    
    Tests:
    - WorkspaceProbeTests: real temp dirs + real ScriptState files — status
      classification matrix, prefix filtering, size, missing-root
    - SelfHealBackgroundTaskTests: RunAsync starts the heal loop + drains on cancel
    - TentacleFlavorSelfHealWiringTests: the flavor schedules a SelfHeal task
      (the dead-code-is-now-live regression guard)
    
    * Harden disk self-heal: retention TTL floor, symlink-safe sizing, corrupt-state reclaim, backoff
    
    The wiring that takes the disk self-heal live (the 30s DiskPressureHealAction over
    {tempRoot}/squid-tentacle-*) is a second sweeper over the same directories the
    24h orphan sweep owns, and the probe/policy had several robustness gaps. Address
    them before the sweep runs in production:
    
    - Honour the operator's orphan-workspace TTL as a retention floor: under mild
      (non-critical) pressure the new sweep no longer evicts inside the post-mortem
      window the existing age-based orphan sweep promises; under critical pressure the
      floor is bypassed so reclaiming disk still wins.
    - Protect a just-created workspace via a 60s fresh-grace floor (even under
      critical pressure), closing the StartScript TOCTOU gap between CreateDirectory
      and the first state Save / running-script registration.
    - Classify a present-but-corrupt state file as Unknown instead of skipping the
      workspace forever — otherwise a dead-but-corrupt workspace is permanently
      un-reclaimable, exactly when disk pressure most needs the space.
    - Skip reparse points in the recursive size walk so a directory symlink (extracted
      package or runtime ln -s) cannot loop the enumeration or mis-attribute a
      symlink target's size to the workspace.
    - Warn once + back off exponentially when a sweep reclaims everything it is
      allowed to but the disk stays under pressure (non-workspace usage), instead of
      churning every 30s.
    - Expose the retention counts + low-disk trigger behind env vars with pinned
      const names (SelfHealOptions), mirroring the orphan-TTL env-var pattern.
    
    All floors default off in DefaultWorkspaceCleanupPolicy/DiskPressure so the
    parameterless constructors behave exactly as before; only the live
    ForLocalWorkspaces wiring injects them — non-breaking.
    
    Tests: SelfHealOptions env-var/default pins; corrupt-state, recursive nested-size
    and symlink-safety probe cases; non-critical-TTL + critical-bypass + fresh-grace
    policy cases; backoff growth/cap/reset; and a high-fidelity wired-chain E2E
    (real probe + policy + ScriptStateStore + Directory.Delete) proving real reclaim
    while a Running workspace and the in-TTL window survive.
    
    * Pin SelfHealOptions bounds + parse branches, invariant-culture percentage parse
    
    Make MinKeepCount/MaxKeepCount public (mirroring LocalScriptService's orphan-TTL
    bounds) and extract the env-value parsing into internal testable helpers so the
    reject/fallback branches are directly pinned (Rule 9 branch coverage). Parse the
    free-percentage with invariant culture so an operator's '0.20' reads the same on a
    comma-decimal locale server.
Loading