Skip to content

Enforce machine-policy cleanup of unavailable targets

Placeholder ppxd requested to merge feat/machine-cleanup-enforcement into main

Summary

First of the Machine Policy enforcement series: the policy's "Clean up — delete unavailable deployment targets after N" setting was stored and shown in the UI but never consumed (DeleteMachinesBehavior / DeleteMachinesAfterSeconds had zero runtime effect). This wires it.

Safe-by-default — a deliberate double opt-in for a destructive operation:

  • Per policy: DeleteMachinesBehavior must be DeleteUnavailableMachines (default DoNotDelete → never eligible) AND the target must have been continuously unavailable for the configured grace period.
  • Globally (Rule 11 three-mode): SQUID_MACHINE_CLEANUP_ENFORCEMENT — default warn = dry-run (logs what would be deleted, deletes nothing), off skips, strict actually deletes. So the default build behaviour is unchanged (no auto-deletion); a removal requires opting in twice.

To measure continuous downtime, Machine gains a nullable UnavailableSince — stamped on first entry to Unavailable, preserved while it stays unavailable, cleared on recovery (additive migration; pre-existing rows are NULL and never eligible until their next unavailable transition). Eligible machines are removed through the same IMachineService.DeleteMachinesAsync path the operator-facing delete uses (Halibut trust reconfigured identically). A daily recurring job dispatches the sweep via mediator (Rule 14/16).

Why non-breaking

  • New nullable column (additive DbUp migration); pre-existing machines NULL.
  • New files + additive UnavailableSince tracking in the health-record path; no public surface removed/changed.
  • Default per-policy DoNotDelete + default global warn (dry-run) ⇒ zero machines deleted by default.
  • No enum/wire/contract changes; no Octopus wording.

Test plan

  • Unit — eligibility matrix (MachineCleanupEvaluator: DoNotDelete, non-unavailable, null go-bad instant, within/at/past grace, non-positive grace), env-var pin + three-mode resolution, UnavailableSince transition function. (29 tests)
  • Integration (real Postgres) — off/warn/strict end-to-end, DoNotDelete/within-grace/null-instant/healthy keep the machine, strict deletes via the real path, UnavailableSince DB round-trip. (8 tests)
  • Full unit sweep 5718 passed; full integration 293 passed (2 pre-existing Redis-env failures unrelated, green in CI); full solution build 0 errors.

Merge request reports