Skip to content

Default the SelfCert guard to Warn and enforce it at startup

Placeholder ppxd requested to merge fix/selfcert-default-warn into main

Summary

1.9.5's SelfCert guard broke a working production on upgrade: the deployment still carried the committed certificate, so every deploy and machine health check failed with an Autofac resolution exception (...HalibutClientFactory → λ:Halibut.HalibutRuntime → Refusing to use Halibut server identity FAF04764...), while the web UI stayed up.

Two defects compounded:

  • The Strict default was the wrong call for brownfield. The master-key guard it borrowed the posture from rejects an empty key — configuration that never protected anything, so refusing it breaks nothing that worked. This guard rejects a working identity that the entire fleet's trust is pinned to; refusing it by default turns an upgrade into an outage that only a fleet-wide re-trust (or the env var) can end. The default drops to warn: un-rotated deployments keep working and get the full rotation warning at every startup; SQUID_SELFCERT_ENFORCEMENT=strict remains the opt-in for rotated fleets.
  • Enforcement lived in the wrong place. It ran only inside the lazily-resolved HalibutRuntime factory, and the sole startup consumer (PollingTrustDistributor.Start) resolves it on a background task whose catch-all logs and continues — so the strict rejection was swallowed into a startup log line and the server came up half-alive. Enforcement moves to SelfCertStartupCheck, an IStartable that runs synchronously inside Build(): strict now refuses to start with the guard's own message; the default warn lands once in the startup log where operators look.

The startup check is deliberately narrower than the factory: a missing or unloadable identity is skipped there (the factory keeps failing lazily with actionable loader context), so the check can never turn a server that used to start into one that does not.

Operational notes

  • Deployments already on 1.9.5 don't need to wait for this release: setting SQUID_SELFCERT_ENFORCEMENT=warn on the API pod restores deploys immediately.
  • The warning is still a real vulnerability notice: the committed certificate's private key and password are public. Rotating it (replace SelfCert:Base64/Password, then re-register or re-trust agents against the new thumbprint) remains the actual fix; strict is the posture to adopt once rotated.

Test plan

  • Unit: default resolves Warn with the env unset; committed cert allowed under Off/Warn, rejected under Strict with thumbprint + escape hatch named
  • Wiring: strict + committed identity fails container build with the thumbprint in the message (fail-fast, no half-alive server); default + committed identity builds, resolves the runtime, and warns exactly once; missing identity still fails lazily naming SelfCert:Base64; corrupt Base64 does not fail the build even under strict; a deployment-specific identity builds and resolves under strict
  • Mutation (each fails tests): default back to Strict (2), startup check unregistered (2), startup check swallowing the strict throw (1), EnsureConfigured removed from the factory (1)
  • Hermetic under SQUID_SELFCERT_ENFORCEMENT unset/off/warn/strict — 19/19 in all
  • Full unit suite 6387/6387; Calamari suite 593/593
  • Verified no other IStartable resolves HalibutRuntime synchronously; the other Strict-default guards (master key, space scope, user roles) all predate 1.9.5 and reject broken-only configuration

Merge request reports

Loading