Default the SelfCert guard to Warn and enforce it at startup
Summary
1.9.5's SelfCert guard broke a working production on upgrade: the deployment still carried the committed certificate, so every deploy and machine health check failed with an Autofac resolution exception (...HalibutClientFactory → λ:Halibut.HalibutRuntime → Refusing to use Halibut server identity FAF04764...), while the web UI stayed up.
Two defects compounded:
-
The Strict default was the wrong call for brownfield. The master-key guard it borrowed the posture from rejects an empty key — configuration that never protected anything, so refusing it breaks nothing that worked. This guard rejects a working identity that the entire fleet's trust is pinned to; refusing it by default turns an upgrade into an outage that only a fleet-wide re-trust (or the env var) can end. The default drops to
warn: un-rotated deployments keep working and get the full rotation warning at every startup;SQUID_SELFCERT_ENFORCEMENT=strictremains the opt-in for rotated fleets. -
Enforcement lived in the wrong place. It ran only inside the lazily-resolved
HalibutRuntimefactory, and the sole startup consumer (PollingTrustDistributor.Start) resolves it on a background task whose catch-all logs and continues — so the strict rejection was swallowed into a startup log line and the server came up half-alive. Enforcement moves toSelfCertStartupCheck, anIStartablethat runs synchronously insideBuild(): strict now refuses to start with the guard's own message; the default warn lands once in the startup log where operators look.
The startup check is deliberately narrower than the factory: a missing or unloadable identity is skipped there (the factory keeps failing lazily with actionable loader context), so the check can never turn a server that used to start into one that does not.
Operational notes
-
Deployments already on 1.9.5 don't need to wait for this release: setting
SQUID_SELFCERT_ENFORCEMENT=warnon the API pod restores deploys immediately. - The warning is still a real vulnerability notice: the committed certificate's private key and password are public. Rotating it (replace
SelfCert:Base64/Password, then re-register or re-trust agents against the new thumbprint) remains the actual fix;strictis the posture to adopt once rotated.
Test plan
-
Unit: default resolves Warnwith the env unset; committed cert allowed under Off/Warn, rejected under Strict with thumbprint + escape hatch named -
Wiring: strict + committed identity fails container build with the thumbprint in the message (fail-fast, no half-alive server); default + committed identity builds, resolves the runtime, and warns exactly once; missing identity still fails lazily naming SelfCert:Base64; corrupt Base64 does not fail the build even under strict; a deployment-specific identity builds and resolves under strict -
Mutation (each fails tests): default back to Strict (2), startup check unregistered (2), startup check swallowing the strict throw (1), EnsureConfiguredremoved from the factory (1) -
Hermetic under SQUID_SELFCERT_ENFORCEMENTunset/off/warn/strict— 19/19 in all -
Full unit suite 6387/6387; Calamari suite 593/593 -
Verified no other IStartableresolvesHalibutRuntimesynchronously; the other Strict-default guards (master key, space scope, user roles) all predate 1.9.5 and reject broken-only configuration