Skip to content

Harden Halibut polling deployment

Placeholder ppxd requested to merge halibut-k8s-deployment-hardening into main

409 on name conflict, body-code va…idation, install-script probe, K8s runbook

Closes the root causes and guardrails for the 2026-04-18 production incident where a Tentacle registered successfully (HTTP 200) but polling loop-failed with opaque EOF errors because:

  1. server returned HTTP 200 + body{code:500} on duplicate-machine-name, which the client mistook for success;
  2. the generated install script embedded a polling URL that was externally unreachable (SLB health check mis-configured + node security group missing 100.64.0.0/10 ingress rule);
  3. no documentation / runbook existed to tell operators which network preconditions Halibut polling actually requires.

Code changes (non-breaking — MachineNameConflictException derives from InvalidOperationException so existing catch sites keep working):

• MachineNameConflictException with MachineName + SpaceId context • EnsureUniqueNameAsync throws the new exception instead of raw InvalidOperationException • GlobalExceptionFilter maps MachineNameConflictException → HTTP 409 • TentacleRegistrationClient now validates body code (not only HTTP status), plus enforces MachineId > 0 && ServerThumbprint != empty before declaring "Registration successful" • Optional HttpMessageHandlerFactory test seam on TentacleRegistrationClientOptions • New ITentacleCommsUrlProbe + TentacleCommsUrlProbe: TCP+TLS probe with actionable remediation hints baked into Detail • MachineScriptService.GenerateTentacleInstallScriptAsync runs the probe and attaches result to response; UI can now render a warning banner instead of silently handing out a broken script • GenerateTentacleInstallScriptData carries new TentacleCommsProbeInfo

Docs: • CLAUDE.md adds "Kubernetes Deployment — Exposing Halibut Polling" section with full yaml, SG rule, diagnostic commands, failure matrix • deploy/k8s-network-requirements.md runbook for operators (independent of developer-facing CLAUDE.md), covering all 5 debugging steps used in the 2026-04-18 incident

Tests (all green: UnitTests 4052, Tentacle.Tests 1002, Calamari 115, Watchdog 17 = 5186 total): • RegisterSsh + RegisterTentaclePolling duplicate-name → 409 regression tests • TentacleRegistrationClient: Http200+bodyCode500, Http409, Http200+ emptyData, Http200+goodData, 4xx-not-retried • TentacleCommsUrlProbe: empty URL, malformed URL, unreachable host (asserts remediation hint mentions 100.64.0.0/10), in-proc TLS server with matching + mismatching thumbprint • MachineInstallScriptService: polling-mode invokes probe, listening- mode skips, unreachable probe surfaces to response

Merge request reports