Skip to content

Serialize InFlightScriptStore reads on the per-task lock stripe

Placeholder ppxd requested to merge fix/inflight-script-store-read-concurrency into main

Summary

  • InFlightScriptStore.TryGetTicketAsync (the crash-resume re-attach probe) queried the shared scoped IRepository/DbContext without taking the per-serverTaskId lock stripe that the writes (RecordDispatchedAsync/ClearAsync) already use.
  • A parallel batch runs every target's re-attach probe on the one DbContext the Hangfire worker owns for that task. Two concurrent probes — or a probe racing a record/clear — drove two operations onto the same context, and EF threw A second operation was started on this context instance. Intermittent crash in multi-target deployments using crash-resume re-attach.
  • Fix: gate the read with the same stripe as the writes, so all of a task's DbContext access serialises onto one writer.

Why this surfaced now

The KubernetesResumeCheckpointE2ETests/TentacleMixedModeE2ETests.MixedMode_SingleDeployment_BothTargetsExecute E2E (two targets in parallel) hits this reattach path and exposed the race intermittently — it's a narrow timing window, so most CI runs win it. Pre-existing on main; not introduced by any recent change.

Non-breaking + generic

  • Same method signature and return; the read just serialises on the existing stripe mechanism. No API/behavior change for callers — only correctness under concurrency.
  • The write path (MutateAsync) is untouched; the fix mirrors its existing inline gate pattern.
  • The stripe is bounded (64, no per-task growth) and a task always maps to the same stripe, so a task's reads + writes fully serialise; distinct tasks colliding on a stripe serialise harmlessly.

Test plan

  • New integration regression InFlightScriptStoreTests.ConcurrentReadsAndWrites_OnOneSharedContext_DoNotThrow — resolves one store (one DbContext) and fires concurrent reads+writes. Verified red→green: without the fix it fails with the exact production error (A second operation was started on this context instance); with the fix it passes. (The other tests here use a fresh Run scope per call and so never share a context — this one deliberately doesn't.)
  • Checkpoint + resume-reattach integration sweep: 11/11. InFlightScript unit: 11/11.
  • dotnet build — 0 errors.

Merge request reports