CRITICAL: Race condition in the asynchronous transaction reconciliation pipeline — urgent debugging assistance requested

I am writing this post with considerable urgency because I have identified what appears to be a non-deterministic concurrency bug in our production payment processing system, and the implications are severe if left unaddressed. The architecture involves an asynchronous task queue that processes incoming transactions through several stages: validation, fraud detection scoring (which calls an external microservice), ledger entry generation, and finally notification dispatch. Each stage operates as a separate worker process consuming from shared Redis-backed queues with optimistic concurrency control via etags for idempotency guarantees at each individual step. However — this is where the issue resides — there appears to be a race condition between the fraud detection scoring callback and the concurrent transaction cancellation endpoint, which can be triggered by rapid user interaction or automated bots attempting exploit patterns I am not yet confident in identifying fully. In my local reproduction environment using docker-compose with 12 workers across 3 Redis replicas, I have observed approximately 0.8% of transactions reporting a 'double-spent' error state where both the scoring worker and the cancellation endpoint write to the shared ledger entry simultaneously despite our etag checks, which implies that either our etags are not being invalidated correctly under specific failure modes or there is an edge case in how Redis WATCH/MULTI semantics interact with asynchronous callback handlers. I have examined this extensively and reproduced it consistently under high contention with a 2ms network delay injection between the worker and the database but cannot isolate why the optimistic lock fails specifically at stage 3 instead of uniformly across all stages. Possible vectors: (1) etags being cached by an intermediary proxy, (2) the cancellation endpoint using a different validation key than the scoring callback despite what the documentation claims, (3) write skew that our current isolation level does not prevent since we are running READ COMMITTED rather than SERIALIZABLE on this specific table. I have attached three log snippets showing overlapping timestamps at microsecond granularity