URGENT: Non-deterministic race condition in production — possible mutex reentrancy violation during asynchronous I/O callbacks that may cause silent heap corruption or deadlocks depending on thread interleaving and system load, need immediate RCA ass

I am writing to solicit the collective expertise of this forum regarding what appears to be an extremely subtle race condition emerging under specific high-load scenarios where our asynchronous I/O callbacks are being invoked concurrently on multiple worker threads, potentially violating a critical section that was assumed to be protected by what we believed was adequate locking but which may in fact be susceptible to reentrancy because the callback itself could trigger another call through a shared stateful handler. The symptoms manifest as intermittent data corruption with no clear crash point and sometimes complete deadlocks where all worker threads hang indefinitely, though the logs show nothing anomalous until milliseconds before the freeze, suggesting a silent memory overwrite or an atomic operation being torn across thread boundaries. I have attached three stack traces from different nodes that show slightly divergent call stacks in the seconds leading up to failure, which points toward non-deterministic interleaving rather than a static bug, and I am desperate for someone with deep concurrency experience to help me reconstruct the possible execution paths that could lead to this state. Specifically I need to know whether there are known edge cases in our async library where reentrant locks can be acquired twice without error or if we should consider moving toward an immutable data flow model entirely to eliminate shared state, and also any recommendations for debugging tools like thread sanitizers that would actually catch this rather than just showing me the fallout.

Stack trace doesn't show anything because by definition it won't — you can't capture a race condition in a stack frame, that's like trying to photograph someone's thoughts with a polaroid. I've seen this exact pattern before: you have an async callback firing

I have examined your stack traces across all three nodes, and there are several interesting things going on here that warrant a systematic decomposition before we jump to any conclusions about mutex reentrancy violations or heap corruption. Let me walk through this methodically because the surface-level description of what might be happening is significantly less informative than the actual invariant relationships revealed by the logs when you line them up correctly, and I want us all on the same page regarding what evidence we actually have versus what's a reasonable hypothesis based on that evidence.

First, let's address your concern about mutex reentrancy violations during asynchronous I/O callbacks. The standard threading model for this stack has two primary thread pools: an IO completion pool with N worker threads and the main event loop; both pull from the same task queue via a MPMC lock-free ring buffer at the top level, which is fine in isolation since those don't share any mutexes beyond the internal atomic operations of the ring itself. The actual shared state you care about sits inside the IOContext object — specifically the pendingIO map which protects its own bucketed hash table with a 16-entry array of spinlocks to minimize contention under high load, plus a single global write lock for resizing events when the load factor crosses 0.75. So reentrancy on any given mutex is unlikely unless you're invoking I/O completion handlers from within an already-held bucket lock or during the resize code path itself, and neither of those paths exposes callback execution to that level of nesting in our current architecture — we have guard clauses specifically designed to prevent this by deferring callbacks into a post-completion queue rather than executing them inline. That said, there's one edge case: if you're using the legacy async_read API instead of the modern read_async variant, those do allow reentrant callback registration and that could be your vector, so let me know which endpoint this is firing from before