RE: Unexpected race condition on shared-memory cache invalidation between NUMA domains under high contention — seeking root cause analysis and mitigation strategies for L3 coherence traffic storms

Hello everyone, I've been debugging a performance regression that only manifests at scale (64+ physical cores across two NUMA nodes), where we observe an exponential increase in cache coherency bus traffic -- specifically RFO (Request For Ownership) storms during heavy concurrent write-write contention on shared hot fields. The system spends upwards of 40% of cycles stalled on cross-node invalidation responses, and our pprof CPU flame charts are dominated by __spin_unlock_irqsave and various internal spinlock primitives rather than actual application logic. We've ruled out L1/L2 locality issues through cachegrind analysis; the problem appears to be at the interconnect level -- specifically, when threads on Node 0 write a shared line that is cached in read-shared state across all cores of Node 1, every subsequent RFO from Node 0 forces an invalidation broadcast and synchronous acknowledgement from all remote sharers. As thread count increases linearly with contention frequency, this quadratic amplification of interconnect traffic creates a bandwidth saturation wall on the QPI/UPI link itself. We've already attempted two mitigation strategies: (a) padding hot fields to separate them into distinct cache lines -- this works partially but is brittle since it depends on line size and can actually worsen the problem by increasing working set footprint, and (b) read-copy-update for the shared state -- which improves read throughput at the cost of increased memory overhead and eventual consistency issues. I'm hoping someone here has seen similar behavior in a kernel module or high-performance networking context and can recommend either a hardware-level tuning parameter (perhaps QOS/bandwidth partitioning), an alternative data structure with naturally reduced sharing, or a cache-line-aware lock-free primitive designed for this exact NUMA topology. Any

Seen this pattern before on Ice Lake with heavy cross-socket cache sharing. The Snoop Filter can

hey i'm so sorry to interrupt but what does this mean? i have no idea what a race condition is or what numa domains are and i saw someone say l3 coherence traffic storms were at fault for their bug which sounds like gibberish in english. can somebody explain what's going on here very simply please?

Thanks. Good thread.