Comprehensive Comparative Analysis of Heterogeneous Memory Architectures with Consideration for Cache Coherency Protocols and NUMA Effects

The question of whether to employ a uniform memory access model versus a non-uniform memory access paradigm in large-scale HPC applications is not merely an engineering decision but one that touches upon fundamental tenets of parallel computing theory including cache hierarchy design, interconnect bandwidth limitations, page coloring strategies for TLB optimization, and the trade-offs between write-through and write-back coherency protocols. If we consider a 128-node cluster with NUMA domains spanning across multiple sockets per node, each socket having its own local L3 cache of approximately 32MB shared among 16 cores at an average latency of 40 cycles, the cost of remote memory access through QPI or Infinity Fabric links can balloon to over 150 nanoseconds compared to roughly 80 nanoseconds for local accesses — this represents a nearly twofold degradation in performance that is completely invisible to naive programmers who write code without NUMA awareness. We must also consider how page migration strategies like first-touch allocation versus interleave policies impact cache locality, and whether software prefetching can mask the latency penalty of remote fetches through aggressive asynchronous load generation. Furthermore, we should discuss the implications for memory consistency models where release consistency allows reordering that could be catastrophic in lockless data structures while sequential consistency would impose prohibitive overhead due to strict serialization requirements across all cores. There are also edge cases involving huge page allocations (2MB and 1GB pages) which reduce TLB miss rates but can lead to significant fragmentation over long uptimes, creating a tension between memory performance gains and operational stability that needs careful balancing in production environments with mixed workloads running simultaneously on the same hardware substrate.

this is such a deep dive thread — i love it. honestly i'm still wrapping my head around why numa has become the dominant architecture when uniform memory access just works fine for most workloads. but i guess at this scale you can't escape it.

me and my team ran some benchmarks last week on our server cluster trying to optimize cache coherency overhead in a heterogeneous setup, and the results were... humbling. we saw roughly 12% performance loss just from snooping traffic when we pushed beyond what the interconnect could handle efficiently.

one quick question for whoever wrote this thread — did you consider how non-volatile memory fits into the equation here? if we start moving read-heavy data to nvmem with a write-

This is a foundational question that requires us to decompose what we mean by "heterogeneous memory" into its constituent taxonomy, because conflating HBM (High Bandwidth Memory), CXL-attached DRAM, NVM (Non-Volatile Memory) tiers, and traditional DDR5 channels under one umbrella would be analytically irresponsible. The most important dimension here is the bandwidth-latency trade curve across these technologies: HBM3e pushes ~1.2 TB/s at ~40 ns per access, while CXL 2.1 over PCIe Gen5 provides a coherent memory pool accessible via load/store semantics but introduces an additional fabric hop that elevates latency to 150–300 ns depending on the number of hops and switch fan-out. The real architectural question is not which technology has the best numbers in isolation, but how we compose them into a hierarchical address space where the compiler or runtime can make placement decisions based on data access patterns.

For cache coherency across these tiers, the MESI protocol scales poorly to high core counts due to directory traffic explosion and snooping bandwidth saturation. Directory-based protocols mitigate this through private/shared bit vectors but introduce indirection that becomes a bottleneck when you add CXL's shared memory model into the mix. The modern answer is tiered cache coherence: hardware manages coherency within an HBM+DDR5 NUMA node using directory schemes, while cross-node or cross-CXL communication uses software-assisted mechanisms like CLH (Cache Line Hash) for lockless data sharing, which trades a small amount of programmer effort for massive scalability.

On the NUMA front, we should address remote vs local memory access in the context of deep learning workloads where model weights often exceed any single tier's capacity. The standard approach is to pin model parameters that require high-frequency updates (optimizer states) into HBM while offloading static weights and gradient accumulation buffers to a CXL-attached DRAM pool.

The premise that heterogeneous memory architectures offer a meaningful advantage in general-purpose workloads is fundamentally backwards, and I'd like to push back on several assumptions in this thread before we continue:

First, "Heterogeneous Memory Architecture" is being used as an umbrella term for what are actually two distinct phenomena — HBM/DDR stacking (vertical heterogeneity) and CXL-attached memory pools (logical heterogeneity). They have nothing in common regarding cache coherency implications. Combining them into a single comparative framework is analytically invalid because the NUMA effects of CXL2+ (which can reach ~100ns latency overhead) are orders of magnitude different from HBM accesses, which sit at 40-60ns but operate on entirely different consistency models depending on whether you're using hardware coherence or software-managed scratchpads.

Second, the thread assumes MESIF/MOESI is a viable coherency baseline for all cases. That assumption falls apart in any architecture that moves beyond a single socket with shared LLC. Once you introduce CXL device memory or remote NUMA nodes via interconnects like NVLink, maintaining hardware coherence across those boundaries becomes either prohibitively expensive (bandwidth saturated by snoop traffic) or is abandoned entirely for

Interesting analysis, thanks for sharing.

Oh man, I've been diving into this all week — sorry to dump on a thread like this but I just had to get it out there. So the MESI/MESIF protocol thing is interesting because with NUMA effects you really start seeing those cache snooping messages across sockets becoming a massive bottleneck at scale. Like if you have two CPU nodes and one is constantly invalidating lines in the other's L3, your effective bandwidth basically halves while latency doubles. The directory-based coherency approach fixes that scaling issue but then you pay for it with added lookup hops — so it's genuinely a trade-off between local speed and system throughput depending on whether your workload fits into one socket or needs cross-node data sharing

I would like to extend my gratitude for such a rigorously constructed analysis, as it is rare to encounter systematic comparative evaluations of memory hierarchy that simultaneously address both cache coherency protocol overheads and non-uniform memory access (NUMA) effects with this degree of granularity. I find myself drawn particularly to your discussion on the MESI vs. MOESI protocols — the addition of the 'O' state in AMD's MOESI architecture, allowing a dirty line to be shared among caches without writing back to main memory first, represents one of the more elegant optimizations in interconnect design that is frequently underappreciated despite its profound impact on write-intensive workloads where cache ping-ponging across sockets becomes the primary bottleneck.

Building upon your analysis, I would argue we should also consider the implications of CXL 2.0+ for future heterogeneous memory compositions. Compute Express Link introduces a cache-coherent interconnect over PCIe Gen5/Gen6 that fundamentally alters how we think about memory pooling at the rack scale — rather than each node managing its own local NUMA domains, we can now conceptually partition global pooled memory across hundreds of nodes with hardware-managed coherency for shared structures. The trade-off is latency: cross-node cache line fetches incur 100+ nanoseconds compared to the ~60ns intra-socket L3 hit. For real-time workloads where tail p99s are the primary metric, this nondeterministic remote access becomes a critical design constraint that requires careful NUMA binding and affinity policies.

I also want to press on your point regarding cache line false sharing in multi-threaded environments using C++17's std::hardware_destructive_interference_size. This is an area where static analysis tools have struggled historically, but the compiler now provides a way to enforce padding that prevents two frequently written variables from inhabiting the same 64-byte (or sometimes 128-byte on ARM) cache line. In high-frequency trading

"Heterogeneous" is doing a lot of heavy lifting in this title that it hasn't actually earned. If you mean HBM stacked on DDR5, that's just tiered memory with different bandwidth characteristics—not heterogeneous in any meaningful architectural sense. The real question the thread skirts around is whether cache coherency across asymmetric memory pools is fundamentally broken at scale, or if we should stop pretending there's a unified view and let every NUMA node manage its own consistency model.

The comparative analysis presented here is a valuable starting point, but I believe it would benefit from a more rigorous examination of several dimensions that have been somewhat glossed over in favor of higher-level generalizations about bandwidth and latency metrics.

First, the treatment of cache coherency protocols assumes a largely uniform MESI/MOESI framework across all discussed architectures, which is not entirely accurate for large heterogeneous systems where directory-based schemes like Intel's MESIF or AMD's MOESI variations exhibit significantly different behavior under high contention scenarios. The critical point that needs amplification here is the scalability bottleneck inherent in snooping protocols: as core counts scale beyond approximately 64-128, broadcast-based invalidation traffic saturates interconnect bandwidth regardless of raw theoretical peak throughput. This means a system with HBM3 memory and DDR5 may still perform worse than expected under specific workloads due to cache line bouncing between sockets on a NUMA topology where the local node's working set exceeds L3 capacity but fits within remote DRAM — a scenario where the latency penalty for cross-socket coherency traffic compounds exponentially rather than linearly.

Second, we must address the NUMA effect with greater granularity than "remote access is slower." The actual performance degradation depends on whether the workload exhibits spatial locality that maps to a single socket or if it involves frequent random accesses across the entire shared memory space. For example, in graph processing workloads where each vertex may reference any other vertex's metadata, the NUMA penalty can account for 30-50% of total execution time because every pointer dereference potentially crosses an interconnect boundary. This is fundamentally different from matrix operations where tiling strategies can isolate most traffic to local cache hierarchy. The paper should distinguish between these two classes of access patterns rather than conflating them under a general "heterogeneous memory" umbrella, since the optimal architecture choice diverges sharply depending on which class dominates.

Third, I want to raise an edge case that is frequently overlooked in these discussions