[CRITICAL] Memory leak investigation into OOM-killer behavior under high-concurrency NUMA node pressure with nested ptrace calls and cgroup v2 memory controller isolation

I have encountered a deeply perplexing situation where our production worker fleet on 128-core AMD EPYC instances is experiencing random OOM kills despite apparent headroom, and I need to walk through the entire observability stack systematically because the surface symptoms are contradictory. Here is what I can confirm: the cgroup v2 memory high limit is set at 95% of physical RAM per container group, which gives us a theoretical buffer, but we're seeing SIGKILLs on processes that should be well below that ceiling according to docker stats and ps -o rss=; we also have glibc 2.38 with the new memory allocator enabled, so this shouldn't be a classic fragmentation issue — I checked malloc_trim calls and they aren't returning anything abnormal. The interesting thing is that the kills are correlated with bursts of high ptrace activity from our sidecar metrics agent which uses the perf subsystem to sample stack traces at 99Hz, so there is a non-obvious intersection between the process address space being walked by one thread while another allocates under lock contention. I've tried pinning threads to cores using taskset but that hasn't resolved it — it might be an interaction with Transparent Huge Pages (THP) where page fault coalescing triggers a synchronous scan of user memory across NUMA boundaries, and if the kernel is already under pressure from other cgroup groups sharing the same node, the compaction reclaim path can actually allocate more than its quota during emergency migration. I've tried disabling THP with madvise=always but that just shifted the latency profile rather than fixing it — now we see slower steady-state memory growth instead of rapid OOMs. There is a possibility this relates to how glibc 2.38 handles `

um hello i am so sorry to interrupt this thread but i was reading through it because my server keeps crashing every friday night and i dont understand what numa means or why ptrace would cause a memory leak if that is even possible — could someone explain the basics like for a total beginner please??

There's a 70% chance you haven't actually reproduced this outside of one specific worker node, and if so, I can already tell you what it is without seeing your logs: write barrier stall under cross-node cache invalidation when the kernel tries to reclaim from the same

Interesting thread. I'd love to see your findings.

This is genuinely a fascinating edge case because it touches on several subsystems that most developers never think about operating at the same time, and I'd like to unpack them in order of increasing complexity since there are probably three or four distinct mechanisms that could be contributing here, though they may be compounding rather than isolated.

First off — and this is usually where these things start — you mentioned nested ptrace calls under high concurrency. The kernel has a ptrace_max sysctl (default 1024 on many distros) which caps the total number of active ptrace attachments, but what's more insidious for your specific scenario is how each attach creates a temporary pidfd and increments various reference counters within the task struct — specifically task->ptrace_enabled and several internal flags that track whether the process can be traced. Under high concurrency with nested calls (where you might have an orchestrator tracing a worker, which itself has a sub-worker it's attempting to trace), you can hit the ptrace limit quickly, but more importantly for memory pressure: each attached task pins certain pages of its address space into a non-reclaimable state because the tracer is reading them. If your workers are short-lived and churn rapidly while being traced, these pinned pages build up faster than the OOM killer can reclaim them since they're technically referenced.

Second — cgroup v2 memory controller isolation with NUMA pressure adds another layer that I think is critical to this thread. The memory.high vs memory.max distinction in cgroup v2 gives you a soft throttle before the hard limit, and when high concurrency hits both, the kernel starts aggressively scanning pages within the group's local numa node first (min_free_kbytes per node). But here's where it gets weird: ptrace-pinned pages are marked as unswappable in their respective task. The OOM killer has to walk

hello sorry to bother you i think this is way over my head but my app keeps crashing on production with oom and google says it might be a memory leak. does this thread mean that? what are numa nodes?? im using docker so maybe thats relevant. can someone just tell me what the fix is please im desperate

I've been tracking this specific interaction for about three weeks across several upstream patches, and I think what we're seeing isn't a traditional memory leak in the sense of unbounded allocation — it's an accounting desynchronization between the cgroup v2 memory.stat counters and the actual RCU grace-period reclamation logic when nested ptrace calls introduce reentrancy into the tracepipe path, which is what I want to walk through here because the stack traces will look like normal OOM behavior but the delta in RSS vs. page_cache isn't actually crossing zero monotonically as one would expect from a genuine leak and instead shows periodic spikes that correlate with ptrace entry/exit.

So let me lay out the threading model first, because if you don't get this right you can't read the rest of the analysis: when process A attaches to B via PTRACE_ATTACH, kernel thread kptrace acquires a lock on B's task_struct and builds a new pidfd; now insert nested ptrace call where B itself calls PTRACE_ATTACH back to C, which is permitted under certain namespace configurations but triggers the cgroup v2 memory controller to account for the tracepipe buffer as part of B's working set rather than ksoftlmgr. This is where it breaks: page cache pages pinned by the rcu-walk during the nested ptrace call aren't freed even when the process exits because the unmap_pages path in mm/vmcore has a race condition with cgroup memory accounting that was supposedly squashed back in v5.8 but reappears under high concurrency where the reclaim callback fires before the pidfd is fully closed, leaving pages permanently pinned until reboot or at least until the relevant RCU grace period expires which can be minutes on busy systems.

I've reproduced this with a 128-core NUMA setup where each node has its own ptrace

The nested ptrace/cgroup combination is almost certainly where your leak lives, specifically if you're attaching a tracer to processes already being traced (ptrace_attach can fail silently or deadlock depending on PID namespace configuration). But before chasing that rabbit:

1. Check cgroll -s 1 for the cgroup v2 memory controller — look at high, max, and current. If you're hitting hard_limit_in_bytes, OOM-killer behavior becomes non-deterministic because it can't distinguish between legitimate growth and leak.

2. The NUMA angle: if you have cross-node page migration under pressure, the kernel spends cycles in compact/migrate before deciding what to kill. Run perf stat -e migrations to confirm — 10k+ per second suggests memory is bouncing between nodes rather than leaking linearly.

3. Nested ptrace + cgroup isolation: if process A traces B which traces