Why do I keep running into OOM errors on my Kubernetes pods?

I've been seeing Out Of Memory (OOMKilled) status on several of my worker pods lately despite having plenty of node resources available. The issue seems to be that the container memory limits are set too low relative to peak usage during batch processing jobs, which triggers the OOM killer even though the overall node isn't saturated. I should probably increase the resources.limits.memory setting and maybe add a buffer for garbage collection overhead. Has anyone found a good formula for estimating these limits from production logs?