**Thread ** Python performance bottleneck — GIL vs multiprocessing — when to use which?

• GIL (Global Interpreter Lock) prevents true parallelism for CPU-bound threads in CPython; use multiprocessing or concurrent.futures.ProcessPoolExecutor for parallel computation across multiple cores.*
• For I/O-bound tasks, threading is fine since the GIL is released during blocking operations — networking requests and file reads don't conflict with other threads.*
• Rule of thumb: CPU intensive = multiprocessing; I/O bound = threading or asyncio. For hybrid workloads, subprocesses for computation + async for network calls is common in web services.*

The simplest rule of thumb: CPU-bound and pure Python → multiprocessing or a C extension; IO-bound or mixed with network/disk → asyncio + threads for blocking calls.

GIL specifics worth noting:

• I/O releases GIL automatically during the system call
• Extensions in C can manually release (Py_BEGIN_ALLOW_THREADS) — this is why NumPy/Pandas scale well despite Python being locked
• The new "free-threaded" build (3.13+) exists but isn't production-ready for most workloads

Common gotcha: multiprocessing serializes data between processes via pickle, which can become a bottleneck itself if you pass large objects around. For shared state with high contention, consider shared_memory or offloading to C/Rust.

This is one of those questions where the correct answer depends entirely on what your code does, because both approaches solve fundamentally different problems that are often conflated in discussion threads. Let me be exhaustive about this since it's a common source of confusion and many people reach for multiprocessing as a hammer when they need a scalpel or vice versa.

The GIL (Global Interpreter Lock) For context: the GIL is a mutex protecting access to Python objects, which means only one thread can execute Python bytecode at any given moment. This was designed decades ago to make C extensions safe and simple — you don't have to worry about every single object being thread-safe because the lock serializes everything at the interpreter level. The tradeoff is that CPU-bound threads cannot run in parallel, even on a 128-core machine.

When threading IS useful (GIL-friendly scenarios) Thread your code when:

• Your bottleneck is I/O-bound — network requests, database queries, file operations. While one thread waits for the OS to complete an operation, it releases the GIL and another thread can acquire it. The Python interpreter effectively context-switches between threads during waiting periods. This gives you concurrency without process overhead.

• You're calling a C extension that explicitly releases the GIL via Py_BEGIN_ALLOW_THREADS / Py_END_ALLOW_THREADS. NumPy, SciPy, and many other libraries do this for their computationally intensive operations. The Python layer waits while the underlying C/Fortran code runs in parallel across all cores — which is why you can achieve true parallelism within a single process using these libs.

• You want shared state between workers without IPC (Inter-Process Communication) overhead. Threads share memory naturally; processes do not.

When multiprocessing IS useful (GIL-unfriendly scenarios) Multiprocessing bypasses the GIL entirely because each worker is its own Python interpreter with its own lock