Multiprocessing

Overview

Threads exist within a single OS process; that’s why they can share access to common objects. We can do concurrent computing at the process level, also. Unlike threads, separate processes cannot directly access variables set up by other processes. This independence is helpful because each process has its own GIL and its own private pool of resources. On a modern multi-core processor, a process may have its own core, permitting concurrent work with other cores.

The multiprocessing API was originally designed to mimic the threading API. However, the multiprocessing interface has evolved, and in recent versions of Python, it supports more features more robustly. The multiprocessing library is designed for when CPU-intensive jobs need to happen in parallel and multiple cores are available. Multiprocessing is not as useful when the processes spend a majority of their time waiting on I/O (for example, network, disk, database, or keyboard), but it is the way to go for parallel computation.

Example

The multiprocessing module spins up new operating system processes to do the work. This means there is an entirely separate copy of the Python interpreter running for each process. Let’s try to parallelize a compute-heavy operation using similar constructs to those provided by the threading API, as follows: