Multiprocessing Pools

Overview

Because each process is kept meticulously separate by the operating system, interprocess communication becomes an important consideration. We need to pass data between these separate processes. One really common example is having one process write a file that another process can read. When the two processes are reading and writing a file, and running concurrently, we have to be sure the reader is waiting for the writer to produce data. The operating system pipe structure can accomplish this. Within the shell, we can write ps -ef | grep python and pass output from the ps command to the grep command. The two commands run concurrently. For Windows PowerShell users, there are similar kinds of pipeline processing, using different command names.

Data serialization

The multiprocessing package provides some additional ways to implement interprocess communication. Pools can seamlessly hide the way data is moved between processes. Using a pool looks much like a function call: you pass data into a function, it is executed in another process or processes, and when the work is done, a value is returned. It is important to understand how much work is being done to support this: objects in one process are pickled and passed into an operating system process pipe. Then, another process retrieves data from the pipe and unpickles it. The requested work is done in the subprocess and a result is produced. The result is pickled and passed back through the pipe. Eventually, the original process unpickles and returns it. Collectively, we call this pickling, transferring, and unpickling serializing the data.

The serialization to communicate between processes takes time and memory. We want to get as much useful computation done with the smallest serialization cost. The ideal mix depends on the size and complexity of the objects being exchanged, meaning that different data structure designs will have different performance levels.

Note: Performance predictions are difficult to make. It’s essential to profile the application to ensure the concurrency design is effective.

Example

Armed with this knowledge, the code to make all this machinery work is surprisingly simple. Let’s look at the problem of calculating all the prime factors of a list of random numbers. This is a common part of a variety of cryptography algorithms.

It requires months, possibly years of processing power to factor the 232-digit numbers used by some encryption algorithms. The following implementation, while readable, is not at all efficient; it would take years to factor even a 100-digit number. That’s okay because we want to see it using lots of CPU time factoring 9-digit numbers: