What is the multiprocessing module in Python?
What is multiprocessing?
Let’s break the word into 2 halves, i.e., multi and processing. The word multi means using more than one. The word processing means getting work done using a processor in the computer. So, the combination means using more than one processor to get the work done.
Multiprocessing has 4 main concepts:
- Process class
- Pool class
- Pipes and Queues
- Locks
This shot covers the Process and Pool classes. In Part 2, Pipes and Queues and Locks are covered.
The Process class
In multiprocessing, processes are spawned by creating an object of the Process class. To start the execution of the process, use the .start() method of the Process object.
from multiprocessing import Processimport mathdef factorial(num):print(f"Factorial of {num} is {math.factorial(num)}")if __name__ == '__main__':p = Process(target=factorial, args=(5,))p.start()p.join()
You can optionally give a name to the process by giving a value to the name parameter during object creation.
To access the current running process inside the function, use the current_process method.
from multiprocessing import Process, current_processimport mathdef factorial(num):print(f"Factorial of {num} is {math.factorial(num)}")curr_process = current_process()print(f"Process Name is '{curr_process.name}'")aliveness = "alive" if curr_process.is_alive else "not alive"print(f"{curr_process.name} is {aliveness}")if __name__ == '__main__':p = Process(target=factorial, args=(5,), name="Factorial process")p.start()p.join()
The Pool class
The pool class lets you create a pool of processes. The maximum number of processes created in the pool is the number of processors in the system. The number of processors in the system can be obtained using os.cpu_count() or multiprocessing.cpu_count().
There are 2 basic methods that the Pool class provides:
map
The method signature is as follows:
map(self, func, iterable, chunksize=None)
This method chops an iterable into several chunks, which it submits to the process pool as separate tasks. The method calls blocks until the function is completed.
apply
The method signature is as follows:
apply(self, func, args=(), kwds={})
This method applies the given function for a given number of arguments. This method calls blocks until the function is completed.
There are 2 variations of the above methods, i.e., map_async() and apply_async, that return the control immediately rather than waiting to complete the function.
| Method | Accepts Multiple arguments | Blocking | Ordered Results |
|---|---|---|---|
| map | no | yes | yes |
| apply | yes | yes | no |
| map_async | no | no | yes |
| apply_async | yes | no | no |
from multiprocessing import Poolimport mathdef factorial(num):return math.factorial(num)def mp_map():with Pool() as p:print(p.map(factorial, range(5)))def mp_map_async():results = []def map_callback(result):results.append(result)p = Pool()p.map_async(factorial, range(5), callback=map_callback)p.close()p.join()print(results)def mp_apply():with Pool() as p:print([p.apply(factorial, args=(i,)) for i in range(5)])def mp_apply_async():results = []def apply_callback(result):results.append(result)p = Pool()_ = [p.apply_async(factorial, args=(i,), callback=apply_callback) for i in range(5)]p.close()p.join()print(results)if __name__ == '__main__':print("Map Method")mp_map()print("-"*10)print("Async Map Method")mp_map_async()print("-"*10)print("Apply Method")mp_apply()print("-"*10)print("Async Apply Method")mp_apply_async()
