Table of Contents
- Overview
- Prerequisites
- Installation
- Introduction to
multiprocessing - Using
multiprocessingfor Concurrency - Using Pools for Parallel Processing
- Conclusion
Overview
Python’s multiprocessing module allows us to leverage the power of multiple processes for concurrent execution. This offers a way to speed up CPU-bound tasks or work with blocking I/O operations more efficiently. In this tutorial, you will learn how to use multiprocessing to achieve concurrency, how to create and manage multiple processes, and how to use process pools for parallel processing tasks. By the end of this tutorial, you will be able to write efficient and scalable Python code that takes advantage of multiprocessing techniques.
Prerequisites
To follow along with this tutorial, you should have a basic understanding of Python programming and be familiar with functions, classes, and modules. It’s also helpful to have an understanding of concurrency concepts and how multiple processes can improve performance.
Installation
Python’s multiprocessing module is included in the standard library, so no additional installation is required. However, if you are using a Python version older than 2.6, you will need to install the multiprocessing module separately. You can do this by running the following command:
python
pip install multiprocessing
Introduction to multiprocessing
Python’s multiprocessing module is designed to allow the execution of multiple processes in separate memory spaces, enabling true parallelism. It provides an API similar to the threading module but uses processes instead of threads.
One key advantage of using processes over threads is that processes are not subject to the Global Interpreter Lock (GIL), which can limit the performance of Python threads when executing CPU-bound tasks. By using processes, we can fully utilize multiple CPU cores and achieve better performance for CPU-bound tasks.
Using multiprocessing for Concurrency
To start using the multiprocessing module, we need to import it at the beginning of our script:
python
import multiprocessing
Running Functions in Parallel
The multiprocessing module provides the Process class, which allows us to create and manage individual processes in Python. To run a function in a separate process, we can create an instance of the Process class and pass the target function as an argument:
```python
import multiprocessing
def my_func():
# code to be executed in the new process
if __name__ == '__main__':
p = multiprocessing.Process(target=my_func)
p.start()
p.join()
``` In the example above, we create a new process `p` and specify `my_func` as the target function to be executed. We then start the process with `p.start()` and wait for it to finish executing with `p.join()`.
Sharing Data between Processes
In some cases, we may need to share data between different processes. The multiprocessing module provides several mechanisms for doing this, such as shared memory and message passing.
To share data using shared memory, we can use the Value and Array classes:
```python
import multiprocessing
def my_func(shared_value, shared_array):
# code to be executed in the new process
if __name__ == '__main__':
value = multiprocessing.Value('i', 0)
array = multiprocessing.Array('d', [1.0, 2.0, 3.0])
p = multiprocessing.Process(target=my_func, args=(value, array))
p.start()
p.join()
``` In the example above, we create a shared integer value using `multiprocessing.Value('i', 0)` and a shared array of doubles using `multiprocessing.Array('d', [1.0, 2.0, 3.0])`. These shared objects can be passed as arguments to the target function and accessed by multiple processes.
Synchronization and Locks
To avoid race conditions and ensure data integrity when working with shared resources, we can use synchronization primitives such as locks. The multiprocessing module provides the Lock class for this purpose:
```python
import multiprocessing
def my_func(shared_value, lock):
lock.acquire()
try:
# critical section
shared_value.value += 1
finally:
lock.release()
if __name__ == '__main__':
value = multiprocessing.Value('i', 0)
lock = multiprocessing.Lock()
processes = []
for _ in range(10):
p = multiprocessing.Process(target=my_func, args=(value, lock))
processes.append(p)
p.start()
for p in processes:
p.join()
print(value.value) # Output: 10
``` In the example above, we create a lock using `multiprocessing.Lock()`. Inside the target function, we acquire the lock using `lock.acquire()` before entering the critical section, and release it using `lock.release()` afterwards.
Using Pools for Parallel Processing
The multiprocessing.Pool class provides a convenient way to parallelize the execution of a function across multiple input values. It automatically distributes the workload among a pool of worker processes.
To use a Pool, we need to import it from the multiprocessing module:
```python
import multiprocessing
if __name__ == '__main__':
with multiprocessing.Pool() as pool:
results = pool.map(my_func, [1, 2, 3, 4, 5])
print(results)
``` In the example above, we create a `Pool` and use the `map` method to apply the `my_func` function to each item in the input list. The `map` function blocks until all the results are ready and returns a list containing the results in the same order.
Handling Exceptions in Pool Processes
When using a Pool, exceptions raised by the worker processes are propagated back to the parent process. We can catch and handle these exceptions by wrapping the map call in a try-except block:
```python
import multiprocessing
def my_func(x):
if x == 3:
raise ValueError("Oops! Invalid value.")
return x * 2
if __name__ == '__main__':
with multiprocessing.Pool() as pool:
try:
results = pool.map(my_func, [1, 2, 3, 4, 5])
except ValueError as e:
print("An error occurred:", str(e))
else:
print(results)
``` In the example above, the `my_func` function raises a `ValueError` if the input value is equal to 3. We catch this exception in the `try-except` block and handle it accordingly.
Conclusion
In this tutorial, you learned how to use Python’s multiprocessing module for concurrency. You learned how to create and manage multiple processes, share data between processes, and use process pools for parallel processing tasks. By utilizing these techniques, you can improve the performance of CPU-bound tasks and work with blocking I/O operations more efficiently.
You should now have a solid understanding of the multiprocessing module and be able to apply it to your own Python projects to take advantage of concurrency and parallel processing capabilities.
Remember that multiprocessing is not always the solution to every performance problem. It is best suited for CPU-bound tasks and can sometimes introduce additional complexity. Before implementing multiprocessing in your code, consider whether the performance gains outweigh the added complexity and potential overhead.
Continue exploring the multiprocessing module’s documentation to discover more features and options that can help you optimize your Python code.