☀️ Parallel processing in Python

Parallel processing in Python is a programming approach that allows developers to design programs in which different parts execute simultaneously, reducing the overall runtime. Python provides several libraries for parallel processing, including multiprocessing, threading, and concurrent.futures. Below are examples demonstrating how to use each of these methods:

1. Using the multiprocessing Library

The multiprocessing library allows you to create separate processes that run concurrently.

pythonCopy codeimport multiprocessing

def worker(num):
    """Calculate the square of a number."""
    print(f"Square of {num}: {num * num}")

if __name__ == '__main__':
    # Create a list to hold processes
    processes = []
    for i in range(10):
        p = multiprocessing.Process(target=worker, args=(i,))
        processes.append(p)
        p.start()

    # Wait for all processes to complete
    for process in processes:
        process.join()

---

2. Using the threading Library

The threading library creates threads that share the same memory space, which can be suitable for I/O-bound tasks.

pythonCopy codeimport threading

def worker(num):
    """Calculate the square of a number."""
    print(f"Square of {num}: {num * num}")

if __name__ == '__main__':
    # Create a list to hold threads
    threads = []
    for i in range(10):
        t = threading.Thread(target=worker, args=(i,))
        threads.append(t)
        t.start()

    # Wait for all threads to complete
    for thread in threads:
        thread.join()

---

3. Using the concurrent.futures Library

The concurrent.futures library provides a higher-level interface for parallel execution, allowing you to use a thread or process pool.

pythonCopy codeimport concurrent.futures

def worker(num):
    """Calculate the square of a number."""
    print(f"Square of {num}: {num * num}")

if __name__ == '__main__':
    # Use a process pool to execute tasks in parallel
    with concurrent.futures.ProcessPoolExecutor() as executor:
        executor.map(worker, range(10))

---

Key Differences Between Methods

1. multiprocessing:
   • Creates separate processes.
   • Suitable for CPU-bound tasks.
   • Avoids the Global Interpreter Lock (GIL).
2. threading:
   • Creates threads within the same process.
   • Suitable for I/O-bound tasks.
   • Limited by the GIL for CPU-bound tasks.
3. concurrent.futures:
   • High-level abstraction over multiprocessing and threading.
   • Easy to use for both processes and threads.

---

Choosing the Right Method

• For CPU-bound tasks (e.g., numerical computations): Use multiprocessing or concurrent.futures.ProcessPoolExecutor.
• For I/O-bound tasks (e.g., network requests, file operations): Use threading or concurrent.futures.ThreadPoolExecutor.

These tools provide flexibility and efficiency, allowing you to implement parallel processing effectively in Python for tasks ranging from simple calculations to complex real-world applications.