Mastering Concurrency and Parallelism in Python: Threading, Multiprocessing, and Asyncio

Introduction link

Concurrency and parallelism are key concepts for developing high-performance applications. Python provides several modules that enable concurrent and parallel execution of code. We’ll discuss three primary methods: threading, multiprocessing, and asyncio.

Threading link

Threading is a technique for achieving concurrency. In Python, threads allow you to run multiple operations concurrently in the same process space.

Basic Threading link

  import threading

def print_cube(num):
    """Function to print cube of given num"""
    print("Cube: {}".format(num * num * num))

def print_square(num):
    """Function to print square of given num"""
    print("Square: {}".format(num * num))

# Creating thread objects
t1 = threading.Thread(target=print_square, args=(10,))
t2 = threading.Thread(target=print_cube, args=(10,))

# Starting threads
t1.start()
t2.start()

# Waiting for both threads to complete
t1.join()
t2.join()

print("Done!")
  

In this example, two threads t1 and t2 run concurrently, which allows print_square and print_cube to execute simultaneously.

Multiprocessing link

Multiprocessing is used for spreading tasks over multiple processors, aiming to achieve parallelism (simultaneous execution).

Basic Multiprocessing link

  from multiprocessing import Process, current_process

def worker():
    """Worker function"""
    print('Worker:', current_process().name)

if __name__ == '__main__':
    # Number of CPUs
    num_cpus = 4
    processes = []
    for i in range(num_cpus):
        process = Process(target=worker)
        processes.append(process)
        process.start()

    for process in processes:
        process.join()

    print("Processing complete!")
  

This example shows how to create and run multiple processes. By using the Process class from multiprocessing, we can execute the function worker concurrently on multiple CPUs.

Asyncio Module link

asyncio is used for writing concurrent code using the async/await syntax.

Using Asyncio link

  import asyncio

async def count_to_ten():
    """Asynchronously count to ten"""
    for i in range(1, 11):
        print(i)
        await asyncio.sleep(1)  # Simulate an I/O operation

async def main():
    await count_to_ten()

# Running the coroutine
asyncio.run(main())
  

Here, count_to_ten is an asynchronous function that counts from 1 to 10, pausing for a second between numbers using await asyncio.sleep(1), which mimics a blocking I/O operation. asyncio.run(main()) is used to run the main coroutine that drives the count_to_ten coroutine.

Conclusion link

Concurrency and parallelism are powerful strategies for optimizing performance in Python applications. By understanding and utilizing threading, multiprocessing, and asyncio, you can significantly improve the efficiency of your code, especially in I/O-bound and CPU-bound operations. This guide provides a comprehensive overview of these methods, showing you how to implement them in real-world scenarios.