Labs / Python parallel computing

A Quick Guide to Python Parallel Computing: 12 Python Multiprocessing Programming - Process Pool

Author: Li Zisheng · Source: FreakStudio open documentation · License: CC BY 4.0

Contents
A Quick Guide to Python Parallel Computing: 12 Python Multiprocessing Programming - Process Pool

Abstract:

In Python multiprocessing programming, the Pool class is used to create a process pool that can execute multiple tasks in parallel. Through methods such as map and apply, functions and parameters are distributed to child processes, which improves CPU utilization, simplifies process management, and facilitates resource reuse. It is suitable for CPU-intensive scenarios.

A Quick Guide to Python Parallel Computing: 12 Python Multiprocessing Programming - Process Pool

For more information about us, you can refer to the cloud document: Freak Embedded Studio Cloud Docs, or visit our wiki: https://github.com/leezisheng/Doc/wik

FreakStudio's Blog

Previous Recommendations:

You who study embedded systems still don't know Object Oriented? ?!

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 00 Introduction to Object Oriented Design Methodology

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 01 Basic Concepts of Object Oriented Programming

The Most Beginner-Friendly Object Oriented Programming Tutorial Across the Web: 02 Python Implementation of Classes and Objects - Creating Classes with Python

The Most Beginner-Friendly Object Oriented Programming Tutorial Across the Network: 03 Python Implementation of Classes and Objects – Adding Attributes to Custom Classes

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 04 Python Implementation of Classes and Objects - Adding Methods to Custom Classes

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 05 Python Implementation of Classes and Objects - PyCharm Code Tags

The most beginner-friendly Object Oriented programming tutorial on the entire network: 06 Python implementation of classes and objects - data encapsulation of custom classes

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 07 Python Implementation of Classes and Objects - Type Annotations

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 08 Python Implementation of Classes and Objects - @property Decorator

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 09 Python Implementation of Classes and Objects - Relationships Between Classes

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 10 Python Implementation of Classes and Objects - Class Inheritance and Liskov Substitution Principle

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 11 Python Implementation of Classes and Objects - Subclass Calling Superclass Methods

The Most Beginner-Friendly Object Oriented Programming Tutorial Across the Network: 12 Python Implementation of Classes and Objects - Using Python's logging Module to Output Program Runtime Logs

The Best Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 13 Python Implementation of Classes and Objects – Installation and Usage of Sourcetrail, a Visual Code Reading Tool

The Most Beginner-Friendly Object Oriented Programming Course on the Whole Network: The Most Beginner-Friendly Object Oriented Programming Course on the Whole Network: 14 Python Implementation of Classes and Objects - Static Methods and Class Methods of Classes

The most beginner-friendly Object Oriented programming tutorial on the entire network: 15 Python implementation of classes and objects - the __slots__ magic method

The Most Beginner-Friendly Object Oriented Programming Tutorial Across the Network: 16 Python Implementation of Classes and Objects - Polymorphism, Method Overriding and Open Closed Principle

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 17 Python Implementation of Classes and Objects - Duck Typing and "file-like object"

The Most Beginner-Friendly Object Oriented Programming Course on the Whole Network: 18 Python Implementation of Classes and Objects - Multiple Inheritance and Serial Port Data Curve Plotting with PyQtGraph

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 19 Python Implementation of Classes and Objects – Using PyCharm to Automatically Generate File Comments and function Comments

The Most Beginner-Friendly Object Oriented Programming Course on the Whole Network: 20 Python Implementation of Classes and Objects – Implementation of Composition Relationship and CSV File Saving

The Most Beginner-Friendly Object Oriented Programming Course on the Whole Network: 21 Python Implementation of Classes and Objects - Organization of Multiple Files: Modules and Packages

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 22 Python Implementation of Classes and Objects - Exceptions and Syntax Errors

The Most Beginner-Friendly Object Oriented Programming Tutorial Across the Network: 23 Python Implementation of Classes and Objects - Throwing Exceptions

The Best Beginner-Friendly Object Oriented Programming Course on the Whole Network: 24 Python Implementation of Classes and Objects - Exception Catching and Handling

The Most Beginner-Friendly Object Oriented Programming Course on the Whole Network: 25 Python Implementation of Classes and Objects - Python Determines the Type of Input Data

The most beginner-friendly Object Oriented programming tutorial on the entire network: 26 Python implementation of classes and objects - context managers and the with statement

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 27 Python Implementation of Classes and Objects - Implementation of Exception Hierarchy and Custom Exception Classes in Python

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 28 Python Implementation of Classes and Objects - A Comprehensive Summary of Python Programming Principles, Philosophy and Specifications

The Most Beginner-Friendly Object Oriented Programming Tutorial Across the Network: 29 Python Implementation of Classes and Objects – Assertions, Defensive Programming and the Usage of the help function

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 30 Python's Built-in Data Types - The object Root Class

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 31 Python's Built-in Data Types - Object and Type

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 32 Python's Built-in Data Types - Class and Instance

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 33 Python's Built-in Data Types - The Relationship Between Object and Type

The Most Beginner-Friendly Object Oriented Programming Course on the Whole Network: 34 Python's Built-in Data Types - Commonly Used Composite Data Types in Python: Tuples and Named Tuples

The most beginner-friendly Object Oriented programming tutorial across the network: 35 Python's built-in data types - docstrings and the __doc__ attribute

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 36 Python's Built-in Data Type - Dictionary

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 37 Common Composite Data Types in Python - Lists and List Comprehensions

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 38 Common Composite Data Types in Python - Implementing stack, Queues and Deques with Lists

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 39 Common Composite Data Types in Python - Sets

The most beginner-friendly Object Oriented programming tutorial across the entire network: 40 Common composite data types in Python - enumerations and the usage of the enum module

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 41 Common Composite Data Types in Python - Queues (FIFO, LIFO, Priority Queues, Deques and Circular Queues)

The most beginner-friendly Object Oriented programming tutorial on the entire network: 42 Commonly used composite data types in Python - collections container data types

The Most Beginner-Friendly Object Oriented Programming Tutorial Across the Network: 43 Common Composite Data Types in Python - Extending Built-in Data Types

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 44 Python Built-in functions and Magic Methods – Overriding Magic Methods of Built-in Types

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 45 Implementing Common Data Structures in Python - Linked List, Tree, Hash Table, Graph and Heap

The Most Beginner-Friendly Object Oriented Programming Course on the Whole Network: 46 Python function Methods and Interfaces - function and Event-Driven Frameworks

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 47 Python function Methods and Interfaces - Callback Function

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 48 Python function Methods and Interfaces - Positional Arguments, Default Arguments, Variable Arguments and Keyword Arguments

The Most Beginner-Friendly Object Oriented Programming Tutorial Across the Network: 49 Python function Methods and Interfaces - The Difference Between function and Methods and Lambda Anonymous function

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 50 Python function Methods and Interfaces - Interfaces and Abstract Base Classes

The most beginner-friendly Object Oriented programming tutorial across the network: 51 Python function methods and interfaces - Implementing interfaces with Zope

The most beginner-friendly Object Oriented programming tutorial on the entire network: 52 Python function methods and interfaces - Protocol and interfaces

The most beginner-friendly Object Oriented programming tutorial across the entire network: 53 Python Strings and Serialization - Strings and Character Encodings

The Most Beginner-Friendly Object Oriented Programming Course on the Whole Network: 54 Python Strings and Serialization - String Formatting and the format Method

The most beginner-friendly Object Oriented programming tutorial on the entire network: 55 Python Strings and Serialization - Byte Sequence Types and Mutable Byte Strings

The most beginner-friendly Object Oriented programming tutorial on the entire network: 56 Python Strings and Serialization - Applications of Regular Expression and the re Module

The most beginner-friendly Object Oriented programming tutorial on the entire network: 57 Python Strings and Serialization - Serialization and Deserialization

The most beginner-friendly Object Oriented programming tutorial on the entire network: 58 Python Strings and Serialization – Definition and Implementation of Serialized Web Objects

The Most Beginner-Friendly Object Oriented Programming Course on the Entire Internet: 59 Python Parallelism and Concurrency – Parallelism, Concurrency, Threads and Processes

Master Python Parallel Computing in One Article: 00 Basic Concepts of Parallel Computing

A Quick Guide to Python Parallel Computing: 01 Python Multithreaded Programming – Basic Concepts, Switching Process, GIL Lock Mechanism, and Producer-Consumer Model

A Quick Guide to Python Parallel Computing: 02 Python Multithreaded Programming — the threading Module, Thread Creation & Query, and Daemon Threads

A Quick Guide to Python Parallel Computing: 03 Python Multithreaded Programming - Multithread Synchronization (Part 1) — Based on Mutex Locks, Recursion Locks and Semaphores

A Quick Guide to Python Parallel Computing: 04 Python Multithreaded Programming – Multithread Synchronization (Part 2) — Based on Condition Variables, Events and Barriers

A Quick Guide to Python Parallel Computing: 05 Python Multithreaded Programming – Scheduled Thread Execution

A Quick Guide to Python Parallel Computing: 06 Python Multithreaded Programming – Communication Based on Queues

A Quick Guide to Python Parallel Computing: 07 Python Multithreaded Programming – Using Thread Pools and Performance Evaluation of Multithreading

A Quick Guide to Python Parallel Computing: 08 Python Multiprocessing Programming – Process Creation & Naming, ID Retrieval, Daemon Process Setup and Process Termination

A Quick Guide to Python Parallel Computing: 09 Python Multiprocessing Programming – Data Synchronization Between Processes: Based on Mutex Locks, Recursion Locks, Semaphores, Condition Variables, Events and Barriers

Master Python Parallel Computing in One Read: 10 Python Multiprocessing Programming - Data Sharing Between Processes - Based on Shared Memory and Data Managers

A Quick Guide to Python Parallel Computing: 11 Python Multiprocessing Programming — Secure Data Transmission Between Processes Based on Queues and Pipes

For more exciting content, you can check out:

C Language 1.5 Programming Practice: Pure C Modularization × Inheritance × Polymorphism Framework

Speed Up Your Python: A Quick Guide to Python Parallel Computing

Mastering the Debugging Principles of CM3 Microcontrollers in One Article

After half a month of dedicated grinding, the ultimate roundup of embedded technology stack is now ready

The "Secret Martial Arts Manual" for Computer Science Competitions

An open-source project collection for MicroPython: awesome-micropython, which covers MicroPython tool libraries in various aspects

Avnet ZUBoard 1CG Development Board — A New Choice for Deep learning

Engineers should not blindly worship open-source code, but also pay attention to mastering basic skills

What? A motor drive module with customizable color schemes? !

What? Three new expansion boards just released for the XIAO main controller!

A Step-by-Step Guide to Publishing Third-Party Libraries for Arduino

A 10,000-Word Detailed Guide: Step-by-Step Tutorial on Publishing Third-Party Libraries for MicroPython/Python

【Electronic DIY Magic Tool】Compatible with All Kinds of 5-Wire Stepper Motors! Supports seeed XIAO Series Development Boards

Master the Electronic Design Contest in One Read: The Must-Know Winning Guide for Electronics Enthusiasts

Master the Optoelectronic Design Competition in One Read: A Must-See for Electronics Enthusiasts!

"Brainless" Operation Guide for Engineering Competitions: A Step-by-Step Roadmap from Knowledge Learning, hardware Selection, Code Debugging to Report Writing

How to Win the National Award of the Datang Cup with Zero Foundation? All the Senior's Competition Preparation Tips Are Here

【Extensively Compiled In-Depth Guide】College Student Competition Crash Course: Topic Selection × Team Formation × Roadshow, Get Competition-Ready in 48 Hours

Chaos in Contemporary College Students' Competitions: A Survival Guide from "Involution" to "Heirloom Projects"

Charging over 6,000 yuan for a single conference shoot? Here are all the shooting tips and steps

A female college student's on-site research on losing 5,000 yuan from a street stall: Where in Chengdu has the largest crowd and is easiest to make money? I'll tell you!

Making Money with Polaroid Cameras at Chengdu’s Popular Check-in Spots: A Super Practical Street-Vendor-Level Side Hustle Guide for Tourists

Impressions of Chengdu: An Electronic Girl Created a Touch of Romance by Herself

General Relays vs. Latching Relays vs. MOSFETs: A Complete Analysis of Working Principles and Circuit Design

Goodbye TP4056! Domestic SY3501D single chip integrates charging, discharging and boosting functions with only 7 components! Open source PCB files attached

POB: Programming Oriented to Boss — A Novel Reality-Driven Programming Paradigm

Document Acquisition:

You can download the document via the following link:

https://github.com/leezisheng/Doc

This document is a learning guide on parallel computing and Python concurrent programming, which covers the basic concepts of parallel computing, as well as the core knowledge points of Python multi-threaded programming, multi-process programming and coroutine programming:

A Quick Guide to Python Parallel Computing: 12 Python Multiprocessing Programming - Process Pool

Main Text

1. Basic Concept of Process Pool

Python process pool Pool is similar to the Python thread pool explained earlier. Although using multiprocessing can improve efficiency, creating processes consumes a large amount of computer resources (the resources occupied by creating a process Process are far greater than those occupied by creating a thread Thread). Threads are the smallest running units of a computer. If even threads require a thread pool, there is no reason for processes not to use a process pool, right?

The multiprocessing library provides the Pool class for implementing simple multi-process tasks. The Pool class allocates a specified number of processes for users to invoke. When a new request is submitted to the Pool, a new process will be created to execute the request if the pool is not yet full; if the pool is full, the request will be queued to wait until a process in the pool finishes running, after which new processes will be created to handle these pending requests.

The Pool class is defined as follows:

Pool([processes[, initializer[, initargs[, maxtasksperchild[, context]]]]])
(1) processes is the number of worker processes to use. If processes is None, the value returned by os.cpu_count() is used;
(2) initializer is a callable to be executed when each worker process starts, defaulting to None,
initargs is the argument tuple to pass to initializer;
(3) context can be used to specify the context of the started worker processes. Usually a process pool is created with the function multiprocessing.Pool() or the Pool() method of a context object. In both cases, context is set appropriately.

Note that only the process that created the process pool object can call its methods.

The main methods of the Pool class include:

A Quick Guide to Python Parallel Computing: 12 Python Multiprocessing Programming - Process Pool
A Quick Guide to Python Parallel Computing: 12 Python Multiprocessing Programming - Process Pool
A Quick Guide to Python Parallel Computing: 12 Python Multiprocessing Programming - Process Pool

Note: Although the second parameter is an iterator, in practical use, the program will not run the subprocess until the entire queue is ready.

A Quick Guide to Python Parallel Computing: 12 Python Multiprocessing Programming - Process Pool

Meanwhile, the methods apply_async () and map_async () return an instance obj of AsyncResult. This instance has the following methods:

A Quick Guide to Python Parallel Computing: 12 Python Multiprocessing Programming - Process Pool

2. Synchronous invocation of the apply () method

In the following code, we create three processes in the process pool, and use the apply () method to execute synchronously work () function until this task is completed and we get res, and finally print the result in the main process. Synchronous means that when a process is executing a request, if the request takes some time to return information, the process will wait all the time and will not continue to execute until it receives the returned information.

import os,time
from multiprocessing import Pool

def work(n):
    print('%s run' %os.getpid())
    time.sleep(0.5)
    return n**2
if __name__ == '__main__':
    p=Pool(3)
    res_l=[]
    for i in range(10):
        res=p.apply(work,args=(i,))
        res_l.append(res)
    print(res_l)
A Quick Guide to Python Parallel Computing: 12 Python Multiprocessing Programming - Process Pool

As can be seen, although the pool has three idle processes, the subsequent ten work functions are all executed serially in sequence without parallel computing.

3. Asynchronous invocation via the apply_async () method

In the following code, we create three processes in the process pool, and use the apply_async () method to asynchronously execute the work () function. After the result is returned, we put the result into a list, return the process, and then execute a new task. It should be noted that the three processes in the process pool will not be started or ended at the same time; instead, one process will be released immediately after it finishes execution, and this process will then receive a new task.

import os
import time
import random
from multiprocessing import Pool

def work(n):
    print('%s run' %os.getpid())
    time.sleep(random.random())
    return n**2

if __name__ == '__main__':
    p=Pool(3)
    res_l=[]

    for i in range(10):
        res=p.apply_async(work,args=(i,))
        res_l.append(res)
    p.close()
    p.join()
    for res in res_l:
        print(res.get()) 

The following are the operation results:

A Quick Guide to Python Parallel Computing: 12 Python Multiprocessing Programming - Process Pool

Multiple work () functions can be seen executing in parallel.With 10 tasks and 3 processes, since the process pool is configured to allow a maximum of 3 concurrent processes, tasks 1,2 and 3 are executed simultaneously. As can be seen from the code output, after tasks 1,2 and 3 finish running, the for loop enters a blocked state, and only after one of tasks 1,2 and 3 completes will the for loop proceed to execute tasks 4,5 and 6, ensuring that no more than 3 tasks run at the same time (the process pool multiprocessing. Pool and the thread pool ThreadPoolExecutor follow the same principle).

Note here that for tasks submitted asynchronously via apply_async, the main process needs to call the join () method to wait for all tasks in the process pool to finish processing, after which you can use the get method to collect the results returned by apply_async. Asynchronous means that the process does not need to wait idly all the time; instead, it continues to execute subsequent operations regardless of the status of other processes. The system will notify the process to handle the result once a message is returned, which improves execution efficiency.

4. Invocation via the map method

The following example demonstrates how to execute a parallel application via a process pool. We create a process pool with 4 processes, then use the map () method to perform a simple calculation. map () blocks until the results are obtained, and this method executes each element of the iterable data as a task in the process pool.

import multiprocessing

def function_square(data):
    result = data*data
    return result

if __name__ == '__main__':
    inputs = list(range(100))
    pool = multiprocessing.Pool(processes=4)
    # The pool.map() method submits some independent tasks to the process pool
    pool_outputs = pool.map(function_square, inputs)
    pool.close()
    pool.join()
    # The calculation results are stored in pool_outputs().
    # Finally, print the results:
    print ('Pool    :', pool_outputs)

The following are the operation results:

A Quick Guide to Python Parallel Computing: 12 Python Multiprocessing Programming - Process Pool

5. callback callback function

In the following scenarios, we need to use a callback function: once any task in the process pool is completed, it immediately notifies the main process to process the task result, and the main process calls the callback function to process the result. We can put time-consuming (blocking) tasks into the process pool, and then specify a callback function (executed by the main process), so that when the main process executes the callback function, it omits the I/O process and directly obtains the result of the task.

Here we take the multi-process Progress Bar example for illustration. First, we need to use pip to install the tqdm library. tqdm is a Python Progress Bar library that can add a progress prompt in a Python long loop. Users only need to wrap any iterator with it, and it is a fast, highly scalable Progress Bar tool library.

tqdm The basic parameters of the object include:

desc ('str'): prefix passed to the progress bar
mininterval (float): minimum update interval [default: 0.1] seconds
set_postfix: set information
elapsed: time elapsed
remaining: time remaining
rate_fmt: rate

The sample code is as follows:

from tqdm import tqdm
from multiprocessing import Pool
import time
import os
# Create a progress bar object
# and set the progress to advance in steps of 10
pbar = tqdm(total=10)
# Set the description text of the progress bar, used to show a summary of the task
pbar.set_description('Sleep')
# Use the lambda operator to convert the progress bar update function into a callback object
update = lambda *args: pbar.update()

def work(n):
    print('%s run' %os.getpid())
    time.sleep(1)
    return n**2

if __name__ == '__main__':
    p=Pool(3)
    res_l=[]
    for i in range(10):
        # Update the progress bar in the callback function
        res=p.apply_async(work,args=(i,), callback=update)
        res_l.append(res)
    p.close()
    p.join()
    for res in res_l:
        print(res.get())

The following is the running result. As you can see, every time the process pool finishes processing a task, it will call tqdm 's update () method once, thereby realizing the display of the multi-process Progress Bar.

A Quick Guide to Python Parallel Computing: 12 Python Multiprocessing Programming - Process Pool