Labs / Python parallel computing

A Quick Guide to Python Parallel Computing: 13 Python Asynchronous Programming – Basic Concepts, Event Loop and Callback Mechanism

Author: Li Zisheng · Source: FreakStudio open documentation · License: CC BY 4.0

Contents
A Quick Guide to Python Parallel Computing: 13 Python Asynchronous Programming – Basic Concepts, Event Loop and Callback Mechanism

Abstract:

Asynchronous programming in Python builds coroutines based on async/await, which run within an event loop. Coroutines generate Tasks, suspend when encountering await, resume execution via callback triggers once I/O operations complete, and schedule concurrent tasks in a non-blocking manner through the event loop to achieve high concurrency in a single thread.

A Quick Guide to Python Parallel Computing: 13 Python Asynchronous Programming – Basic Concepts, Event Loop and Callback Mechanism

For more information about us, you can refer to the cloud document: Freak Embedded Studio Cloud Docs, or visit our wiki: https://github.com/leezisheng/Doc/wik

FreakStudio's Blog

Previous Recommendations:

You who study embedded systems still don't know Object Oriented? ?!

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 00 Introduction to Object Oriented Design Methodology

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 01 Basic Concepts of Object Oriented Programming

The Most Beginner-Friendly Object Oriented Programming Tutorial Across the Web: 02 Python Implementation of Classes and Objects - Creating Classes with Python

The Most Beginner-Friendly Object Oriented Programming Tutorial Across the Network: 03 Python Implementation of Classes and Objects – Adding Attributes to Custom Classes

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 04 Python Implementation of Classes and Objects - Adding Methods to Custom Classes

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 05 Python Implementation of Classes and Objects - PyCharm Code Tags

The most beginner-friendly Object Oriented programming tutorial on the entire network: 06 Python implementation of classes and objects - data encapsulation of custom classes

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 07 Python Implementation of Classes and Objects - Type Annotations

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 08 Python Implementation of Classes and Objects - @property Decorator

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 09 Python Implementation of Classes and Objects - Relationships Between Classes

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 10 Python Implementation of Classes and Objects - Class Inheritance and Liskov Substitution Principle

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 11 Python Implementation of Classes and Objects - Subclass Calling Superclass Methods

The Most Beginner-Friendly Object Oriented Programming Tutorial Across the Network: 12 Python Implementation of Classes and Objects - Using Python's logging Module to Output Program Runtime Logs

The Best Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 13 Python Implementation of Classes and Objects – Installation and Usage of Sourcetrail, a Visual Code Reading Tool

The Most Beginner-Friendly Object Oriented Programming Course on the Whole Network: The Most Beginner-Friendly Object Oriented Programming Course on the Whole Network: 14 Python Implementation of Classes and Objects - Static Methods and Class Methods of Classes

The most beginner-friendly Object Oriented programming tutorial on the entire network: 15 Python implementation of classes and objects - the __slots__ magic method

The Most Beginner-Friendly Object Oriented Programming Tutorial Across the Network: 16 Python Implementation of Classes and Objects - Polymorphism, Method Overriding and Open Closed Principle

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 17 Python Implementation of Classes and Objects - Duck Typing and "file-like object"

The Most Beginner-Friendly Object Oriented Programming Course on the Whole Network: 18 Python Implementation of Classes and Objects - Multiple Inheritance and Serial Port Data Curve Plotting with PyQtGraph

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 19 Python Implementation of Classes and Objects – Using PyCharm to Automatically Generate File Comments and function Comments

The Most Beginner-Friendly Object Oriented Programming Course on the Whole Network: 20 Python Implementation of Classes and Objects – Implementation of Composition Relationship and CSV File Saving

The Most Beginner-Friendly Object Oriented Programming Course on the Whole Network: 21 Python Implementation of Classes and Objects - Organization of Multiple Files: Modules and Packages

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 22 Python Implementation of Classes and Objects - Exceptions and Syntax Errors

The Most Beginner-Friendly Object Oriented Programming Tutorial Across the Network: 23 Python Implementation of Classes and Objects - Throwing Exceptions

The Best Beginner-Friendly Object Oriented Programming Course on the Whole Network: 24 Python Implementation of Classes and Objects - Exception Catching and Handling

The Most Beginner-Friendly Object Oriented Programming Course on the Whole Network: 25 Python Implementation of Classes and Objects - Python Determines the Type of Input Data

The most beginner-friendly Object Oriented programming tutorial on the entire network: 26 Python implementation of classes and objects - context managers and the with statement

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 27 Python Implementation of Classes and Objects - Implementation of Exception Hierarchy and Custom Exception Classes in Python

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 28 Python Implementation of Classes and Objects - A Comprehensive Summary of Python Programming Principles, Philosophy and Specifications

The Most Beginner-Friendly Object Oriented Programming Tutorial Across the Network: 29 Python Implementation of Classes and Objects – Assertions, Defensive Programming and the Usage of the help function

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 30 Python's Built-in Data Types - The object Root Class

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 31 Python's Built-in Data Types - Object and Type

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 32 Python's Built-in Data Types - Class and Instance

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 33 Python's Built-in Data Types - The Relationship Between Object and Type

The Most Beginner-Friendly Object Oriented Programming Course on the Whole Network: 34 Python's Built-in Data Types - Commonly Used Composite Data Types in Python: Tuples and Named Tuples

The most beginner-friendly Object Oriented programming tutorial across the network: 35 Python's built-in data types - docstrings and the __doc__ attribute

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 36 Python's Built-in Data Type - Dictionary

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 37 Common Composite Data Types in Python - Lists and List Comprehensions

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 38 Common Composite Data Types in Python - Implementing stack, Queues and Deques with Lists

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 39 Common Composite Data Types in Python - Sets

The most beginner-friendly Object Oriented programming tutorial across the entire network: 40 Common composite data types in Python - enumerations and the usage of the enum module

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 41 Common Composite Data Types in Python - Queues (FIFO, LIFO, Priority Queues, Deques and Circular Queues)

The most beginner-friendly Object Oriented programming tutorial on the entire network: 42 Commonly used composite data types in Python - collections container data types

The Most Beginner-Friendly Object Oriented Programming Tutorial Across the Network: 43 Common Composite Data Types in Python - Extending Built-in Data Types

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 44 Python Built-in functions and Magic Methods – Overriding Magic Methods of Built-in Types

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 45 Implementing Common Data Structures in Python - Linked List, Tree, Hash Table, Graph and Heap

The Most Beginner-Friendly Object Oriented Programming Course on the Whole Network: 46 Python function Methods and Interfaces - function and Event-Driven Frameworks

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 47 Python function Methods and Interfaces - Callback Function

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 48 Python function Methods and Interfaces - Positional Arguments, Default Arguments, Variable Arguments and Keyword Arguments

The Most Beginner-Friendly Object Oriented Programming Tutorial Across the Network: 49 Python function Methods and Interfaces - The Difference Between function and Methods and Lambda Anonymous function

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 50 Python function Methods and Interfaces - Interfaces and Abstract Base Classes

The most beginner-friendly Object Oriented programming tutorial across the network: 51 Python function methods and interfaces - Implementing interfaces with Zope

The most beginner-friendly Object Oriented programming tutorial on the entire network: 52 Python function methods and interfaces - Protocol and interfaces

The most beginner-friendly Object Oriented programming tutorial across the entire network: 53 Python Strings and Serialization - Strings and Character Encodings

The Most Beginner-Friendly Object Oriented Programming Course on the Whole Network: 54 Python Strings and Serialization - String Formatting and the format Method

The most beginner-friendly Object Oriented programming tutorial on the entire network: 55 Python Strings and Serialization - Byte Sequence Types and Mutable Byte Strings

The most beginner-friendly Object Oriented programming tutorial on the entire network: 56 Python Strings and Serialization - Applications of Regular Expression and the re Module

The most beginner-friendly Object Oriented programming tutorial on the entire network: 57 Python Strings and Serialization - Serialization and Deserialization

The most beginner-friendly Object Oriented programming tutorial on the entire network: 58 Python Strings and Serialization – Definition and Implementation of Serialized Web Objects

The Most Beginner-Friendly Object Oriented Programming Course on the Entire Internet: 59 Python Parallelism and Concurrency – Parallelism, Concurrency, Threads and Processes

Master Python Parallel Computing in One Article: 00 Basic Concepts of Parallel Computing

A Quick Guide to Python Parallel Computing: 01 Python Multithreaded Programming – Basic Concepts, Switching Process, GIL Lock Mechanism, and Producer-Consumer Model

A Quick Guide to Python Parallel Computing: 02 Python Multithreaded Programming — the threading Module, Thread Creation & Query, and Daemon Threads

A Quick Guide to Python Parallel Computing: 03 Python Multithreaded Programming - Multithread Synchronization (Part 1) — Based on Mutex Locks, Recursion Locks and Semaphores

A Quick Guide to Python Parallel Computing: 04 Python Multithreaded Programming – Multithread Synchronization (Part 2) — Based on Condition Variables, Events and Barriers

A Quick Guide to Python Parallel Computing: 05 Python Multithreaded Programming – Scheduled Thread Execution

A Quick Guide to Python Parallel Computing: 06 Python Multithreaded Programming – Communication Based on Queues

A Quick Guide to Python Parallel Computing: 07 Python Multithreaded Programming – Using Thread Pools and Performance Evaluation of Multithreading

A Quick Guide to Python Parallel Computing: 08 Python Multiprocessing Programming – Process Creation & Naming, ID Retrieval, Daemon Process Setup and Process Termination

A Quick Guide to Python Parallel Computing: 09 Python Multiprocessing Programming – Data Synchronization Between Processes: Based on Mutex Locks, Recursion Locks, Semaphores, Condition Variables, Events and Barriers

Master Python Parallel Computing in One Read: 10 Python Multiprocessing Programming - Data Sharing Between Processes - Based on Shared Memory and Data Managers

A Quick Guide to Python Parallel Computing: 11 Python Multiprocessing Programming — Secure Data Transmission Between Processes Based on Queues and Pipes

A Quick Guide to Python Parallel Computing: 12 Python Multiprocessing Programming - Process Pool

For more exciting content, you can check out:

C Language 1.5 Programming Practice: Pure C Modularization × Inheritance × Polymorphism Framework

Speed Up Your Python: A Quick Guide to Python Parallel Computing

Mastering the Debugging Principles of CM3 Microcontrollers in One Article

After half a month of dedicated grinding, the ultimate roundup of embedded technology stack is now ready

The "Secret Martial Arts Manual" for Computer Science Competitions

An open-source project collection for MicroPython: awesome-micropython, which covers MicroPython tool libraries in various aspects

Avnet ZUBoard 1CG Development Board — A New Choice for Deep learning

Engineers should not blindly worship open-source code, but also pay attention to mastering basic skills

What? A motor drive module with customizable color schemes? !

What? Three new expansion boards just released for the XIAO main controller!

A Step-by-Step Guide to Publishing Third-Party Libraries for Arduino

A 10,000-Word Detailed Guide: Step-by-Step Tutorial on Publishing Third-Party Libraries for MicroPython/Python

【Electronic DIY Magic Tool】Compatible with All Kinds of 5-Wire Stepper Motors! Supports seeed XIAO Series Development Boards

Master the Electronic Design Contest in One Read: The Must-Know Winning Guide for Electronics Enthusiasts

Master the Optoelectronic Design Competition in One Read: A Must-See for Electronics Enthusiasts!

"Brainless" Operation Guide for Engineering Competitions: A Step-by-Step Roadmap from Knowledge Learning, hardware Selection, Code Debugging to Report Writing

How to Win the National Award of the Datang Cup with Zero Foundation? All the Senior's Competition Preparation Tips Are Here

【Extensively Compiled In-Depth Guide】College Student Competition Crash Course: Topic Selection × Team Formation × Roadshow, Get Competition-Ready in 48 Hours

Chaos in Contemporary College Students' Competitions: A Survival Guide from "Involution" to "Heirloom Projects"

Charging over 6,000 yuan for a single conference shoot? Here are all the shooting tips and steps

A female college student's on-site research on losing 5,000 yuan from a street stall: Where in Chengdu has the largest crowd and is easiest to make money? I'll tell you!

Making Money with Polaroid Cameras at Chengdu’s Popular Check-in Spots: A Super Practical Street-Vendor-Level Side Hustle Guide for Tourists

Impressions of Chengdu: An Electronic Girl Created a Touch of Romance by Herself

General Relays vs. Latching Relays vs. MOSFETs: A Complete Analysis of Working Principles and Circuit Design

Goodbye TP4056! Domestic SY3501D single chip integrates charging, discharging and boosting functions with only 7 components! Open source PCB files attached

POB: Programming Oriented to Boss — A Novel Reality-Driven Programming Paradigm

Document Acquisition:

You can download the document via the following link:

https://github.com/leezisheng/Doc

This document is a learning guide on parallel computing and Python concurrent programming, which covers the basic concepts of parallel computing, as well as the core knowledge points of Python multi-threaded programming, multi-process programming and coroutine programming:

10 1.1

Main Text

1. Basic Concepts of Asynchronous Programming

1.1 Difference between Concurrency and Parallelism

First, let's first get acquainted with the two basic concepts: sequence vs. concurrency, and serialization vs. parallelism.

1.2 The Relationship Between Concurrency and Parallelism

1.3 Concept of Asynchronous Model

Apart from the sequential execution model and the parallel execution model, there exists a third type of model called the asynchronous model. In an asynchronous model, multiple events are allowed to occur (be processed) simultaneously. When a program invokes a time-consuming function (method), it does not block the execution flow of the program, and the program continues to execute downward. When the function finishes execution, the program can receive a completion notification or access the execution result (if there is a return value or a return value is required).Simply put, asynchronous programming is a programming paradigm that allows a program to continue executing other tasks while waiting for an operation (such as an I/O operation) to complete.

Let's take an example to illustrate the difference between synchronous and asynchronous operations. In this example, the program retrieves two files over the network and then performs a merge operation on them:

1.3.1 Difference Between Concurrency and Parallelism

First, let's first understand the two basic concepts: sequence and concurrency, as well as serial and parallel.

(1) Sequential and concurrent refer to whether multiple tasks can be started simultaneously:

① Sequence: The current task can only start execution after the previous task that has started execution is completed;

② Concurrency: The current task can start executing regardless of whether the previous task that has started execution has been completed or not.

① Serial: There is only one task execution unit, which physically can only execute individual tasks sequentially one by one;

② Parallelism: There are multiple task execution units, so multiple tasks can be executed simultaneously from a physical perspective.

① The core of concurrency lies in the ability to handle multiple tasks — it does not necessarily mean executing multiple tasks simultaneously, and the focus here is on the abstract scheduling of tasks;

② The key to parallelism lies in your ability to handle multiple tasks simultaneously, which focuses on the actual execution of tasks.

1.3.2 The Relationship Between Concurrency and Parallelism

(1) Both concurrency and parallelism can involve multiple threads. The key lies in whether these threads can be executed simultaneously by multiple CPUs: if yes, it is parallelism; concurrency, by contrast, means that multiple threads are executed by a single CPU in a time-sharing, round-robin manner.

(2) Parallelism requires a multi-core CPU to be achieved, while concurrency can be realized with a single-core CPU (concurrency appears parallel at the macro level but is serial at the micro level).

(3) Parallelism is a subset of concurrency; concurrency does not necessarily entail parallelism, but parallelism definitely falls under concurrency.

1.3.3 Concept of Asynchronous Model

Apart from the sequential execution model and the parallel execution model, there exists a third type, known as the asynchronous model. In an asynchronous model, multiple events are allowed to occur (be processed) simultaneously. When a program invokes a time-consuming function (method), it does not block the execution flow of the program, and the program continues to execute downward. When the function finishes execution, the program can receive a completion notification or access the execution result (if there is a return value or a return value is required).Simply put, asynchronous programming is a programming paradigm that allows a program to continue executing other tasks while waiting for an operation (such as an I/O operation) to complete.

Let's take an example to illustrate the difference between synchronous and asynchronous operations. In this example, the program retrieves two files over the network and then performs a merge operation on them:

1 2

For the above example in an asynchronous system, the solution is to spin up an additional thread for processing: the first thread fetches the first file, the second thread fetches the second file, and the second thread does not need to wait for the first thread to finish executing before running. After both threads have obtained their corresponding results, we then synchronously process the operation of merging the results.

Let's look at another scenario: a single-threaded approach reads files from the OS (Operating System) and needs to perform mathematical operations. In an asynchronous system, the program initiates a request to read a file from the OS. Since the read operation takes a relatively long time, while waiting for the file reading to complete, the program returns control to the CPU to carry out the mathematical operations.

In asynchronous programming, a function is usually provided for relatively time-consuming functions, and an additional parameter for callback is included in the parameters of this function. This function is often called a callback function. When the relatively time-consuming function finishes executing, the result is returned through the callback function.

1.3.4 Relationship Between Asynchronous Programming Model and Processes & Threads

Here we take multithreading as an example: multithreading refers to the concurrent or parallel execution of multiple instructions (threads) at the same time.

A simple example is opening two browser windows to download two files at the same time. Each window uses a separate thread to download the file, and there is no need for either to wait for the other to finish; instead, the downloads proceed in parallel.

The following diagram illustrates the process of concurrently executing a multi-threaded application:

1 2 3

From the above introduction, we can see that multithreading is all about the concurrent execution of functionalities, while asynchronous programming is about the non-blocking execution between functions (which means the main thread that invokes a method does not need to wait synchronously for another thread to complete, so that the main thread can handle other tasks). We can apply asynchrony to both single-threaded and multi-threaded scenarios, so multithreading is just one implementation form of asynchronous programming.

A major difference between asynchronous programming and multithreaded programming models is that in multithreading, the operating system determines when to suspend or resume a certain activity on the timeline, whereas in the asynchronous concurrency model, programmers must assume that a thread may be suspended and replaced at any time.

Programmers can break down tasks into numerous small steps that can be executed at intervals, so that if one task requires the output of another, the dependent task must receive its corresponding input.

We can use thread pools or process pools to handle asynchronous tasks.

2. Key Implementation Approaches of Asynchronous Programming: Event Loop and Callback Mechanism

2.1 Event Loop Mechanism

The event loop is a core concept in asynchronous programming, which enables a program to remain responsive while processing I/O or other time-consuming operations. In psychology, this can be analogized to the human "autopilot mode", under which we can switch between different tasks without losing our focus.

In a computing system, an entity that can generate events is called an event source, and an entity that can process events is called an event handler. In addition, there are some third-party entities called event loops, whose role is to manage all events, run continuously in a loop throughout the entire program execution, track the sequence of event occurrences and place them in a queue, and invoke the corresponding event handler to process events when the main thread is idle. The event processing part is the only active part of the system; when one event is processed, the flow proceeds to handle the next event.For example, in a Graphical User Interface (GUI) application, the program can perform data processing tasks in the background while continuously detecting user click and input events.

The working process of the event loop can be divided into several steps:

(1) Registering events and callbacks: The program registers the events it is interested in and the corresponding callback functions with the event loop.

(2) Wait for events to occur: The event loop runs and waits for events to occur; during this process, the program continues to execute other tasks.

(3) Event triggering and callback execution: Once a registered event occurs, the event loop will invoke the corresponding callback function to process these events.

This process is similar to how the human brain handles sensory input: while our brain waits for important information, it is capable of processing other streams of information, and will respond immediately as soon as the important information arrives.

2.2 Callback Mechanism

Callback Mechanism is a common Design pattern in asynchronous programming, which allows a predetermined function to be executed automatically when a certain task is completed. This is analogous to conditioned reflex in psychology: when a specific condition (event) is met, the associated behavior (callback) is executed automatically.

At the technical level, a callback is a function pointer or reference that is invoked by an event handler when a specific event occurs. It allows developers to defer the execution of a certain part of the code (i. e., the callback function) until the specific event takes place. Callbacks provide a flexible way to handle asynchronous events, while decoupling the occurrence of events from the processing logic, making the code more Modularization.

2.3 Specific Implementation

Python's Asyncio module provides methods for managing events, coroutines, tasks and threads, as well as primitives for writing concurrent code. The main components and concepts of this module include:

A Quick Guide to Python Parallel Computing: 13 Python Asynchronous Programming – Basic Concepts, Event Loop and Callback Mechanism

Asyncio provides the following methods for managing the event loop:

Methods

Function

loop = get_event_loop()

Get the event loop of the current context

loop.call_later

(time_delay, callback, argument)

Delay the execution of the callback method by time_delay seconds

loop.call_soon

(callback, argument)

Call the callback as soon as possible, the call_soon () function ends,

The main thread will call the callback immediately after returning to the event loop

loop.time()

Returns the internal time of the current time loop as a float type

asyncio.set_event_loop()

Set the event loop for the current context

asyncio.new_event_loop()

Create a new time loop based on this strategy and return it

loop.run_forever()

It will keep running until stop () is called

Here, we have defined three asynchronous tasks that are executed sequentially in the order shown in the diagram below.

1 2 3

The code is as follows:


import asyncio
import random
import time

# end_time: defines the maximum time function_1() may run,
#           and is passed as an argument to function_2() via the call_later method
# loop:     the event loop obtained earlier through the get_event_loop() method

def function_1(end_time, loop):
    # Print the function name
    print ("function_1 called")
    # Delay 0 to 1 s
    time.sleep(random.randint(0, 1))
    if (loop.time() + 1.0) < end_time:
        loop.call_later(1, function_2, end_time, loop)
    else:
        loop.stop()

def function_2(end_time, loop):
    print ("function_2 called ")
    # Delay 0 to 1 s
    time.sleep(random.randint(0, 1))
    if (loop.time() + 1.0) < end_time:
        loop.call_later(1, function_3, end_time, loop)
    else:
        loop.stop()
        
def function_3(end_time, loop):
    print ("function_3 called")
    # Delay 0 to 1 s
    time.sleep(random.randint(0, 1))
    if (loop.time() + 1.0) < end_time:
        loop.call_later(1, function_1, end_time, loop)
    else:
        loop.stop()

# Get this event loop
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
# Define the end time
end_loop = loop.time() + 9
# Call the function_1() function via the call_soon method
loop.call_soon(function_1, end_loop, loop)
# Set a long period during which it keeps responding
loop.run_forever()
# If the running time exceeds the set limit, the event loop terminates
loop.close()

The running results are as follows:

function_1 function_2 function_3