Labs / Python parallel computing

A Quick Guide to Python Parallel Computing: 09 Python Multiprocessing Programming – Data Synchronization Between Processes: Based on Mutex Locks, Recursion Locks, Semaphores, Condition Variables, Events and Barriers

Author: Li Zisheng · Source: FreakStudio open documentation · License: CC BY 4.0

Contents
A Quick Guide to Python Parallel Computing: 09 Python Multiprocessing Programming – Data Synchronization Between Processes: Based on Mutex Locks, Recursion Locks, Semaphores, Condition Variables, Events and Barriers

Abstract:

Multi-process synchronization mechanisms include mutexes, recursive locks, semaphores, condition variables, events, barriers, etc. : mutexes are used to protect shared resources, recursive locks support repeated locking by the same process, semaphores can control the number of resources accessed, condition variables are used for processes to wait for specific conditions, events are used for Inter-Process Communication and synchronization, and barriers are used for multiple processes to synchronize at specific points to ensure collaborative operation.

347622978

For more information about us, you can refer to the cloud document: Freak Embedded Studio Cloud Docs, or visit our wiki: https://github.com/leezisheng/Doc/wik

FreakStudio's Blog

Previous Recommendations:

You who are learning embedded systems, don’t you even know Object Oriented yet? ?!

The Most Beginner-Friendly Object Oriented Programming Tutorial Across the Network: 00 Introduction to Object Oriented Design Methodology

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 01 Basic Concepts of Object Oriented Programming

The Most Beginner-Friendly Object Oriented Programming Course on the Whole Network: 02 Python Implementation of Classes and Objects - Creating Classes with Python

The Most Beginner-Friendly Object Oriented Programming Course on the Whole Network: 03 Python Implementation of Classes and Objects – Adding Attributes to Custom Classes

The Most Beginner-Friendly Object Oriented Programming Course on the Whole Network: 04 Python Implementation of Classes and Objects - Adding Methods to Custom Classes

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 05 Python Implementation of Classes and Objects - PyCharm Code Labels

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 06 Python Implementation of Classes and Objects - Data Encapsulation of Custom Classes

The most beginner-friendly Object Oriented programming tutorial on the entire network: 07 Python implementation of classes and objects - type annotations

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 08 Python Implementation of Classes and Objects - @property Decorator

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 09 Python Implementation of Classes and Objects - Relationships Between Classes

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 10 Python Implementation of Classes and Objects - Class Inheritance and Liskov Substitution Principle

The Most Beginner-Friendly Object Oriented Programming Tutorial Across the Network: 11 Python Implementation of Classes and Objects - Subclass Calling Parent Class Methods

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 12 Python Implementation of Classes and Objects - Using Python's logging Module to Output Program Runtime Logs

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 13 Python Implementation of Classes and Objects – Installation and Usage of Sourcetrail, a Visual Code Reading Tool

The Most Beginner-Friendly Object Oriented Programming Course on the Internet: The Most Beginner-Friendly Object Oriented Programming Course on the Internet: 14 Python Implementation of Classes and Objects - Static Methods and Class Methods of Classes

The most beginner-friendly Object Oriented programming tutorial across the entire network: 15 Python implementation of classes and objects - the __slots__ magic method

The Most Beginner-Friendly Object Oriented Programming Tutorial Across the Internet: 16 Python Implementation of Classes and Objects - Polymorphism, Method Overriding and Open Closed Principle

The most beginner-friendly Object Oriented programming tutorial across the network: 17 Python implementation of classes and objects - duck typing and "file-like object"

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 18 Python Implementation of Classes and Objects - Multiple Inheritance and Drawing Curve Graphs of Serial Port Data with PyQtGraph

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 19 Python Implementation of Classes and Objects – Automatically Generate File Comments and Function Comments with PyCharm

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 20 Python Implementation of Classes and Objects - Implementation of Composition Relationship and CSV File Saving

The Most Beginner-Friendly Object Oriented Programming Tutorial Online: 21 Python Implementation of Classes and Objects - Multi-File Organization: Module and Package

The most beginner-friendly Object Oriented programming tutorial on the entire network: 22 Python implementation of classes and objects - exceptions and syntax errors

The Most Beginner-Friendly Object Oriented Programming Course on the Whole Network: 23 Python Implementation of Classes and Objects - Throwing Exceptions

The Most Beginner-Friendly Object Oriented Programming Tutorial Across the Network: 24 Python Implementation of Classes and Objects - Exception Capture and Handling

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 25 Python Implementation of Classes and Objects - Python Determines Input Data Types

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 26 Python Implementation of Classes and Objects - Context Managers and the with Statement

The Most Beginner-Friendly Object Oriented Programming Tutorial Across the Network: 27 Python Implementation of Classes and Objects - Implementation of Exception Hierarchy and Custom Exception Classes in Python

The Most Beginner-Friendly Object Oriented Programming Course on the Whole Network: 28 Python Implementation of Classes and Objects - A Comprehensive Summary of Python Programming Principles, Philosophy and Specifications

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 29 Python Implementation of Classes and Objects – Assertions, Defensive Programming and the Usage of the help function

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 30 Python's Built-in Data Types - The object Root Class

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 31 Python's Built-in Data Types - Object and Type

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 32 Python's Built-in Data Types - Class and Instance

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 33 Built-in Data Types in Python – The Relationship Between Object and Type

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 34 Python's Built-in Data Types - Commonly Used Composite Data Types in Python: Tuples and Named Tuples

The most beginner-friendly Object Oriented programming tutorial on the entire network: 35 Python's built-in data types - docstrings and the __doc__ attribute

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 36 Python's Built-in Data Type - Dictionary

The Most Beginner-Friendly Object Oriented Programming Course on the Whole Network: 37 Common Composite Data Types in Python - Lists and List Comprehensions

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 38 Common Composite Data Types in Python – Implementing stack, Queue and Deque with Lists

The Most Beginner-Friendly Object Oriented Programming Course on the Whole Network: 39 Common Composite Data Types in Python - Sets

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 40 Common Composite Data Types in Python – Usage of Enumerations and the enum Module

The Most Beginner-Friendly Object Oriented Programming Tutorial Across the Network: 41 Common Composite Data Types in Python - Queues (FIFO, LIFO, Priority Queue, Deque and Circular Queue)

The most beginner-friendly Object Oriented programming tutorial across the network: 42 Commonly used composite data types in Python - collections container data types

The Most Beginner-Friendly Object Oriented Programming Tutorial Across the Network: 43 Common Composite Data Types in Python - Extending Built-in Data Types

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 44 Python Built-in functions and Magic Methods – Overriding Magic Methods of Built-in Types

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 45 Implementing Common Data Structures in Python - Linked List, Tree, Hash Table, Graph and Heap

The most beginner-friendly Object Oriented programming tutorial across the network: 46 Python function methods and interfaces - functions and event-driven frameworks

The most beginner-friendly Object Oriented programming tutorial across the network: 47 Python function methods and interfaces - Callback function

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 48 Python function Methods and Interfaces - Positional Arguments, Default Arguments, Variable Arguments and Keyword Arguments

The Most Beginner-Friendly Object Oriented Programming Tutorial Across the Network: 49 Python function Methods and Interfaces – The Difference Between function and Methods, and Lambda Anonymous Functions

The most beginner-friendly Object Oriented programming tutorial on the entire network: 50 Python function methods and interfaces - Interfaces and Abstract Base Classes

The most beginner-friendly Object Oriented programming tutorial on the entire network: 51 Python function methods and interfaces - Implementing interfaces with Zope

The most beginner-friendly Object Oriented programming tutorial across the network: 52 Python function methods and interfaces - Protocol and interfaces

The most beginner-friendly Object Oriented programming tutorial across the entire network: 53 Python Strings and Serialization - Strings and Character Encodings

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 54 Python Strings and Serialization - String Formatting and the format Method

The most beginner-friendly Object Oriented programming tutorial on the entire network: 55 Python Strings and Serialization - Byte Sequence Types and Mutable Byte Strings

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 56 Python Strings & Serialization - Regular Expression and re Module Application

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 57 Python Strings and Serialization - Serialization and Deserialization

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 58 Python Strings and Serialization – Definition and Implementation of Serialized Web Objects

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 59 Python Parallelism and Concurrency - Parallelism & Concurrency and Threads & Processes

Master Python Parallel Computing in One Article: 00 Basic Concepts of Parallel Computing

A Quick Guide to Python Parallel Computing: 01 Python Multithreaded Programming – Basic Concepts, Switching Process, GIL Lock Mechanism, and Producer-Consumer Model

A Quick Guide to Python Parallel Computing: 02 Python Multithreaded Programming - threading Module, Thread Creation & Query, and Daemon Threads

A Quick Guide to Python Parallel Computing: 03 Python Multithreaded Programming - Multithread Synchronization (Part 1) — Based on Mutex Locks, Recursion Locks and Semaphores

A Quick Guide to Python Parallel Computing: 04 Python Multithreaded Programming - Multithread Synchronization (Part 2) — Based on Condition Variables, Events and Barriers

A Quick Guide to Python Parallel Computing: 05 Python Multithreaded Programming – Scheduled Execution of Threads

A Quick Guide to Python Parallel Computing: 06 Python Multithreaded Programming – Communication Based on Queues

A Quick Guide to Python Parallel Computing: 07 Python Multithreaded Programming – Usage of Thread Pools and Performance Evaluation of Multithreading

A Quick Guide to Python Parallel Computing: 08 Python Multiprocessing Programming – Process Creation & Naming, ID Retrieval, Daemon Process Setup and Process Termination

More exciting content can be found here:

Speed Up Your Python: A Quick Guide to Python Parallel Computing

Understand the Debugging Principle of CM3 Microcontroller in One Article

After half a month of grinding work, the ultimate summary of embedded technology stack is now released

The "Secret Martial Arts Manual" for Computer Science Competitions

An open-source project collection for MicroPython: awesome-micropython, which covers MicroPython tool libraries in various aspects

Avnet ZUBoard 1CG Development Board — A New Choice for Deep learning

Engineers should not blindly worship open-source code; they also need to focus on mastering fundamental skills.

What? A motor drive module with customizable color schemes? !

What? Three brand-new expansion boards have just been released for the XIAO main controller!

A Step-by-Step Guide to Publishing Your Third-Party Library for Arduino

A 10,000-Word Comprehensive Guide: Step-by-Step Instructions to Publish Third-Party Libraries for MicroPython/Python

[Electronic DIY Essential] Works with All 5-Wire Stepper Motors! Compatible with Seeed XIAO Series Development Boards

Master the Electronic Design Competition in One Read: Award-Winning Tips Every Electronics Student Must See

Master the Optoelectronic Design Competition in One Read: A Must-See for Electronics Enthusiasts!

A Guide to Engineering Competitions: A Step-by-Step Roadmap from Knowledge Learning, hardware Selection, Code Debugging to Report Writing

How to win the national award of Datang Cup with zero foundation? All the senior's competition preparation tips are here

Charging over 6,000 yuan for filming a single conference? Here are all the filming tips and procedures

Document Acquisition:

You can download the document via the following link:

https://github.com/leezisheng/Doc

This document is a learning guide on parallel computing and Python concurrent programming, which covers the core knowledge points including the basic concepts of parallel computing, Python multi-threaded programming, multi-process programming and coroutine programming:

10 1.1

Body Text

Data is not shared between processes except through shared memory and shared processes, but they share the same file system, so accessing the same file or the same printing terminal poses no problem. However, sharing leads to competition, and the result of competition is disorder, which requires synchronization between processes to control.

Process synchronization primitives include: Lock, Event, Condition, Semaphore, Rlock, and Barrier. The relevant synchronization primitives are quite similar to those in the threading library.

The following code shows that multiple processes share the same print terminal without synchronization. It runs concurrently with high efficiency, but the competition for the same print terminal leads to garbled print output.

from multiprocessing import Process
import os,time
def work():
    print('%s is running' %os.getpid())
    time.sleep(2)
    print('%s is done' %os.getpid())

if __name__ == '__main__':
    for i in range(3):
        p=Process(target=work)
        p.start()

The following is the running effect. It can be seen that the two statements that should have been output in turn by the three processes are now output out of order.

23476 26032 16172

1. Process Data Synchronization Based on Mutex Locks

As shown below work function utilizes acquire () and release () to control the read and write permissions of shared data:

from multiprocessing import Process,Lock
import os,time
def work(lock):
    lock.acquire()
    print('%s is running' %os.getpid())
    time.sleep(2)
    print('%s is done' %os.getpid())
    lock.release()
if __name__ == '__main__':
    lock=Lock()
    for i in range(3):
        p=Process(target=work,args=(lock,))
        p.start()
A Quick Guide to Python Parallel Computing: 09 Python Multiprocessing Programming – Data Synchronization Between Processes: Based on Mutex Locks, Recursion Locks, Semaphores, Condition Variables, Events and Barriers

It can be seen that the output is in order; the locking implementation changes the execution from concurrent to serial, which sacrifices runtime efficiency but avoids race conditions.

A Quick Guide to Python Parallel Computing: 09 Python Multiprocessing Programming – Data Synchronization Between Processes: Based on Mutex Locks, Recursion Locks, Semaphores, Condition Variables, Events and Barriers

2. Process Data Synchronization Based on Recursion Lock

The sample code is as follows:

from multiprocessing import Process,RLock
import multiprocessing
import time

mutex = RLock()

def test(mutex):
    if mutex.acquire():
        print("I am %s" % multiprocessing.current_process().name)
        if mutex.acquire():
            print("I am %s" % multiprocessing.current_process().name)
            mutex.release()
            time.sleep(1)
        mutex.release()

if __name__ == '__main__':
    for i in range(0, 10):
        p = Process(target=test,args=(mutex,))
        p.start()

The output is as follows, and it can be seen that the output is normal:

Process-1 Process-10
A Quick Guide to Python Parallel Computing: 09 Python Multiprocessing Programming – Data Synchronization Between Processes: Based on Mutex Locks, Recursion Locks, Semaphores, Condition Variables, Events and Barriers

3. Process Data Synchronization Based on Semaphores

The following is sample code, and we can see from the running results that the two print statements output sequentially.

from multiprocessing import Process,Semaphore
import multiprocessing
import time

# Create a semaphore; a semaphore is an internal counter
# used to indicate how many concurrent reads the current shared resource allows
semaphore = Semaphore(1)

def test(semaphore):
    # Test the semaphore that controls this resource.
    if semaphore.acquire():
        # If the semaphore value is positive, use of the resource is allowed. The thread decrements the semaphore by 1.
        print("I am %s,pid:%s" % (multiprocessing.current_process().name,multiprocessing.current_process().pid))
        print("%s am runing" % multiprocessing.current_process().name)
        semaphore.release()

if __name__ == '__main__':
    for i in range(0, 10):
        p = Process(target=test,args=(semaphore,))
        p.start()
Process-1 1 10

4. Process Data Synchronization Based on Condition Variables

Here, we take the producer-consumer model as an example. In the previous multithreading example of thread synchronization based on condition variables, we used global variables as the cache, and we do the same in this example.

Some students may ask: Didn't we say that data within processes is independent? That's correct. In multi-process programming, different processes cannot share data by default. However, Python provides several mechanisms to achieve data sharing between multiple processes, one of which is shared memory. Shared memory allows multiple processes to share a single storage region; when one process writes information into the shared memory, other processes can read it conveniently.In Python, you can use Value, Array to store data in shared memory, and you can also use the multiprocessing module's sharedctypes to customize shared-memory ctypes objects.

A Quick Guide to Python Parallel Computing: 09 Python Multiprocessing Programming – Data Synchronization Between Processes: Based on Mutex Locks, Recursion Locks, Semaphores, Condition Variables, Events and Barriers

In the following example, the value object is used as a cache: as long as the cache is not full, the producer keeps producing data into it; as long as the cache is not empty, the consumer keeps taking data out of it (and then destroys the taken data).

When the buffer queue is not empty, the producer will notify the consumer; when the buffer queue is not full, the consumer will notify the producer.

from multiprocessing import Process,Condition,Value
import multiprocessing
import time

condition = Condition()
products = Value('i', 0)

# Producer process
def Producer(condition,products):
    while True:
        if condition.acquire():
        # The consumer modifies the shared resource by acquiring the lock
            if products.value < 10:
            # If the number of products is less than 10, keep producing, and notify consumers with the notify method
            # As long as the cache is not full, the producer keeps producing into the cache;
                products.value += 1;
                print("Producer(%s):deliver one, now products:%s" % (multiprocessing.current_process().name, products.value))
                # When the buffer queue is not empty, the producer will notify the consumers
                condition.notify()
            else:
            # If it is full, the producer enters a waiting state until it is woken up
                print("Producer(%s):already 10, stop deliver, now products:%s" %(multiprocessing.current_process().name, products.value))
                condition.wait()
            # Release the resource
            condition.release()
            time.sleep(1)

# Consumer process
def Consumer(condition,products):
    while True:
        if condition.acquire():
            if products.value > 1:
            # As long as the buffer is not empty, the consumer keeps taking items from it (then destroys them)
                products.value -= 1
                print("Consumer(%s):consume one, now products:%s" % (multiprocessing.current_process().name, products.value))
                # When the buffer queue is not full, the consumer will notify the producer.
                condition.notify()
            else:
            # The cache is empty, the consumer thread waits
                print("Consumer(%s):only 1, stop consume, products:%s" % (multiprocessing.current_process().name, products.value))
                condition.wait()
            # Release the resource
            condition.release()
            time.sleep(1)

if __name__ == '__main__':
    ProducerProcess = Process(target=Producer,args=(condition,products))
    ConsumerProcess = Process(target=Consumer, args=(condition, products))
    ProducerProcess.start()
    ConsumerProcess.start()

The following shows the running status, which indicates normal operation.

1 2

5. Event-based Process Data Synchronization

Sample code and execution result are shown below, which is similar to that in multi-threading scenarios.

from multiprocessing import Process,Event
import multiprocessing

event = Event()

def worker(event_obj, i):
    print('{i}号进程等待事件信号'.format(i=i))
    event_obj.wait()
    print('{i}号进程收到事件信号'.format(i=i))

if __name__ == '__main__':
    for i in range(5):
        p = Process(target=worker, args=(event, i))
        p.start()
    event.set()
0 1 2 3 4

6. Barrier-based process data synchronization

The following code demonstrates how to use the barrier () function to synchronize two processes. We have 4 processes, where processes 1 and 2 are managed by the barrier statement, while processes 3 and 4 have no synchronization policy.

import multiprocessing
from multiprocessing import Barrier, Lock, Process
import time
from datetime import datetime

def test_with_barrier(synchronizer, serializer):
    name = multiprocessing.current_process().name
    synchronizer.wait()
    now = time.time()
    time.sleep(1)
    with serializer:
        print("process %s ----> %s" % (name, datetime.fromtimestamp(now)))

def test_without_barrier():
    name = multiprocessing.current_process().name
    now = time.time()
    print("process %s ----> %s" % (name, datetime.fromtimestamp(now)))

if __name__ == '__main__':
    synchronizer = Barrier(2)
    serializer = Lock()
    Process(name='p1 - test_with_barrier', target=test_with_barrier, args=(synchronizer,serializer)).start()
    Process(name='p2 - test_with_barrier', target=test_with_barrier, args=(synchronizer,serializer)).start()
    Process(name='p3 - test_without_barrier', target=test_without_barrier).start()
    Process(name='p4 - test_without_barrier', target=test_without_barrier).start()

Run the code, and you will see that Process 1 and Process 2 print at the same time:

1 2 p1 2024-01-20 11 53 35.831692 p2 3 4

The following diagram illustrates how a barrier synchronizes two processes:

P1 P2