Labs / Python parallel computing

Master Python Parallel Computing in One Read: 10 Python Multiprocessing Programming - Data Sharing Between Processes - Based on Shared Memory and Data Managers

Author: Li Zisheng · Source: FreakStudio open documentation · License: CC BY 4.0

Contents
Master Python Parallel Computing in One Read: 10 Python Multiprocessing Programming - Data Sharing Between Processes - Based on Shared Memory and Data Managers

Abstract:

In Python inter-process communication, shared memory enables efficient data sharing via Value and Array: it directly manipulates physical memory to avoid data copying, requires synchronization locks for thread safety, and supports numeric values, arrays and custom structs (with the help of the ctypes module). By contrast, the Manager manages shared objects (such as dict and list) through a service process, which supports more data types but comes with lower performance. Shared memory is suitable for high-frequency small-data scenarios, while the Manager applies to complex data structures; both require a process synchronization mechanism to ensure data consistency, striking a balance between efficiency and flexibility.

Master Python Parallel Computing in One Read: 10 Python Multiprocessing Programming - Data Sharing Between Processes - Based on Shared Memory and Data Managers

For more information about us, you can refer to the cloud document: Freak Embedded Studio Cloud Docs, or visit our wiki: https://github.com/leezisheng/Doc/wik

FreakStudio's Blog

Previous Recommendations:

You who study embedded systems still don't know Object Oriented? ?!

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 00 Introduction to Object Oriented Design Methodology

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 01 Basic Concepts of Object Oriented Programming

The Most Beginner-Friendly Object Oriented Programming Tutorial Across the Web: 02 Python Implementation of Classes and Objects - Creating Classes with Python

The Most Beginner-Friendly Object Oriented Programming Tutorial Across the Network: 03 Python Implementation of Classes and Objects – Adding Attributes to Custom Classes

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 04 Python Implementation of Classes and Objects - Adding Methods to Custom Classes

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 05 Python Implementation of Classes and Objects - PyCharm Code Tags

The most beginner-friendly Object Oriented programming tutorial on the entire network: 06 Python implementation of classes and objects - data encapsulation of custom classes

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 07 Python Implementation of Classes and Objects - Type Annotations

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 08 Python Implementation of Classes and Objects - @property Decorator

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 09 Python Implementation of Classes and Objects - Relationships Between Classes

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 10 Python Implementation of Classes and Objects - Class Inheritance and Liskov Substitution Principle

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 11 Python Implementation of Classes and Objects - Subclass Calling Superclass Methods

The Most Beginner-Friendly Object Oriented Programming Tutorial Across the Network: 12 Python Implementation of Classes and Objects - Using Python's logging Module to Output Program Runtime Logs

The Best Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 13 Python Implementation of Classes and Objects – Installation and Usage of Sourcetrail, a Visual Code Reading Tool

The Most Beginner-Friendly Object Oriented Programming Course on the Whole Network: The Most Beginner-Friendly Object Oriented Programming Course on the Whole Network: 14 Python Implementation of Classes and Objects - Static Methods and Class Methods of Classes

The most beginner-friendly Object Oriented programming tutorial on the entire network: 15 Python implementation of classes and objects - the __slots__ magic method

The Most Beginner-Friendly Object Oriented Programming Tutorial Across the Network: 16 Python Implementation of Classes and Objects - Polymorphism, Method Overriding and Open Closed Principle

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 17 Python Implementation of Classes and Objects - Duck Typing and "file-like object"

The Most Beginner-Friendly Object Oriented Programming Course on the Whole Network: 18 Python Implementation of Classes and Objects - Multiple Inheritance and Serial Port Data Curve Plotting with PyQtGraph

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 19 Python Implementation of Classes and Objects – Using PyCharm to Automatically Generate File Comments and function Comments

The Most Beginner-Friendly Object Oriented Programming Course on the Whole Network: 20 Python Implementation of Classes and Objects – Implementation of Composition Relationship and CSV File Saving

The Most Beginner-Friendly Object Oriented Programming Course on the Whole Network: 21 Python Implementation of Classes and Objects - Organization of Multiple Files: Modules and Packages

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 22 Python Implementation of Classes and Objects - Exceptions and Syntax Errors

The Most Beginner-Friendly Object Oriented Programming Tutorial Across the Network: 23 Python Implementation of Classes and Objects - Throwing Exceptions

The Best Beginner-Friendly Object Oriented Programming Course on the Whole Network: 24 Python Implementation of Classes and Objects - Exception Catching and Handling

The Most Beginner-Friendly Object Oriented Programming Course on the Whole Network: 25 Python Implementation of Classes and Objects - Python Determines the Type of Input Data

The most beginner-friendly Object Oriented programming tutorial on the entire network: 26 Python implementation of classes and objects - context managers and the with statement

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 27 Python Implementation of Classes and Objects - Implementation of Exception Hierarchy and Custom Exception Classes in Python

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 28 Python Implementation of Classes and Objects - A Comprehensive Summary of Python Programming Principles, Philosophy and Specifications

The Most Beginner-Friendly Object Oriented Programming Tutorial Across the Network: 29 Python Implementation of Classes and Objects – Assertions, Defensive Programming and the Usage of the help function

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 30 Python's Built-in Data Types - The object Root Class

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 31 Python's Built-in Data Types - Object and Type

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 32 Python's Built-in Data Types - Class and Instance

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 33 Python's Built-in Data Types - The Relationship Between Object and Type

The Most Beginner-Friendly Object Oriented Programming Course on the Whole Network: 34 Python's Built-in Data Types - Commonly Used Composite Data Types in Python: Tuples and Named Tuples

The most beginner-friendly Object Oriented programming tutorial across the network: 35 Python's built-in data types - docstrings and the __doc__ attribute

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 36 Python's Built-in Data Type - Dictionary

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 37 Common Composite Data Types in Python - Lists and List Comprehensions

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 38 Common Composite Data Types in Python - Implementing stack, Queues and Deques with Lists

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 39 Common Composite Data Types in Python - Sets

The most beginner-friendly Object Oriented programming tutorial across the entire network: 40 Common composite data types in Python - enumerations and the usage of the enum module

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 41 Common Composite Data Types in Python - Queues (FIFO, LIFO, Priority Queues, Deques and Circular Queues)

The most beginner-friendly Object Oriented programming tutorial on the entire network: 42 Commonly used composite data types in Python - collections container data types

The Most Beginner-Friendly Object Oriented Programming Tutorial Across the Network: 43 Common Composite Data Types in Python - Extending Built-in Data Types

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 44 Python Built-in functions and Magic Methods – Overriding Magic Methods of Built-in Types

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 45 Implementing Common Data Structures in Python - Linked List, Tree, Hash Table, Graph and Heap

The Most Beginner-Friendly Object Oriented Programming Course on the Whole Network: 46 Python function Methods and Interfaces - function and Event-Driven Frameworks

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 47 Python function Methods and Interfaces - Callback Function

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 48 Python function Methods and Interfaces - Positional Arguments, Default Arguments, Variable Arguments and Keyword Arguments

The Most Beginner-Friendly Object Oriented Programming Tutorial Across the Network: 49 Python function Methods and Interfaces - The Difference Between function and Methods and Lambda Anonymous function

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 50 Python function Methods and Interfaces - Interfaces and Abstract Base Classes

The most beginner-friendly Object Oriented programming tutorial across the network: 51 Python function methods and interfaces - Implementing interfaces with Zope

The most beginner-friendly Object Oriented programming tutorial on the entire network: 52 Python function methods and interfaces - Protocol and interfaces

The most beginner-friendly Object Oriented programming tutorial across the entire network: 53 Python Strings and Serialization - Strings and Character Encodings

The Most Beginner-Friendly Object Oriented Programming Course on the Whole Network: 54 Python Strings and Serialization - String Formatting and the format Method

The most beginner-friendly Object Oriented programming tutorial on the entire network: 55 Python Strings and Serialization - Byte Sequence Types and Mutable Byte Strings

The most beginner-friendly Object Oriented programming tutorial on the entire network: 56 Python Strings and Serialization - Applications of Regular Expression and the re Module

The most beginner-friendly Object Oriented programming tutorial on the entire network: 57 Python Strings and Serialization - Serialization and Deserialization

The most beginner-friendly Object Oriented programming tutorial on the entire network: 58 Python Strings and Serialization – Definition and Implementation of Serialized Web Objects

The Most Beginner-Friendly Object Oriented Programming Course on the Entire Internet: 59 Python Parallelism and Concurrency – Parallelism, Concurrency, Threads and Processes

Master Python Parallel Computing in One Article: 00 Basic Concepts of Parallel Computing

A Quick Guide to Python Parallel Computing: 01 Python Multithreaded Programming – Basic Concepts, Switching Process, GIL Lock Mechanism, and Producer-Consumer Model

A Quick Guide to Python Parallel Computing: 02 Python Multithreaded Programming — the threading Module, Thread Creation & Query, and Daemon Threads

A Quick Guide to Python Parallel Computing: 03 Python Multithreaded Programming - Multithread Synchronization (Part 1) — Based on Mutex Locks, Recursion Locks and Semaphores

A Quick Guide to Python Parallel Computing: 04 Python Multithreaded Programming – Multithread Synchronization (Part 2) — Based on Condition Variables, Events and Barriers

A Quick Guide to Python Parallel Computing: 05 Python Multithreaded Programming – Scheduled Thread Execution

A Quick Guide to Python Parallel Computing: 06 Python Multithreaded Programming – Communication Based on Queues

A Quick Guide to Python Parallel Computing: 07 Python Multithreaded Programming – Using Thread Pools and Performance Evaluation of Multithreading

A Quick Guide to Python Parallel Computing: 08 Python Multiprocessing Programming – Process Creation & Naming, ID Retrieval, Daemon Process Setup and Process Termination

A Quick Guide to Python Parallel Computing: 09 Python Multiprocessing Programming – Data Synchronization Between Processes: Based on Mutex Locks, Recursion Locks, Semaphores, Condition Variables, Events and Barriers

For more exciting content, you can check out:

C Language 1.5 Programming Practice: Pure C Modularization × Inheritance × Polymorphism Framework

Speed Up Your Python: A Quick Guide to Python Parallel Computing

Mastering the Debugging Principles of CM3 Microcontrollers in One Article

After half a month of dedicated grinding, the ultimate roundup of embedded technology stack is now ready

The "Secret Martial Arts Manual" for Computer Science Competitions

An open-source project collection for MicroPython: awesome-micropython, which covers MicroPython tool libraries in various aspects

Avnet ZUBoard 1CG Development Board — A New Choice for Deep learning

Engineers should not blindly worship open-source code, but also pay attention to mastering basic skills

What? A motor drive module with customizable color schemes? !

What? Three new expansion boards just released for the XIAO main controller!

A Step-by-Step Guide to Publishing Third-Party Libraries for Arduino

A 10,000-Word Detailed Guide: Step-by-Step Tutorial on Publishing Third-Party Libraries for MicroPython/Python

【Electronic DIY Magic Tool】Compatible with All Kinds of 5-Wire Stepper Motors! Supports seeed XIAO Series Development Boards

Master the Electronic Design Contest in One Read: The Must-Know Winning Guide for Electronics Enthusiasts

Master the Optoelectronic Design Competition in One Read: A Must-See for Electronics Enthusiasts!

"Brainless" Operation Guide for Engineering Competitions: A Step-by-Step Roadmap from Knowledge Learning, hardware Selection, Code Debugging to Report Writing

How to Win the National Award of the Datang Cup with Zero Foundation? All the Senior's Competition Preparation Tips Are Here

【Extensively Compiled In-Depth Guide】College Student Competition Crash Course: Topic Selection × Team Formation × Roadshow, Get Competition-Ready in 48 Hours

Chaos in Contemporary College Students' Competitions: A Survival Guide from "Involution" to "Heirloom Projects"

Charging over 6,000 yuan for a single conference shoot? Here are all the shooting tips and steps

A female college student's on-site research on losing 5,000 yuan from a street stall: Where in Chengdu has the largest crowd and is easiest to make money? I'll tell you!

Making Money with Polaroid Cameras at Chengdu’s Popular Check-in Spots: A Super Practical Street-Vendor-Level Side Hustle Guide for Tourists

Document Acquisition:

You can download the document via the following link:

https://github.com/leezisheng/Doc

This document is a learning guide on parallel computing and Python concurrent programming, which covers the basic concepts of parallel computing, as well as the core knowledge points of Python multi-threaded programming, multi-process programming and coroutine programming:

Master Python Parallel Computing in One Read: 10 Python Multiprocessing Programming - Data Sharing Between Processes - Based on Shared Memory and Data Managers

Main Text

1. Shared Memory

Shared memory, as the name suggests, allows two unrelated processes to access the same logical memory. It is a highly efficient way for two running processes to share and transfer data between each other. The memory shared between different processes usually refers to the same segment of physical memory. Processes can attach this same physical memory segment to their own address space, so that all processes can access the addresses within the shared memory. If a process writes data to the shared memory, the changes made will immediately affect any other process that has access to this same shared memory segment.

Master Python Parallel Computing in One Read: 10 Python Multiprocessing Programming - Data Sharing Between Processes - Based on Shared Memory and Data Managers

Its features include:

Although shared memory enables fast data transfer between processes, it has the drawback that the content in the memory may be overwritten or tampered with by other processes, so inter-process synchronization methods are required to ensure data security.

multiprocessing provides Value and Array, which allow data to be stored in a shared memory map for access by all processes.

1.1 Shared Memory - Numeric Type

First, let's take a look at the data format of value:

Value(typecode_or_type, *args, lock=True)  

Function:

The sample code is shown below, and you can see that when the child process modifies the value, the value in the parent process changes accordingly:

import multiprocessing

def func(num):
    num.value = 11

if __name__ == "__main__":
    # i denotes an integer; the main process and child process share this value.
    num = multiprocessing.Value("i", 10)
    print(num.value)

    p = multiprocessing.Process(target=func, args=(num,))
    p.start()
    p.join()

    print(num.value)
Master Python Parallel Computing in One Read: 10 Python Multiprocessing Programming - Data Sharing Between Processes - Based on Shared Memory and Data Managers

1.2 Shared Memory - Array Type

Array Data Format:

Array(typecode_or_type, size_or_initializer, *, lock=True) 

Function:

The parameters of the Value and Array data storage types are as follows (partial):

Master Python Parallel Computing in One Read: 10 Python Multiprocessing Programming - Data Sharing Between Processes - Based on Shared Memory and Data Managers

The sample code is as follows, from which you can see that when the child process modifies the elements of the array, the main process will also see the corresponding changes:

import multiprocessing

def func(num):
    num[2] = 9999  # The child process modifies the array, and the main process sees the change

if __name__ == "__main__":
    num = multiprocessing.Array("i", [1, 2, 3, 4, 5])  # The main process and child process share this array
    print(num[:])

    p = multiprocessing.Process(target=func, args=(num,))
    p.start()
    p.join()

    print(num[:])
Master Python Parallel Computing in One Read: 10 Python Multiprocessing Programming - Data Sharing Between Processes - Based on Shared Memory and Data Managers

1.3 Custom Data Structures

Here we use the classes in ctypes to define structures. The class used to define a structure needs to inherit from the ctypes base class Structure, and then the composition of the structure is defined by defining the _fields_ attribute of the class. The _fields_ attribute is generally defined as a two-dimensional tuple, and for each one-dimensional tuple in it, two values need to be defined: the first value is a string, which serves as the variable name inside the structure, and the second value is a ctypes type, which is used to define the data type of the current structure variable.Note that for structs defined in Python, their variable names, class names and the like can differ from those in C language, but the number, data types and order of struct variables must strictly correspond to the definitions in the C source code; otherwise, memory access errors may occur.

The code is as follows:

from multiprocessing import Process, Lock
from multiprocessing.sharedctypes import Value, Array
from ctypes import Structure, c_double

# Custom data structure
class Point(Structure):
    # Defines a structure type, equivalent to a struct declaration in C.
    # This structure defines two structure variables: x corresponds to an int type, y corresponds to a double type.
    _fields_ = [('x', c_double), ('y', c_double)]

def modify(n, x, s, A):
    n.value **= 2
    x.value **= 2
    s.value = s.value.upper()
    for a in A:
        a.x **= 2
        a.y **= 2

if __name__ == '__main__':
    lock = Lock()

    n = Value('i', 7)  # Define an int
    x = Value(c_double, 1.0/3.0, lock=False)  # Define a ctypes type
    s = Array('c', b'hello world', lock=lock)  # Define a string
    A = Array(Point, [(1.875, -6.25), (-5.75, 2.0), (2.375, 9.5)], lock=lock)  
# Define an array of Point type

    p = Process(target=modify, args=(n, x, s, A))
    p.start()
    p.join()

    print(n.value)
    print(x.value)
    print(s.value)
    print([(a.x, a.y) for a in A])
Master Python Parallel Computing in One Read: 10 Python Multiprocessing Programming - Data Sharing Between Processes - Based on Shared Memory and Data Managers

2. Data Manager

Apart from the shared memory approach, another method for inter-process object sharing is to utilize a service process:

The manager object returned by Manager () controls a service process, which holds Python objects and allows other processes to manipulate them via proxies.(Note: Manual locking is required for Manager.)

Managers have the following characteristics:

Manager () The types supported by the returned manager are as follows:

list, dict, Namespace, Lock, RLock, Semaphore, BoundedSemaphore, Condition, Event, Barrier, Queue, Value and Array.

The sample code is as follows: the main process defines a dictionary and a list. In the child process, you can add and modify the content of the dictionary, insert new data into the list, so as to realize data sharing between processes, that is, multiple processes can modify the same piece of data together.

from multiprocessing import Process, Manager

def fun1(dic,lis,index):
    dic[index] = 'a'
    dic['2'] = 'b'
    lis.append(index)

if __name__ == '__main__':
    with Manager() as manager:
        # Note how the dictionary is declared; it cannot be defined directly with {}
        dic = manager.dict()
        # [0,1,2,3,4]
        l = manager.list(range(5))

        process_list = []
        for i in range(10):
            p = Process(target=fun1, args=(dic,l,i))
            p.start()
            process_list.append(p)

        for res in process_list:
            res.join()
        print(dic)
        print(l)
Master Python Parallel Computing in One Read: 10 Python Multiprocessing Programming - Data Sharing Between Processes - Based on Shared Memory and Data Managers