Labs / Python parallel computing

A Quick Guide to Python Parallel Computing: 06 Python Multithreaded Programming – Communication Based on Queues

Author: Li Zisheng · Source: FreakStudio open documentation · License: CC BY 4.0

Contents
A Quick Guide to Python Parallel Computing: 06 Python Multithreaded Programming – Communication Based on Queues

Abstract:

A queue is a linear data structure that supports First-In-First-Out (FIFO) operations and is commonly used to decouple producers and consumers. In scenarios with slow production and fast consumption, queues act as buffers to balance the speed discrepancy. LifoQueue is a stack-style queue that follows the Last-In-First-Out (LIFO) principle, making it suitable for use cases such as undo operations. PriorityQueue, on the other hand, sorts elements by priority, which fits scenarios requiring preferential processing like task scheduling.These three types of queues are implemented via the `Queue`, `LifoQueue` and `PriorityQueue` classes respectively, which provide methods such as `put ()` and `get ()`, and are important thread-safe tools in concurrent programming.

347622978

For more information about us, you can refer to the cloud document: Freak Embedded Studio Cloud Docs, or visit our wiki: https://github.com/leezisheng/Doc/wik

FreakStudio's Blog

Previous Recommendations:

You who are learning embedded systems, don’t you even know Object Oriented yet? ?!

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 00 Introduction to Object Oriented Design Methodology

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 01 Basic Concepts of Object Oriented Programming

The Most Beginner-Friendly Object Oriented Programming Course on the Whole Network: 02 Python Implementation of Classes and Objects - Creating Classes with Python

The Most Beginner-Friendly Object Oriented Programming Course on the Whole Network: 03 Python Implementation of Classes and Objects – Adding Attributes to Custom Classes

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 04 Python Implementation of Classes and Objects - Adding Methods to Custom Classes

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 05 Python Implementation of Classes and Objects - PyCharm Code Tab

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 06 Python Implementation of Classes and Objects - Data Encapsulation of Custom Classes

The most beginner-friendly Object Oriented programming tutorial across the entire network: 07 Python implementation of classes and objects - type annotations

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 08 Python Implementation of Classes and Objects - @property Decorator

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 09 Python Implementation of Classes and Objects - Relationships Between Classes

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 10 Python Implementation of Classes and Objects - Class Inheritance and Liskov Substitution Principle

The Most Beginner-Friendly Object Oriented Programming Tutorial Across the Network: 11 Python Implementation of Classes and Objects - Subclass Calling Parent Class Methods

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 12 Python Implementation of Classes and Objects - Using Python's logging Module to Output Program Runtime Logs

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 13 Python Implementation of Classes and Objects – Installation and Usage of Sourcetrail, a Visual Code Reading Tool

The Most Beginner-Friendly Object Oriented Programming Course on the Internet: The Most Beginner-Friendly Object Oriented Programming Course on the Internet: 14 Python Implementation of Classes and Objects - Static Methods and Class Methods of Classes

The most beginner-friendly Object Oriented programming tutorial across the entire network: 15 Python implementation of classes and objects - the __slots__ magic method

The Most Beginner-Friendly Object Oriented Programming Tutorial Across the Internet: 16 Python Implementation of Classes and Objects - Polymorphism, Method Overriding and Open Closed Principle

The most beginner-friendly Object Oriented programming tutorial across the network: 17 Python implementation of classes and objects - duck typing and "file-like object"

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 18 Python Implementation of Classes and Objects - Multiple Inheritance and Drawing Curve Graphs of Serial Port Data with PyQtGraph

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 19 Python Implementation of Classes and Objects – Using PyCharm to Automatically Generate File Comments and function Comments

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 20 Python Implementation of Classes and Objects - Implementation of Composition Relationship and CSV File Saving

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 21 Python Implementation of Classes and Objects - Multi-File Organization: Module and Package

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 22 Python Implementation of Classes and Objects - Exceptions and Syntax Errors

The Most Beginner-Friendly Object Oriented Programming Course on the Whole Network: 23 Python Implementation of Classes and Objects - Throwing Exceptions

The Most Beginner-Friendly Object Oriented Programming Tutorial Across the Network: 24 Python Implementation of Classes and Objects - Exception Capture and Handling

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 25 Python Implementation of Classes and Objects - Python Determines Input Data Types

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 26 Python Implementation of Classes and Objects - Context Managers and the with Statement

The Most Beginner-Friendly Object Oriented Programming Tutorial Across the Network: 27 Python Implementation of Classes and Objects - Implementation of Exception Hierarchy and Custom Exception Classes in Python

The Most Beginner-Friendly Object Oriented Programming Course on the Whole Network: 28 Python Implementation of Classes and Objects - A Comprehensive Summary of Python Programming Principles, Philosophy and Specifications

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 29 Python Implementation of Classes and Objects – Assertions, Defensive Programming and the Usage of the help function

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 30 Python's Built-in Data Types - The object Root Class

The most beginner-friendly Object Oriented programming tutorial across the entire network: 31 Built-in data types in Python - Object and Type

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 32 Python's Built-in Data Types - Class and Instance

The most beginner-friendly Object Oriented programming tutorial on the entire network: 33 Built-in data types in Python – the relationship between objects and types

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 34 Python's Built-in Data Types - Commonly Used Composite Data Types in Python: Tuples and Named Tuples

The most beginner-friendly Object Oriented programming tutorial on the entire network: 35 Python's built-in data types - docstrings and the __doc__ attribute

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 36 Python's Built-in Data Type - Dictionary

The Most Beginner-Friendly Object Oriented Programming Course on the Whole Network: 37 Common Composite Data Types in Python - Lists and List Comprehensions

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 38 Common Composite Data Types in Python – Implementing stack, Queue and Deque with Lists

The Most Beginner-Friendly Object Oriented Programming Course on the Whole Network: 39 Common Composite Data Types in Python - Sets

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 40 Common Composite Data Types in Python – Usage of Enumerations and the enum Module

The Most Beginner-Friendly Object Oriented Programming Tutorial Across the Network: 41 Common Composite Data Types in Python - Queues (FIFO, LIFO, Priority Queue, Deque and Circular Queue)

The most beginner-friendly Object Oriented programming tutorial across the network: 42 Commonly used composite data types in Python - collections container data types

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 43 Common Composite Data Types in Python - Extending Built-in Data Types

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 44 Python Built-in functions and Magic Methods – Overriding Magic Methods of Built-in Types

The Most Beginner-Friendly Object Oriented Programming Course on the Whole Network: 45 Implementing Common Data Structures in Python – Linked List, Tree, Hash Table, Graph and Heap

The most beginner-friendly Object Oriented programming tutorial across the network: 46 Python function methods and interfaces - functions and event-driven frameworks

The most beginner-friendly Object Oriented programming tutorial across the network: 47 Python function methods and interfaces - Callback function

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 48 Python function Methods and Interfaces - Positional Arguments, Default Arguments, Variable Arguments and Keyword Arguments

The Most Beginner-Friendly Object Oriented Programming Tutorial Across the Network: 49 Python function Methods and Interfaces – The Difference Between function and Methods, and Lambda Anonymous Functions

The most beginner-friendly Object Oriented programming tutorial on the entire network: 50 Python function methods and interfaces - Interfaces and Abstract Base Classes

The most beginner-friendly Object Oriented programming tutorial on the entire network: 51 Python function methods and interfaces - Implementing interfaces with Zope

The most beginner-friendly Object Oriented programming tutorial across the network: 52 Python function methods and interfaces - Protocol and interfaces

The most beginner-friendly Object Oriented programming tutorial across the entire network: 53 Python Strings and Serialization - Strings and Character Encodings

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 54 Python Strings and Serialization - String Formatting and the format Method

The most beginner-friendly Object Oriented programming tutorial on the entire network: 55 Python Strings and Serialization - Byte Sequence Types and Mutable Byte Strings

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 56 Python Strings & Serialization - Regular Expression and re Module Application

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 57 Python Strings and Serialization - Serialization and Deserialization

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 58 Python Strings and Serialization – Definition and Implementation of Serialized Web Objects

The Most Beginner-Friendly Object Oriented Programming Tutorial on the Whole Network: 59 Python Parallelism and Concurrency - Parallelism & Concurrency and Threads & Processes

Master Python Parallel Computing in One Article: 00 Basic Concepts of Parallel Computing

A Quick Guide to Python Parallel Computing: 01 Python Multithreaded Programming – Basic Concepts, Switching Process, GIL Lock Mechanism, and Producer-Consumer Model

A Quick Guide to Python Parallel Computing: 02 Python Multithreaded Programming - threading Module, Thread Creation & Query, and Daemon Threads

A Quick Guide to Python Parallel Computing: 03 Python Multithreaded Programming - Multithread Synchronization (Part 1) — Based on Mutex Locks, Recursion Locks and Semaphores

A Quick Guide to Python Parallel Computing: 04 Python Multithreaded Programming - Multithread Synchronization (Part 2) — Based on Condition Variables, Events and Barriers

A Quick Guide to Python Parallel Computing: 05 Python Multithreaded Programming – Scheduled Execution of Threads

More exciting content can be found here:

Speed Up Your Python: A Quick Guide to Python Parallel Computing

Understand the Debugging Principle of CM3 Microcontroller in One Article

After half a month of dedicated work, the ultimate summary of embedded technology stack is now released

The "Martial Arts Manual" for Computer Science Competitions

An open-source project collection for MicroPython: awesome-micropython, which covers MicroPython tool libraries in various aspects

Avnet ZUBoard 1CG Development Board — A New Choice for Deep learning

Engineers should not blindly worship open-source code; they also need to focus on mastering fundamental skills.

What? A motor drive module with customizable color schemes? !

What? Three brand-new expansion boards have just been released for the XIAO main controller!

A Step-by-Step Guide to Publishing Your Third-Party Library for Arduino

A 10,000-Word Comprehensive Guide: Step-by-Step Instructions to Publish Third-Party Libraries for MicroPython/Python

Document Acquisition:

You can download the document via the following link:

https://github.com/leezisheng/Doc

This document is a learning guide on parallel computing and Python concurrent programming, which covers the core knowledge points including the basic concepts of parallel computing, Python multi-threaded programming, multi-process programming and coroutine programming:

A Quick Guide to Python Parallel Computing: 06 Python Multithreaded Programming – Communication Based on Queues

Body Text

1. Basic Concepts and Applications of Queues

A queue is a First-In-First-Out (FIFO) linear list, which is usually implemented with a linked list or an array in practical applications. A queue only allows insertion operations at the rear end (called rear) and deletion operations at the front end (called front). The operation mode of a queue is similar to that of a stack, and the only difference is that a queue only allows new data to be added at the rear end.In Python, queues can be imported via the built-in module queue, and the specific import method is: from queue import Queue, The queue module provides a FIFO data structure suitable for multi-threaded programming, which can be used to safely pass messages or data between producer and consumer threads.

A Quick Guide to Python Parallel Computing: 06 Python Multithreaded Programming – Communication Based on Queues

Queue has the following four commonly used methods:

Here we take the producer-consumer model as an example, where producers send products to a queue and consumers receive products from the queue. The code is as follows:

from threading import Thread, Event
from queue import Queue
import time
import random
class producer(Thread):

    # First, we create a producer class.
    # Since we use a queue to hold the numbers, we no longer need a list to store them.
    def __init__(self, queue):
        Thread.__init__(self)
        self.queue = queue

    def run(self) :
        # The producer class produces integers, then puts them into the queue in a for loop
        for i in range(10):
            item = random.randint(0, 256)
            # The producer uses Queue.put(item [,block[, timeout]]) to insert data into the queue.
            # Queue is synchronized; it has a built-in locking mechanism before inserting data.
            self.queue.put(item)
            # Two situations can occur:
            # (1) If block is True and timeout is None:
            #   It may block until a free slot is available. If timeout is a positive integer,
            #   it blocks until that time and then raises an exception.
            # (2) If block is False:
            #   If the queue has a free slot, the item is inserted immediately; otherwise an exception is raised immediately (timeout is ignored).
            #   In this example, put() checks whether the queue is full, then calls wait() to start waiting.
            print('Producer notify: item N° %d appended to queue by %s' % (item, self.name))
            time.sleep(1)


class consumer(Thread):
    def __init__(self, queue):
        Thread.__init__(self)
        self.queue = queue


    def run(self):
        while True:
            # The consumer uses Queue.get([block[, timeout]]) to retrieve data from the queue,
            # The queue also uses locking internally. If the queue is empty, the consumer blocks.
            item = self.queue.get()
            print('Consumer notify : %d popped from queue by %s' % (item, self.name))
            # The consumer takes an integer from the queue, then marks it as processed with the task_done() method.
            self.queue.task_done()


if __name__ == '__main__':
    queue = Queue()
    t1 = producer(queue)
    t2 = consumer(queue)
    t3 = consumer(queue)
    t4 = consumer(queue)
    t1.start()
    t2.start()
    t3.start()
    t4.start()
    t1.join()
    t2.join()
    t3.join()
    t4.join()

The code runs as follows:

Thread-1 Thread-2 197

The interaction between producers and consumers can be illustrated in the following diagram:

A Quick Guide to Python Parallel Computing: 06 Python Multithreaded Programming – Communication Based on Queues

Other methods of Queue include:

0

join () Method and task_done () Method

Queue.task_done () After completing a task, Queue.task_done () function sends a signal to the queue that the task has been completed.

Queue.join () essentially means waiting until the queue is empty before performing other operations.

If a thread fetches an item from the queue once, but does not call task_done (), then join will be unable to determine whether the queue has been fully processed. If you call join () at the end, it will never return and will hang indefinitely. You can think of it this way: every time you call task_done, one element is removed from the queue. When join is executed at the end, it checks whether the queue length is zero to confirm the queue is finished, then proceeds with the main thread.

2. Slow Production - Fast Consumption

In the scenario of fast production and slow consumption, we can directly use task_done () together with join () to let empty () determine whether the queue has ended. Of course, we can correctly judge whether the queue has been emptied, but for the get method in the thread, if nothing tells it that the queue is empty, get will continue to block. In that case, we need to add a judgment in the get procedure: if empty () holds, break to exit the loop, otherwise get () will still keep blocking.

However, if the speed of producers is comparable to that of consumers, or the production rate is lower than the consumption rate, then it is unreliable to rely on task_done () to decrement the queue count; the queue will frequently be in a state of undersupply and often remain empty, so it is also unreliable to use empty for judgment. This situation will lead to the consequence that join can detect that the queue has finished processing, but threads cannot rely on empty () to determine whether they can exit. We can insert a specific "marker" at the end of each thread in the consumption queue. During consumption, we check whether such a "marker" is obtained; if so, we can determine that the queue has ended, since all production queues have finished and no new entries will be added.

3. Last-In-First-Out Queue LifoQueue

Contrary to the Queue mentioned above, the last data stored is taken out first, while the first data stored is taken out last. This kind of first-in-last-out queue is similar to a stack.

The sample code is as follows:

import queue
import threading
# You can set the queue length; when the queue is full it blocks automatically
q = queue.LifoQueue(5)
def put():
    for i in range(3):
        q.put(i)
        print("数据 %d 被存入到队列中" % i)
    q.join()
def get():
    for i in range(3):
        value = q.get()
        print("数据 %d 从队列中取出" % value)
        q.task_done()
if __name__ == '__main__':
    t1 = threading.Thread(target=put, args=())
    t1.start()
    t2 = threading.Thread(target=get, args=())
    t2.start()

The code output is as follows:

Demo/00 0 1 2

From the operation results, we find that the order in which we write data is in positive sequence, but the order in which we retrieve data is in reverse sequence. This indicates that the most recently stored data comes out last, which is a first-in, last-out queue.

4. Priority Queue (PriorityQueue)

PriorityQueue (priority queue) means that a priority is assigned when data is stored;when fetching data, the one with the highest priority is taken out. When storing data into the priority queue PriorityQueue, the smaller the set value, the higher the priority.(Note: When storing and fetching data using priority, the data in the queue must be of the same type; for example: class grade ranking, height ranking, etc.).

Sample Code:

import queue

q = queue.PriorityQueue()
q.put([1, 'ace'])
q.put([40, 333])
q.put([3, 'afd'])
q.put([5, '4asdg'])
# 1 is the highest priority,

while not q.empty():  # Run when not empty
    print(q.get())

q = queue.PriorityQueue()

q.put('我')
q.put('你')
q.put('他')
q.put('她')
q.put('ta')

while not q.empty():
    print(q.get())

Output result:

1 3 5 4asdg 40 333

By priority: Whether it is numbers, letters, lists, tuples, etc. (dictionaries and sets have not been tested), data is stored and retrieved based on priority, and the data in the queue must be of the same type, all priority matching is performed according to the order of the ascii code table of the actual data, and Chinese characters follow the unicode table.

A Quick Guide to Python Parallel Computing: 06 Python Multithreaded Programming – Communication Based on Queues