A Quick Guide to Python Parallel Computing: 08 Python Multiprocessing Programming — multiprocessing Module, Process Creation & Naming, Process ID Retrieval, Daemon Process Creation and Process Termination
This section describes the basic usage of the multiprocessing module in Python, covering how to create and name child processes, retrieve process IDs, set daemon processes, and terminate processes safely, to help implement multiprocess concurrent programming for improved program performance.
For more information about us, you can refer to the cloud document: Freak Embedded Studio Cloud Docs, or visit our wiki:https://github.com/leezisheng/Doc/wik
This document is a learning guide on parallel computing and Python concurrent programming, which covers the core knowledge points including the basic concepts of parallel computing, Python multi-threaded programming, multi-process programming and coroutine programming:
Body Text
Multithreading in Python cannot take advantage of multi-core processors. If you want to fully utilize the resources of a multi-core CPU, in most cases you need to use multiprocessing in Python. Python provides the multiprocessing module to spawn child processes and execute our custom tasks (such as function) in these child processes, and the programming interface of this module is similar to that of the multithreading module threading.The multiprocessing module boasts a rich set of features: it supports subprocesses, inter-process communication and data sharing, as well as various forms of synchronization, and provides components including Process, Queue, Pipe, Lock and more.
1. Process Creation
First, let's take a look at the multiprocessing.Process object:
class multiprocessing.Process(group=None, target=None, name=None, args=(), kwargs={}, *, daemon=None)
Wherein:
group: should always be None; it exists only for compatibility with threading.Thread.
target: the callable object invoked by the run() method. It defaults to None, meaning nothing is called.
name: the process name (see name for details).
args: the argument tuple for the target call. The args parameter defaults to (), and can be used to specify a list or tuple of arguments to pass to target.
Kwargs: the dictionary of keyword arguments for the target call.
daemon: sets the process daemon flag to True or False. If None (the default), the flag is inherited from the creating process.
1.1 Call the constructor of the Process class to create a process
The Process class provides the __init__() constructor, which can be used to create processes. Common methods for starting a process include:
The attributes of a process include:
Here, we conduct tests on process creation and startup, with the code provided as follows:
from multiprocessing import Process
def fun1(name):
print('测试%s多进程' %name)
if __name__ == '__main__':
process_list = []
for i in range(5): #Start 5 child processes to run the fun1 function
p = Process(target=fun1,args=('Python',)) #Instantiate the process object
p.start()
process_list.append(p)
for i in process_list:
p.join()
print('结束测试')
The above code spawns 5 child processes to execute the function. As we can observe from the output, the results are printed simultaneously, which implements true parallel operations where multiple CPUs execute tasks at the same time.
We know that a process is the smallest resource allocation unit in Python, meaning that data and memory are not shared between processes. Every time a process is started, resources need to be allocated independently and accessible data copied, so the cost of starting and destroying processes is relatively high. Note that you need to use the process object to call the join () method. Without join (), after the main process exits, child processes will remain in the idle state, and you have to kill them manually.
1.2 Create a process class by inheriting the Process class
If a subclass overrides the constructor, it must ensure that it calls the base class constructor before performing any other operations on the process Process.__init__(); meanwhile, for the run () method, you can refer to the official documentation:
Here, the test code is as follows:
from multiprocessing import Process
class MyProcess(Process): #Inherit from the Process class
def __init__(self,name):
super(MyProcess,self).__init__()
self.name = name
def run(self):
print('测试%s多进程' % self.name)
if __name__ == '__main__':
process_list = []
for i in range(5): #Start 5 child processes to run the fun1 function
p = MyProcess('Python') #Instantiate the process object
p.start()
process_list.append(p)
for i in process_list:
p.join()
print('结束测试')
The results are as follows, and the effect is the same as that of the first method:
Note: On Windows, Process () must be placed under the if __name__ == '__main__': block
By instantiating a process object in the main Code Block, we can prevent infinite recursion calls. The best practice is to define the target function in a separate script file and then import it for use.
2. Naming of Processes
In the example from the previous section, we created a process and assigned the target function and function variables to it. However, if we can assign a name to the process, it will be more convenient for debugging.
current_process () can return the Process object corresponding to the current process. The following code creates a named process and an unnamed process in the main program, both with the target function being foo () function.
Process Identifier (PID), whose full English name is Process Identifier, is also commonly referred to as process ID. As the name suggests, it is the unique identity tag for each process: once a program starts running, the system will automatically assign a unique PID to the corresponding process. After the process terminates, the PID will be reclaimed by the system and may be reassigned to a newly launched program.
You can use current_process (). pid here to get the PID of the current process. If you change current_process (). name in the above code to current_process (). pid, you will get the following result:
4. Daemon
A daemon is a background process that resides in memory. It detaches from terminal control and is unaffected by terminal signals such as Ctrl+ C. Typically, daemons are used to periodically perform certain tasks or continuously wait to handle certain occurring events. When the parent process terminates, the daemon terminates along with it (essentially, the daemon ends before the parent process, which is responsible for reclaiming resources).
When you need to handle relatively large tasks without requiring manual intervention, executing them as background processes is a very common programming model. Such processes can run concurrently with other processes. Through the background process option of Python's multiprocessing module, we can make processes run in the background.
To run the process in the background, we set the daemon parameter to True:
Processes not running in the background will produce output, while those running in the background will not; background processes will terminate automatically once the main process ends.
5. Process Termination
We can use the terminate () method to kill a process immediately. Additionally, we can use the is_alive () method to check whether a process is still running:
In this example, create a process whose target function is foo (). After starting it, we kill it via the terminate () method.
We create a thread, then use the is_alive () method to monitor its lifecycle. After that, we call the terminate () method to terminate the process. Finally, we verify that the process has ended by reading its ExitCode status code.