Multiprocessing issue: function calling

Question

from multiprocessing import Process

a=[]

def one():
    for i in range(3):
        a.append(i)

def main():
    p1=Process(target=one)
    p1.start()
if __name__=='__main__':
    main()

print('After calling from Multi-process')
print(a)

one()
print('Calling outside Multi-process')
print(a)

Output:

After calling from Multi-process:
[]
Calling outside Multi-process:
[0, 1, 2]

Why elements are not getting appended to a when calling the function one from Process?

Each `Process` runs in its own memory-space, so you can't share global variables like `a` among them — in other words each one has it's own separate `a` variable. — martineau, Dec 08 '20 at 10:44

score 0 · Answer 1 · answered Dec 08 '20 at 10:26

Process of multi-processing creates sub-processes where the all relevant memories are copied over and separately modified, that is, it doesn't share the memory locations for the global variables.

If you really want to make this work, you can use Threading instead of Process. That does share the global memory locations as opposed to making multiple copies of global variables.

Do from threading import Thread and p1=Thread(target=one) instead.

score 0 · Answer 2 · answered Dec 08 '20 at 10:35

I checked the documentation and I have some understanding, which may be helpful to you.

The python documentation has the following description of the Process class in the multiprocessing package:

Contexts and start methods Depending on the platform, multiprocessing supports three ways to start a process. These start methods are

spawn The parent process starts a fresh python interpreter process. The child process will only inherit those resources necessary to run the process object’s run() method. In particular, unnecessary file descriptors and handles from the parent process will not be inherited. Starting a process using this method is rather slow compared to using fork or forkserver.

Available on Unix and Windows. The default on Windows and macOS.

fork The parent process uses os.fork() to fork the Python interpreter. The child process, when it begins, is effectively identical to the parent process. All resources of the parent are inherited by the child process. Note that safely forking a multithreaded process is problematic.

Available on Unix only. The default on Unix.

forkserver When the program starts and selects the forkserver start method, a server process is started. From then on, whenever a new process is needed, the parent process connects to the server and requests that it fork a new process. The fork server process is single threaded so it is safe for it to use os.fork(). No unnecessary resources are inherited.

Available on Unix platforms which support passing file descriptors over Unix pipes.

From the description, we can know that the Process class uses os.fork to run the method specified by target in several cases.

os.fork will completely copy all objects of the parent process except for resources such as file descriptors for use by the child process.

Therefore, the list a operated by the one method running in the child process is the unique memory space of the child process itself, and will not change the list a owned by the parent process.

To verify, we can simply modify the one method, example:

def one():
    for i in range(3):
        a.append(i)
        print('pid:', os.getpid(), 'ppid:', os.getppid(), 'list:', a)

Then we run this script again, and we will get the following results:

After calling from Multi-process
[]                                   // Print directly in the parent process
pid: 6990 ppid: 1419 list: [0]       // Call the `one` method in the parent process
pid: 6990 ppid: 1419 list: [0, 1]    // Call the `one` method in the parent process
pid: 6990 ppid: 1419 list: [0, 1, 2] // Call the `one` method in the parent process
Calling outside Multi-process
[0, 1, 2]                            // Print directly in the parent process
pid: 6991 ppid: 6990 list: [0]       // Call the `one` method in the child process
pid: 6991 ppid: 6990 list: [0, 1]    // Call the `one` method in the child process
pid: 6991 ppid: 6990 list: [0, 1, 2] // Call the `one` method in the child process

Multiprocessing issue: function calling

2 Answers2