Python3 多进程共享变量实现方法(亲测)
一、錯誤的實現方式
最初以為是沒添加global聲明導致修改未生效,但實際操作發現global方式在多進程中也只能讀不能寫。錯誤示例代碼如下:
import multiprocessing# 聲明一個全局變量 share_var = ["start flag"]def sub_process(process_name):# 企圖像單個進程那樣通過global聲明使用全局變量global share_varshare_var.append(process_name)# 但是很可惜,在多進程中這樣引用只能讀,修改其他進程不會同步改變for item in share_var:print(f"{process_name}-{item}")passdef main_process():process_list = []# 創建進程1process_name = "process 1"tmp_process = multiprocessing.Process(target=sub_process,args=(process_name,))process_list.append(tmp_process)# 創建進程2process_name = "process 2"tmp_process = multiprocessing.Process(target=sub_process, args=(process_name,))process_list.append(tmp_process)# 啟動所有進程for process in process_list:process.start()for process in process_list:process.join()if __name__ == "__main__":main_process()執行結果如下,可以看到進程1中的修改未表現在進程2中(不過要注意,和多線程一樣,如果運算量再大一點進程1并不一定比進程2先執行):
?
二、共享普通類型變量實現方法
import multiprocessing# 不能將共享變量和共享鎖定義成全局變量然后通過global引用那樣會報錯,只能傳過來 def sub_process(process_name,share_var,share_lock):# 獲取鎖share_lock.acquire()share_var.append(process_name)# 釋放鎖share_lock.release()for item in share_var:print(f"{process_name}-{item}")passdef main_process():# 單個值聲明方式。typecode是進制類型,value是初始值# share_var = multiprocessing.Manager().Value(typecode, value)# 數組聲明方式。typecode是數組變量中的變量類型,sequence是數組初始值# share_var = multiprocessing.Manager().Array(typecode, sequence)# 字典聲明方式# share_var = multiprocessing.Manager().dict()# 列表聲明方式share_var = multiprocessing.Manager().list()share_var.append("start flag")# 聲明一個共享鎖share_lock = multiprocessing.Manager().Lock()process_list = []process_name = "process 1"tmp_process = multiprocessing.Process(target=sub_process,args=(process_name,share_var,share_lock))process_list.append(tmp_process)process_name = "process 2"tmp_process = multiprocessing.Process(target=sub_process, args=(process_name,share_var,share_lock))process_list.append(tmp_process)for process in process_list:process.start()for process in process_list:process.join()if __name__ == "__main__":main_process()執行結果如下,可以看到進程1中的修改已表現在進程2中(不過要注意,和多線程一樣,如果運算量再大一點進程1并不一定比進程2先執行):
typecode如果是數值或單個字符,可為以下類型(注意有引號):
| Type Code | C Type | Python Type |
| 'c' | char | character |
| 'b' | signed char | int |
| 'B' | unsigned char | int |
| 'u' | Py_UNICODE | unicode character |
| 'h' | signed short | int |
| 'H' | unsigned short | int |
| 'i' | signed int | int |
| 'I' | unsigned int | int |
| 'l' | signed long | int |
| 'L' | unsigned long | int |
| 'f' | float | float |
| 'd' | double | flo |
?
?
?
?
?
?
?
?
?
?
?
?
?
如果是字符串類型,typecode可為以下第一列形式(注意無引號):
| ctypes type | C type | Python type |
| c_bool | _Bool | bool (1) |
| char | ?char | 1-character string |
| c_wchar | wchar_t | 1-character unicode string |
| c_byte | char | int/long |
| c_ubyte | unsigned char | int/long |
| c_short | short | int/long |
| c_ushort | unsigned short | int/long |
| c_int | int | int/long |
| c_uint | unsigned in | int/long |
| c_long | long | int/long |
| c_ulong | unsigned long | int/long |
| c_longlong | __int64 or long long | int/long |
| c_ulonglong | unsigned __int64 or unsigned long long | int/long |
| c_float | float | float |
| c_double | double | float |
| c_longdouble | long double | float |
| c_char_p | char * (NUL terminated) | string or None |
| c_wchar_p | wchar_t * (NUL terminated) | unicode or None |
| c_void_p | void * | int/long or None |
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
三、共享實例化對象實現方法
同事還想共享一個文件對象,然后問上邊的方法是不是只能共享字典、列表,沒法共享對象。
回頭一看,Value和Array中typecode要求是c語言中存在的類型,其他只有dict()和list()方法沒有其他方法,所以似乎上邊的方法共享實例化對象是不行的。
import multiprocessing import threading# 實例化一個全局文件對象 file_obj = open("1.txt","a") share_lock = threading.Lock()def sub_process(process_name):global file_obj,share_lockshare_lock.acquire()file_obj.writelines(f"{process_name}")share_lock.release()passdef main_process():process_list = []# 創建進程1process_name = "process 1"tmp_process = multiprocessing.Process(target=sub_process,args=(process_name,))process_list.append(tmp_process)# 創建進程2process_name = "process 2"tmp_process = multiprocessing.Process(target=sub_process, args=(process_name,))process_list.append(tmp_process)# 啟動所有進程for process in process_list:process.start()for process in process_list:process.join()if __name__ == "__main__":main_process()?
3.2 共享需要修改實例化對象實現方法----使用BaseManager
global方式不能修改變量(如要修改其成員變量),在大多時候也是可以了,但總讓人覺得不是一種完美的實現方法。有沒有可以修改的實現方法呢,答案是有的,可以使用BaseManager。示例代碼如下。
import multiprocessing from multiprocessing.managers import BaseManager import threading# 鎖可以通過global也可以在Process中傳無所謂 share_lock = threading.Lock()# 定義一個要共享實例化對象的類 class Test():def __init__(self):self.test_list = ["start flag"]def test_function(self,arg):self.test_list.append(arg)def print_test_list(self):for item in self.test_list:print(f"{item}")def sub_process(process_name,obj):global share_lockshare_lock.acquire()obj.test_function(f"{process_name}")share_lock.release()obj.print_test_list()passdef main_process():# 如果是想注冊open方法這樣操作# manager = BaseManager()# # 一定要在start前注冊,不然就注冊無效# manager.register('open', open)# manager.start()# obj = manager.open("1.txt","a")# 為了更加直接我們直接以一個Test類的實例化對象來演示manager = BaseManager()# 一定要在start前注冊,不然就注冊無效manager.register('Test', Test)manager.start()obj = manager.Test()process_list = []# 創建進程1process_name = "process 1"tmp_process = multiprocessing.Process(target=sub_process,args=(process_name,obj))process_list.append(tmp_process)# 創建進程2process_name = "process 2"tmp_process = multiprocessing.Process(target=sub_process, args=(process_name,obj))process_list.append(tmp_process)# 啟動所有進程for process in process_list:process.start()for process in process_list:process.join()if __name__ == "__main__":main_process()執行結果如下,可以看到進程1中的修改已表現在進程2中(不過要注意,和多線程一樣,如果運算量再大一點進程1并不一定比進程2先執行):
總結
以上是生活随笔為你收集整理的Python3 多进程共享变量实现方法(亲测)的全部內容,希望文章能夠幫你解決所遇到的問題。
- 上一篇: Codeforces 494E. Sha
- 下一篇: websocket python爬虫_p