python多进程调用DLL报错

Question

基于python3.7，使用msl.loadlib中的两个库Client64和Server32实现了基于64位环境调用32位DLL的操作。单线程正常，但是在运行多进程时报错如下：ValueError: ctypes objects containing pointers cannot be pickled

查了资料，原因是ctypes指针无法被pickle模块序列化。因为在解析DLL函数时，使用了ctypes的指针：from ctypes import c_byte, c_int, c_short, c_ushort, pointer, c_char_p, byref。

 def calcKEY(self, seed_L1):
        """
        char* ipSeed,
        unsigned short sizeSeed,
        char* opKey,
        unsigned short maxSizeKey,
        unsigned short* opSizeKey
        :param seed_L1:
        :return:seedKey
        """
        bSeed = (c_byte * 4)(seed_L1[0], seed_L1[1], seed_L1[2], seed_L1[3])
        self.lib.ASAP1A_CCP_ComputeKeyFromSeed(pointer(bSeed), c_ushort(4), pointer(self.key), c_ushort(4), pointer(c_ushort(4)))
        seedKey = int.from_bytes(self.key, 'big', signed=False)
        return seedKey

• 在CSDN查询到了类似的问题，但没有给具体的解决方案，放上链接[https://wenku.csdn.net/answer/06399e2ba44e405091b200fa0d0569d6?ops_request_misc=%257B%2522request%255Fid%2522%253A%25223627d9a2501cf07a13214df16b298948%2522%252C%2522scm%2522%253A%252220140713.130102334.pc%255Fcommercial.%2522%257D&request_id=3627d9a2501cf07a13214df16b298948&biz_id=&utm_medium=distribute.pc_search_result.none-task-chatgpt-2~all~insert_commercial~default-3-06399e2ba44e405091b200fa0d0569d6-null-null.142^v100^pc_search_result_base8&utm_term=%E5%A4%9A%E8%BF%9B%E7%A8%8B%E8%B0%83%E7%94%A8DLL%E6%96%87%E4%BB%B6%E6%8A%A5%E9%94%99%20ValueError%3A%20ctypes%20objects%20containing%20pointers%20cannot%20be%20pickled&spm=1018.2226.3001.4187&ydreferer=aHR0cHM6Ly9zby5jc2RuLm5ldC9zby9zZWFyY2g%2Fc3BtPTEwMDAuMjExNS4zMDAxLjQ0OTgmcT0lRTUlQTQlOUElRTglQkYlOUIlRTclQTglOEIlRTglQjAlODMlRTclOTQlQThETEwlRTYlOTYlODclRTQlQkIlQjYlRTYlOEElQTUlRTklOTQlOTklMjBWYWx1ZUVycm9yJTNBJTIwY3R5cGVzJTIwb2JqZWN0cyUyMGNvbnRhaW5pbmclMjBwb2ludGVycyUyMGNhbm5vdCUyMGJlJTIwcGlja2xlZCZ0PSZ1PQ%3D%3D]

• 查询了GPT，给出的建议是使用共享内存：multiprocessing.Array，还是无法实现。

Answer 1

阿里嘎多学长整理AIGC生成，因移动端显示问题导致当前答案未能完全显示，请使用PC端查看更加详细的解答过程

python多进程调用DLL报错

问题总结：在使用多进程时，调用DLL函数报错ValueError: ctypes objects containing pointers cannot be pickled，因为ctypes指针无法被pickle模块序列化。

解决方案：由于ctypes指针无法被pickle模块序列化，因此不能使用多进程方式调用DLL函数。可以使用以下方法来解决问题：

1. 使用fork方式rather than spawning new processes. This allows the child process to inherit the parent process's memory, including the DLL handles.

import multiprocessing
from multiprocessing import Process

class Foo:
    def __init__(self):
        # Load the DLL here
        self.dll = self.load_dll()

    def load_dll(self):
        # Load the DLL using ctypes
        return ctypes.CDLL('path/to/dll.so')

    def run(self):
        # Call the DLL function
        self.dll.my_function()

if __name__ == '__main__':
    foo = Foo()
    parent_process = multiprocessing.current_process()
    child_process = Process(target=foo.run)
    child_process.start()
    print("Parent process:", parent_process)
    print("Child process:", child_process)
    child_process.join()

1. 使用共享内存来共享DLL handle。可以使用multiprocessing.Manager来创建共享内存，并在多进程下共享DLL handle。

import multiprocessing

class Foo:
    def __init__(self, dll):
        # Load the DLL here
        self.dll = dll

    def run(self):
        # Call the DLL function
        self.dll.my_function()

if __name__ == '__main__':
    manager = multiprocessing.Manager()
    dll = manager.Value('c', ctypes.CDLL('path/to/dll.so'))
    foo = Foo(dll)
    parent_process = multiprocessing.current_process()
    child_process = Process(target=foo.run)
    child_process.start()
    print("Parent process:", parent_process)
    print("Child process:", child_process)
    child_process.join()

1. 使用ipc通信来共享DLL handle。可以使用multiprocessing.Pipe来创建IPC通信管道，并在多进程下共享DLL handle。

import multiprocessing

class Foo:
    def __init__(self, dll):
        # Load the DLL here
        self.dll = dll

    def run(self):
        # Call the DLL function
        self.dll.my_function()

if __name__ == '__main__':
    parent_process = multiprocessing.current_process()
    child_process = Process(target=foo.run)
    manager = multiprocessing.Manager()
    pipe = multiprocessing.Pipe()
    manager.connect(pipe)
    dll = manager.Value('c', ctypes.CDLL('path/to/dll.so'))
    foo = Foo(dll)
    child_process.start()
    print("Parent process:", parent_process)
    print("Child process:", child_process)
    child_process.join()

以上三个方法都可以解决问题，但是可能存在其他的限制和限制。