问题遇到的现象和发生背景
因为我的代码计算量比较大,于是使用了多进程,但是测试和验证了几遍甚至与demo进行对比,都发现不行。demo可以有效,但是我的实测代码却无效。
问题就是:python多进程情况下只有主进程在跑,但是子进程一动不动。
实现的模块是:concurrent.futures.ProcessPoolExecutor
.....准备数据parameter_dict
with concurrent.futures.ProcessPoolExecutor() as executor:
results = executor.map(calculation,tuple(parameter_dict.values()))
for data in results:
data_tuple = data[0],data[1][0][0],data[2],data[3][0][0],data[4],data[5],data[6]
similar_df.loc[len(similar_df.index)] = data_tuple
我的代码模板大概是这样的
#!/usr/bin/env python
# -*- coding: utf-8 -*-
# time: 2022/12/1 18:59
# file: test.py
import concurrent
from concurrent.futures import as_completed
def cal(parm_list):
# xxx1计算
# xxx2计算
# xxx3计算
return 0
def calculation1(datalist):
"""
下面的每个计算之间互相独立,计算量大
:return:
"""
parm_list = []
for k in datalist:
# 讲过一系列的筛选和计算之后得出要计算的数据k1
# ........
k1 = k
parm_list.append(k1)
with concurrent.futures.ProcessPoolExecutor(max_workers=6) as executor:
results = executor.map(cal,parm_list)
# 结果拼接
for r in results:
print("exe result {num}".format(num=r))
# 数据返回
return 0
def calculation2(datalist):
"""
下面的每个计算之间互相独立,计算量大
:return:
"""
parm_list = []
for k in datalist:
# 讲过一系列的筛选和计算之后得出要计算的数据k1
# ........
k1 = k
parm_list.append(k1)
with concurrent.futures.ProcessPoolExecutor(max_workers=6) as executor:
results = executor.map(cal, parm_list)
# 结果拼接
for r in results:
print("exe result {num}".format(num=r))
# 数据返回
return 0
def work1():
for j in range(9):
# ................预处理数据 为计算准备数据
pass
calculation1()
calculation2()
#保存数据....
def work2():
for j in range(9):
# .............. 预处理数据 为计算准备数据
pass
calculation1()
calculation2()
#保存数据....
def main():
for i in range(2):
work1()
work2()
# .....................
pass
if __name__ == '__main__':
main()
场景
pycharm
进程之间不互相通信,只是每个进程运行结束后我把结果拼接起来而已。
#我想要达到的结果
子进程也跑起来。