weixin_46641005
2020-05-13 16:47
采纳率: 100%
浏览 668

多进程在运行的时候只有一个子进程会运行,怎么解决?

我用python写的多进程爬虫,在linux服务器上运行的时候,不管我设置多少个进程,每次都只有一个子进程运行。
主函数是这样的,mainFunction是我爬虫的主要函数,现在每次都只有第一个进程运行,就是参数是0的那一个。
因为爬虫I/O不算密集,所以没用多线程。另外进程之间是独立的。
我的目标是10个进程同时运行,我有10个API的key,数据也分成10块来抓,这里mainFunction里的参数就是为了对应要用的key和要抓的数据。

if __name__ == '__main__':
    print('Started at ', datetime.now())

    processes = []
    for i in range(0,10):
        t = multiprocessing.Process(target = mainFunction, args = (i,))
        processes.append(t)
    for t in processes:
        t.start()
        t.join()

    print('Finished at ', datetime.now())


麻烦大佬们帮忙找找原因,谢谢!

  • 点赞
  • 写回答
  • 关注问题
  • 收藏
  • 邀请回答

1条回答 默认 最新

  • bobhuang 2020-05-13 18:14
    已采纳

    这段代码,每次循环都是start一个线程,并等待结束后再开始下一个线程。

        for t in processes:
            t.start()
            t.join()
    

    改为:

        for t in processes:
            t.start()
        for t in processes:
            t.join()
    

    完整的测试代码:

    import time
    import multiprocessing
    from datetime import datetime
    
    def mainFunction(idx,running):
        print('thread {0} is running.'.format(idx))
        while(running.value):
            pass
        print('thread {0} is closing .'.format(idx))
    
    if __name__ == '__main__':
        running = multiprocessing.Value('i',1)
        print('Started at ', datetime.now())
        processes = []
        for i in range(0,10):
            t = multiprocessing.Process(target = mainFunction, args = (i,running,))
            processes.append(t)
        for t in processes:
            t.start()
    
        time.sleep(10)
        running.value = 0
    

    运行结果:

    ('Started at ', datetime.datetime(2020, 5, 14, 11, 9, 47, 862220))
    thread 0 is running.
    thread 1 is running.
    thread 2 is running.
    thread 3 is running.
    thread 4 is running.
    thread 5 is running.
    thread 7 is running.
    thread 6 is running.
    thread 9 is running.
    thread 8 is running.
    thread 1 is closing.
    thread 7 is closing.
    thread 2 is closing.
    thread 3 is closing.
    thread 0 is closing.
    thread 5 is closing.
    thread 4 is closing.
    thread 9 is closing.
    thread 6 is closing.
    thread 8 is closing.
    ('Finished at ', datetime.datetime(2020, 5, 14, 11, 9, 57, 989041))
    

    图片说明

    点赞 评论

相关推荐 更多相似问题