weixin_46641005 2020-05-13 16:47 采纳率: 100%
浏览 1022
已采纳

多进程在运行的时候只有一个子进程会运行,怎么解决?

我用python写的多进程爬虫,在linux服务器上运行的时候,不管我设置多少个进程,每次都只有一个子进程运行。
主函数是这样的,mainFunction是我爬虫的主要函数,现在每次都只有第一个进程运行,就是参数是0的那一个。
因为爬虫I/O不算密集,所以没用多线程。另外进程之间是独立的。
我的目标是10个进程同时运行,我有10个API的key,数据也分成10块来抓,这里mainFunction里的参数就是为了对应要用的key和要抓的数据。

if __name__ == '__main__':
    print('Started at ', datetime.now())

    processes = []
    for i in range(0,10):
        t = multiprocessing.Process(target = mainFunction, args = (i,))
        processes.append(t)
    for t in processes:
        t.start()
        t.join()

    print('Finished at ', datetime.now())


麻烦大佬们帮忙找找原因,谢谢!

  • 写回答

1条回答 默认 最新

  • bobhuang 2020-05-13 18:14
    关注

    这段代码,每次循环都是start一个线程,并等待结束后再开始下一个线程。

        for t in processes:
            t.start()
            t.join()
    

    改为:

        for t in processes:
            t.start()
        for t in processes:
            t.join()
    

    完整的测试代码:

    import time
    import multiprocessing
    from datetime import datetime
    
    def mainFunction(idx,running):
        print('thread {0} is running.'.format(idx))
        while(running.value):
            pass
        print('thread {0} is closing .'.format(idx))
    
    if __name__ == '__main__':
        running = multiprocessing.Value('i',1)
        print('Started at ', datetime.now())
        processes = []
        for i in range(0,10):
            t = multiprocessing.Process(target = mainFunction, args = (i,running,))
            processes.append(t)
        for t in processes:
            t.start()
    
        time.sleep(10)
        running.value = 0
    

    运行结果:

    ('Started at ', datetime.datetime(2020, 5, 14, 11, 9, 47, 862220))
    thread 0 is running.
    thread 1 is running.
    thread 2 is running.
    thread 3 is running.
    thread 4 is running.
    thread 5 is running.
    thread 7 is running.
    thread 6 is running.
    thread 9 is running.
    thread 8 is running.
    thread 1 is closing.
    thread 7 is closing.
    thread 2 is closing.
    thread 3 is closing.
    thread 0 is closing.
    thread 5 is closing.
    thread 4 is closing.
    thread 9 is closing.
    thread 6 is closing.
    thread 8 is closing.
    ('Finished at ', datetime.datetime(2020, 5, 14, 11, 9, 57, 989041))
    

    图片说明

    本回答被题主选为最佳回答 , 对您是否有帮助呢?
    评论

报告相同问题?

悬赏问题

  • ¥100 有人会搭建GPT-J-6B框架吗?有偿
  • ¥15 求差集那个函数有问题,有无佬可以解决
  • ¥15 【提问】基于Invest的水源涵养
  • ¥20 微信网友居然可以通过vx号找到我绑的手机号
  • ¥15 寻一个支付宝扫码远程授权登录的软件助手app
  • ¥15 解riccati方程组
  • ¥15 display:none;样式在嵌套结构中的已设置了display样式的元素上不起作用?
  • ¥15 使用rabbitMQ 消息队列作为url源进行多线程爬取时,总有几个url没有处理的问题。
  • ¥15 Ubuntu在安装序列比对软件STAR时出现报错如何解决
  • ¥50 树莓派安卓APK系统签名