python使用多gpu问题

Question

有人用过多GPU跑网络吗？torch.distributed.run命令，如下：

python -m torch.distributed.run --standalone --nnodes=1 --nproc_per_node 8 xxx.py

这里我使用了8块gpu，但是每次这么用都会出现如下warning，然后程序就暂停了没跑起来：

WARNING:__main__:*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************

不知道是什么问题

Answer 1

DDP神坑，基本无解
换launch看看？
python -m torch.distributed.launch
或者看下这个

distributed training not starting · Issue #65121 · pytorch/pytorch · GitHub 🐛 Bug I'm using Huggingface's Transformers package, trying to start distributed training. But it is not starting as I run the distributed script. To Reproduce export CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch --nproc_per... https://github.com/pytorch/pytorch/issues/65121

?
贴个总的你自己找一找吧
https://github.com/pytorch/pytorch/issues?q=Setting+OMP_NUM_THREADS+environment+variable+for+each+process+to+be+1+in+default%2C+