服务器有2张3090的显卡,但是每次训练时只能使用一张,指定
os.environ['CUDA_VISIBLE_DEVICES'] = '0, 1'
也没有用,心想那我就把batchsize改小点 32得了,于是也顺利的得到理想的权重文件,然后准备绘制一个混淆矩阵,当我加载权重文件时发现,现存不足,报如下错误:
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 5.76 GiB (GPU 0; 23.69 GiB total capacity; 13.02 GiB already allocated; 4.03 GiB free; 18.15 GiB reserved in total by PyTorch) If reserved memory is
>> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
但是为什么啊,我的显卡信息如下:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.182.03 Driver Version: 470.182.03 CUDA Version: 11.4 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... Off | 00000000:73:00.0 Off | N/A |
| 60% 71C P2 239W / 350W | 23008MiB / 24259MiB | 18% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 1 NVIDIA GeForce ... Off | 00000000:D5:00.0 Off | N/A |
| 30% 49C P8 22W / 350W | 8MiB / 24268MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
也不知道为什么,绘制不出混淆矩阵好难过,望各位指点一二