训练yolox_l时经过几十个epoch后报错：CUDA error: an illegal memory access was encountered

前几十个epoch没什么问题，可以正常训练。试了很多次，有时候四十几次就停止，不知道为什么。。
这是用服务器训练的，单卡多卡都会出现这样问题
但是用自己的电脑训练就不会这样
报错如下：


2021-10-22 16:07:42 | INFO     | yolox.core.trainer:318 - Save weights to ./YOLOX_outputs/yolox_l
2021-10-22 16:07:43 | INFO     | yolox.core.trainer:188 - ---> start train epoch75
terminate called after throwing an instance of 'c10::CUDAError'
  what():  CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Exception raised from create_event_internal at ../c10/cuda/CUDACachingAllocator.cpp:1055 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7f564e880a22 in /home/vision2021_meas/anaconda3/envs/yolox/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x10983 (0x7f564eae1983 in /home/vision2021_meas/anaconda3/envs/yolox/lib/python3.8/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0x1a7 (0x7f564eae3027 in /home/vision2021_meas/anaconda3/envs/yolox/lib/python3.8/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10::TensorImpl::release_resources() + 0x54 (0x7f564e86a5a4 in /home/vision2021_meas/anaconda3/envs/yolox/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #4: <unknown function> + 0xa27e1a (0x7f56a53d4e1a in /home/vision2021_meas/anaconda3/envs/yolox/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #5: <unknown function> + 0xa27eb1 (0x7f56a53d4eb1 in /home/vision2021_meas/anaconda3/envs/yolox/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #6: <unknown function> + 0x1a6b5a (0x55f420004b5a in /home/vision2021_meas/anaconda3/envs/yolox/bin/python)
frame #7: <unknown function> + 0x110cbc (0x55f41ff6ecbc in /home/vision2021_meas/anaconda3/envs/yolox/bin/python)
frame #8: <unknown function> + 0x1105b9 (0x55f41ff6e5b9 in /home/vision2021_meas/anaconda3/envs/yolox/bin/python)
frame #9: <unknown function> + 0x1105a3 (0x55f41ff6e5a3 in /home/vision2021_meas/anaconda3/envs/yolox/bin/python)
frame #10: <unknown function> + 0x1105a3 (0x55f41ff6e5a3 in /home/vision2021_meas/anaconda3/envs/yolox/bin/python)
frame #11: <unknown function> + 0x1105a3 (0x55f41ff6e5a3 in /home/vision2021_meas/anaconda3/envs/yolox/bin/python)
frame #12: <unknown function> + 0x1105a3 (0x55f41ff6e5a3 in /home/vision2021_meas/anaconda3/envs/yolox/bin/python)
frame #13: <unknown function> + 0x1105a3 (0x55f41ff6e5a3 in /home/vision2021_meas/anaconda3/envs/yolox/bin/python)
frame #14: _PyEval_EvalFrameDefault + 0x65b0 (0x55f42003a160 in /home/vision2021_meas/anaconda3/envs/yolox/bin/python)
frame #15: _PyEval_EvalCodeWithName + 0xd52 (0x55f42002af72 in /home/vision2021_meas/anaconda3/envs/yolox/bin/python)
frame #16: _PyFunction_Vectorcall + 0x594 (0x55f42002ba44 in /home/vision2021_meas/anaconda3/envs/yolox/bin/python)
frame #17: PyObject_Call + 0x7d (0x55f41ff9587d in /home/vision2021_meas/anaconda3/envs/yolox/bin/python)
frame #18: _PyEval_EvalFrameDefault + 0x1f0e (0x55f420035abe in /home/vision2021_meas/anaconda3/envs/yolox/bin/python)
frame #19: _PyEval_EvalCodeWithName + 0x260 (0x55f42002a480 in /home/vision2021_meas/anaconda3/envs/yolox/bin/python)
frame #20: _PyFunction_Vectorcall + 0x534 (0x55f42002b9e4 in /home/vision2021_meas/anaconda3/envs/yolox/bin/python)
frame #21: PyObject_Call + 0x7d (0x55f41ff9587d in /home/vision2021_meas/anaconda3/envs/yolox/bin/python)
frame #22: _PyEval_EvalFrameDefault + 0x1f0e (0x55f420035abe in /home/vision2021_meas/anaconda3/envs/yolox/bin/python)
frame #23: _PyFunction_Vectorcall + 0x1b7 (0x55f42002b667 in /home/vision2021_meas/anaconda3/envs/yolox/bin/python)
frame #24: PyObject_Call + 0x7d (0x55f41ff9587d in /home/vision2021_meas/anaconda3/envs/yolox/bin/python)
frame #25: _PyEval_EvalFrameDefault + 0x1f0e (0x55f420035abe in /home/vision2021_meas/anaconda3/envs/yolox/bin/python)
frame #26: _PyFunction_Vectorcall + 0x1b7 (0x55f42002b667 in /home/vision2021_meas/anaconda3/envs/yolox/bin/python)
frame #27: _PyEval_EvalFrameDefault + 0x4c0 (0x55f420034070 in /home/vision2021_meas/anaconda3/envs/yolox/bin/python)
frame #28: _PyEval_EvalCodeWithName + 0x260 (0x55f42002a480 in /home/vision2021_meas/anaconda3/envs/yolox/bin/python)
frame #29: _PyFunction_Vectorcall + 0x534 (0x55f42002b9e4 in /home/vision2021_meas/anaconda3/envs/yolox/bin/python)
frame #30: _PyEval_EvalFrameDefault + 0x4c0 (0x55f420034070 in /home/vision2021_meas/anaconda3/envs/yolox/bin/python)
frame #31: _PyFunction_Vectorcall + 0x1b7 (0x55f42002b667 in /home/vision2021_meas/anaconda3/envs/yolox/bin/python)
frame #32: _PyEval_EvalFrameDefault + 0x71b (0x55f4200342cb in /home/vision2021_meas/anaconda3/envs/yolox/bin/python)
frame #33: _PyEval_EvalCodeWithName + 0x260 (0x55f42002a480 in /home/vision2021_meas/anaconda3/envs/yolox/bin/python)
frame #34: _PyFunction_Vectorcall + 0x594 (0x55f42002ba44 in /home/vision2021_meas/anaconda3/envs/yolox/bin/python)
frame #35: _PyEval_EvalFrameDefault + 0x15a9 (0x55f420035159 in /home/vision2021_meas/anaconda3/envs/yolox/bin/python)
frame #36: _PyEval_EvalCodeWithName + 0x260 (0x55f42002a480 in /home/vision2021_meas/anaconda3/envs/yolox/bin/python)
frame #37: PyEval_EvalCode + 0x23 (0x55f42002bd33 in /home/vision2021_meas/anaconda3/envs/yolox/bin/python)
frame #38: <unknown function> + 0x2414a2 (0x55f42009f4a2 in /home/vision2021_meas/anaconda3/envs/yolox/bin/python)
frame #39: <unknown function> + 0x252292 (0x55f4200b0292 in /home/vision2021_meas/anaconda3/envs/yolox/bin/python)
frame #40: PyRun_StringFlags + 0x7a (0x55f4200b2eca in /home/vision2021_meas/anaconda3/envs/yolox/bin/python)
frame #41: PyRun_SimpleStringFlags + 0x3c (0x55f4200b2f2c in /home/vision2021_meas/anaconda3/envs/yolox/bin/python)
frame #42: Py_RunMain + 0x15b (0x55f4200b389b in /home/vision2021_meas/anaconda3/envs/yolox/bin/python)
frame #43: Py_BytesMain + 0x39 (0x55f4200b3ce9 in /home/vision2021_meas/anaconda3/envs/yolox/bin/python)
frame #44: __libc_start_main + 0xe7 (0x7f56a829ebf7 in /lib/x86_64-linux-gnu/libc.so.6)
frame #45: <unknown function> + 0x1f7847 (0x55f420055847 in /home/vision2021_meas/anaconda3/envs/yolox/bin/python)

Traceback (most recent call last):
  File "tools/train.py", line 127, in <module>
    launch(
  File "/home/vision2021_meas/mycfhs/yolox/yolox/core/launch.py", line 82, in launch
    mp.start_processes(
  File "/home/vision2021_meas/anaconda3/envs/yolox/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 188, in start_processes
    while not context.join():
  File "/home/vision2021_meas/anaconda3/envs/yolox/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 130, in join
    raise ProcessExitedException(
torch.multiprocessing.spawn.ProcessExitedException: process 5 terminated with signal SIGABRT
/home/vision2021_meas/anaconda3/envs/yolox/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 149 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

写回答
好问题 0 提建议
追加酬金
关注问题
分享
邀请回答
编辑收藏删除
收藏举报

3条回答默认最新

关注

码龄粉丝数原力等级 --

被采纳

被点赞

采纳率
爱晚乏客游 2021-10-22 17:07
关注
你用的官网的代码吗？如果是的话加上 -o参数看看。
多显卡训练出现这个问题的吗还是啥？或者是显存不足，你可以试试看下把batch size改小点或者将图片输入改小点试试看。
还有一个就是使用resume参数将从上次停止的地方继续训练。如果都不行的话就只能去官网下面提这个问题，详细说明环境，参数设置等，看下作者有什么方法解决没有。

解决无用
评论打赏
分享
举报

评论

按下Enter换行，Ctrl+Enter发表内容

报告相同问题？

关注问题

【提问-pytorch运行报错】CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling `cublasCreate(handle)` pytorch
2021-08-20 11:18

回答 1 已采纳 label是类别，比如说（“100，“110”,“111“），你的100是第0类，110是第1类，111是第2类，你这么写变成了100是第100类，110是第110类，111是第111类了，原本你只有
运行代码出现KeyError：‘epoch’报错？ python
2020-09-13 11:08

回答 2 已采纳你的checkpoint中有没有保存epoch https://www.cnblogs.com/baby-lily/p/10930591.html
报错：error: (-215:Assertion failed) !_src.empty() in function 'cv::cvtColor' opencv python tensorflow
2022-12-15 17:43

回答 1 已采纳你可以用一点小技巧，剔除掉有问题的图片，只读取OK的图片，中间部分修改后的代码如下，望采纳 for im in img_path: try: # 读取图
【debug】报错RuntimeError: CUDA error: an illegal memory access was encountered
2023-08-07 10:04

zy_destiny的博客 mmseg工程报错RuntimeError: CUDA error: an illegal memory access was encountered解决。
RuntimeError: CUDA out of memory. python pytorch 深度学习
2023-04-04 09:44

回答 4 已采纳该回答通过自己思路及引用到各个渠道搜索综合及思考,得到内容具体如下：根据您提供的信息，这个错误是由于GPU内存不足引起的。在训练yolov7标准版时，需要更多的GPU内存，而您的GPU内存已经被占用
paddle 卷积神经网络训练时报错InvalidArgumentError: The input tensor X of SumOp must have same shape.. paddle 深度学习神经网络
2022-03-24 03:18

回答 1 已采纳你传入的张量是252，2但需要的是21，2考虑使用reshape把维度变化一下飞桨的框架我没用过，pytorch是这么操作的，你搜一搜类似的改变张量shape的函数
训练轮次后报错IndexError: string index out of range python
2023-04-11 16:11

回答 4 已采纳你的代码在访问preds[idx]或labels[idx]的一个idx时，该idx超出了这些数组的范围。如果以上回答对您有所帮助，点击一下采纳该答案～谢谢
RuntimeError: CUDA error: an illegal memory access was encountered
2022-07-19 12:31

吨吨不打野的博客很多都是靠感觉的参考： pytorch的github issue：RuntimeError: CUDA error: an illegal memory access was encountered 这个回答好像有效的人比较多，一次惨痛的debug的经历-RuntimeError: CUDA error: an ...
CMD报错：python: can't open file 'interact.py': [Errno 2] No such file or directory github python 深度学习
2021-12-31 16:02

回答 1 已采纳问题1：你的这些函数中缺少附件，所以有些函数没有，运行及会报错，建议你找一个完整的。问题2：你pycharm中所编译的python程序就是你本机的pip文件中的python,所以就不需要了问题3：你应
使用Keras编写的LSTM，训练时出现loss: nan - val_loss: nan，该如何调整？ keras lstm 有问必答深度学习
2022-05-03 18:40

回答 2 已采纳原数据第一列是时间形式的20220503这种，在读数据之后进行下面操作，你的数据是简单的1 2 34这种，数据可能在处理过程丢失或者变成nan了，你可以一步步看看每次处理后当前的数据现在是什么格式 d
如何在进行数据训练是报KeyError: 'val_mape？ keras tensorflow 神经网络
2023-01-02 06:19

回答 6 已采纳出现 KeyError: 'val_mape' 的原因可能是在代码中没有计算并输出 val_mape 这个指标。在训练模型时使用了 ModelCheckpoint 回调函数来保存模型权重，并使用了
RuntimeError: CUDA error: an illegal memory access was encountered 解决思路
2023-09-09 10:11

yyywxk的博客 RuntimeError: CUDA error: an illegal memory access was encountered GPU负载
train.py: error怎么解决 python 深度学习神经网络
2021-06-04 16:22

回答 1 已采纳配置文件路径的参数没传，运行时输入一下命令： python train.py --cfg '配置文件的路径名'
非法的内存访问 RuntimeError: CUDA error: an illegal memory access was encountered
2021-11-28 22:32

跨不过的博客报错 RuntimeError: CUDA error: an illegal memory access was encountered CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be ...
1:RuntimeError: CUDA error: an illegal memory access was encounteredCompile with TORCH_USE_CUDA_DSA`
2024-03-02 15:36

じ☆ve 緈辐的博客报错显示不精确，按照软件推荐方法设置set TORCH_USE_CUDA_DSA=1，设置之后就会报错详细的问题，从而进行修改。
yolov8自定义数据训练报错排查（CUDA error: an illegal memory access was encountered）
2023-08-03 11:10

CodingInCV的博客可以正常训练，出问题的版本是pytorch 1.13.1。若有碰到相似问题的，不妨换一个pytorch版本。github上有人说是由于数据集中标签有问题，不过我处理了一遍数据，并没有这种情况。
Pytorch分布式报错1.non-zero exit status，2.cuDNN error:CUDNN_STATUS_INTERNAL_，3.CUDA error:illegal memory
2021-07-26 15:53

xiangyong58的博客 One epoch之后报错，信息如下： RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not...
Pytorch在训练过程中常见的问题
2020-05-13 11:45

萌萌滴太阳的博客 1 Input type ...仔细看错误信息，CUDA和CPU，输入数据x和模型中的权重值类型不一样，一般来说是因为模型的参数不在GPU中，而输入数据在GPU中，通过添加model.cuda()将模型转移到GPU上以解决这个问题。 2 Inp
YOLO相关问题记录
2021-11-21 19:47

小锋学长生活大爆炸的博客 view size is not compatible with input tensor’s size and stride CUDA error: an illegal memory access was encountered can't convert cuda:0 device type tensor to numpy. 知识点 YOLOv5超参介绍 YOLOv5n6....
没有解决我的问题, 去提问

问题事件

关注

码龄粉丝数原力等级 --

被采纳

被点赞

采纳率
已结题（查看结题原因） 10月26日
关注

码龄粉丝数原力等级 --

被采纳

被点赞

采纳率
修改了问题 10月22日
关注

码龄粉丝数原力等级 --

被采纳

被点赞

采纳率
创建了问题 10月22日

悬赏问题

¥15 C#算法问题, 不知道怎么处理这个数据的转换
¥15 YoloV5 第三方库的版本对照问题
¥15 请完成下列相关问题！
¥15 drone 推送镜像时候 purge: true 推送完毕后没有删除对应的镜像,手动拷贝到服务器执行结果正确在样才能让指令自动执行成功删除对应镜像，如何解决？
¥15 求daily translation（DT）偏差订正方法的代码
¥15 js调用html页面需要隐藏某个按钮
¥15 ads仿真结果在圆图上是怎么读数的
¥20 Cotex M3的调试和程序执行方式是什么样的？
¥20 java项目连接sqlserver时报ssl相关错误
¥15 一道python难题3

训练yolox_l时经过几十个epoch后报错：CUDA error: an illegal memory access was encountered

3条回答 默认 最新

问题事件

悬赏问题

3条回答默认最新