mmdet3d模型部署问题

Question

博主您好，我看到您有发布关于mmdeploy项目使用方法的内容，所以想向您请教一下我在转化mmdet3d模型中所遇到的问题

问题描述

模型文件：centerpoint_0075voxel_second_secfpn_dcn_circlenms_4x8_cyclic_20e_nus_20220810_025930-657f67e0.pth
模型配置文件：centerpoint_voxel0075_second_secfpn_head-dcn-circlenms_8xb4-cyclic-20e_nus-3d.py
问题：使用mmdeploy尝试将模型文件转换为onnx文件时频繁报错，使用的是onnxruntime-gpu推理引擎，不是很明白RuntimeError: get_indice_pairs is not implemented on CPU为什么会发生。而使用官方例程中的centerpoint_pillar02_second_secfpn_head-circlenms_8xb4-cyclic-20e_nus-3d模型及其配置文件就可以顺利转化，这也说明我的环境大致是每问题的

#报错信息
(mmdeploy) archerr@archerr-MS-7D17:~/mmdeploy-1.3.1$ python3 tools/deploy.py configs/mmdet3d/voxel-detection/voxel-detection_onnxruntime_dynamic.py $MODEL_CONFIG $MODEL_PATH $TEST_DATA --work-dir centerpoint --device cuda
02/26 22:32:22 - mmengine - INFO - Start pipeline mmdeploy.apis.pytorch2onnx.torch2onnx in subprocess
/home/archerr/mmdetection3d-main/mmdet3d/evaluation/functional/kitti_utils/eval.py:10: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
def get_thresholds(scores: np.ndarray, num_gt, num_sample_pts=41):
02/26 22:32:23 - mmengine - WARNING - Failed to search registry with scope "mmdet3d" in the "Codebases" registry tree. As a workaround, the current "Codebases" registry in "mmdeploy" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmdet3d" is a correct scope, or whether the registry is initialized.
02/26 22:32:23 - mmengine - WARNING - Failed to search registry with scope "mmdet3d" in the "mmdet3d_tasks" registry tree. As a workaround, the current "mmdet3d_tasks" registry in "mmdeploy" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmdet3d" is a correct scope, or whether the registry is initialized.
Loads checkpoint by local backend from path: centerpoint_0075voxel_second_secfpn_dcn_circlenms_4x8_cyclic_20e_nus_20220810_025930-657f67e0.pth
02/26 22:32:26 - mmengine - INFO - DeformConv2dPack pts_bbox_head.task_heads.0.feature_adapt_cls is upgraded to version 2.
02/26 22:32:26 - mmengine - INFO - DeformConv2dPack pts_bbox_head.task_heads.0.feature_adapt_reg is upgraded to version 2.
02/26 22:32:26 - mmengine - INFO - DeformConv2dPack pts_bbox_head.task_heads.1.feature_adapt_cls is upgraded to version 2.
02/26 22:32:26 - mmengine - INFO - DeformConv2dPack pts_bbox_head.task_heads.1.feature_adapt_reg is upgraded to version 2.
02/26 22:32:26 - mmengine - INFO - DeformConv2dPack pts_bbox_head.task_heads.2.feature_adapt_cls is upgraded to version 2.
02/26 22:32:26 - mmengine - INFO - DeformConv2dPack pts_bbox_head.task_heads.2.feature_adapt_reg is upgraded to version 2.
02/26 22:32:26 - mmengine - INFO - DeformConv2dPack pts_bbox_head.task_heads.3.feature_adapt_cls is upgraded to version 2.
02/26 22:32:26 - mmengine - INFO - DeformConv2dPack pts_bbox_head.task_heads.3.feature_adapt_reg is upgraded to version 2.
02/26 22:32:26 - mmengine - INFO - DeformConv2dPack pts_bbox_head.task_heads.4.feature_adapt_cls is upgraded to version 2.
02/26 22:32:26 - mmengine - INFO - DeformConv2dPack pts_bbox_head.task_heads.4.feature_adapt_reg is upgraded to version 2.
02/26 22:32:26 - mmengine - INFO - DeformConv2dPack pts_bbox_head.task_heads.5.feature_adapt_cls is upgraded to version 2.
02/26 22:32:26 - mmengine - INFO - DeformConv2dPack pts_bbox_head.task_heads.5.feature_adapt_reg is upgraded to version 2.
02/26 22:32:26 - mmengine - WARNING - DeprecationWarning: get_onnx_config will be deprecated in the future.
02/26 22:32:26 - mmengine - INFO - Export PyTorch model to ONNX: centerpoint/end2end.onnx.
02/26 22:32:26 - mmengine - WARNING - Can not find torch.nn.functional.scaled_dot_product_attention, function rewrite will not be applied
02/26 22:32:26 - mmengine - WARNING - Can not find torch._C._jit_pass_onnx_autograd_function_process, function rewrite will not be applied
/home/archerr/.conda/envs/mmdeploy/lib/python3.8/site-packages/mmcv/ops/sparse_ops.py:91: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
if ndim == 2:
/home/archerr/.conda/envs/mmdeploy/lib/python3.8/site-packages/mmcv/ops/sparse_ops.py:93: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
elif ndim == 3:
Process Process-2:
Traceback (most recent call last):
File "/home/archerr/.conda/envs/mmdeploy/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/home/archerr/.conda/envs/mmdeploy/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/archerr/mmdeploy-1.3.1/mmdeploy/apis/core/pipeline_manager.py", line 107, in call
ret = func(*args, **kwargs)
File "/home/archerr/mmdeploy-1.3.1/mmdeploy/apis/pytorch2onnx.py", line 98, in torch2onnx
export(
File "/home/archerr/mmdeploy-1.3.1/mmdeploy/apis/core/pipeline_manager.py", line 356, in wrap
return self.call_function(func_name, *args, **kwargs)
File "/home/archerr/mmdeploy-1.3.1/mmdeploy/apis/core/pipeline_manager.py", line 326, in call_function
return self.call_function_local(func_name, *args, **kwargs)
File "/home/archerr/mmdeploy-1.3.1/mmdeploy/apis/core/pipeline_manager.py", line 275, in call_function_local
return pipe_caller(*args, **kwargs)
File "/home/archerr/mmdeploy-1.3.1/mmdeploy/apis/core/pipeline_manager.py", line 107, in call
ret = func(*args, **kwargs)
File "/home/archerr/mmdeploy-1.3.1/mmdeploy/apis/onnx/export.py", line 138, in export
torch.onnx.export(
File "/home/archerr/.conda/envs/mmdeploy/lib/python3.8/site-packages/torch/onnx/init.py", line 316, in export
return utils.export(model, args, f, export_params, verbose, training,
File "/home/archerr/.conda/envs/mmdeploy/lib/python3.8/site-packages/torch/onnx/utils.py", line 107, in export
_export(model, args, f, export_params, verbose, training, input_names, output_names,
File "/home/archerr/.conda/envs/mmdeploy/lib/python3.8/site-packages/torch/onnx/utils.py", line 724, in _export
_model_to_graph(model, args, verbose, input_names,
File "/home/archerr/mmdeploy-1.3.1/mmdeploy/apis/onnx/optimizer.py", line 27, in model_to_graph__custom_optimizer
graph, params_dict, torch_out = ctx.origin_func(*args, **kwargs)
File "/home/archerr/.conda/envs/mmdeploy/lib/python3.8/site-packages/torch/onnx/utils.py", line 493, in _model_to_graph
graph, params, torch_out, module = _create_jit_graph(model, args)
File "/home/archerr/.conda/envs/mmdeploy/lib/python3.8/site-packages/torch/onnx/utils.py", line 437, in _create_jit_graph
graph, torch_out = _trace_and_get_graph_from_model(model, args)
File "/home/archerr/.conda/envs/mmdeploy/lib/python3.8/site-packages/torch/onnx/utils.py", line 388, in _trace_and_get_graph_from_model
torch.jit._get_trace_graph(model, args, strict=False, _force_outplace=False, _return_inputs_states=True)
File "/home/archerr/.conda/envs/mmdeploy/lib/python3.8/site-packages/torch/jit/_trace.py", line 1166, in _get_trace_graph
outs = ONNXTracedModule(f, strict, _force_outplace, return_inputs, _return_inputs_states)(*args, **kwargs)
File "/home/archerr/.conda/envs/mmdeploy/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/archerr/.conda/envs/mmdeploy/lib/python3.8/site-packages/torch/jit/_trace.py", line 127, in forward
graph, out = torch._C._create_graph_by_tracing(
File "/home/archerr/.conda/envs/mmdeploy/lib/python3.8/site-packages/torch/jit/_trace.py", line 118, in wrapper
outs.append(self.inner(*trace_inputs))
File "/home/archerr/.conda/envs/mmdeploy/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/archerr/.conda/envs/mmdeploy/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1090, in _slow_forward
result = self.forward(*input, **kwargs)
File "/home/archerr/mmdeploy-1.3.1/mmdeploy/apis/onnx/export.py", line 123, in wrapper
return forward(*arg, **kwargs)
File "/home/archerr/mmdeploy-1.3.1/mmdeploy/codebase/mmdet3d/models/mvx_two_stage.py", line 83, in mvxtwostagedetector__forward
_, pts_feats = self.extract_feat(batch_inputs_dict=batch_inputs_dict)
File "/home/archerr/mmdeploy-1.3.1/mmdeploy/codebase/mmdet3d/models/mvx_two_stage.py", line 42, in mvxtwostagedetector__extract_feat
pts_feats = self.extract_pts_feat(
File "/home/archerr/mmdetection3d-main/mmdet3d/models/detectors/mvx_two_stage.py", line 214, in extract_pts_feat
x = self.pts_middle_encoder(voxel_features, voxel_dict['coors'],
File "/home/archerr/.conda/envs/mmdeploy/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/archerr/.conda/envs/mmdeploy/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1090, in _slow_forward
result = self.forward(*input, **kwargs)
File "/home/archerr/.conda/envs/mmdeploy/lib/python3.8/contextlib.py", line 75, in inner
return func(*args, **kwds)
File "/home/archerr/mmdetection3d-main/mmdet3d/models/middle_encoders/sparse_encoder.py", line 145, in forward
x = self.conv_input(input_sp_tensor)
File "/home/archerr/.conda/envs/mmdeploy/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/archerr/.conda/envs/mmdeploy/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1090, in _slow_forward
result = self.forward(*input, **kwargs)
File "/home/archerr/.conda/envs/mmdeploy/lib/python3.8/site-packages/mmcv/ops/sparse_modules.py", line 135, in forward
input = module(input)
File "/home/archerr/.conda/envs/mmdeploy/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/archerr/.conda/envs/mmdeploy/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1090, in _slow_forward
result = self.forward(*input, **kwargs)
File "/home/archerr/.conda/envs/mmdeploy/lib/python3.8/site-packages/mmcv/ops/sparse_conv.py", line 157, in forward
outids, indice_pairs, indice_pair_num = ops.get_indice_pairs(
File "/home/archerr/.conda/envs/mmdeploy/lib/python3.8/site-packages/mmcv/ops/sparse_ops.py", line 99, in get_indice_pairs
return get_indice_pairs_func(indices, batch_size, out_shape,
RuntimeError: get_indice_pairs is not implemented on CPU
02/26 22:32:27 - mmengine - ERROR - /home/archerr/mmdeploy-1.3.1/mmdeploy/apis/core/pipeline_manager.py - pop_mp_output - 80 - mmdeploy.apis.pytorch2onnx.torch2onnx with Call id: 0 failed. exit.

Answer 1

blackteaEM 晚上好🌙🌙🌙
本答案参考通义千问

问题分析：

1. 报错原因：RuntimeError: get_indice_pairs is not implemented on CPU 表明在CPU上运行时，mmcv.ops.sparse_ops.get_indice_pairs 函数没有实现，这通常与模型中包含的稀疏操作有关，这些操作可能需要GPU支持。

2. 模型差异：您的问题模型 centerpoint_0075voxel_second_secfpn_dcn_circlenms_4x8_cyclic_20e_nus_20220810_025930-657f67e0.pth 可能包含了一些需要GPU支持的模块或功能，而官方示例模型可以成功转换，可能是因为它们的结构或配置更简单，不需要CPU上的稀疏运算。

解决方案：

1. 确认环境：

   • 确保你的系统安装了支持CPU稀疏运算的库，如scipy或sparse。
   • 如果没有，尝试安装：pip install scipy sparse
   • 确认mmcv和mmdeploy版本是否支持CPU稀疏运算。

2. 模型移植：

   • 针对不支持CPU的模块，可能需要手动迁移或替换为CPU版本的实现。查看mmcv.ops.sparse_ops.get_indice_pairs源码，确认是否可以直接在CPU上运行。
   • 如果找不到合适的CPU替代方案，可能需要调整模型结构，移除或替换依赖于GPU的模块，使其能够在CPU上运行。

3. 检查模型配置：

   • 查看模型配置文件（centerpoint_voxel0075_second_secfpn_head-dcn-circlenms_8xb4-cyclic-20e_nus-3d.py），确认是否有特别指出需要GPU的设置。
   • 确认模型中的所有层是否都支持CPU推理。

4. 使用兼容工具：

   • 如果可能，考虑使用一个针对CPU优化的版本的MMDetection3D模型，或者寻找一个支持CPU的模型作为起点。

5. 代码修改：

   • 如果上述方法无法解决，可能需要查看mvxtwostagedetector__forward函数和extract_pts_feat函数的具体实现，看看是否有可以替换为CPU版本的部分。
   • 如果有，找到对应的get_indice_pairs调用，并尝试使用@torch.jit.unused装饰器来标记它在CPU上不需要执行。

import torch

# 假设indice_pairs是一个需要在CPU上计算的属性
if torch.cuda.is_available():
    indice_pairs = ops.get_indice_pairs(indices, batch_size, out_shape)
else:
    # 使用torch.jit.unused标记在CPU上不需要计算
    @torch.jit.unused
    indice_pairs = ops.get_indice_pairs(indices, batch_size, out_shape)

1. 运行限制：
   • 如果模型确实依赖于GPU功能，你可能需要在部署时明确指定使用GPU，如--device cuda，而不是默认的CPU。

建议：

• 在尝试转换之前，先在本地环境中确认模型能否在CPU上正常运行。
• 如果模型复杂度高且CPU版本实现困难，考虑使用GPU部署或寻找其他支持CPU的轻量级模型。