About BatchSize failure of TensorRT-8.0.1.6 when running trtexec tool on NVIDIA Jetson Xavier NX

Question

Description

Error Info (When I run “/usr/src/tensorrt/bin/trtexec --loadEngine=XXX.engine --batch=40 --warmUp=200 --iterations=2000”)

Error[3]: [executionContext.cpp::enqueue::276] Error Code 3: Internal Error (Parameter check failed at: runtime/api/executionContext.cpp::enqueue::276, condition: batchSize > 0 && batchSize <= mEngine.getMaxBatchSize(). Note: Batch size was: 40, but engine max batch size was: 1
)
[12/12/2023-21:35:03] [E] Error occurred during inference

Here is my script to convert the engine according to https://docs.nvidia.com/deeplearning/tensorrt/archives/tensorrt-801/quick-start-guide/index.html

/usr/src/tensorrt/bin/trtexec --onnx=${1} --saveEngine=${2}  --buildOnly --workspace=10240 --explicitBatch --minShapes=input:1x3x224x224 --optShapes=input:40x3x224x224 --maxShapes=input:40x3x224x224 --fp16

Note: By the way, my Jetson has a fixed version, so I have to solve it in this version. I look forward to your reply, thank you.

Environment

TensorRT Version: 8.0.1.6

NVIDIA GPU: NVIDIA Jetson Xavier NX

CUDA Version: 10.2

CUDNN Version: 8.2.1.32

Operating System: ubuntu18.04

Jetpack: 4.6-b199

Answer 1

问题描述：

在NVIDIA Jetson Xavier NX上使用TensorRT 8.0.1.6版本的trtexec工具时，当运行参数设置为--batch=40时会出现错误，报错信息提示batchSize > 0 && batchSize <= mEngine.getMaxBatchSize()参数检查失败，而实际上batch size为40，但是引擎的最大批处理大小为1。

解决方案：

这个错误的原因是引擎的最大批处理大小小于指定的批处理大小。在这种情况下，我们需要重新生成引擎，并将--maxBatch设置为更高的值。可以尝试将--optShapes和--maxShapes参数设置为相同的值，以避免出现此问题。如果这不起作用，可以尝试逐步增加--maxBatch的值，直到满足实际需求。另外，也可以通过调整模型的输入形状来使引擎支持更高的批处理大小。
Problem:

When using the trtexec tool of TensorRT 8.0.1.6 on NVIDIA Jetson Xavier NX, an error occurs when the --batch=40 option is set. The error message indicates that "Parameter check failed at: runtime/api/executionContext.cpp::enqueue::276, condition: batchSize > 0 && batchSize <= mEngine.getMaxBatchSize(). Note: Batch size was: 40, but engine max batch size was: 1".

Solution:

The reason for this error is that the maximum batch size of the engine is less than the specified batch size. In this case, we need to regenerate the engine and set --maxBatch to a higher value. You can try setting the --optShapes and --maxShapes parameters to the same value to avoid this problem. If this doesn't work, you can try gradually increasing the value of --maxBatch until it meets your actual needs. Alternatively, you can adjust the input shape of the model to make the engine support higher batch sizes.