20210619

[cuda10.2导致的bug]

好不容易将solov2_trt编译通过,运行期间又出妖蛾子了.

编译生成了 demo_solov2_coco 可执行文件(build/examples目录下)
平台上提供了 solov2_lite_coco_704.onnx 文件,需要使用onnx2trt转换成trt engine.
参考
< https://github.com/onnx/onnx-tensorrt >
按照安装步骤,然后运行

onnx2trt my_model.onnx -o my_engine.trt

转好了trt engine,就可以运行
./demo_solov2_coco -trt_f model.trt -data_f a.mp4

然后,果然有坑!
报错如下:
OpenCV: FFMPEG: tag 0x47504a4d/'MJPG' is not supported with codec id 7 and format 'mp4 / MP4 (MPEG-4 Part 14)'
OpenCV: FFMPEG: fallback to use tag 0x7634706d/'mp4v'
[06/19/2021-19:52:38] [F] [TRT] Assertion failed: cublasStatus == CUBLAS_STATUS_SUCCESS
../rtSafe/cublas/cublasLtWrapper.cpp:279
Aborting...
[06/19/2021-19:52:38] [E] [TRT] FAILED_EXECUTION: std::exception
[06/19/2021-19:52:38] [E] Execute Failed!
[06/19/2021-19:52:38] [I] input data size: 1486848Infer time is 0ms
[06/19/2021-19:52:38] [F] [TRT] Assertion failed: cublasStatus == CUBLAS_STATUS_SUCCESS
../rtSafe/cublas/cublasLtWrapper.cpp:279
Aborting...
[06/19/2021-19:52:38] [E] [TRT] FAILED_EXECUTION: std::exception
[06/19/2021-19:52:38] [E] Execute Failed!
[06/19/2021-19:52:38] [I] Infer time is 0ms
[06/19/2021-19:52:39] [F] [TRT] Assertion failed: cublasStatus == CUBLAS_STATUS_SUCCESS
../rtSafe/cublas/cublasLtWrapper.cpp:279
Aborting...
[06/19/2021-19:52:39] [E] [TRT] FAILED_EXECUTION: std::exception
[06/19/2021-19:52:39] [E] Execute Failed!
[06/19/2021-19:52:39] [I] Infer time is 0ms
[06/19/2021-19:52:39] [F] [TRT] Assertion failed: cublasStatus == CUBLAS_STATUS_SUCCESS
../rtSafe/cublas/cublasLtWrapper.cpp:279

经查,这个问题是cuda10.2的问题,取官网下载补丁(patch2)安装即可.
官网关于patch的介绍
Patch 2 (Released Nov 17, 2020)
This patch fixes an issue in cuBLAS library batched GEMM APIs which caused silent corruption of data in uncommon cases with large batch counts in mixed precision and fast math.

然后,成功踩完了所有的坑.


@刘看山

确实是c++14的问题.
在我的环境下,用平台上的方法没打开c++14编译.

最终,编译通过了

Solution

CMakeLists.txt 更换如下内容

#c++ 14 force!                                                                 
#set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++14")                           
set(CMAKE_CXX_STANDARD 14)                                                      
set(CMAKE_CXX_STANDARD_REQUIRED ON)                                             
set(CMAKE_CXX_EXTENSIONS OFF)

多次测试发现,只要使用 set(CMAKE_CXX_STANDARD 14)      即可,后两句在我的配置环境下不需要

改了之后不再报错.
(马后炮: 原来的方法似乎没有强制使用c++14进行编译)

参考
https://stackoverflow.com/questions/42834844/how-to-get-cmake-to-pass-either-std-c14-c1y-or-c17-c1z-based-on-gcc-vers

另外,编译还需安装cudnn8(对应cuda-10.2)
接下来,还应该有坑..

@刘看山
更新了编译环境,如下:
-- The C compiler identification is GNU 8.4.0
-- The CXX compiler identification is GNU 8.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-std=c++14
-- TensorRT Header => /home/yangjb/TensorRT/include
-- TensorRT Lib => /home/yangjb/TensorRT/lib
-- Project Lib => /home/yangjb/work/DL/strangeAI/solov2_trt/lib
-- Project source => /home/yangjb/work/DL/strangeAI/solov2_trt/src
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found CUDA: /usr/local/cuda (found suitable exact version "10.2")
-- Found OpenCV: /usr/local (found suitable version "4.5.3", minimum required is "4")
WARING Can not find opencv lib. It will use the default path => /usr/local/lib
-- OpenCV_INCLUDE_DIRS => /usr/local/include/opencv4
-- OpenCV_LIBRARY_DIRS => /usr/local/lib
-- OpenCV_LIBS => opencv_calib3d;opencv_core;opencv_dnn;opencv_features2d;opencv_flann;opencv_gapi;opencv_highgui;opencv_imgcodecs;opencv_imgproc;opencv_ml;opencv_objdetect;opencv_photo;opencv_stitching;opencv_video;opencv_videoio;opencv_alphamat;opencv_aruco;opencv_barcode;opencv_bgsegm;opencv_bioinspired;opencv_ccalib;opencv_cudaarithm;opencv_cudabgsegm;opencv_cudacodec;opencv_cudafilters;opencv_cudaimgproc;opencv_cudalegacy;opencv_cudaobjdetect;opencv_cudaoptflow;opencv_cudawarping;opencv_cudev;opencv_datasets;opencv_dnn_objdetect;opencv_dnn_superres;opencv_dpm;opencv_face;opencv_freetype;opencv_fuzzy;opencv_hdf;opencv_hfs;opencv_img_hash;opencv_intensity_transform;opencv_line_descriptor;opencv_mcc;opencv_optflow;opencv_phase_unwrapping;opencv_plot;opencv_quality;opencv_rapid;opencv_reg;opencv_rgbd;opencv_saliency;opencv_shape;opencv_structured_light;opencv_superres;opencv_surface_matching;opencv_text;opencv_videostab;opencv_wechat_qrcode;opencv_xfeatures2d;opencv_ximgproc;opencv_xobjdetect;opencv_xphoto
-- Found CUDA: /usr/local/cuda (found version "10.2")
WARING Can not find CUDA lib. It will use the default path => /usr/local/cuda/lib64
-- CUDA_INCLUDE_DIRS : /usr/local/cuda/include
-- CUDA_LIBRARY_DIRS : /usr/local/cuda/lib64
-- Found GTest: /usr/local/lib/libgtest.a
-- Configuring done
-- Generating done
-- Build files have been written to: /home/yangjb/work/DL/strangeAI/solov2_trt/build

make 还是报类似的错误

In file included from /home/yangjb/work/DL/strangeAI/solov2_trt/include/chest/solov2.h:34,
from /home/yangjb/work/DL/strangeAI/solov2_trt/examples/demo_solov2.cc:31:
/home/yangjb/work/DL/strangeAI/solov2_trt/include/cuda_common.h:164:28: error: ‘std::enable_if_t’ has not been declared
template <typename T, std::enable_if_t<!std::is_void::value, int> = 0>
^~~~~~~~~~~
/home/yangjb/work/DL/strangeAI/solov2_trt/include/cuda_common.h:164:39: error: expected ‘>’ before ‘<’ token
template <typename T, std::enable_if_t<!std::is_void::value, int> = 0>
^
/home/yangjb/work/DL/strangeAI/solov2_trt/include/cuda_common.h:170:28: error: ‘std::enable_if_t’ has not been declared
template <typename T, std::enable_if_t<std::is_void::value, int> = 0>
^~~~~~~~~~~
/home/yangjb/work/DL/strangeAI/solov2_trt/include/cuda_common.h:170:39: error: expected ‘>’ before ‘<’ token
template <typename T, std::enable_if_t<std::is_void::value, int> = 0>
^
/home/yangjb/work/DL/strangeAI/solov2_trt/include/cuda_common.h:171:20: error: redefinition of ‘template<class T, > void trtonnx2::cuda::CudaMemcpy(T*, const T*, size_t, cudaMemcpyKind)’
static inline void CudaMemcpy(T *dest, const T *src, size_t size,

踩坑了又

@刘看山 g++8刚才试了一下,cuda10.0不支持
教成里能不能提供一下编译通过的环境版本

solov2_trt工具链编译安装问题

按照教程安装了opencv4.5.3, tensorRT7.2.3.4以及thor.

cmake已经通过
-- TensorRT Header => /home/yangjb/TensorRT/include -- TensorRT Lib => /home/yangjb/TensorRT/lib -- Project Lib => /home/yangjb/work/DL/strangeAI/solov2_trt/lib -- Project source => /home/yangjb/work/DL/strangeAI/solov2_trt/src -- Found CUDA: /usr/local/cuda-10.0 (found suitable exact version "10.0") WARING Can not find opencv lib. It will use the default path => /usr/local/lib -- OpenCV_INCLUDE_DIRS => /usr/local/include/opencv4 -- OpenCV_LIBRARY_DIRS => /usr/local/lib -- OpenCV_LIBS => opencv_calib3d;opencv_core;opencv_dnn;opencv_features2d;opencv_flann;opencv_gapi;opencv_highgui;opencv_imgcodecs;opencv_imgproc;opencv_ml;opencv_objdetect;opencv_photo;opencv_stitching;opencv_video;opencv_videoio;opencv_alphamat;opencv_aruco;opencv_barcode;opencv_bgsegm;opencv_bioinspired;opencv_ccalib;opencv_cudaarithm;opencv_cudabgsegm;opencv_cudacodec;opencv_cudafilters;opencv_cudaimgproc;opencv_cudalegacy;opencv_cudaobjdetect;opencv_cudaoptflow;opencv_cudawarping;opencv_cudev;opencv_datasets;opencv_dnn_objdetect;opencv_dnn_superres;opencv_dpm;opencv_face;opencv_freetype;opencv_fuzzy;opencv_hdf;opencv_hfs;opencv_img_hash;opencv_intensity_transform;opencv_line_descriptor;opencv_mcc;opencv_optflow;opencv_phase_unwrapping;opencv_plot;opencv_quality;opencv_rapid;opencv_reg;opencv_rgbd;opencv_saliency;opencv_shape;opencv_structured_light;opencv_superres;opencv_surface_matching;opencv_text;opencv_videostab;opencv_wechat_qrcode;opencv_xfeatures2d;opencv_ximgproc;opencv_xobjdetect;opencv_xphoto -- Found CUDA: /usr/local/cuda-10.0 (found version "10.0") WARING Can not find CUDA lib. It will use the default path => /usr/local/cuda/lib64 -- CUDA_INCLUDE_DIRS : /usr/local/cuda-10.0/include -- CUDA_LIBRARY_DIRS : /usr/local/cuda/lib64 -- Configuring done -- Generating done -- Build files have been written to: /home/yangjb/work/DL/strangeAI/solov2_trt/build

make 报错

In file included from /home/yangjb/work/DL/strangeAI/solov2_trt/include/buffers.h:40:0,
from /home/yangjb/work/DL/strangeAI/solov2_trt/include/tensorrt_wrapper.h:42,
from /home/yangjb/work/DL/strangeAI/solov2_trt/src/tensorrt_wrapper.cc:28:
/home/yangjb/work/DL/strangeAI/solov2_trt/include/cuda_common.h:164:28: error: ‘std::enable_if_t’ has not been declared
template <typename T, std::enable_if_t<!std::is_void::value, int> = 0>
^~~~~~~~~~~
/home/yangjb/work/DL/strangeAI/solov2_trt/include/cuda_common.h:164:39: error: expected ‘>’ before ‘<’ token
template <typename T, std::enable_if_t<!std::is_void::value, int> = 0>
^
/home/yangjb/work/DL/strangeAI/solov2_trt/include/cuda_common.h:170:28: error: ‘std::enable_if_t’ has not been declared
template <typename T, std::enable_if_t<std::is_void::value, int> = 0>
^~~~~~~~~~~
/home/yangjb/work/DL/strangeAI/solov2_trt/include/cuda_common.h:170:39: error: expected ‘>’ before ‘<’ token
template <typename T, std::enable_if_t<std::is_void::value, int> = 0>
^
/home/yangjb/work/DL/strangeAI/solov2_trt/include/cuda_common.h:171:20: error: redefinition of ‘template<class T, > void trtonnx2::cuda::CudaMemcpy(T*, const T*, size_t, cudaMemcpyKind)’
static inline void CudaMemcpy(T *dest, const T *src, size_t size,
^~~~~~~~~~
/home/yangjb/work/DL/strangeAI/solov2_trt/include/cuda_common.h:165:20: note: ‘template<class T, > void trtonnx2::cuda::CudaMemcpy(T*, const T*, size_t, cudaMemcpyKind)’ previously declared here
static inline void CudaMemcpy(T *dest, const T *src, size_t size,
^~~~~~~~~~
/home/yangjb/work/DL/strangeAI/solov2_trt/include/cuda_common.h:191:28: error: ‘std::enable_if_t’ has not been declared
template <typename T, std::enable_if_t<!std::is_void::value, int> = 0>
^~~~~~~~~~~
/home/yangjb/work/DL/strangeAI/solov2_trt/include/cuda_common.h:191:39: error: expected ‘>’ before ‘<’ token
template <typename T, std::enable_if_t<!std::is_void::value, int> = 0>
^
/home/yangjb/work/DL/strangeAI/solov2_trt/include/cuda_common.h:199:28: error: ‘std::enable_if_t’ has not been declared
template <typename T, std::enable_if_t<std::is_void::value, int> = 0>
^~~~~~~~~~~
/home/yangjb/work/DL/strangeAI/solov2_trt/include/cuda_common.h:199:39: error: expected ‘>’ before ‘<’ token
template <typename T, std::enable_if_t<std::is_void::value, int> = 0>
^
/home/yangjb/work/DL/strangeAI/solov2_trt/include/cuda_common.h:200:20: error: redefinition of ‘template<class T, > void trtonnx2::cuda::CudaMemcpy(T*, const T*, size_t, cudaMemcpyKind, cudaStream_t)’
static inline void CudaMemcpy(T *dest, const T *src, size_t size,
^~~~~~~~~~
/home/yangjb/work/DL/strangeAI/solov2_trt/include/cuda_common.h:192:20: note: ‘template<class T, > void trtonnx2::cuda::CudaMemcpy(T*, const T*, size_t, cudaMemcpyKind, cudaStream_t)’ previously declared here
static inline void CudaMemcpy(T *dest, const T *src, size_t size,
^~~~~~~~~~
/home/yangjb/work/DL/strangeAI/solov2_trt/include/cuda_common.h:223:28: error: ‘std::enable_if_t’ has not been declared
template <typename T, std::enable_if_t<!std::is_void::value, int> = 0>
^~~~~~~~~~~
/home/yangjb/work/DL/strangeAI/solov2_trt/include/cuda_common.h:223:39: error: expected ‘>’ before ‘<’ token
template <typename T, std::enable_if_t<!std::is_void::value, int> = 0>
^
/home/yangjb/work/DL/strangeAI/solov2_trt/include/cuda_common.h:229:28: error: ‘std::enable_if_t’ has not been declared
template <typename T, std::enable_if_t<std::is_void::value, int> = 0>
^~~~~~~~~~~
/home/yangjb/work/DL/strangeAI/solov2_trt/include/cuda_common.h:229:39: error: expected ‘>’ before ‘<’ token
template <typename T, std::enable_if_t<std::is_void::value, int> = 0>
^
/home/yangjb/work/DL/strangeAI/solov2_trt/include/cuda_common.h:230:20: error: redefinition of ‘template<class T, > void trtonnx2::cuda::CudaMemset(T*, int, size_t, cudaStream_t)’
static inline void CudaMemset(T *dest, int value, size_t size,
^~~~~~~~~~
/home/yangjb/work/DL/strangeAI/solov2_trt/include/cuda_common.h:224:20: note: ‘template<class T, > void trtonnx2::cuda::CudaMemset(T*, int, size_t, cudaStream_t)’ previously declared here
static inline void CudaMemset(T *dest, int value, size_t size,
^~~~~~~~~~
/home/yangjb/work/DL/strangeAI/solov2_trt/include/cuda_common.h:235:28: error: ‘std::enable_if_t’ has not been declared
template <typename T, std::enable_if_t<!std::is_void::value, int> = 0>
^~~~~~~~~~~
/home/yangjb/work/DL/strangeAI/solov2_trt/include/cuda_common.h:235:39: error: expected ‘>’ before ‘<’ token
template <typename T, std::enable_if_t<!std::is_void::value, int> = 0>
^
/home/yangjb/work/DL/strangeAI/solov2_trt/include/cuda_common.h:240:28: error: ‘std::enable_if_t’ has not been declared
template <typename T, std::enable_if_t<std::is_void::value, int> = 0>
^~~~~~~~~~~
/home/yangjb/work/DL/strangeAI/solov2_trt/include/cuda_common.h:240:39: error: expected ‘>’ before ‘<’ token
template <typename T, std::enable_if_t<std::is_void::value, int> = 0>
^
/home/yangjb/work/DL/strangeAI/solov2_trt/include/cuda_common.h:241:20: error: redefinition of ‘template<class T, > void trtonnx2::cuda::CudaMemset(T*, int, size_t)’
static inline void CudaMemset(T *dest, int value, size_t size) {
^~~~~~~~~~
/home/yangjb/work/DL/strangeAI/solov2_trt/include/cuda_common.h:236:20: note: ‘template<class T, > void trtonnx2::cuda::CudaMemset(T*, int, size_t)’ previously declared here
static inline void CudaMemset(T *dest, int value, size_t size) {
^~~~~~~~~~
/home/yangjb/work/DL/strangeAI/solov2_trt/include/cuda_common.h:246:28: error: ‘std::enable_if_t’ has not been declared
template <typename T, std::enable_if_t<!std::is_void::value, int> = 0>
^~~~~~~~~~~
/home/yangjb/work/DL/strangeAI/solov2_trt/include/cuda_common.h:246:39: error: expected ‘>’ before ‘<’ token
template <typename T, std::enable_if_t<!std::is_void::value, int> = 0>
^
/home/yangjb/work/DL/strangeAI/solov2_trt/include/cuda_common.h:252:28: error: ‘std::enable_if_t’ has not been declared
template <typename T, std::enable_if_t<std::is_void::value, int> = 0>
^~~~~~~~~~~
/home/yangjb/work/DL/strangeAI/solov2_trt/include/cuda_common.h:252:39: error: expected ‘>’ before ‘<’ token
template <typename T, std::enable_if_t<std::is_void::value, int> = 0>
^
/home/yangjb/work/DL/strangeAI/solov2_trt/include/cuda_common.h:253:20: error: redefinition of ‘template<class T, > void trtonnx2::cuda::CudaMalloc(T**, size_t)’
static inline void CudaMalloc(T **dest, size_t size) {
^~~~~~~~~~
/home/yangjb/work/DL/strangeAI/solov2_trt/include/cuda_common.h:247:20: note: ‘template<class T, > void trtonnx2::cuda::CudaMalloc(T**, size_t)’ previously declared here
static inline void CudaMalloc(T **dest, size_t size) {
^~~~~~~~~~
/home/yangjb/work/DL/strangeAI/solov2_trt/include/cuda_common.h:261:28: error: ‘std::enable_if_t’ has not been declared
template <typename T, std::enable_if_t<!std::is_void::value, int> = 0>
^~~~~~~~~~~
/home/yangjb/work/DL/strangeAI/solov2_trt/include/cuda_common.h:261:39: error: expected ‘>’ before ‘<’ token
template <typename T, std::enable_if_t<!std::is_void::value, int> = 0>
^
/home/yangjb/work/DL/strangeAI/solov2_trt/include/cuda_common.h:267:28: error: ‘std::enable_if_t’ has not been declared
template <typename T, std::enable_if_t<std::is_void::value, int> = 0>
^~~~~~~~~~~
/home/yangjb/work/DL/strangeAI/solov2_trt/include/cuda_common.h:267:39: error: expected ‘>’ before ‘<’ token
template <typename T, std::enable_if_t<std::is_void::value, int> = 0>
^
/home/yangjb/work/DL/strangeAI/solov2_trt/include/cuda_common.h:268:20: error: redefinition of ‘template<class T, > void trtonnx2::cuda::CudaHostRegister(T*, size_t)’
static inline void CudaHostRegister(T *dest, size_t size) {
^~~~~~~~~~~~~~~~
/home/yangjb/work/DL/strangeAI/solov2_trt/include/cuda_common.h:262:20: note: ‘template<class T, > void trtonnx2::cuda::CudaHostRegister(T*, size_t)’ previously declared here
static inline void CudaHostRegister(T *dest, size_t size) {
^~~~~~~~~~~~~~~~
In file included from /home/yangjb/work/DL/strangeAI/solov2_trt/include/tensorrt_wrapper.h:42:0,
from /home/yangjb/work/DL/strangeAI/solov2_trt/src/tensorrt_wrapper.cc:28:
/home/yangjb/work/DL/strangeAI/solov2_trt/include/buffers.h: In member function ‘void trtonnx2::BufferManager::freeCudaBuffers()’:
/home/yangjb/work/DL/strangeAI/solov2_trt/include/buffers.h:222:52: warning: deleting ‘void*’ is undefined [-Wdelete-incomplete]
delete mManagedBuffers[i]->deviceBuffer.data();
^
CMakeFiles/solov2_trt.dir/build.make:1668: recipe for target 'CMakeFiles/solov2_trt.dir/src/tensorrt_wrapper.cc.o' failed
make[2]: *** [CMakeFiles/solov2_trt.dir/src/tensorrt_wrapper.cc.o] Error 1
CMakeFiles/Makefile2:72: recipe for target 'CMakeFiles/solov2_trt.dir/all' failed
make[1]: *** [CMakeFiles/solov2_trt.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2

环境配置:

• ubuntu18.04
• gcc 7.5.0
• NVIDIA-SMI 450.102.04 Driver Version: 450.102.04
• cuda10.0
• opencv4.5.3

希望得到大佬门的一些提示,感谢!

实测torch1.6版本导出ONNX会出错,torch升级到1.7就可以了.非常感谢!