2021-06-05 10:10:45.955409: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-06-05 10:10:48.441744: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-06-05 10:10:48.449493: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library nvcuda.dll
2021-06-05 10:10:49.418418: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 1650 Ti computeCapability: 7.5
coreClock: 1.485GHz coreCount: 16 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 178.84GiB/s
2021-06-05 10:10:49.418570: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-06-05 10:10:49.461584: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2021-06-05 10:10:49.461670: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
2021-06-05 10:10:49.484312: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll
2021-06-05 10:10:49.491009: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll
2021-06-05 10:10:49.543460: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll
2021-06-05 10:10:49.557900: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll
2021-06-05 10:10:49.560458: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
2021-06-05 10:10:49.560875: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-06-05 10:10:49.562573: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-06-05 10:10:49.563781: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 1650 Ti computeCapability: 7.5
coreClock: 1.485GHz coreCount: 16 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 178.84GiB/s
2021-06-05 10:10:49.563941: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll
2021-06-05 10:10:49.564020: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2021-06-05 10:10:49.564098: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll
2021-06-05 10:10:49.564177: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll
2021-06-05 10:10:49.564255: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll
2021-06-05 10:10:49.564333: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll
2021-06-05 10:10:49.564414: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll
2021-06-05 10:10:49.564495: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
2021-06-05 10:10:49.564601: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-06-05 10:10:50.383146: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-06-05 10:10:50.383232: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0
2021-06-05 10:10:50.383279: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N
2021-06-05 10:10:50.383857: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2903 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1650 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5)
2021-06-05 10:10:50.385197: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
Create YOLOv3 model with 9 anchors and 1 classes.
Train on 2135 samples, val on 237 samples, with batch size 10.
2021-06-05 10:10:52.113656: I tensorflow/core/profiler/lib/profiler_session.cc:136] Profiler session initializing.
2021-06-05 10:10:52.113723: I tensorflow/core/profiler/lib/profiler_session.cc:155] Profiler session started.
2021-06-05 10:10:52.114477: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1365] Profiler found 1 GPUs
2021-06-05 10:10:52.115359: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cupti64_110.dll'; dlerror: cupti64_110.dll not found
2021-06-05 10:10:52.115736: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cupti.dll'; dlerror: cupti.dll not found
2021-06-05 10:10:52.115997: E tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1415] function cupti_interface_->Subscribe( &subscriber_, (CUpti_CallbackFunc)ApiCallback, this)failed with error CUPTI could not be loaded or symbol could not be found.
2021-06-05 10:10:52.116132: I tensorflow/core/profiler/lib/profiler_session.cc:172] Profiler session tear down.
2021-06-05 10:10:52.116193: E tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1496] function cupti_interface_->Finalize()failed with error CUPTI could not be loaded or symbol could not be found.
WARNING:tensorflow:`period` argument is deprecated. Please use `save_freq` to specify the frequency in number of batches seen.
D:\anaconda\envs\tensorflow2\lib\site-packages\tensorflow\python\keras\engine\training.py:1844: UserWarning: `Model.fit_generator` is deprecated and will be removed in a future version. Please use `Model.fit`, which supports generators.
warnings.warn('`Model.fit_generator` is deprecated and '
2021-06-05 10:10:52.667257: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
Epoch 1/500
2021-06-05 10:11:01.677405: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:592] layout failed: Invalid argument: Subshape must have computed start >= end since stride is negative, but is 0 and 2 (computed from start 0 and end 9223372036854775807 over shape with rank 2 and stride-1)
2021-06-05 10:11:02.525055: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll
2021-06-05 10:11:04.488338: I tensorflow/core/platform/windows/subprocess.cc:308] SubProcess ended with return code: 0
2021-06-05 10:11:04.544183: I tensorflow/core/platform/windows/subprocess.cc:308] SubProcess ended with return code: 0
2021-06-05 10:11:04.662407: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll
2021-06-05 10:11:05.360511: E tensorflow/stream_executor/cuda/cuda_blas.cc:226] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2021-06-05 10:11:05.631732: E tensorflow/stream_executor/cuda/cuda_blas.cc:226] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2021-06-05 10:11:05.645877: E tensorflow/stream_executor/cuda/cuda_blas.cc:226] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2021-06-05 10:11:05.645954: W tensorflow/stream_executor/stream.cc:1455] attempting to perform BLAS operation using StreamExecutor without BLAS support
Traceback (most recent call last):
File "D:/PyCharm_2020.2.3/VOCdevkit/VOC2007/keras-yolo3-master/train.py", line 109, in <module>
_main()
File "D:/PyCharm_2020.2.3/VOCdevkit/VOC2007/keras-yolo3-master/train.py", line 22, in _main
train(model, annotation_path, input_shape, anchors, len(class_names), log_dir=log_dir)
File "D:/PyCharm_2020.2.3/VOCdevkit/VOC2007/keras-yolo3-master/train.py", line 44, in train
initial_epoch=0)
File "D:\anaconda\envs\tensorflow2\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1861, in fit_generator
initial_epoch=initial_epoch)
File "D:\anaconda\envs\tensorflow2\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1100, in fit
tmp_logs = self.train_function(iterator)
File "D:\anaconda\envs\tensorflow2\lib\site-packages\tensorflow\python\eager\def_function.py", line 828, in __call__
result = self._call(*args, **kwds)
File "D:\anaconda\envs\tensorflow2\lib\site-packages\tensorflow\python\eager\def_function.py", line 888, in _call
return self._stateless_fn(*args, **kwds)
File "D:\anaconda\envs\tensorflow2\lib\site-packages\tensorflow\python\eager\function.py", line 2943, in __call__
filtered_flat_args, captured_inputs=graph_function.captured_inputs) # pylint: disable=protected-access
File "D:\anaconda\envs\tensorflow2\lib\site-packages\tensorflow\python\eager\function.py", line 1919, in _call_flat
ctx, args, cancellation_manager=cancellation_manager))
File "D:\anaconda\envs\tensorflow2\lib\site-packages\tensorflow\python\eager\function.py", line 560, in call
ctx=ctx)
File "D:\anaconda\envs\tensorflow2\lib\site-packages\tensorflow\python\eager\execute.py", line 60, in quick_execute
inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.InternalError: 2 root error(s) found.
(0) Internal: Blas SGEMM launch failed : m=432640, n=32, k=64
[[node model_2/conv2d_2/Conv2D (defined at /PyCharm_2020.2.3/VOCdevkit/VOC2007/keras-yolo3-master/train.py:44) ]]
[[model_2/yolo_loss/while_2/LoopCond/_26/_146]]
(1) Internal: Blas SGEMM launch failed : m=432640, n=32, k=64
[[node model_2/conv2d_2/Conv2D (defined at /PyCharm_2020.2.3/VOCdevkit/VOC2007/keras-yolo3-master/train.py:44) ]]
0 successful operations.
0 derived errors ignored. [Op:__inference_train_function_23106]
Function call stack:
train_function -> train_function
2021-06-05 10:11:05.935333: W tensorflow/core/kernels/data/generator_dataset_op.cc:107] Error occurred when finalizing GeneratorDataset iterator: Failed precondition: Python interpreter state is not initialized. The process may be terminated.
[[{{node PyFunc}}]]
Process finished with exit code 1
