tensorflow断点续训报错

Question

tensorflow断点续训报错，求解

checkpoint_save_path = "./checkpoint/mnist.ckpt"
if os.path.exists(checkpoint_save_path + '.index'):
    print('-------------load the model-----------------')
    model.load_weights(checkpoint_save_path)

cp_callback = tf.keras.callbacks.ModelCheckpoint(filepath=checkpoint_save_path,
                                                 save_weights_only=True,
                                                 save_best_only=True)
history = model.fit(x_train, y_train, batch_size=32, epochs=5, validation_data=(x_test, y_test), validation_freq=1,
                    callbacks=[cp_callback])

运行结果：

2022-06-16 15:10:06.532407: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at save_restore_v2_ops.cc:283 : UNKNOWN: Failed to rename: ./checkpoint\mnist.ckpt_temp/part-00000-of-00001.data-00000-of-00001 to: ./checkpoint\mnist.ckpt.data-00000-of-00001 : ���ļ��Ѵ���ʱ���޷��������ļ���
; Unknown error
Traceback (most recent call last):
  File "D:/tensorflow/class4/class4/MNIST_FC/p16_mnist_train_ex3.py", line 29, in <module>
    callbacks=[cp_callback])
  File "D:\Python37\lib\site-packages\keras\utils\traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "D:\Python37\lib\site-packages\tensorflow\python\eager\execute.py", line 55, in quick_execute
    inputs, attrs, num_outputs)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 137: invalid start byte

Answer 1

执行model.load_weights(filepath)后，filepath这个文件会被占用，无法删除或重命名，导致后面的回调函数ModelCheckpoint(filepath)无法自动保存权重文件。暂时想到的方法是，model.load_weights(filepath1)和ModelCheckpoint(filepath2)使用两个不同的文件名，经测试可行。只是每次运行都需要修改model.load_weights(filepath)的参数。