onhuko 2023-04-13 17:09 采纳率: 58.3%
浏览 69
已结题

关于#tensorflow#的问题,如何解决?

tensorflow 报错内容如下 这是什么问题


class Encoder(tf.keras.Model) :
    def __init__(self, vocab_size, embedding_dim, enc_units, batch_sz) :
        super(Encoder, self).__init__()
        self.batch_sz = batch_sz

        self.enc_units = enc_units

        self.embedding = keras.layers.Embedding(vocab_size, embedding_dim)

        self.gru = keras.layers.GRU(self.enc_units, return_sequences=True, return_state=True, recurrent_initializer="glorot_uniform")

    @tf.function
    def call(self, x, hidden) :
        x = self.embedding(x)
        output, state = self.gru(x, initial_state=hidden)
        return output, state
    def initialize_hidden_state(self) :
        return tf.zeros((self.batch_sz, self.enc_units))


class BahdanauAttentionMechanism(tf.keras.layers.Layer) :

    def __init__(self, units) :
        super(BahdanauAttentionMechanism, self).__init__()

        self.W1 = layers.Dense(units)

        self.W2 = layers.Dense(units)

        self.V = layers.Dense(1)
    @tf.function
    def call(self, query, values) :
        hidden_with_time_axis = tf.expand_dims(query, 1)

        score = self.V(tf.nn.tanh(self.W1(values) + self.W2(hidden_with_time_axis)))

        attention_weights = tf.nn.softmax(score, axis=1)

        context_vector = attention_weights * values
        context_vector = tf.math.reduce_sum(context_vector, axis=1)

        return context_vector, attention_weights



class Decoder(tf.keras.Model) :
    def __init__(self, vocab_size, embedding_dim, dec_units, batch_sz) :
        super(Decoder, self).__init__()
        
        self.batch_sz = batch_sz

        self.dec_units = dec_units

        self.embedding = layers.Embedding(vocab_size, embedding_dim) 


        self.gru = layers.GRU(self.dec_units, return_sequences=True, return_state=True, recurrent_initializer="glorot_uniform")

        self.fc = layers.Dense(vocab_size)

        self.attention = BahdanauAttentionMechanism(self.dec_units)
    @tf.function
    def call(self, x, hidden, enc_output) :
        context_vector, attention_weights = self.attention(hidden, enc_output)
    
        x = self.embedding(x)
        x = tf.concat([tf.expand_dims(context_vector, 1), x], axis=-1)
        output, state = self.gru(x)

        output = tf.reshape(output, (-1, output.shape[2]))

        x = self.fc(output)
        return x, state, attention_weights
Current allocation summary follows.
Current allocation summary follows.
2023-04-13 17:02:03.448253: W tensorflow/core/framework/op_kernel.cc:1780] OP_REQUIRES failed at matmul_op_impl.h:728 : RESOURCE_EXHAUSTED: OOM when allocating tensor with shape[64,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
2023-04-13 17:02:03.449923: W tensorflow/core/common_runtime/bfc_allocator.cc:491] ***************************************************************************************************x
2023-04-13 17:02:03.450152: W tensorflow/core/framework/op_kernel.cc:1780] OP_REQUIRES failed at resource_variable_ops.cc:725 : RESOURCE_EXHAUSTED: OOM when allocating tensor with shape[64,1,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
  File "C:\Users\DELL\Desktop\h\p1.py", line 326, in <module>
    train_model(q_hidden, encoder, decoder, q_index, BATCH_SIZE, dataset, steps_per_epoch, optimizer, checkpoint, checkpoint_prefix, summary_writer)
  File "C:\Users\DELL\Desktop\h\p1.py", line 208, in train_model
    batch_loss = optimizer_loss(q, a, q_hidden, encoder, decoder, q_index, BATCH_SIZE, optimizer)
  File "C:\Users\DELL\Desktop\h\p1.py", line 189, in optimizer_loss
    batch_loss, grads = grad_loss(q, a, q_hidden, encoder, decoder, q_index, BATCH_SIZE)
  File "C:\Users\DELL\Desktop\h\p1.py", line 175, in grad_loss
    predictions, a_hidden, _ = decoder(a_input, a_hidden, q_output)
  File "C:\Users\DELL\AppData\Roaming\Python\Python310\site-packages\keras\utils\traceback_utils.py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "C:\Users\DELL\AppData\Roaming\Python\Python310\site-packages\tensorflow\python\eager\execute.py", line 54, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.ResourceExhaustedError: Exception encountered when calling layer "decoder" "                 f"(type Decoder).

Graph execution error:

Detected at node 'dense_5/Tensordot/MatMul' defined at (most recent call last):
    File "C:\Users\DELL\Desktop\h\p1.py", line 319, in <module>
      a_output, _, _ = decoder(tf.random.uniform((64, 1)), q_hidden, q_output)
    File "C:\Users\DELL\AppData\Roaming\Python\Python310\site-packages\keras\utils\traceback_utils.py", line 65, in error_handler
      return fn(*args, **kwargs)
    File "C:\Users\DELL\AppData\Roaming\Python\Python310\site-packages\keras\engine\training.py", line 557, in __call__
      return super().__call__(*args, **kwargs)
    File "C:\Users\DELL\AppData\Roaming\Python\Python310\site-packages\keras\utils\traceback_utils.py", line 65, in error_handler
      return fn(*args, **kwargs)
    File "C:\Users\DELL\AppData\Roaming\Python\Python310\site-packages\keras\engine\base_layer.py", line 1097, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "C:\Users\DELL\AppData\Roaming\Python\Python310\site-packages\keras\utils\traceback_utils.py", line 96, in error_handler
      return fn(*args, **kwargs)
    File "C:\Users\DELL\Desktop\h\p1.py", line 138, in call
      context_vector, attention_weights = self.attention(hidden, enc_output)
    File "C:\Users\DELL\AppData\Roaming\Python\Python310\site-packages\keras\utils\traceback_utils.py", line 65, in error_handler
      return fn(*args, **kwargs)
    File "C:\Users\DELL\AppData\Roaming\Python\Python310\site-packages\keras\engine\base_layer.py", line 1097, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "C:\Users\DELL\AppData\Roaming\Python\Python310\site-packages\keras\utils\traceback_utils.py", line 96, in error_handler
      return fn(*args, **kwargs)
    File "C:\Users\DELL\Desktop\h\p1.py", line 109, in call
      score = self.V(tf.nn.tanh(self.W1(values) + self.W2(hidden_with_time_axis)))
    File "C:\Users\DELL\AppData\Roaming\Python\Python310\site-packages\keras\utils\traceback_utils.py", line 65, in error_handler
      return fn(*args, **kwargs)
    File "C:\Users\DELL\AppData\Roaming\Python\Python310\site-packages\keras\engine\base_layer.py", line 1097, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "C:\Users\DELL\AppData\Roaming\Python\Python310\site-packages\keras\utils\traceback_utils.py", line 96, in error_handler
      return fn(*args, **kwargs)
    File "C:\Users\DELL\AppData\Roaming\Python\Python310\site-packages\keras\layers\core\dense.py", line 244, in call
      outputs = tf.tensordot(inputs, self.kernel, [[rank - 1], [0]])
Node: 'dense_5/Tensordot/MatMul'
OOM when allocating tensor with shape[64,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[{{node dense_5/Tensordot/MatMul}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.
 [Op:__forward_call_3520]

Call arguments received by layer "decoder" "                 f"(type Decoder):
  • x=tf.Tensor(shape=(64, 1), dtype=int32)
  • hidden=tf.Tensor(shape=(64, 1024), dtype=float32)
  • enc_output=tf.Tensor(shape=(64, 74, 1024), dtype=float32)

  • 写回答

6条回答 默认 最新

  • teellyy 2023-04-13 20:18
    关注

    这个错误提示是说在计算过程中,尝试在GPU上分配一个shape为[64,1024]的float型tensor时内存不足,导致程序崩溃。同样的原因也导致了另一个位置出现了类似的错误提示。

    解决这个问题的思路有以下几种:

    1.减小batch_size
    减小batch_size可以减少每次计算需要的内存,从而避免内存不足的情况。不过这样可能会降低训练效率。

    2.减小模型参数量
    减小模型参数量可以减少每次计算需要的内存,可以通过使用更小的embedding_dim或者减小模型层数等方式实现。但是这样会牺牲模型的表达能力。

    3.将模型移动到CPU上计算
    如果GPU内存真的非常有限,可以考虑将模型移动到CPU上计算。不过这样会大大降低训练速度,需谨慎考虑。

    4.增加GPU显存
    如果以上方法都不能解决问题,可以考虑增加GPU显存。这可以通过更换显卡或者在代码中调用tf.config.experimental.set_memory_growth方法来实现。

    下面是增加显存的示例代码:

    import tensorflow as tf
    gpu_options = tf.compat.v1.GPUOptions(per_process_gpu_memory_fraction=0.7)
    sess = tf.compat.v1.Session(config=tf.compat.v1.ConfigProto(gpu_options=gpu_options))
    tf.compat.v1.keras.backend.set_session(sess)
    
    
    

    将这段代码加入到你的程序中,并尝试减小per_process_gpu_memory_fraction的值,看是否能解决问题。

    本回答被题主选为最佳回答 , 对您是否有帮助呢?
    评论
查看更多回答(5条)

报告相同问题?

问题事件

  • 系统已结题 4月22日
  • 已采纳回答 4月14日
  • 创建了问题 4月13日

悬赏问题

  • ¥15 数电设计题 没有设计思路 不知道用什么芯片进行设计 求提供设计思路
  • ¥15 在动态多目标优化问题中,第一幅图展示的是问题DF6的相关定义和绘制的POS和POF图,请问图中公式PS(t)和PF(t)是如何推导的
  • ¥60 设计一种优化算法结合案例给出智能仓储四向穿梭车的调度计划
  • ¥15 Errno2:No such file or directory,在当前文件确实没有该图片,怎么解决?
  • ¥15 博世摄像头数据存储的问题(iscsi)
  • ¥15 如何实现对学生籍贯信息管理系统的选择排序
  • ¥15 写一个51单片机的时钟代码
  • ¥15 git clone报错
  • ¥15 3d-slicer超声造影动态图像导入报错
  • ¥15 化工过程分析与合成问题求解决