Pytorch框架nn.RNN训练时反向传播报错

Question

邀请@Mr.Winter` ：

基于Pytorch框架，在训练nn.RNN模型时，反向传播报错。代码简化为以下：

Python版本3.9，torch版本1.13.1

```python

import torch

rnn = torch.nn.RNN(input_size=1, hidden_size=1, num_layers=1)

train_set_x = torch.tensor([[[1]],[[2]],[[3]],[[4]],[[5]]], dtype=torch.float32)
train_set_y = torch.tensor([[[2]],[[4]],[[6]],[[8]],[[10]]], dtype=torch.float32)

h0 = torch.tensor([[0]], dtype=torch.float32)
h_cur = h0

loss = torch.nn.MSELoss()
opt = torch.optim.Adadelta(rnn.parameters(), lr = 0.01)


for i in range(5):
    opt.zero_grad()
    train_output, h_next = rnn(train_set_x[i], h_cur)
    rnn_loss = loss(train_output,train_set_y[i])
    rnn_loss.backward()
    opt.step()
    print(train_output)
    h_cur = h_next

报错内容

```python
RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.

按照提示修改代码：

import torch

rnn = torch.nn.RNN(input_size=1, hidden_size=1, num_layers=1)

train_set_x = torch.tensor([[[1]],[[2]],[[3]],[[4]],[[5]]], dtype=torch.float32)
train_set_y = torch.tensor([[[2]],[[4]],[[6]],[[8]],[[10]]], dtype=torch.float32)

h0 = torch.tensor([[0]], dtype=torch.float32)
h_cur = h0

loss = torch.nn.MSELoss()
opt = torch.optim.Adadelta(rnn.parameters(), lr = 0.01)

with torch.autograd.set_detect_anomaly(True):
    for i in range(5):
        opt.zero_grad()
        train_output, h_next = rnn(train_set_x[i], h_cur)
        rnn_loss = loss(train_output,train_set_y[i])
        rnn_loss.backward(retain_graph=True)
        opt.step()
        print(train_output)
        h_cur = h_next

仍然报错：

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1, 1]], which is output 0 of AsStridedBackward0, is at version 3; expected version 2 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

让人费解的是上面的提示是因为“置位”操作导致的，但是我全程没有用过“+=”这种操作。
最最让人费解的是，通过print可以看到前两个训练循环是有输出的，到第三个循环就开始报错了：

tensor([[0.1129]], grad_fn=<SqueezeBackward1>)
tensor([[-0.1872]], grad_fn=<SqueezeBackward1>)
C:\Users\Lenovo\Desktop\DL\LSTM_poem\lib\site-packages\torch\autograd\__init__.py:197: UserWarning: Error detected in AddmmBackward0. Traceback of forward call that caused the error:
  File "C:\Users\Lenovo\Desktop\DL\LSTM_poem\test.py", line 17, in <module>
    train_output, h_next = rnn(train_set_x[i], h_cur)
....后面省略

在C站上有很多提到过这个问题的解决方式，尝试解决都不灵。

Answer 1

基于最新版ChatGPT4的回答，望采纳!!!有其他问题也可以询问我哦、”(最新版更智能，功能更加强大)

这个问题是由于在循环中，计算图被保留了，但在每次循环时，其实我们需要重新构建计算图。因此，需要将 retain_graph=True 改为 retain_graph=False 或直接删除 retain_graph 参数。另外，由于在每个循环中，我们都在使用相同的RNN对象，因此需要在每次循环时将梯度清零，以防止梯度累加。同时，由于 RNN 的特性，其输入数据的维度应该是 (sequence_length, batch_size, input_size)，因此我们需要将输入数据 train_set_x 和 train_set_y 的维度调整为 (1, 1, 1)。

以下是修改后的代码：

import torch

rnn = torch.nn.RNN(input_size=1, hidden_size=1, num_layers=1)

train_set_x = torch.tensor([[[1]],[[2]],[[3]],[[4]],[[5]]], dtype=torch.float32)
train_set_y = torch.tensor([[[2]],[[4]],[[6]],[[8]],[[10]]], dtype=torch.float32)

h0 = torch.tensor([[0]], dtype=torch.float32)
h_cur = h0

loss = torch.nn.MSELoss()
opt = torch.optim.Adadelta(rnn.parameters(), lr = 0.01)

for i in range(5):
    opt.zero_grad()
    train_output, h_next = rnn(train_set_x[i].view(1, 1, 1), h_cur)
    rnn_loss = loss(train_output, train_set_y[i].view(1, 1, 1))
    rnn_loss.backward()
    opt.step()
    print(train_output)
    h_cur = h_next.detach()

在这个修改后的代码中，我们在每次循环时，都将输入数据 train_set_x[i] 和 train_set_y[i] 通过 .view(1, 1, 1) 方法调整为 (1, 1, 1) 的维度。另外，我们在每次循环结束时使用 h_cur = h_next.detach()，以将 h_next 从计算图中分离出来，避免在后续的循环中引起计算图保留的问题。这样，在每次循环时，我们都会重新构建计算图，避免了之前出现的错误。