Encountered An Error: RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation.

[Jason-Young-AI]

Error

/home/xx/CondaEnv/MiniConda3/envs/VEnv/lib/python3.12/site-packages/torch/autograd/graph.py:769: UserWarning: Error detected in MmBackward0. Traceback of forward call that caused the error:
  File "/home/xx/CondaEnv/MiniConda3/envs/VEnv/bin/app", line 33, in <module>
    sys.exit(load_entry_point('App', 'console_scripts', 'app')())
  File "/home/app/commands/main.py", line 53, in main
    arguments.run(arguments)
  File "/home/app/commands/applications.py", line 32, in deep_learning_train_run
    train(
  File "/home/app/applications/deep_learning.py", line 321, in train
    exact_train(0,
  File "/home/app/applications/deep_learning.py", line 195, in exact_train
    (loss, logs) = task.train(minibatch)
  File "/home/app/applications/tasks/node_prediction.py", line 285, in train
    output = self.model(minibatch.x, minibatch.edge_index, minibatch.mask_x_position)
  File "/home/xx/CondaEnv/MiniConda3/envs/VEnv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/xx/CondaEnv/MiniConda3/envs/VEnv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/app/applications/models/node_prediction/gcn_np.py", line 26, in forward
    x = self.layer_2(x, edge_index)
  File "/home/xx/CondaEnv/MiniConda3/envs/VEnv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/xx/CondaEnv/MiniConda3/envs/VEnv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/xx/CondaEnv/MiniConda3/envs/VEnv/lib/python3.12/site-packages/torch_geometric/nn/conv/gcn_conv.py", line 261, in forward
    x = self.lin(x)
  File "/home/xx/CondaEnv/MiniConda3/envs/VEnv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/xx/CondaEnv/MiniConda3/envs/VEnv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/xx/CondaEnv/MiniConda3/envs/VEnv/lib/python3.12/site-packages/torch_geometric/nn/dense/linear.py", line 147, in forward
    return F.linear(x, self.weight, self.bias)
 (Triggered internally at /opt/conda/conda-bld/pytorch_1724789563135/work/torch/csrc/autograd/python_anomaly_mode.cpp:111.)
  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
Traceback (most recent call last):
  File "/home/xx/CondaEnv/MiniConda3/envs/VEnv/bin/app", line 33, in <module>
    sys.exit(load_entry_point('App', 'console_scripts', 'app')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/app/commands/main.py", line 53, in main
    arguments.run(arguments)
  File "/home/app/commands/applications.py", line 32, in deep_learning_train_run
    train(
  File "/home/app/applications/deep_learning.py", line 321, in train
    exact_train(0,
  File "/home/app/applications/deep_learning.py", line 214, in exact_train
    loss.backward(retain_graph=retain_graph)
  File "/home/xx/CondaEnv/MiniConda3/envs/VEnv/lib/python3.12/site-packages/torch/_tensor.py", line 521, in backward
    torch.autograd.backward(
  File "/home/xx/CondaEnv/MiniConda3/envs/VEnv/lib/python3.12/site-packages/torch/autograd/__init__.py", line 289, in backward
    _engine_run_backward(
  File "/home/xx/CondaEnv/MiniConda3/envs/VEnv/lib/python3.12/site-packages/torch/autograd/graph.py", line 769, in _engine_run_backward
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [128, 156]], which is output 0 of AsStridedBackward0, is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

Source Code

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.nn import Embedding
from torch_geometric.nn import GCNConv

class GCN_NP(nn.Module):

    def __init__(self, node_dict_size, node_dim, hidden_dim, dropout, output_embedding = False):
        super(GCN_NP, self).__init__()
        self.output_embedding = output_embedding
        self.node_embedding_layer = Embedding(node_dict_size, node_dim)
        self.dropout = dropout
        self.layer_1 = GCNConv(node_dim, hidden_dim)
        self.layer_2 = GCNConv(hidden_dim, node_dict_size)
        self.initialize_parameters()

    def forward(self, x, edge_index, mask_x_position):
        x = self.node_embedding_layer(x).squeeze(1)
        x = F.dropout(x, p=self.dropout, training=self.training)
        x = self.layer_1(x, edge_index)
        x = F.relu(x)
        x = F.dropout(x, p=self.dropout, training=self.training)
        x = self.layer_2(x, edge_index)
        # if self.output_embedding:
        #    return torch.mean(x, dim=0).unsqueeze(0)
        return F.log_softmax(x[mask_x_position], dim=1)

    def initialize_parameters(self):
        nn.init.normal_(self.node_embedding_layer.weight, mean=0, std=self.node_embedding_layer.embedding_dim ** -0.5)

Environment

I use MiniConda to create environment:

• torch-geometric: 2.6.1
• pytorch: 2.4.1
• pytorch-cuda: 12.1

Try

1. I tried adding the line x = x.clone(), as mentioned in issue #7283, but it dit not work.
2. The code runs without errors when I install a lower version of torch (version: 2.0.1). However, I wonder why this bug occurs when torch version higher than 2.0.1.