cuDNN error on test data after processing 40k reads

[lpryszcz]

Hi, I encountered below error after processing 40,500 reads from the test data. Any idea?

Process Process-2:
Traceback (most recent call last):
  File "/home/lpryszcz/src/miniforge3/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/home/lpryszcz/src/miniforge3/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/lpryszcz/src/mAFiA/RODAN/basecall.py", line 182, in mp_gpu
    out = model.forward(event)
  File "/home/lpryszcz/src/mAFiA/RODAN/models.py", line 226, in forward
    x = self.convlayers(x)
  File "/home/lpryszcz/src/mAFiA/mafia-venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/lpryszcz/src/mAFiA/mafia-venv/lib/python3.10/site-packages/torch/nn/modules/container.py", line 204, in forward
    input = module(input)
  File "/home/lpryszcz/src/mAFiA/mafia-venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/lpryszcz/src/mAFiA/RODAN/models.py", line 141, in forward
    x = self.pointwise(x)
  File "/home/lpryszcz/src/mAFiA/mafia-venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/lpryszcz/src/mAFiA/mafia-venv/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 313, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/home/lpryszcz/src/mAFiA/mafia-venv/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 309, in _conv_forward
    return F.conv1d(input, weight, bias, self.stride,
RuntimeError: Unable to find a valid cuDNN algorithm to run convolution

[ADHDrian]

Hi @lpryszcz , which CUDA version do you have? We have tested 12.4 as well as 11.x before. Also, I suggest using python3.9 to set up the venv as specified in the .toml.