CUDA error

[jodyphelan]

I'm trying to run on some fast5 files but it terminates shortly after running:

$ python3 basecall.py --directory ../fast5_pass/ --output some.fasta

...
...
...

  File "/mnt/storageG1/jody/miniconda3/envs/heron/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/mnt/storageG1/jody/miniconda3/envs/heron/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/mnt/storageG1/jody/bonito/heron/backend.py", line 91, in caller_process
    output = caller_network(torch.tensor(signal).cuda())
  File "/mnt/storageG1/jody/miniconda3/envs/heron/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/mnt/storageG1/jody/bonito/heron/network.py", line 111, in forward
    x_evs = block(x_evs)
  File "/mnt/storageG1/jody/miniconda3/envs/heron/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/mnt/storageG1/jody/bonito/heron/model.py", line 283, in forward
    _x += self.residual(x)
RuntimeError: The size of tensor a (36) must match the size of tensor b (37) at non-singleton dimension 2
[W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]