ResourceWarning: unclosed file

[beantowel]

~/anaconda3/lib/python3.6/site-packages/audioread/__init__.py:86: ResourceWarning: unclosed file <_io.BufferedReader name=8>
  if ffdec.available():

After upgrading my pip packages(i don't know which cause it), running programs previously having no warnings pop up this message frequently. How to fix it?

[sampsyo]

Can you provide any way to reproduce this problem? With just a context-free warning message like this, it's impossible to tell what's going on.

[beantowel]

hello! I wrote the lines to reproduce this problem:

import unittest
import librosa

class Bar():
    def __getitem__(self, idx):
        wav = librosa.load('~/Music/test.mp3') # or whatever audio file
        return {'wav': wav}
        
class Foo(unittest.TestCase):
    def test_bar(self):
        b = Bar()
        y = b[0]['wav']
        print(y)

save the code in a file(audioReadWarning.py), then excute using unittest:

$ python -m unittest audioReadWarning.Foo.test_bar
/home/beantowel/anaconda3/lib/python3.6/site-packages/audioread/__init__.py:86: ResourceWarning: unclosed file <_io.BufferedReader name=4>
  if ffdec.available():
/home/beantowel/anaconda3/lib/python3.6/site-packages/audioread/__init__.py:86: ResourceWarning: unclosed file <_io.BufferedReader name=7>
  if ffdec.available():
(array([0., 0., 0., ..., 0., 0., 0.], dtype=float32), 22050)
.
----------------------------------------------------------------------
Ran 1 test in 12.088s

OK

And i found if i directly call the class method, this warning won't pop up

[sampsyo]

Hmm… I notice you're using the library via librosa, not directly. It seems like there's a strong chance this could be a bug in librosa itself, not in audioread, if it's not letting the file close properly. Maybe it would be worth digging to see if you can reproduce this when using audioread directly?

[beantowel]

hello! I used audioread directly as follows:

import unittest
import os
import audioread

class Bar():
    def __getitem__(self, idx):
        y = []
        with audioread.audio_open(os.path.realpath('ANY_AUDIO.mp3')) as input_file:
            for frame in input_file:
                y.append(frame)
        return {'wav': y}
        
class Foo(unittest.TestCase):
    def test_bar(self):
        b = Bar()
        print('before read')
        y = b[0]['wav']
        print('read')
        
# f = Foo()
# f.test_bar()

run it as mentioned before, the warnings pop up the same way.

[sampsyo]

Thanks for narrowing that down! It's odd, though—I ran your code verbatim, and it runs without a warning:

$ python3 -m unittest test.Foo.test_bar
before read
read
.
----------------------------------------------------------------------
Ran 1 test in 0.080s

OK

This is Python 3.7.3 on macOS 10.14.5. I'm using the latest git master source of audioread—could you be using an older version?

[beantowel]

I use Python 3.6.5 and audioread 2.1.8 on Ubuntu 18.04. Slightly changed code as in the attached file (to keep comment short&clean)
test.py.txt

Then i found by turning warnings into exceptions the code would have same behavior:

$ python -W error test.py
before read
Exception ignored in: <_io.FileIO name=4 mode='rb' closefd=True>
ResourceWarning: unclosed file <_io.BufferedReader name=4>
Exception ignored in: <_io.FileIO name=6 mode='rb' closefd=True>
ResourceWarning: unclosed file <_io.BufferedReader name=6>
read

$ python -W error -m unittest test.Foo.test_bar
before read
Exception ignored in: <_io.FileIO name=4 mode='rb' closefd=True>
ResourceWarning: unclosed file <_io.BufferedReader name=4>
Exception ignored in: <_io.FileIO name=6 mode='rb' closefd=True>
ResourceWarning: unclosed file <_io.BufferedReader name=6>
read
before read
Exception ignored in: <_io.FileIO name=4 mode='rb' closefd=True>
ResourceWarning: unclosed file <_io.BufferedReader name=4>
Exception ignored in: <_io.FileIO name=6 mode='rb' closefd=True>
ResourceWarning: unclosed file <_io.BufferedReader name=6>
read
.
----------------------------------------------------------------------
Ran 1 test in 0.077s

OK

$ python -m unittest test.Foo.test_bar
before read
read
before read
/home/beantowel/anaconda3/lib/python3.6/site-packages/audioread/__init__.py:86: ResourceWarning: unclosed file <_io.BufferedReader name=4>
  if ffdec.available():
/home/beantowel/anaconda3/lib/python3.6/site-packages/audioread/__init__.py:86: ResourceWarning: unclosed file <_io.BufferedReader name=6>
  if ffdec.available():
read
.
----------------------------------------------------------------------
Ran 1 test in 0.079s

OK

[sampsyo]

Got it; thanks! I was actually able to reproduce this by forcing my system to use the FFmpeg backend.

But these things are actually super hard to debug, and it will take some work to narrow it down. Any chance you'd be interested in helping out? We'll need to audit all the file handles opened in ffdec.py one by one and check whether they're being closed properly, perhaps by printing out their state at the end of the close method.

[Julian]

(Running into this issue myself from running beet import. Will see if I can see where the resource is being leaked.)

[Julian]

Simplified the reproducer to just:

import audioread
import sys
with audioread.audio_open(sys.argv[1]) as input_file:
    for frame in input_file:
        pass

Adding import tracemalloc; tracemalloc.start() for the tracemalloc module shows it's:

/Users/julian/.dotfiles/.local/share/virtualenvs/beets/lib/python3.8/site-packages/audioread/__init__.py:86: ResourceWarning: unclosed file <_io.BufferedReader name=3>
  if ffdec.available():
Object allocated at (most recent call last):
  File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", lineno 844
    self.stdout = io.open(c2pread, 'rb', bufsize)
/Users/julian/.dotfiles/.local/share/virtualenvs/beets/lib/python3.8/site-packages/audioread/__init__.py:86: ResourceWarning: unclosed file <_io.BufferedReader name=5>
  if ffdec.available():
Object allocated at (most recent call last):
  File "/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/subprocess.py", lineno 849
    self.stderr = io.open(errread, 'rb', bufsize)

[sampsyo]

Hmm; thanks for investigation! Unfortunately this output doesn't quite pin it down… it's pointing to the subprocess module, but not where that module was invoked from in our code. Is there any way to make tracemalloc output full tracebacks?

FWIW, here is where we should be closing the stdout and stderr streams:

audioread/audioread/ffdec.py

Lines 307 to 308 in 8d02710

	self.proc.stdout.close()
	self.proc.stderr.close()