Memory considerations and questions

[auroracramer]

Hello! First of all thanks for all of your work on HEAR, I think it's super valuable for the machine listening community.

I've been working on a fork to extend this evaluation framework to other tasks (namely spatial localization tasks), and have been finding that with the current data loading infrastructure, using memmaps when in_memory=False causes GPU utilization to be very low, and so I've been trying to use in_memory=True so to try and avoid an apparent I/O bottleneck. I've also found though that in_memory=True results in memory usage to inflate unexpectedly over epochs as well as across grid point searches. (IIRC using memmap with massive batch sizes helps avoid GPU under-utilization issues to be avoided, but then encounters OOM issues, but I have to double check this)

As far as I'm aware, I don't think I've introduced anything that would prevent memory from being released (though I could totally be missing something). I am working with a dataset that has 60 second examples (which seems to be related to #101), though for the sequence-wise processing I've added, long examples are chunked to avoid massive batch tensors. I've been trying to add memory optimizations here and there and forcing garbage collection in some places (which can result in seg faults 😞), as well as trying to use various profiling tools to pinpoint the issue, but haven't been successful yet.

All this being said, have there been any similar memory consumption issues you've encountered, and if so, have did you come across any ways to address it?

[turian]

@auroracramer Can you explain what "sequence-wise processing" you are doing? Is this scene classification or a timestamp task?

We definitely were able to run heareval on very long audio. Like beehive had 600sec audio clips. The gtzan datasets are 30 second audios.

I guess I would need more information to understand why it would OOM for you but not for us. I am not sure what you're changing precisely.

Have you tried running heareval on beehive or gtzan? If they are crashing or not is an important step to isolating the issue.

[auroracramer]

@turian

Can you explain what "sequence-wise processing" you are doing? Is this scene classification or a timestamp task?

Basically we have an extra "sequence" dimension so that the model can (potentially) process sequences of embeddings (e.g. with an RNN) and/or compute loss on consecutive outputs (e.g. w/ ADPIT). In our case this is for a timestamp task, but it could also be adapted for scene classification.

We definitely were able to run heareval on very long audio. Like beehive had 600sec audio clips. The gtzan datasets are 30 second audios.

Good to know!

I guess I would need more information to understand why it would OOM for you but not for us. I am not sure what you're changing precisely.

Have you tried running heareval on beehive or gtzan? If they are crashing or not is an important step to isolating the issue.

I haven't yet, but I'll do that and see if it's an issue specific to our setup. I was more curious to see if this was something y'all encountered so if you had already found a solution we wouldn't have to reinvent a solution. We've added a bunch of things here and there so I'd prefer to narrow it down first so as to not take up too much of your time.

Thanks!

[turian]

I am still not grokking this precisely. I'm happy to chat more to discuss. What is your timezime? I am berlin. So afternoons might work best for us? You can find me on telegram at my @mylastname
Or propose another way we can chat.