fix: fix memory leak while downloading large files #1707

[hexadivine]

Added "MediaGenBaseDownload" class in order to fetch chunks using generator and to avoid memory leak issue.

Fixes #1706 🦕

[parthea]

Thanks for the fix @h3x4d1v1n3! Please could you add a test?

[hexadivine]

Sure @parthea

[aria1991]

Hi @h3x4d1v1n3 & @parthea we can use the test below:

import unittest
from unittest.mock import patch, MagicMock
import random
import time

class TestMediaGenBaseDownload(unittest.TestCase):

    @patch('random.random')
    @patch('time.sleep')
    def test_next_chunk(self, mock_sleep, mock_random):
        mock_random.return_value = 0
        mock_sleep.return_value = None

        # mock request object
        request = MagicMock()
        request.headers = {'accept': 'application/json'}
        request.uri = 'https://test.com'
        request.http = MagicMock()

        # mock http object
        request.http.request.return_value = (MagicMock(), b'content')

        downloader = MediaGenBaseDownload(request)

        # loop through the generator
        for chunk, status, done in downloader.next_chunk():
            self.assertEqual(chunk, b'content')
            self.assertEqual(done, True)
            self.assertIsInstance(status, MediaDownloadProgress)

In this test, I have mocked the random and time modules. I have also created a mock request object and http object to simulate the response of the request. The test will loop through the generator function next_chunk and check if the output of the generator is as expected.

[hexadivine]

Thanks for the input @aria1991.
@parthea - I have added the tests as asked. Thanks.

[hexadivine]

@parthea could you check the latest comits for tests. Let me know your assessment...

[hexadivine]

One of my commit do not have verified tag, causing it to fail CLA check. I will hard reset the commit and update the code with new branch.