Skip to content

Commit 3f81acc

Browse files
committed
Fix memory leak in HTTPResourcePath
Fixed an issue where query parameters or fragments in an HTTP URL would cause it to be considered as having a different root URI. This caused a new HTTP session to be created and cached each time an HTTP URL with a different query parameter was accessed. This resulted in a memory leak in datalinker when it was using HTTPResourcePath to access S3 presigned URLs, which always have unique query parameters.
1 parent bc58b9c commit 3f81acc

File tree

3 files changed

+25
-1
lines changed

3 files changed

+25
-1
lines changed

doc/changes/DM-43739.bugfix.md

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
`ResourePath.root_uri()` now strips query parameters and fragments from the URL. This fixes a memory leak where `HttpResourcePath` would create and cache a new HTTP session for each different set of query parameters.

python/lsst/resources/_resourcePath.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -417,7 +417,7 @@ def root_uri(self) -> ResourcePath:
417417
uri : `ResourcePath`
418418
Root URI.
419419
"""
420-
return self.replace(path="", forceDirectory=True)
420+
return self.replace(path="", query="", fragment="", params="", forceDirectory=True)
421421

422422
def split(self) -> tuple[ResourcePath, str]:
423423
"""Split URI into head and tail.

tests/test_http.py

+23
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,29 @@ class GenericHttpTestCase(GenericTestCase, unittest.TestCase):
5959
scheme = "http"
6060
netloc = "server.example"
6161

62+
def test_root_uri(self):
63+
self.assertEqual(ResourcePath("http://server.com").root_uri(), ResourcePath("http://server.com/"))
64+
self.assertEqual(
65+
ResourcePath("http://user:[email protected]:3000/").root_uri(),
66+
ResourcePath("http://user:[email protected]:3000/"),
67+
)
68+
self.assertEqual(
69+
ResourcePath("http://user:[email protected]:3000/some/path").root_uri(),
70+
ResourcePath("http://user:[email protected]:3000/"),
71+
)
72+
self.assertEqual(
73+
ResourcePath("http://user:[email protected]:3000/some/path#fragment").root_uri(),
74+
ResourcePath("http://user:[email protected]:3000/"),
75+
)
76+
self.assertEqual(
77+
ResourcePath("http://user:[email protected]:3000/some/path?param=value").root_uri(),
78+
ResourcePath("http://user:[email protected]:3000/"),
79+
)
80+
self.assertEqual(
81+
ResourcePath("http://user:[email protected]:3000/some/path;parameters").root_uri(),
82+
ResourcePath("http://user:[email protected]:3000/"),
83+
)
84+
6285

6386
class HttpReadWriteWebdavTestCase(GenericReadWriteTestCase, unittest.TestCase):
6487
"""Test with a real webDAV server, as opposed to mocking responses."""

0 commit comments

Comments
 (0)