Memory leak with attributes (#25881)

[Infiziert90]

Describe the bug
#25881
same area, but with different approach ...
In the docs you can find this approach for getting all attributes: [attr for attr in feature], so i did that and after a few seconds my memory was completely full.

When using attributes() it works fine.

How to Reproduce
Use a large dataset with NULL attributes

for feat in layer.getFeatures():
    [attr for attr in feat]

QGIS and OS versions
3.10.0

[roya0045]

can you explain what you are trying to accomplish?

[Infiziert90]

for feat in layer.getFeatures():
    [attr for attr in feat.attributes()]

In the c++ docs you can find for attributes() a note, that you also could accomplish the same when you use the way from my bug report.

https://qgis.org/api/classQgsFeature.html#a5a473f89868c234d53cea67504cd2f2e

But that way is leaking huge amount of memory for me...

[roya0045]

And if you do ?

for feat in layer.getFeatures():
     feat.attributes()

[Infiziert90]

When using attributes() it works fine.

[roya0045]

And if you try

for feat in layer.getFeatures():
     [ attr for attr in feat]

As stated in the doc?

[Infiziert90]

Memory leak

[roya0045]

It might be the list comprehension and your dataset that causes leakage.

[Infiziert90]

The leak is also present when using the normal for loop way.
And yes it is the dataset, but I think with the same problem from #25881

[Infiziert90]

Same behavior with 3.12.0

[Infiziert90]

Steps for reproduction ...

1. open python console
2. console ->

>>>layer = QgsVectorLayer(r"PATH\altlpar0.dbf", "", "ogr")
>>>for _ in range(1000):
...    for feat in layer.getFeatures():
...        t = [attr for attr in feat]
...        # let the gc cleanup t (no difference)
...        # del t

altlpar0.zip

[Infiziert90]

Tested it with 3.16.3 again and it still has the same memory leaking effect

[Infiziert90]

The python docs also says that this way of doing it is possible: https://qgis.org/pyqgis/master/core/QgsFeature.html?highlight=qgsfeature#qgis.core.QgsFeature.attributes (in the note section)

while being at it again, i tried some things and documented it with screenshots, showing that qgis is even holding onto the leaked memory:
Same code as in #32944 (comment)

similiar approach but using the attributes function:

[roya0045]

There is the possibility that the creation of an iterator in python is the issue, though this may come from sip as this process is not present in c++ and only built in python

QGIS/python/core/auto_generated/qgsfeature.sip.in

Line 38 in df28e63

SIP_PYOBJECT __iter__();

[nyalldawson]

I can't reproduce on current versions, is this still an issue?

[Infiziert90]

I can't reproduce on current versions, is this still an issue?

Tried it with 3.40.3-Bratislava under Windows 11 Version 2009 and it still spiked above 12 GB memory usage.
I opened up RAMMap after the process had finished and the console returned back to being interactable, this tool showed me an active usage of over 4 GB with the a total of 12 GB being allocated as private memory by qgis-bin.exe.

Leaks:

layer = QgsVectorLayer(r"PATH\altlpar0.dbf", "", "ogr")
for _ in range(1000):
    for feat in layer.getFeatures():
        t = [attr for attr in feat]

The version below results in a memory usage of 286 MB, with no memory spikes at all.
Presumably because the GC is actually clearing things up.

No Leak:

layer = QgsVectorLayer(r"PATH\altlpar0.dbf", "", "ogr")
for _ in range(1000):
    for feat in layer.getFeatures():
        t = [attr for attr in feat.attributes()]