Memory Leak in parallel-deflate.c

[nob13]

Hi all, thanks for this nice VNC Server library!

I am currently investigating something which looks like a memory leak, however my understanding of AML is limited.

• The leak only happens if the ZRLE-Encoding is used (e.g. RealVNC-Viewer) , but not when Tight-Encoding (e.g. Vinagre)
• The leak happens in Linux with current Git-Version of AML/NeatVNC, so it doesn't have to do with my OSX Experiments

From the Malloc-Debugging, I see that the memory is consumed in parallel-deflate.c: schedule_deflate_job in deflateInit2 and some others. However the corresponding method deflate_job_destroy is never called.

Perhaps this is the same issue as in any1/wayvnc#367

[any1]

Does this fix it?

diff --git a/src/aml.c b/src/aml.c
index 48253fc..fabaab7 100644
--- a/src/aml.c
+++ b/src/aml.c
@@ -791,14 +791,15 @@ static void aml__handle_event(struct aml* self, struct aml_obj* obj)
         */
        aml_ref(obj);
 
-       if (obj->cb && aml_is_started(self, obj)) {
+       if (aml_is_started(self, obj)) {
                /* Single-shot objects must be stopped before the callback so
                 * that they can be restarted from within the callback.
                 */
                if (aml__obj_is_single_shot(obj))
                        aml_stop(self, obj);
 
-               obj->cb(obj);
+               if (obj->cb)
+                       obj->cb(obj);
        }
 
        if (obj->type == AML_OBJ_HANDLER) {

[nob13]

Yes, deflate_job_destroy is now called while being connected with RealVNC

What I still see is an increasing memory consumption when I connect with Real VNC, then disconnect, then connect again and so on, but I haven't measured this.

[any1]

I pushed the change to aml. Can you track down the client object leaks using asan or valgrind?

[nob13]

Thanks! Regarding client objects, It'll take some time, because of holidays.

[nob13]

I had a second look at it, however without any success. If I watch the neatvnc server with btop I can see an increase of memory consumption of around 10MB per conncetion cycle using RealVNC Viewer, which stays there if the client leaves.

However valgrind (even with --leak-check=full --trace-malloc=yes) doesn't show any leaks.

The only interesting thing is this:

==16169== Invalid read of size 4
==16169==    at 0x1242F6: stream_tcp__on_writable (in /my/path/screensharing/unix/build/screensharing)
==16169==    by 0x12436C: stream_tcp__on_event (in /my/path/screensharing/unix/build/screensharing)
==16169==    by 0x20F0D9: aml__handle_event (in /my/path/screensharing/unix/build/screensharing)
==16169==    by 0x20F36C: aml_dispatch (in /my/path/screensharing/unix/build/screensharing)
==16169==    by 0x114C88: sc::Aml::iteration(long) const (in /my/path/screensharing/unix/build/screensharing)
==16169==    by 0x114888: main (in /my/path/screensharing/unix/build/screensharing)
==16169==  Address 0xd53e528 is 8 bytes inside a block of size 4,096 free'd
==16169==    at 0x484988F: free (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==16169==    by 0x123E7B: stream_tcp_destroy (in /my/path/screensharing/unix/build/screensharing)
==16169==    by 0x123A7E: stream_destroy (in /my/path/screensharing/unix/build/screensharing)
==16169==    by 0x118750: client_close (in /my/path/screensharing/unix/build/screensharing)
==16169==    by 0x11F9F6: nvnc_client_close (in /my/path/screensharing/unix/build/screensharing)
==16169==    by 0x11DA72: on_client_event (in /my/path/screensharing/unix/build/screensharing)
==16169==    by 0x1242DD: stream_tcp__on_readable (in /my/path/screensharing/unix/build/screensharing)
==16169==    by 0x124356: stream_tcp__on_event (in /my/path/screensharing/unix/build/screensharing)
==16169==    by 0x20F0D9: aml__handle_event (in /my/path/screensharing/unix/build/screensharing)
==16169==    by 0x20F36C: aml_dispatch (in /my/path/screensharing/unix/build/screensharing)
==16169==    by 0x114C88: sc::Aml::iteration(long) const (in /my/path/screensharing/unix/build/screensharing)
==16169==    by 0x114888: main (in /my/path/screensharing/unix/build/screensharing)
==16169==  Block was alloc'd at
==16169==    at 0x484D953: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==16169==    by 0x1247A3: stream_new (in /my/path/screensharing/unix/build/screensharing)
==16169==    by 0x11DD98: on_connection (in /my/path/screensharing/unix/build/screensharing)
==16169==    by 0x20F0D9: aml__handle_event (in /my/path/screensharing/unix/build/screensharing)
==16169==    by 0x20F36C: aml_dispatch (in /my/path/screensharing/unix/build/screensharing)
==16169==    by 0x114C88: sc::Aml::iteration(long) const (in /my/path/screensharing/unix/build/screensharing)
==16169==    by 0x114888: main (in /my/path/screensharing/unix/build/screensharing)

But this is probably unrelated.

IMHO the bug can be closed, as the main reason is fixed within AML.