DFS connections remain open leading to eventual system crash

[legsak1mbo]

When connecting to a standard Windows/SAMBA file server, calling smbclient_state_free ensures that the connection is closed. Since smbc_free_context is called in the module's destructor we can even get away with not calling it. However, when the connection is to a DFS target said connection remains open even after the script completes.

After connecting to a DFS target, running lsof | grep microsoft on the server yields four open microsoft-ds connections per request. This can cause serious issues because eventually the system will reach it's open files limit and crash.

The only way to close the connections is to restart Apache and this issue can be worked around by setting MaxRequestsPerChild in httpd.conf to something very low (say fifteen) but from a performance standpoint this is less than ideal.

It seems that the connections live as long as PHP does so this issue does not occur when running a script from the command line.

I think that in the case of DFS we need to be doing something more than calling smbc_free_context. Looking through http://fossies.org/linux/misc/samba-4.0.25.tar.gz/samba-4.0.25/source3/libsmb/clientgen.c there is a cli_shutdown function which talks about connections remaining active even when smbc_free_context is called but I'm not sure if this is relevant here.

[aklomp]

This is definitely a bug, but I'm afraid that we're powerless to do anything about it. This project is just an interface to libsmbclient. We can't do any low-level Samba protocol stuff, because we're stuck with only the few high-level calls exported in libsmbclient.h. This issue should be addressed downstream by the Samba devs.

[legsak1mbo]

I thought as much. Have you experienced the same? If so I've opened a bug here https://bugzilla.samba.org/show_bug.cgi?id=11195

Please chime in as appropriate.

[aklomp]

It's possible that the call we do to smbc_free_context() in the destructor fails and returns zero one. I didn't check if it does, but it would be totally within libsmbclient's rights. We don't currently handle that case, because what are our options? We could retry the call, if that would even help, but we can't sit around, stop the world and wait till we get a positive result. So the code might, as the comment indicates, leak the handle in some cases.

(Oops, I just pressed the close + post button by accident. Moving on...)

A good test might be to check if the call indeed returns zero one for DFS connections, and if it does, see if one or two immediate retries get the connection to close.

I can't reproduce the bug on my home setup, unfortunately.

(Edited: smbc_free_context returns zero on success, or one on failure with errno set.)

[aklomp]

An easy way to check if smbc_free_context is returning 1 for failure is to check the PHP error logs: they should have a message saying that the context could not be destroyed.

[legsak1mbo]

I've investigated a little further. I called smbc_setDebug(state->ctx, 9) inside smbclient_state_new and get...

Apr  6 18:08:49 7 httpd: smbc_remove_usused_server: 0x7f29c9b7c9e0 removed.
Apr  6 18:08:49 7 httpd: Performing aggressive shutdown.
Apr  6 18:08:49 7 httpd: Context 0x7f29c9b2be20 successfully freed
Apr  6 18:08:49 7 httpd: Freeing parametrics:

...in the syslog which would suggest to me that libsmbclient does believe that it's successfully freeing the context. I also tried calling smbc_getFunctionPurgeCachedServers(state->ctx)(state->ctx) before smbc_free_context but still no dice - the connections stay open.

[legsak1mbo]

Just thinking out loud, it might be worth adding debug to the options in smbclient_option_set.

[aklomp]

Sounds useful. I was under the impression from reading libsmbclient.h that the only output options for debug info were either printing it to stdout or printing it to stderr. Neither are very useful for a PHP extension, so I just ignored the debug level settings. I assume that the debug info is ending up in your syslog because the service manager running your webserver is logging the webserver's stderr to that location.

If libsmbclient thinks it has freed the context successfully, then our hands are tied. Hopefully the Samba guys can help out.

[legsak1mbo]

I'm not calling smbc_setOptionDebugToStderr so it's just using stdout.