implement interpreter pool to reduce memory usage for threaded mode #17
Conversation
…from being freed in the new interpreter" This reverts commit 929a32d. This was a temporary work around until correct refcnting could be added
also reversed test to make it clear the else is the default behavior
FUSE treats threads as light-weight objects and creates/deletes them frequently. This leads to a new perl interpreter being created for all new threads, but never being reclaimed/reused when a thread goes away. So, a long running threaded mount continues to leak memory as new perl interpreters are continuously created. This commit is a first-pass at an alternate interpreter/thread usage pattern. When a thread does not have an interpreter associated with it, it searches for a currently inactive interpreter to utilize before it attempts to create one.
Accessing MY_CXT on perl < 5.10.0 fetches the object out of a global hash. All perl data-structure accesses are not thread-safe due to refcounting. This means that MY_CXT is not a safe place to store anything that controls access to a specific interpreter.
|
Because this is a threaded design change, I would ask that this pull request is tested on other architectures before it gets merged in to make sure I didn't overlook something. |
|
That's my plan. I'll probably test it over the next few days; NetBSD and Mac OS X, I think, are the only non-Linux UN*X-like OSes I can think of that ship a threaded Perl interpreter. I'll probably have to dick around with FreeBSD for awhile to get a threaded build up without breaking my VM. Not sure if that's even possible with OpenBSD, since it's part of the base system. I'll see what I can figure out. |
|
So far 'make test' seems to work fine on MacOS X 10.6 and NetBSD 6.0.2; is there anything in particular that should be done to beat on it? |
|
I would try some simultaneous operations (such as multiple processes reading files at the same time) to exercise the interpreter pool and actually get it potentially swapping the active interpreters for threads. For my testing I usually spawn this command on multiple shells and then copy a file while it's going: |
|
I've now been running a couple mounts using this patch for 2 and a half weeks without having to remount them. They are currently using less memory than they were after a couple hours without this patch, and we haven't had any issues with using the mounts. I'm satisfied that the code is stable on Fedora Core, not sure if you ever got around to trying any stress testing on other *nix flavors. |
Conflicts: Fuse.xs
|
This one also fixes some deadlocks i had while using the threaded mode. THX! |
|
ping? |
FUSE treats threads as light-weight objects and creates/deletes them
frequently. This leads to a new perl interpreter being created for all new
threads, but never being reclaimed/reused when a thread goes away. So, a long
running threaded mount continues to leak memory as new perl interpreters are
continuously created.
To address that issue, this pull request implements an interpreter pool to satisfy requests when running in threaded mode. Threaded callbacks will attempt to use an inactive interpreter from the pool, or create a new one if none are available.
This pull request has been tested on the following versions of perl (all running on Fedora Core):
5.8.8
5.10.0
5.14.4
5.16.3