Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Feature/self aware filters #493

Open
wants to merge 3 commits into
base: develop
Choose a base branch
from

Conversation

grahamrow
Copy link
Member

Yikes.

Trying to solve a few problems here:

  1. Improved pipeline stability - @martin-gustafsson identified a lot of sketchy behavior of the pipeline. I moved to having all of the filer objects sharing Exit and now Panic events that the filter set upon Exceptions and the experiment periodically checks to see if it should bail. I have no real-world stability improvements to claim.
  2. Move to methods running actual filter code - using class methods allows Processes to be started without any explicit dependence on instance variables. Verbosity increases here since we now have to explicitly pass lots of important constructs, but in principle it is now much easier to write a fast filter in some external language. Currently configuration parameters live in a dictionary, but this could be spit out in ctypes struct later on.
  3. Finally allow things to work using spawn - this is a disaster as references to shared objects such as Queues and RawArrays are not properly synchronized without a multiprocessing Manager entity. Adding this still took a TON of hacks to fix numpy RawArray mappers, return values from IO buffers, etc. etc. etc. Using vanilla multiprocessing is

Still need to convert the following to using class methods:

  • Alazar listener
  • Filters other than averager and buffer

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant