Define inter-process communication

[jhpierce]

How should pre-process, process, and post-process communicate?

Overall, the process must leverage multithreading, since the nature of the work is primarily IO-bound - especially pre and post processing.. I'm picturing using python Queues to pass objects in between phases. Queues will enable us to define the level of parallelism at each layer, as well as max queue size, etc. We want this to be lightning fast locally.

What should the objects passed from Pre-Process to Process look like? And from Process to Post-Process? We'll probably want models representing the results of pre-processing, processing, and post-processing.