Failures in Distributed Algorithms Appear to Hang forever in CCClient

[coltonbh]

With Jan when trying to do a distributed frequency analysis on a TS structure, BigChem workers showed some failures in the logs, and his client would hang forever without getting a response. Why is a failed structure not returned? How are failures handled by celery in these distributed algorithms?

Need to test some failure cases and see how they are handled and make sure the correct behavior is happening. I think "correct" would be returning a "ProgramFailure" object if any of the gradients fail.