post-mortem DHH crash

[chadwhitacre]

Right after https://www.youtube.com/watch?v=p1E-svVd9Xc we crashed. Manifested as a drained Aspen thread pool:

CPU load isn't bad, response times spike.

[chadwhitacre]

If 40 were too many threads I would expect CPU load to reflect that. Spiking response times make me think that we hit some slow queries which backed up our threads. What other parameters are we missing? Network I/O on the box?

[chadwhitacre]

I'm not going to debug this. :-(

At least it's noted, alongside #1541.

[zbynekwinkler]

I think we should strive to fail faster (200s is way too long). I do not think it is possible with python threads - I do not know of a way how to cancel a running thread in python from "outside" after a certain time :(. Logging requests that took too much time would make sense in this context but this info is in some way already available in papertrail logs.