Load troubleshooting

[difranco]

We are falling behind in queue processing on average.
Changed to burst worker mode to mitigate this somewhat.
[master a214d94] change to burst processing to better handle load

Long term we will need more workers. May need to move to a dedicated instance because of this.

[difranco]

Summary:
Unscheduled upgrade to Trusty on tools-login created initial outage and mismatch between prepared python virtualenv and cluster environment. After repairing environment, began falling behind in queue processing. There were too few Trusty instances in the queue-processing pool so Yuvi added some. Catching up on queue processing now. Still monitoring situation.

Relevant threads:
Outage:
https://lists.wikimedia.org/pipermail/labs-l/2015-March/003522.html
https://lists.wikimedia.org/pipermail/labs-l/2015-March/003533.html
Job limits:
https://lists.wikimedia.org/pipermail/labs-l/2015-March/003537.html
Semi-scheduled, still pending NFS outage:
https://lists.wikimedia.org/pipermail/labs-l/2015-March/003536.html