Bed 4262 az panic recovery

[benwaples]

BP-367

Adds panic recovery to help with debugging panics that are causing a crash without logging anything. This will recover from a panic, log it, and keep the app from fully crashing.

Implementation

The big gotcha here is that recover() will not catch panics that occur in a separate goroutine... and azurehound leverages goroutines significantly. To handle this, we used channels to transmit those errors between each goroutine.

The main panicChan is opened in start.go and passed down to each subsequent func that starts a new goroutine. When a panic occurs, panicRecovery will catch it and send it to panicChan, then handleBubbledPanic will log the error (as well as stack) and stop the context.

Implementation exception

Each ...CmdImpl function creates its own panicChan.

Testing

I tested this by dropping panics at multiple different levels of goroutines and running collections.

notes:

panic recovery wasn't added to the client package... yet. I think this is a good starting point and we can refactor the client package to account for panicChan in another PR.

[benwaples]

Im thinking I should move this child context up so that the getAvailableJobs and startJob are not using the parent context.

[benwaples]

It appears that this won't make a difference because both of those requests will have been completed by the time we get to the goroutine where we spawn a child context.

[juggernot325]

@benwaples and I discussed the possibility of moving the channel for panics to a global variable instead of passing it into each function separately. He's going to play around with that to see if it's an option to simplify these changes a bit.

[juggernot325]

This looks great! Thank you for the refactor.