CV2-4233 add retry looper for alegre requests that intermittently fail

[DGaffney]

Description

When running via smooch, every once in a while, a request from Check-API to Alegre fails. There is every indication that this is some sort of intermittent, non-serious issue. When trying to reproduce them, even at large scales with lots of attempts to recreate the issue, Alegre responds correctly consistently. When looking at Alegre and Presto, neither show any indication that there is an issue. The response itself doesn't seem to be loud or fundamentally broken. For the time being, we're introducing a retry - and if that fails, we'll be much more confident that something more fundamental is happening that requires a deeper dive.

References: CV2-4233

How has this been tested?

I've added a direct test that should yield a proper null output after retrying multiple times, and ran a local script with side-effecting printouts to ensure it is doing what is intended.

Things to pay attention to during code review

Nothing really - fairly straightforward retry cycle

Checklist

• I have performed a self-review of my own code
• I have added unit and feature tests, if the PR implements a new feature or otherwise would benefit from additional testing
• I have added regression tests, if the PR fixes a bug
• I have added logging, exception reporting, and custom tracing with any additional information required for debugging
• I considered secure coding practices when writing this code. Any security concerns are noted above.
• I have commented my code in hard-to-understand areas, if any
• I have made needed changes to the README
• My changes generate no new warnings
• If I added a third party module, I included a rationale for doing so and followed our current guidelines

[caiosba]

A couple of suggestions and one request - also, tests must pass - good to go after that! :)

[codeclimate]

Code Climate has analyzed commit 436d10f and detected 0 issues on this pull request.

The test coverage on the diff in this pull request is 100.0% (100% is the threshold).

This pull request will bring the total coverage in the repository to 100.0% (0.0% change).

View more on Code Climate.

[DGaffney]

@caiosba ran into some slightly weird speedbumps yesterday but back to full passage - lemme know if you need anything else before signoff!