[P2P] Optimize block pool requester retry and peer pick up logic

[yzang2019]

Describe your changes and provide context

Problems:
There are some inefficiency in the current p2p block sync module:

• Even If a bpRequester has already downloaded the block from the peer, it can reset the block to nil whenever a retry is triggered. A retry can be triggered whenever a peer got disconnected or timedout, this could lead to a lot of extra network usages and block sync slow down. What usually happens is that during retry, it actually pick bad peers and give up previously downloaded block from good peers, then it end up having to wait for the response for a long time
• Block sync reactor will need next two consecutive blocks to be available in the block pool in order to proceed the applyBlock, if any one is missing, the reactor will stuck and wait for the bprequester download to be finished
• There's no timeout for a single block request, a block request can take more than a few minutes to get the response from a bad/slow peer, which would cause block sync to stuck during this time
• We are not using the good peer list when picking available peers

Solution:

• Introducing a retry reason, there are two major reasons, either we need to retry for a peer being removed due to disconnection, or we need to retry when we failed validating the block in the reactor.
• When we handle the retry operation, we should check if we already have the block, if we already have the block read, we don't need to retry any more, unless the retry reason is bad block
• Introducing an extra timeout for requester routine, when we wait for the block response, we should try a different peer if we haven't got the response after timeout

Testing performed to validate your change

Tested on standalone rpc nodes

[philipsu522]

nit - woudl this case ever be hit? could also use modulo to have more randomness

[yzang2019]

I thought math.randn is inclusive?

[codecov]

Codecov Report

Attention: 15 lines in your changes are missing coverage. Please review.

Comparison is base (72bb29c) 59.11% compared to head (6f4c6d3) 58.09%.
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #170      +/-   ##
==========================================
- Coverage   59.11%   58.09%   -1.03%     
==========================================
  Files         281      249      -32     
  Lines       38940    33907    -5033     
==========================================
- Hits        23021    19698    -3323     
+ Misses      14124    12645    -1479     
+ Partials     1795     1564     -231     

Files	Coverage Δ
internal/eventbus/event_bus.go	89.18% <100.00%> (ø)
internal/p2p/peermanager.go	80.14% <0.00%> (-0.29%)	⬇️
internal/blocksync/pool.go	77.96% <78.94%> (+0.49%)	⬆️

... and 50 files with indirect coverage changes