scraper, project, superblock: Implement automated greylisting

[jamescowens]

This PR is for tracking the progress of implementing automated greylisting.

Please see the following set of notes for design considerations that need to be discussed.

• Basic manual greylisting and scraper machinery for determining automatic greylisting complete
  • • Manual greylisting is an administrative contract type that rides normal transactions
      • Scrapers will now collect statistics on projects that have a greylisted status of either AUTO_GREYLISTED or MANUAL_GREYLISTED. Credits and average credits will be recorded in the project payloads, but the project magnitude will be zero, and they will not contribute to CPID magnitude
      • The Scraper code does not deal DIRECTLY with greylisting rules as this is an individual node responsibility
      • The convergence rules in terms of required number of projects use ACTIVE projects and do not include greylisted projects even though the stats are being collected for greylisted projects. This is because a project may be greylisted and available or literally not available at all, so convergences cannot be always expected at the project level for greylisted projects.
• TODO
  • Wire up automatic greylisting
    • These exist along superblock boundaries, like superblocks themselves (claims with a valid superblock contract) and beacon activation
    • Need to compute ZCD and WAS
    • ZCD rule is <= 7 zero credit days out of 40, WAS rule is last 7 days average project credits / 40 days average project credits >= 0.1
    • Since this needs to be on superblock boundaries, we can slightly change the rules for implementation to be in superblocks rather than days. Since almost all of the time superblocks are very close to one day, this is almost the same.
    • Implies an algorithm that operates over 40 days of superblock history
    • No stats for a whitelisted project in a superblock (i.e. because the project is hard down) needs to be counted in ZCD, with zero entry for WAS averaging, even though last project convergence may be from the 48 hour stats carryover. This may require a tweak to the scraper convergence code
    • Choice of
      • stateless methods that repeatedly iterate over 40 superblock history to apply rules
      • methods over a cache structure that stores a subset of information from up to 40 superblocks relevant to the rule computation
      • Advantage of stateless is simplicity
      • Disadvantage is that it is fairly expensive, as the superblock registry has to be iterated over and processed – this involves disk I/0.
      • Advantage of caching is speed
      • Disadvantage of caching is complexity
      • Once client is synced this is only called when the superblock is staked and processed by the client, ~1 per 24 hours.
      • During sync will be approx 1 per 960 blocks
    • Need to define order of precedence of manual greylist versus automatic. Status of manual greylist must always override automatic as the whole point to manual greylisting once this is put into place is to deal with corner case issues that are not handled by the ZCD and WAS rules.
      • Manual greylisting is granular to each block (i.e. an administrative contract of type project)
      • Automatic greylisting is granular to the superblock (valid staked superblock claim)
      • Walking this through…
        • Example 1

block → MAN_GREYLISTED

superblock → AUTO_GREYLISTED → status still MAN_GREYLISTED

superblock → removed from AUTO_GREYLISTED → status still MAN_GREYLISTED

block → removal from MAN_GREYLISTED → ACTIVE

• • • • • Example 2

block → MAN_GREYLISTED

superblock → AUTO_GREYLISTED → status still MAN_GREYLISTED

block → removed from MAN_GREYLISTED → AUTO_GREYLISTED

superblock → removed from AUTO_GREYLISTED → ACTIVE

• • • • I think this means we have to do the cache. The most convenient way to deal with this order of precedence is to store the underlying AUTO_GREYLIST status in the cache and have methods to utilize this information
        - Have status in cache of something like AUTO_GREYLIST_QUAL which means project has met the conditions for AUTO_GREYLIST by the rules, but was already MAN_GREYLISTED
        - This would be checked for each contract injection to change MAN_GREYLIST status, to decide whether new status is either AUTO_GREYLIST or ACTIVE
        - The AUTO_GREYLIST_QUAL is a flag on the project at the current (head) state
        - Maybe this really belongs in the in memory superblock structure? This does not need to be in the on-disk (chain) superblock structure at the cost of some computations.
        - There is an existing superblock cache (SuperblockIndex g_superblock_index) that currently stores the last three superblocks and could be expanded to 40 superblocks as an easy way to do the cache. This means more computation on top of the cache but much faster because it operates on in memory structures rather than reading from the superblock registry (disk I/O). It also means more memory usage.
        - Maybe best to modify the cache to be a hybrid and store more limited information for superblocks 4 – 40. But this makes the cache updating more complicated.
        - The memory usage of the additional superblocks is minimal compared to the current size of other data structures with the current active beacon count and chain length; however, when the benefactor contracts are implemented, this will no longer be true.
    • Create more detailed automated greylist reporting
      • • Simple listing status on project not sufficient, because users will want to know the details of why a project is greylisted (this is ZCD/WAS reporting)
          • Probably should be something that operates on the project grid in the GUI and allows “clicking” the project whitelisting status and then having a pop-up window that displays the details of ZCD/WAS.

[div72]

Scrapers will now collect statistics on projects that have a greylisted status of either AUTO_GREYLISTED or MANUAL_GREYLISTED.

Makes sense for automatic greylisted projects for de-greylisting, but why are statistics collected for manually greylisted projects? The greylister can then operate on projects with either ACTIVE or AUTO_GREYLISTED status.

Also considering the WAS & ZCD calculation is done per day, I am not sure if bothering with caching is worth it. Might be worthwhile to make some dumb implementation and do a benchmark.

Could adding a separate -projectnotify parameter be useful? I've been thinking about making a mailing list for new polls, adding project state changes doesn't sound bad.

[jamescowens]

Excellent question. Depending on the reasons for the manual greylist, statistics may still be available for a project. If so they should continue to be collected, because the ZCD and WAS rules would then apply if the manual greylist status was removed.

What are you thinking in terms of functionality for the -projectnotify parameter?

[div72]

If so they should continue to be collected, because the ZCD and WAS rules would then apply if the manual greylist status was removed.

Good point.

block → removal from MAN_GREYLISTED → ACTIVE

Manual greylisting should instantly take effect, but should ungreylisting do so too? It should be simpler to make manual ungreylisting put the project in an auto greylist state. It'll take until the next superblock for the project to become active but that's ok imo. I'm imagining a FSM like this:

                              ⣀⡤⠤⠒⠒⠒⠉⠉⠉⠉⠉⠉⠙⠒⠒⠢⠤⣄⡀                         
                            ⡴⠋⠁                    ⠉⠳⡄                       
                            ⢣⣀    AUTO_GREYLIST    ⢀⣠⠇                       
                             ⠈⠙⠒⠤⠤⣀⣀⣀⣀⣀⣀⣀⣀⣀⣀⡠⠤⠴⠚⠉                         
                          ⠠⠤⣔⠶⢢               ⡠⣤⢄⣀⡀                       
                        ⢀⡠⠔⠉   ⠁             ⠈  ⠑⠢⡀                       
                     ⠘⠴⠮⠥⠤                         ⠈⠑⠤⡀                      
   ⣀⣠⠤⠤⠖⠒⠒⠒⠒⠒⠒⠒⠒⠒⠲⠤⠤⣄⣀                               ⣈⣑⢄⡀⡔                          
⣠⠖⠋⠁                   ⠈⠙⠲⣄                              ⣉⡭⠤⠖⠒⠒⠒⠒⠦⠤⣄⡀                          
⡇      MANUAL_GREYLIST     ⢸   ⠒⠾⣛⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒  ⢸⡁  ACTIVE ⢀⡹                           
⠙⠲⢤⣀                   ⣀⡤⠖⠋                              ⠈⠙⠒⠒⠒⠒⠒⠒⠒⠚⠉                           
   ⠈⠉⠙⠒⠒⠲⠤⠤⠤⠤⠤⠤⠤⠖⠒⠒⠋⠉⠁                                               

What are you thinking in terms of functionality for the -projectnotify parameter?

Similar to other notify commands. It should be triggered on project status changes(added to whitelist, removed, greylisted etc.), should call a script with the contract hash(or superblock hash in case of an automatic greylist).

[jamescowens]

That is a good simplification actually.

[jamescowens]

Well... back to the drawing board. With "Initial implementation of AutoGreylist" I have created an auto greylist class that executes the ZCD and WAS rules. It is not wired into the whitelist or the superblock yet, but you can see the results using getautogreylist rpc.

The idea on this is to use the AutoGreylist as an override in the Whitelist class, marking the status of projects that meet automatic greylisting status as AUTO_GREYLISTED. This would take precedence over ACTIVE or even MANUALLY_GREYLISTED, There are no project entry (registry) status updates made for the automatic greylist. It is maintained as a global singleton cache which is intended to be refreshed when a new superblock is staked. So when a project comes off the auto greylist, it will revert automatically to the original status dictated by the last valid project entry status.

There is a big problem with the input to the AutoGreylist class, however. The scrapers filter the projects to select only the statistics of actively beaconed crunchers to save processing time and network bandwidth/node memory. This means that the TC reported in the project files provided by the scrapers for greylisted projects, indeed all projects, when summed, only are across active beacon holders for that project, not ALL crunchers. The AutoGreylist class uses the information in the historical superblocks, which is a reduced form of the scraper statistics, so it telescopes the same problem.

Unfortunately, this causes serious issues with the current rules. For example, here is an output after the SB at 3473369 was posted on mainnet:

{
"auto_greylist_projects": [
{
"project:": "SiDock@home",
"zcd": 8,
"WAS": 0.05824032176973353
},
{
"project:": "asteroids@home",
"zcd": 6,
"WAS": 0
},
{
"project:": "gpugrid",
"zcd": 4,
"WAS": 0.02413605533979817
},
{
"project:": "rosetta@home",
"zcd": 10,
"WAS": 0.01227473310563061
}
]
}

I haven't fully traced the WAS, but the ZCD for asteroids, I went and traced the void AutoGreylist::RefreshWithSuperblock(SuperblockPtr superblock_ptr_in) as it updated the greylist. What became immediately apparent is that the asteroids TC for the latest superblock at the time of the run (3473369), was actually LESS than the TC for the previous SB (3472381). How can this be. It is absolutely possible. For example, beaconholders that contributed to the TC for that project may expire between SB's and actually cause the TC summed across all active beacon holders to decline. This is clearly not what is intended by the rules.

The rules using the script based greylist checker operate on the total TC for the entire project across ALL CPID's, whether registered beacon holders or not, and as such are much more stable.

I was hoping to get away with minimal changing to the scraper plumbing and the project stats objects that are put in the manifests of the scrapers to avoid additional points of possible problems, but it looks like I am going to have to bite the bullet and

1. Sum the TC for all of the users in the source stats export file for a project as it is being processed by the scraper, and
2. Modify the manifest processing to handle that information, which will have to be provided as another manifest object.
3. Modify the superblock to store the project-wide TC's for all projects across ALL CPID's, not just the TC project sum for active beaconed CPIDs.
4. Have the AutoGreylist class use this info instead of the existing project TC sums in the superblock.

This is going to cause the scraper processing to go up some.

Ugh.

[jamescowens]

Alright. I am nearing the end of the work on the automated greylisting functionality. There are two important issues that have come out of my testing that I would like some opinions on:

1. Unlike @RoboticMind's script checker that operates on the BOINC site data, the scrapers will only start collecting overall total credit data to evaluate the auto greylisting status when the project is first whitelisted. This means the dataset is constrained at the beginning, so the rules are constrained. For example, if there have been only three superblocks since the whitelisting, then there can't possibly be 7 ZCD, so that rule must passes. This rule will not have a chance to fail until there are 7 superblocks after the whitelisting. To me this is fine.
2. If the first superblock doesn't collect stats due to a problem with the project after the whitelisting, at the second superblock, the WAS will evaluate to zero and the project will be greylisted on the second superblock. This can be avoided by requiring a minimum of two superblocks back from the baseline before the rule is applied.
3. The WAS definition of the 7 day credit average / 40 day credit average is actually imprecise. One must calculate this by measuring the CHANGE in credit between the superblocks at the beginning and end of the appropriate intervals. So does 7 mean 7 SB's, which is actually 6 superblocks of deltas back from the present, because that includes 7 superblocks, (endpoint inclusive) or is it 7 superblocks back from the present (baseline), which measures seven superblocks of change, which is actually 8 superblocks inclusive? Same corresponding question for the 40 superblock denominator. @RoboticMind in his script seems to take the latter point of view based on my read of it... i.e. 7 superblocks of delta for the numerator of WAS, which is actually 8 superblocks inclusive, and 40 superblocks of delta, which is actually 41 superblocks inclusive for the denominator.

Thoughts?