Improve API reliability

[onmax]

This PR introduces changes that improve reliability, simplify code, and keep it minimal.

Warning

Actions with schedule only run on main. Therefore I have created #101 to merge just the action.

Background

When we started to create the Validators API, we had to assume different scenarios and a lot of edge cases. We also lack of different external libraries (like @nimiq/utils/albatross-policy) which made the code complex and hard to maintain.

With the present PR we aim to remove the unused code, simplify code, and add more tools/UI elements to debug easier.

Breaking changes

Please, refer to the changelog.md to see list of breaking changes.

How the API works

The API itself still works the same, we have:

1. Fetcher: Retrieves information about ended epochs
   The code has been untouched, as this worked perfectly.
2. Fetcher current epoch: Retrieves information about current epoch
   This step before was unclear. Now, we have a proper system to retrieve data about the current epoch. We retrieve data for any kind of validator. Please read below "kind of validators"
3. Score computation
4. API endpoint to retrieve endpoints. The /api/v1/validators has minimal changes, where the rest of endpoints have been heavily modified. We have never documented them so I would say it ok, however I would like to ask the community for feedback.

Note

More information about how the project works internally in the README.md of this PR!

Kind of validators

Kind	Elected	Tracked
ElectedTrackedValidator	✅	✅
ElectedUntrackedValidator	✅	❌
UnelectedTrackedValidator	❌	✅
UnelectedUntrackedValidator	❌	❌

tracked: Weather the validator has submitted information in GitHub.

Note

Having a UnelectedUntrackedValidator should only be when the validator has been selected for the first time in history.

The big change is how we store the information in the DB.

Before we only stored activity for elected validators. This was prune to errors since it made the code quite complex because we needed to fill in the missing data from the database at runtime.

Now, we have shifted and we store one activity per epoch per validator. The unelected validators will have missed: -1, rewarded: -1, dominanceViaSlots: -1 and likelihood: -1. The rest of fields will be filled: balance, dominanceViaBalance, stakers (new field).

Warning

We should create a system for deleted validators and stop tracking those. This is part of future work.

Range

The range is composed by the following fields:

• fromXXX: The information for the first block/epoch we consider (~9 months). We have: fromEpoch, fromBlockNumber, fromTimestamp
• toXXX: The information for the last block/epoch we consider. We have: toEpoch, toBlockNumber, toTimestamp
• snapshotXXX: The information for current epoch. (It is not called currentEpoch as this could be missleading when talking about a range that is in the past). We have: snapshotEpoch, snapshotBlockNumber, snapshotTimestamp. If you have a better name, please suggest it.

Import JSON validators information

Cloudflare cannot access to the filesystem. Therefore, this PR also introduces a new mechanism to fetch the validators using ungh api.

Monorepo

This PR also simplifies monorepo configuration, having a catalog of dependencies, dev-related dependencies only in the top level (eslint, vitest, scripts), among other changes...

Out of scope

Please check out the issues tab to see which new issues will NOT be addressed in this PR

This PR closes the following issues

• missingEpochs includes the current epoch #69
• Add "fallback" score for validators that are not selected #92
• After new epoch, make sure we return value #90
• [VTS] - Ignore macroblock for window_size calculation #77
• add dominance field #32
• Missing blocksPerBatch in policy #65
• Sorty by score #42
• incorreect hasDefaultLogo #41
• /validators/{{ADDRESS}} #33

[Albermonte]

If you click on the sync button it loads for ever, seems it doesnt detect when the sunc is finished and just continue syncing.

[sync]  ERROR  {"message":"[POST] \"https://df20f5a1.validators-api-testnet.pages.dev/api/_hub/database/raw\": <no response> fetch failed","statusCode":500}

[sync] ℹ Client disconnected, aborting sync process

 ERROR  [unhandledRejection] {"message":"[POST] \"https://df20f5a1.validators-api-testnet.pages.dev/api/_hub/database/raw\": <no response> fetch failed","statusCode":500}

    at createError (node_modules/.pnpm/[email protected]/node_modules/h3/dist/index.mjs:66:12)
    at report (server/api/[version]/sync.get.ts:25:1)
    at Timeout._onTimeout (server/api/[version]/sync.get.ts:68:1)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)

An then starts to sync again

Cannot continue testing since every validator returns a not found error when clicking on them