The Thousand Validators Programme is an initiative by Web3 Foundation and Parity Technologies to use the funds held by both organizations to nominate validators in the community.
The nominating backend will routinely change its nominations at every era. The backend does this by short-listing candidates by validity and then sorts validators by their weighted score in descending order. Validators with a higher weighted score are selected for any possible slots. As validators are nominated and actively validate, their weighted scores decrease, allowing other validators to be selected in subsequent rounds of assessment. If a validator is active during a single nomination period (the time after a new nomination and before the next one) and does not break any of the requirements, it will have its rank increased by 1. Validators with higher rank have performed well within the program for a longer period of time. The backend nominates as many validators as it reasonably can in such a manner to allow each nominee an opportunity to be elected into the active set.
A monorepo containing TypeScript microservices for the Thousand Validators Program.
The following is a monorepo of packages for the Thousand Validators Program. Each package is a microservice that can be run independently or together with other microservices.
The monorepo is managed using Yarn workspaces, and contains the following packages:
packages/common: A package containing common code shared across all microservices.packages/core: A package containing the core logic of the Thousand Validators Program.packages/gateway: A package for an API gateway that exposes the backend with a REST API.packages/telemetry: A package for a telemetry client that monitors uptimepackages/worker: A packages for job queue workers that perform background tasks.
There's a few ways of running the backend with docker containers, either in kubernetes, or with docker-compose.
There is the Current / Monolith way of running instances, and the Microservice way of running instances.
Current / Monolith Architecture:
Microservice Architecture:
The following are different ways of running in either Current or Microservice architecture with either Kusama or Polkadot, and either Development or Production:
Kusama Current- Running as a monolith with production values
Polkadot Current- Running as a monolith with production values
Kusama Microservice- Running as microservices with production values
Polkadot Microservice- Running as microservices with production values
Polkadot Current Dev- Running as a monolith with development values
Kusama Current Dev- Running as a monolith with development values
Kusama Microservice Dev- Running as microservices with development values
Polkadot Microservice Dev- Running as microservices with development values
Each package contains a Dockerfile, which is used for running in production, and Dockerfile-dev, which is used for development. The development images will use run with nodemon so that each time files is saved/changed it will rebuild the image and restart the container. Any changes for the regular run Dockerfile will need a manual rebuilding of the docker image.
The difference of running as either Current or Microservice is in which docker containers get run with docker-compose (Microservices have services separated out as their own containers, and additionally rely on Redis for messages queues). Outside of this everything else (whether it's run as a Kusama or Polkadot instance) is determined by the JSON configuration files that get generated.
git clone https://github.com/w3f/1k-validators-be.git
cd 1k-validators-beEnsure the following are installed on your machine:
The following are scripts that can be run with yarn that will run all the required installations, config generations and build and run the docker containers. If these are not used you will need to do each separately in the following sections.
First run:
yarn install
Kusama Current / Monolith Production:
yarn docker:kusama-current:startKusama Current / Monolith Dev:
yarn docker:kusama-current-dev:startPolkadot Current / Monolith Production:
yarn docker:polkadot-current:startPolkadot Current / Monolith Dev:
yarn docker:polkadot-current-dev:startKusama Microservice Production:
yarn docker:kusama-microscervice:startKusama Microservice Dev:
yarn docker:kusama-microservice-dev:startPolkadot Microservice Production:
yarn docker:polkadot-current:startPolkadot Microservice Dev:
yarn docker:polkadot-current-dev:startyarn installyarn buildBefore running the microservices with docker-compose, you must create configuration files for each service to be run.
Kusama Current Config:
This will create a configuration file for a Kusama instance that mirrors what is currently deployed. This runs the core service within one container, and includes the gateway and telemetry client, all run in the same node.js process.
yarn create-config-kusama-currentPolkadot Current Config:
This will create a configuration file for a Kusama instance that mirrors what is currently deployed. This runs the core service within one container, and includes the gateway and telemetry client, all run in the same node.js process.
yarn create-config-polkadot-currentKusama Microservice Config:
This will create configuration files for a Kusama instance for each microservice that runs with production values. This runs core, gateway, telemetry, and worker as separate processes in their own container - each one needs it's own configuration file.
yarn create-config-kusama-microservicePolkadot Microservice Config:
This will create configuration files for a Polkadot instance for each microservice that runs with production values. This runs core, gateway, telemetry, and worker as separate processes in their own container - each one needs it's own configuration file.
yarn create-config-polkadot-microserviceEither is from the same docker-compose.current.yml file, and runs only the core container, mongo container, and mongo-express container.
Build and run as detached daemon:
docker compose -f docker-compose.current.yml up -d --buildEither is from the same docker-compose.microservice.yml file. This runs core, gateway, telemetry, and worker as separate processes in their own container - each one needs it's own configuration file. It additionally runs a redis, mongo, and mongo-express container.
Build and run as detached daemon:
docker compose -f docker-compose.microservice.yml up -d --buildRunning Kusama Current Dev, Polkadot Current Dev, Kusama Microservice Dev, or Polkadot Microservice Dev
Either is from the same docker-compose.yml file.
Build and run as detached daemon:
docker compose -f docker-compose.yml up -d --buildTo view the aggregated logs of all the containers:
yarn docker:logsor
docker compose logs -f To view the logs of an individual service:
Core:
yarn docker:logs:coreor
docker logs 1k-validators-be-1kv-core-1 -fGateway:
yarn docker:logs:gatewayor
docker logs 1k-validators-be-1kv-gateway-1 -f Telemetry:
yarn docker:logs:telemetryor
docker logs 1k-validators-be-1kv-telemetry-1 -f Worker:
yarn docker:logs:workeror
docker logs 1k-validators-be-1kv-worker-1 -f To stop all containers:
yarn docker:stopor
docker-compose downWhen running as a monolith, the Express REST API is exposed on port 3300. When running as microservices, the Express REST API is exposed on port 3301.
You can then query an endpoint like /candidates by going to http://localhost:3300/candidates or http://localhost:3301/candidates in your browser.
To view the Mongo Express GUI to interact with the MongoDB Database, go to http://localhost:8888/ in your browser. Or run yarn open:mongo-express from the root directory.
To view the BullMQ Board GUI to interact with the Job Queue, go to http://localhost:3301/bull in your browser if running as microservices. Or run yarn open:bull from the root directory.