prospective-parachains rework: take II (paritytech#4937)

Resolves paritytech#4800

# Problem
In paritytech#4035, we removed
support for parachain forks and cycles and added support for backing
unconnected candidates (candidates for which we don't yet know the full
path to the latest included block), which is useful for elastic scaling
(parachains using multiple cores).

Removing support for backing forks turned out to be a bad idea, as there
are legitimate cases for a parachain to fork (if they have other
consensus mechanism for example, like BABE or PoW). This leads to
validators getting lower backing rewards (depending on whether they back
the winning fork or not) and a higher pressure on only the half of the
backing group (during availability-distribution for example). Since we
don't yet have approval voting rewards, backing rewards are a pretty big
deal (which may change in the future).

# Description

A backing group is now allowed to back forks. Once a candidate becomes
backed (has the minimum backing votes), we don't accept new forks unless
they adhere to the new fork selection rule (have a lower candidate
hash).
This helps with keeping the implementation simpler, since forks will
only be taken into account for candidates which are not backed yet (only
seconded).
Having this fork selection rule also helps with reducing the work
backing validators need to do, since they have a shared way of picking
the winning fork. Once they see a candidate backed, they can all decide
to back a fork and not accept new ones.
But they still accept new ones during the seconding phase (until the
backing quorum is reached).

Therefore, a block author which is not part of the backing group will
likely not even see the forks (only the winning one).

Just as before, a parachain producing forks will still not be able to
leverage elastic scaling but will still work with a single core. Also,
cycles are still not accepted.

## Some implementation details

`CandidateStorage` is no longer a subsystem-wide construct. It was
previously holding candidates from all relay chain forks and complicated
the code. Each fragment chain now holds their candidate chain and their
potential candidates. This should not increase the storage consumption
since the heavy candidate data is already wrapped in an Arc and shared.
It however allows for great simplifications and increase readability.

`FragmentChain`s are now only creating a chain with backed candidates
and the fork selection rule. As said before, `FragmentChain`s are now
also responsible for maintaining their own potential candidate storage.

Since we no longer have the subsytem-wide `CandidateStorage`, when
getting a new leaf update, we use the storage of our latest ancestor,
which may contain candidates seconded/backed that are still in scope.

When a candidate is backed, the fragment chains which hold it are
recreated (due to the fork selection rule, it could trigger a "reorg" of
the fragment chain).

I generally tried to simplify the subsystem and not introduce
unneccessary optimisations that would otherwise complicate the code and
not gain us much (fragment chains wouldn't realistically ever hold many
candidates)

TODO:
- [x] update metrics
- [x] update docs and comments
- [x] fix and add unit tests
- [x] tested with fork-producing parachain
- [x] tested with cycle-producing parachain
- [x] versi test
- [x] prdoc

polkadot/node/core/prospective-parachains/Cargo.toml

@@ -12,24 +12,18 @@ workspace = true
 [dependencies]
 futures = { workspace = true }
 gum = { workspace = true, default-features = true }
-codec = { workspace = true, default-features = true }
 thiserror = { workspace = true }
 fatality = { workspace = true }
-bitvec = { workspace = true, default-features = true }
 
 polkadot-primitives = { workspace = true, default-features = true }
-polkadot-node-primitives = { workspace = true, default-features = true }
 polkadot-node-subsystem = { workspace = true, default-features = true }
 polkadot-node-subsystem-util = { workspace = true, default-features = true }
 
 [dev-dependencies]
 assert_matches = { workspace = true }
 polkadot-node-subsystem-test-helpers = { workspace = true }
-polkadot-node-subsystem-types = { workspace = true, default-features = true }
 polkadot-primitives-test-helpers = { workspace = true }
+sp-tracing = { workspace = true }
 sp-core = { workspace = true, default-features = true }
-sc-keystore = { workspace = true, default-features = true }
-sp-application-crypto = { workspace = true, default-features = true }
-sp-keyring = { workspace = true, default-features = true }
-sp-keystore = { workspace = true, default-features = true }
+rand = { workspace = true }
 rstest = { workspace = true }

polkadot/node/core/prospective-parachains/src/error.rs

@@ -30,18 +30,6 @@ use fatality::Nested;
 #[allow(missing_docs)]
 #[fatality::fatality(splitable)]
 pub enum Error {
-	#[fatal]
-	#[error("SubsystemError::Context error: {0}")]
-	SubsystemContext(String),
-
-	#[fatal]
-	#[error("Spawning a task failed: {0}")]
-	SpawnFailed(SubsystemError),
-
-	#[fatal]
-	#[error("Participation worker receiver exhausted.")]
-	ParticipationWorkerReceiverExhausted,
-
 	#[fatal]
 	#[error("Receiving message from overseer failed: {0}")]
 	SubsystemReceive(#[source] SubsystemError),
@@ -55,9 +43,6 @@ pub enum Error {
 	#[error(transparent)]
 	ChainApi(#[from] ChainApiError),
 
-	#[error(transparent)]
-	Subsystem(SubsystemError),
-
 	#[error("Request to chain API subsystem dropped")]
 	ChainApiRequestCanceled(oneshot::Canceled),