Add ensemble model

[sfluegel05]

Problem

Currently, we have a range of different approaches for classifying molecules in ChEBI (ELECTRA-based, GNN-based (https://github.com/ChEB-AI/python-chebai-graph) and algorithmic / logic-based (https://github.com/sfluegel05/chemlog2).

All approaches have specific strengths and weaknesses. The goal of an ensemble is to take different methods and aggregate their predictions so that the final result is better than the individual results.

Task

The architecture of the ensemble methods should take the following input:

• For each model and ChEBI class: A prediction (possible values: true, false, error / don't know, out of scope), possibly also a confidence value (real number that indicates how sure the model is about a prediction, e.g. from 0 - no confidence to 1 - very confident)

It should aggregate these values into a single prediction (for each class), taking into account the predictions of each model and the "trustworthiness" of the model (this score is specific to each class, and possibly different for positive and negative predictions).

Example:

Given a ChEBI class, we have received the following predictions:

model A	model B	model C	model D
true	false	true	out of scope

The simplest approach would be to weight all models equally and return true for this class (with a 2-1 vote). However, we should also take the trustworthiness into account. These values might come from the precision / true predictive value (TPV; TP / (TP + FP)) and negative predictive value (NPV; TN / (TN + FN)) of a model on a test set.

metric	model A	model B	model C	model D
TPV	0.7	0.8	0.6	0.7
NPV	0.9	0.99	0.8	1

In other words: If model A and model C predict "true" for this class, they are correct in 70% and 60% of cases (according to their TPV). If model model B predicts "false" for this class, it is correct in 99% of cases (according to the NPV).

An aggregation method would then weight two predictions with "trustworthiness" of 0.7 and 0.6 against one with 0.99. Depending on the aggregation method used, it might decide to trust model B.

Future work

• Extend this method towards a hierarchical ontology-based approach
• Use bagging and boosting to improve perfomance further

[aditya0by0]

• How to combine multiple lightning module and save hyperparameters Lightning-AI/pytorch-lightning#7249

[sfluegel05]

Here is an updated specification of what the final ensemble should look like. I would suggest not to implement all of this in one go, but to implement it in several iterations.

There are two main components:

Controller

Decides which models get used

• Input: A SMILES string, a list of available models, for each model: "activation conditions", i.e., a list of classes one of which has to be predicted by some other model to activate this model
• Output: For each model and class: a binary prediction score (true / false), a float prediction score [0, 1], a confidence score [0,1] (optional)
• Algorithm: (1) call all models that have no activation conditions, get their results. (2) If any model predicts any class that is an activation condition for a model that has not been called yet, call the new model. (3) Repeat until no new models are activated. (4) Hand over all prediction results to the Consolidator

Consolidator

Decides how to aggregate model results into a single prediction

• Input: Trust scores for each model and class (i.e., TPV and NPV, might be incomplete), output of Controller (For each model and class: a binary prediction score (true / false), a float prediction score [0, 1], a confidence score [0,1] (optional))
• Output: For each class: a binary prediction score (true / false / don't know), a confidence score [0,1]
• Algorithm: In its simplest form, the Consolidator should do a weighted majority vote, using the trust scores and prediction scores as weights. More complex decision procedures might be added later

For now, we should skip the Controller and confidence scores. Currently, we have no models that we can't run without activation conditions. Confidence scores will come into play if new models that produce such scores are added (e.g., LNNs).