Interesting paper

[blazespinnaker]

I've been playing around with the ideas proposed in this paper as they are similar to some results that I am seeing.

In particular, I'm using a formation like this:

preds = .. #Assume preds have been generated by some ensemble/model to evaluate / fix up

def test_is_better(preds, disagreement_index, new_prob):
  new_preds = preds.copy()
  new_preds [disagreement_index] = new_prob
  model_evaluator = lightgbm.LGBMRegressor()
  model_evaluator.fit(X_Unlabeled, new_preds)
  for i in range(5):
      model_evaluator_1 =  lightgbm.LGBMRegressor()
      model_evaluator_1.fit(X_Unlabeled, model_evaluator_1.predict(X_Unlabeled))
      model_evaluator =  lightgbm.LGBMRegressor()
      model_evaluator.fit(X_Unlabeled, model_evaluator_1.predict(X_Unlabeled))
  return roc_auc_score(y_holdout, model_evaluator.predict(X_holdout)) > BASE_ROC_AUC_SCORE

I'm finding it works for not just out of distribution data, but also somewhat for in distribution data as well.

What's interesting is that iterating on the model_evaluator is what is important and improves the accuracy of this metric.