Approaching learning problems with ill-defined "notions of best"

[sgbaird]

Paraphrased question by @faris-k

How do you approach problems that don't have a well-defined metric for optimization (i.e., unsupervised learning problems where there aren't any ground truths)?

[sgbaird]

Taking from a Theory of Predictive Modeling course I took at BYU, a learning problem has the following three aspects:

• Data
• A "bucket of models" to choose from (i.e., model(s) and their hyperparameters)
• A "notion of best" (measuring stick)

If the question is about hyperparameter tuning for unsupervised tasks, then the data and bucket of models are likely well-defined, but the notion of best isn't. RMSE and MAE are straightforward "notions of best" for property prediction. However, mat-discover is a project without an obvious notion of best because the goal is to find high-performing, chemically novel materials. High performance is usually straightforward, but novelty can be pretty subjective. Some metrics I came up with were:

• At each iteration, how many new periodic elements are being explored?
• At each iteration, how many new unique chemical formula templates (e.g., $\mathrm{SiO}_2$ is of the form $\mathrm{AB}_2$) are being explored? (figures)

Note that I wasn't explicitly optimizing for these - I was using these metrics to persuade myself/others that it was or wasn't carrying out novel exploration. My choice of embedding and clustering parameters was primarily based on trial and error and intuition. It was probably arbitrary at times (i.e., choose something, since a choice had to be made).

When I presented these ideas, I received suggestions about other methods to compare with and additional metrics to try.

There was also some great discussion and feedback about assessing the performance of an adaptive design scheme at #44.

In the self-driving-lab-demo project, I was trying to make a case for using more sophisticated multi-objective optimization algorithms: in particular, using expected hypervolume improvement instead of scalarized objectives. See facebook/Ax#1210. My takeaway was that algorithms tend to give you what you ask for - if you ask it to minimize RMSE, it tends to give you minimal RMSE values. If you ask it to optimize the expected hypervolume improvement, it tends to give you results with improved Pareto front hypervolumes. It's up to the user to decide what fits the project's high-level goals and vision and consider the importance/cost trade-offs of performing analysis to back up the decision.