WIP (no active development) - New talktorial: ML splitting schemes

[kimheeye]

Details

• Talktorial ID: xxx
• Title: Performance of ligand-based machine learning methods for the classification of active/inactive compounds, considering various validation approaches
• Original authors: Hee-yeong Kim
• Reviewer(s): Andrea Volkamer
• Date of review: 20.12.20

General

• Search for literature belonging to the topic and filter information towards
  splitting schemes.

• Structure the notebook for the student’s presentation (~20min).

• Write a project work with at least 3000 words belonging to the topic.

  • Observe the results for the different ML methods based on the data splitting schemes (single random split, n-fold- and time-split CV, cluster-based split).

Programming Tasks

• Create a EGFR compounds data, comprising bioactivity data and document year, using T001 and filter them by
  Lipinski‘s Rule of five by T002.

Use the ML approaches from T007 and validate them by three different data splitting schemes. Therefore implement the following methods to split the data into train/test and validation set:

• Single Random Split.

• n-fold Cross Validation (CV).

Time-split CV

• Implement a naive time-split CV provided by sklearn TimeSeriesSplit().
• Create a plot to investigate the distribution of data depending on time of publication (document year).
• Assess the train/test split from above implemented time-split CV and specify the splits such that the cut lies between distinct publication years.
• Compare the Classifier’s performance for both approaches.

Cluster-based Split

• Use the Butina algorithm provided by RDKit (functions from T005) to cluster the compounds by their fingeprint representation with Tanimoto similarity measure.
• Cluster the compounds using the sklearn.cluster.KMeans() function and euclidian distance measure, based on their pysicochemical properties, calculated with CalcDescriptors() from RDKit .
• Select the train set from large clusters and use the remaining small clusters and/or singletons to create an external test set.
• Run the Classifiers on the generated train/test set.

[AndreaVolkamer]

@kimheeye really well done! Great notebook, text and illustrations!
Just a few 'cosmetic' comments here:

• Have a look at the template on how (and where) to cite references and figure captions.
• For the functions can you please insert doc strings (see e.g. the functions in [T007])(https://github.com/volkamerlab/teachopencadd/blob/master/teachopencadd/talktorials/T007_compound_activity_machine_learning/talktorial.ipynb)
• In the practical part, it would be good to add a few more markdown descriptions on what is done (and maybe why) and especially what the results and findings are (a short discussion)
• also maybe link or mark somehow, which functions were reused from T007