How to do the logistic regression on my own dataset

[Yang-J-LIN]

Hello, I want to retrain the part of logistic regression. Can I just update the parameters of esm1b.contact_head and use esm1b.predict_contacts to get the output for loss calculation? Part of the code is shown below.

optimizer = torch.optim.Adam(esm1b.contact_head.parameters(), lr=0.001)
esm1b_contacts = esm1b.predict_contacts(esm1b_batch_tokens)
loss = loss_function(esm1b_contacts, targets)

Thanks for your help.

[joshim5]

Hi Yang, yes that could work! If you just want to train the regression, make sure that you just train the regression and set requires_grad=False for everything else: https://github.com/facebookresearch/esm/blob/master/esm/modules.py#L315

You'll also want to include regularization. See Figure 6 for the grid search we performed to choose hyperparameters.

When we trained the logistic regression, we actually used the Liblinear solver from Scikit learn for fast convergence: https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html

[Yang-J-LIN]

Hi Joshua, thanks for your quick reply. I tried to re-train the esm1b.contact_head but cannot get similar results. I guess there might be some problem with the optimization. Could you please tell me how you design loss function and which optimizer do you use? I have tested Adam optimizer, with a 0.001 learning rate and 10/0.01 weight decay. I have tested MSE loss and L1 loss, and a L1 regularization is added ( the same setting with the paper).

[tomsercu]

By default we use sklearn.linear_model.LogisticRegression(penalty="l1", C=0.15, solver="liblinear")
Stack all the pairwise positions (i,j) from upper diagonal from a small set of structures, ie each position is seen as an independent sample for sklearn.

[Leo-T-Zang]

Hi @tomsercu ,
Is that possible you can elabrate more on how to finetune the ContactHead Module with sklearn? Since predict_contacts gives results after sigmoid, how do we combine this with .LogisticRegression? And what does Stack all the pairwise positions (i,j) mean, like if we can .fit(x,y), how is x and y look like?

Also @Yang-J-LIN , did you figure out how to finetune their contacthead?

Thank you all so much!!!!