Role of the 120,353,507 papers labeled NAN

[drchangliu]

The test set includes 146818 papers labeled with (-1) or the hidden label. I understand that they will be used for the final evaluation. Then what is the role of the 120,353,507 papers labeled NAN? Are their true labels known to anyone? Will they be used in the final evaluation? Or will they only contribute to unsupervised learning and graph structures? Thanks.

[weihua916]

Thanks for the great questions!
Those unlabeled papers correspond to non-arXiv papers, and hence their arXiv category labels are not given even in the raw data. Those papers won't be used in the final evaluation, but we believe they provide important graph connectivity information for the prediction task. To give you some sense, if we only consider the citation links between the labeled arXiv papers, the direct graph connectivity between papers would be much sparser (average node degree would be less than 2, according to Table 2 of our paper). Hope this helps!

[tedzhouhk]

May I ask if the hidden labels will be published after the submission deadline? Thanks.

[weihua916]

We are most likely hiding the test labels and have an evaluation server to evaluate model's predictions. We will keep the community updated.

[tedzhouhk]

Thank you so much for the quick reply!