Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement NT-Xent loss (implicit negative samples) #136

Open
Tracked by #109
ziw-liu opened this issue Aug 15, 2024 · 2 comments
Open
Tracked by #109

Implement NT-Xent loss (implicit negative samples) #136

ziw-liu opened this issue Aug 15, 2024 · 2 comments
Labels
enhancement New feature or request representation Representation learning (SSL)
Milestone

Comments

@ziw-liu
Copy link
Collaborator

ziw-liu commented Aug 15, 2024

Implement InfoNCE-style (e.g. NT-Xent from SimCLR) training objective for contrastive learning. This changes sampling strategy so it probably should be done with #123.

@ziw-liu ziw-liu added the enhancement New feature or request label Aug 15, 2024
@mattersoflight mattersoflight changed the title Implicit negative sampling Implement NT-Xent loss Sep 3, 2024
@mattersoflight mattersoflight changed the title Implement NT-Xent loss Implement NT-Xent loss (implicit negative samples) Sep 3, 2024
@mattersoflight
Copy link
Member

mattersoflight commented Sep 3, 2024

@ziw-liu as discussed, first implement NT-Xent loss for the "time and cell identity agnostic" sampling, which can be done with the current dataloader.

Based on the results, we can figure out our strategy for time and cell identity-aware sampling, and how to implement it properly.

Previous implementation.

@ziw-liu
Copy link
Collaborator Author

ziw-liu commented Sep 3, 2024

Just to log our thoughts, a 'proper' implementation would be to treat each track as a sample instead of each node as a sample in the dataset, so that each track contributes one positive pair per epoch, and no batch will contain more than 2 patches from the same track.

@mattersoflight mattersoflight added this to the v0.3.0 milestone Sep 9, 2024
@ziw-liu ziw-liu added the representation Representation learning (SSL) label Sep 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request representation Representation learning (SSL)
Projects
None yet
Development

No branches or pull requests

2 participants