Skip to content

Commit

Permalink
Update README.md
Browse files Browse the repository at this point in the history
  • Loading branch information
luyug authored Jun 16, 2021
1 parent 47f7c99 commit 79e1fe0
Showing 1 changed file with 2 additions and 0 deletions.
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@
Gradient Cached Dense Passage Retrieval (`GC-DPR`) - is an extension of the original [DPR](https://github.com/facebookresearch/DPR) library.
We introduce Gradient Cache technique which enables scaling batch size of the contrastive form loss and therefore the number of in-batch negatives far beyond GPU RAM limitation. With `GC-DPR` , you can reproduce the state-of-the-art open Q&A system trained on 8 x 32GB V100 GPUs with a single 11 GB GPU.

To use Gradient Cache in your own project, checkout our [GradCache package](https://github.com/luyug/GradCache).

## Gradient Cache Technique
- Retriever training quality depends on its effective batch size.
- The contrastive form loss (NLL with in-batch negatives) conditions on the entire batch and requires fitting the entire batch into GPU memory.
Expand Down

0 comments on commit 79e1fe0

Please sign in to comment.