Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove/accelerate experimental features #6

Open
tdozat opened this issue Jun 18, 2017 · 0 comments
Open

Remove/accelerate experimental features #6

tdozat opened this issue Jun 18, 2017 · 0 comments
Assignees

Comments

@tdozat
Copy link
Owner

tdozat commented Jun 18, 2017

For some future ideas I have, the pretrained vocabulary will need approximated counts. One way to do this is to fit it to a zipfian distribution, but people have noticed that natural language vocabularies tend to be best modeled with three zipfian distributions--one for frequent words, one for medium-frequency words, and one for rare words. So I have the model fit the training file's vocabulary to the interpolation of three zipfian distributions.

But, the process of fitting this is slow, not currently useful, and probably confusing for people expecting a parser and not a zipfian regressor. So it should either be removed and only called by the experimental classes once they've been built, or its optimization accelerated and done in numpy so that it's not noticeable to laypeople.

One way of speeding it up might be alternating using Newton's method to optimize the zipfian parts and using Newton's method, Adam, or gradient descent to optimize the softmax, rather than using Adam to optimize the whole thing at once.

@tdozat tdozat self-assigned this Jun 18, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant