Some fixes in LM part regarding ngram history length + MLE ngram #13
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lecture_note.tex
Outdated
| @@ -3568,11 +3568,12 @@ \section{$n$-Gram Language Model} | |||
| conditional probability (Eq.~\eqref{eq:unidir_sentence}~(a)) is only conditioned | |||
| on the $n-1$ preceding symbols only, meaning | |||
| \begin{align*} | |||
| p(w_k | w_{<k}) \approx p(w_k | w_{k-n}, w_{k-n+1}, \ldots, w_{k-1}). | |||
| % p(w_k | w_{<k}) \approx p(w_k | w_{k-n}, w_{k-n+1}, \ldots, w_{k-1}). % history length should be n-1 | |||
There was a problem hiding this comment.
please remove this commented line
lecture_note.tex
Outdated
| \end{align*} | ||
| This results in | ||
| \begin{align*} | ||
| p(S) \approx \prod_{t=1}^T p(w_t | w_{t-n}, \ldots, w_{t-1}). | ||
| p(S) \approx \prod_{t=1}^T p(w_t | w_{t-n+1}, \ldots, w_{t-1}). % history should have n-1 length |
lecture_note.tex
Outdated
| \subsection{Smoothing and Back-Off} | ||
|
|
||
| {\em Note that I am missing many references this section, as I am writing this | ||
| on my travel. I will fill in missing references once I'm back from my | ||
| travel.} | ||
|
|
||
| The biggest issue of having an $n$-gram that never occurs in the training corpus | ||
| is that any sentence containing the $n$-gram will be given a zero probability | ||
| is that any sentence containing such $n$-gram will be given a zero probability |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
So given the definition of n-gram the text is 100% correct but in the formulas there are always histories of length n, which is probably a typo. I have also added small explanation about why relative freq. ngram estimator is optimal from the MLE perspective.