New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Figure out where the rest of the boilerplates are coming from #5

Open

don-han opened this issue Feb 26, 2016 · 1 comment

Member

don-han commented Feb 26, 2016

BIDS-projects/topic-modeling#21 (comment)

Contributor

chewisinho commented Mar 15, 2016

I found the location of the topic words "email address", "spambots", and "JavaScript enabled". They all come from one source: www.sdav-scidac.org

The problem is not with the scraper (we are not getting blocked) and not with the HTML parser - the problem is the website itself.

I have a super simple fix that actually filters out the words really effectively. Just look for this sentence:

"This email address is being protected from spambots. You need JavaScript enabled to view it."

Removing the above sentence from the corpus produced pretty good results.

Things to consider:

What other boilerplates need to be removed?
How should we remove these boilerplates? Should we hard code the fix (because it's astonishingly effective), or should we try adding these words to the stop words, etc.? Thoughts?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment