GSoC 2019 ideas

[pypt]

Student application period has just started, so it's about time for us to come up with some GSoC ideas.

We can reuse some from the last year:

• Build a tool to do some cool visualizations
• Create PostgreSQL-based job queue
  • Probably too hard and not going to happen, but it doesn't hurt to leave it there. Or should we just remove it?
• Implement a method to detect subtopics of a topic
  • A lot of students are asking about this idea but I'm not sure if I'm the one who would be able to be the best mentor for this task as (simply put) I don't know much about the subject. Dongge, our GSoC 2017 student, did implement subtopics using Louvain but it's still unmerged to this day.
• Do your own freehand project

I'd also add easier, low-priority tasks from our side-projects, e.g. the Ultimate Sitemap Parser:

• (easy) Add support for RSS / Atom sitemaps
• (easy) Detection of sitemap if it's not present in robots.txt (has some relevant discussion and stats too; I wonder whether Alex (@kienli) is a student!)
• (medium-hard) yield found links instead of returning them (simple concept but would probably require rethinking and rewriting a lot of stuff)

sentence_splitter Python module:

• Add more supported languages?

feed_seeker:

• Fix fix timeout bug feed_seeker#2 and evaluate and improve accuracy of feed discovery feed_seeker#3?

[pypt]

Another idea for the Ultimate Sitemap Parser:

• Rewrite code to use asyncio: sitemap processing is mostly CPU-bound (all the XML parsing takes a lot of time), but asyncio might improve performance by something like 15%, and also would be useful when Crawl-Delay is defined in robots.txt

[pypt]

Another idea:

readability-lxml is aging fast and is not necessarily still the best library around to extract body of the article from the HTML page. More and more articles get loaded using JavaScript due to an ongoing "frontend everywhere!" frenzy. Lastly, various CDNs, e.g. Cloudflare, are blocking our crawler just because our user agent doesn't have JavaScript enabled.

I think inevitably we'll have to switch to running a chromeless browser, loading each and every downloaded story in it, and then applying a well-supported third-party library, e.g. Mozilla's Readability, to extract article title, author and body.

So, an experimental project for the student could be:

• Set up a chromeless browser
• Set up Readability
• Develop a HTTP service that accepts a parameter URL (and/or HTML body), loads it in the browser, runs Readability's magic, and returns the extracted HTML back to the requester.
• Package everything in a nice Docker image

There are similar projects like this, e.g. https://github.com/schollz/readable, so we'd need to do some research into whether such a thing exists already first.

[hroberts]

another idea:

Create scaffolding for a new version of our api. our existing api is written in perl, is inconsistent among its different major parts, and is goofily un-rest-ish in several places. we would like to replace it with a python based api that uses a modern framework for api specification, implementation, and testing.

The work to replace the entire api is much too large for one summer, so this task would be to build enough of the end points to demonstrate how to apply the chosen framework to the problem.

[jiahao-c]

Another idea:

readability-lxml is aging fast and is not necessarily still the best library around to extract body of the article from the HTML page. More and more articles get loaded using JavaScript due to an ongoing "frontend everywhere!" frenzy. Lastly, various CDNs, e.g. Cloudflare, are blocking our crawler just because our user agent doesn't have JavaScript enabled.

I think inevitably we'll have to switch to running a chromeless browser, loading each and every downloaded story in it, and then applying a well-supported third-party library, e.g. Mozilla's Readability, to extract article title, author and body.

So, an experimental project for the student could be:

• Set up a chromeless browser
• Set up Readability
• Develop a HTTP service that accepts a parameter URL (and/or HTML body), loads it in the browser, runs Readability's magic, and returns the extracted HTML back to the requester.
• Package everything in a nice Docker image

There are similar projects like this, e.g. https://github.com/schollz/readable, so we'd need to do some research into whether such a thing exists already first.

Hello pypt,
I thought chromeless were deprecated. Have you considered Puppeteer or Selenium?

[pypt]

Hey @jhcccc! By "chromeless", I had a general browser without chrome (UI) in mind :) Which specific rendering engine (Gecko, WebKit) and/or tool to use for the task would be up to a student.

[pypt]

Final list:

https://docs.google.com/document/d/1GGbGtFOMS07dog4yzglY5hZCDc41ZQjY1RqRKOlW0B4/edit?usp=sharing

Sent it to Ellen and interested students.