Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updates fcc crawl for new fcc site #126

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Conversation

matthewdking
Copy link
Member

fixes #125

Included in this PR:

  • new FreeCodeCamp arrays of the challenges the user has to complete
  • replace the use of request-promise in the crawl of FCC with puppeteer headless browser. The reason for this was down to the data we needed not being accessible on the first call to the site. If anyone knows a neater way to do it open to suggestions 😄

Will update tests for this in the next few days.

Be good to get this in before FAC15 application deadline (August 17th)

@BartBucknill
Copy link
Collaborator

@matthewdking Thanks again for your amazing effort taking this on!

Bit concerned about the performance hit of waiting for the entire freecodecamp profile to load all it's resources before we process the page but I don't have a great alternative.
The only alternative I see right now would be to query https://www.freecodecamp.org/api/users/get-public-profile?username=[username] and combine this with the data from https://www.freecodecamp.org/services/map-ui to see which challenges the user has completed.
Problem with this is that as this is not a public API it may be changed without warning.

So for now I'm happy to go with Puppeteer. Hopefully one day freecodecamp will sort out a public api and someone will update prereq-check to use it...

@BartBucknill
Copy link
Collaborator

@matthewdking I've only been able to do minimal testing, but the code looks good - as soon as the tests are sorted I'm happy to approve the merge.

@matthewdking matthewdking force-pushed the feature/125 branch 3 times, most recently from 2bf2c7d to a4c0625 Compare August 20, 2018 18:22
@codecov-io
Copy link

codecov-io commented Aug 20, 2018

Codecov Report

Merging #126 into master will decrease coverage by 1.88%.
The diff coverage is 81.08%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #126      +/-   ##
==========================================
- Coverage   93.47%   91.59%   -1.89%     
==========================================
  Files          18       18              
  Lines         322      333      +11     
==========================================
+ Hits          301      305       +4     
- Misses         21       28       +7
Impacted Files Coverage Δ
tests/dummy-data/freecodecamp-html-success.js 100% <ø> (ø) ⬆️
routes/report.js 86.36% <100%> (ø) ⬆️
model/freecodecamp-arrays.js 100% <100%> (ø) ⬆️
model/freecodecamp-crawl.js 76.66% <73.07%> (-23.34%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0cd15b5...f0f2344. Read the comment docs.

@matthewdking matthewdking temporarily deployed to prereq-check August 20, 2018 19:57 Inactive
@matthewdking matthewdking temporarily deployed to prereq-check August 20, 2018 20:13 Inactive
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

FreeCodeCamp new privacy setting
3 participants