Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

(Not ready for merge) Proposal for #714: Parallelizing compiler #715

Closed
wants to merge 2 commits into from
Closed

(Not ready for merge) Proposal for #714: Parallelizing compiler #715

wants to merge 2 commits into from

Conversation

jhmcstanton
Copy link
Contributor

@jhmcstanton jhmcstanton commented Jul 23, 2019 •
edited
Loading

Note: no new tests added, but existing tests all pass.

For my use case this worked well, the build went from 4:04 minutes to 2:26 minutes. This was with 99 images marked as separate Rules, with the remainder of my site on a single thread, and running with 4 capabilities.

If you're OK with this approach then I can clean this up a bit.

@jhmcstanton
Proposal for jaspervdj#714: Parallelizing compiler
7427ba5 
Note: no new tests added, but existing tests all pass.
@jhmcstanton jhmcstanton changed the title Proposal for #714: Parallelizing compiler (Not ready for merge) Proposal for #714: Parallelizing compiler Jul 23, 2019
@jhmcstanton jhmcstanton mentioned this pull request Jul 23, 2019
Closed
@jhmcstanton
watchPar fix: correcting inotify abuse
b83ac90 
The original implementation was calling `watchUpdates` many times,
causing resource exhaustion:
 `site: addWatch: resource exhausted (No space left on device)`
@jaspervdj
Copy link
Owner

I think it would be good to (finally) have parallel compilation in Hakyll. However, I have a slight preference for a traditional "jobs" approach, where we spawn N threads (possibly using the async library since it makes things a lot simple) and these N "workers" take items and complete them. Would you be open to switching to that?

@jhmcstanton
Copy link
Contributor Author

I'm up for any approach really, especially if it simplifies the implementation.

Were you thinking to focus more on chase in the Runtime? Or something else?

@jhmcstanton
Copy link
Contributor Author

jhmcstanton commented Aug 24, 2019 •
edited
Loading

Just a quick update: been going down the route of async-ifying chase. Have it compiling and passing all tests, but I'm currently bumping into some exceptions caused by locked files when running my own site build.

@jhmcstanton jhmcstanton mentioned this pull request Aug 25, 2019
Closed
@jhmcstanton
Copy link
Contributor Author

@jaspervdj the new pull request using async can be found in #725

@LaurentRDC LaurentRDC mentioned this pull request Apr 22, 2021
Merged
@LaurentRDC
Copy link
Contributor

This PR was superseeded by #844 and can therefore be closed.

@Minoru Minoru closed this Jun 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@jhmcstanton @jaspervdj @LaurentRDC @Minoru