Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incremental Build RFC #763

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Incremental Build RFC #763

wants to merge 1 commit into from

Conversation

natemoo-re
Copy link
Member

Summary

Built-in incremental build support

Links

@montsamu
Copy link

montsamu commented Mar 7, 2024

One small datapoint in favor of the currently experimental content cache:

Audiobookaneers.com build:

21:10:55 [build] 5904 page(s) built in 170.55s

21:13:36 [build] 5904 page(s) built in 118.22s

Second build was more than 50 seconds (about 30%) faster. I'm hoping to see similar improvements on the Cloudfront side now that they have a "beta" build cache as well, but so far the gains are a bit more minimal (from 2m24s to 1m54s, though this DID include changes to 3 posts content, though not new posts/tags/etc) and their beta build cache only seems to be holding onto node_modules/.astro anyway, so far.

In the "best case scenario" when I add a post to /src/content/posts, the following pages would need to be rebuilt:

  1. /2024/01/11/some-title/index.html -- the actual new post page
  2. /2024/01/10/previous-title/index.html -- the previous new post needs rebuilt to pick up a "next newer post" link
  3. /2024/01/11/ index.html -- index of the day's posts
  4. /2024/01/11/page/2... -- in unlikely event of more than a page (10) worth of posts for the day, all page/2 page/3 etc. would need to be updated
  5. /2024/01/index.html -- index of the month's posts
  6. /2024/01/page/2... -- as above, the new post would bump all pages of this paginated set
  7. /2024/index.html -- index of the year's posts
  8. /2024/page/2... -- as above
  9. /index.html -- front page, always shows newest posts
  10. /page/2/index.html, /page/3/index.html... on through /page/70/index.html as the new post bumps through the pagination
  11. /rss.xml
  12. /author/myname/index.html -- archive by author
  13. /author/myname/page/2...
  14. /category/somecategory/index.html -- archive by category
  15. /category/somecategory/page/2...
  16. /tag/sometag/index.html -- archive by tag(s)
  17. /tag/sometag/page/2...
  18. /tag/anothertag...

I tell you what, this has discouraged me from implementing a 'recent posts' sidebar widget into the static build, as that would mean that every single page on the site would need to be regenerated. I'll definitely just use client-side JS for that! And it makes me think about investigating the "hybrid" approach for the pagination as this would cut down a lot of what needs to be rebuilt.

@Mark-RD
Copy link

Mark-RD commented Apr 2, 2024

Build caching idea:

//Return a page hash, unique for each page variation
export function getHash() {
  //return CMS.pageVersion;
  //return CMS.lastUpdateTime;
  //return null; //Always rebuild - Sitemap etc
  return Astro.props; //Default
}

//OR

export async function getStaticPaths() {
  return [
    { hash : /* required for partial building, null by default */ ,params: { /* required */ } ,props: { /* optional */ } },
  ];
}
---
TEMPLATE

If every page had a function (similar to getStaticPaths) called getHash that produced a unique hash for a particular page, then on a new build a list of the previous hashes and new hashes could be compared and only the ones with different hashes would be built.

This assumes that the output HTML is deterministic based of its inputs, for example the default implementation of getHash would return: sha256(Astro.props).

  • Where to get the last array of page hashes? Build caches native to platforms
  • Where to download the last full build to add the incremental files? Build caches native to platforms, or adaptors to S3 compatible APIs e.g. Cloudflare R2
  • What if there are no build cache adaptors? Rebuild the full site
  • What if the build cache is empty or expired? Rebuild the full site
  • What if the hash array is empty? Rebuild the full site

@Mark-RD
Copy link

Mark-RD commented Apr 2, 2024

  • What to do if the SSG is so large it times out current static host builds? Allow the build cache to be hydrated on a local machine before pushing to host

@reasonadmin reasonadmin mentioned this pull request Jun 20, 2024

# Background & Motivation

The original [incremental build support](https://github.com/withastro/roadmap/discussions/237) proposal is one of our oldest and most highly upvoted issues. The corresponding [roadmap issue](https://github.com/withastro/roadmap/issues/698) has likewise recieved a lot of attention and positive feedback. From the extensive performance profiling we've done, we know that Rollup is the main bottleneck in Astro's build process.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The original [incremental build support](https://github.com/withastro/roadmap/discussions/237) proposal is one of our oldest and most highly upvoted issues. The corresponding [roadmap issue](https://github.com/withastro/roadmap/issues/698) has likewise recieved a lot of attention and positive feedback. From the extensive performance profiling we've done, we know that Rollup is the main bottleneck in Astro's build process.
The original [incremental build support](https://github.com/withastro/roadmap/discussions/237) proposal is one of our oldest and most highly upvoted issues. The corresponding [roadmap issue](https://github.com/withastro/roadmap/issues/698) has likewise received a lot of attention and positive feedback. From the extensive performance profiling we've done, we know that Rollup is the main bottleneck in Astro's build process.

# Goals

- Improve `astro build` performance
- Restructure our
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- Restructure our
- Restructure our (something?)


# Example

This propsal introduces minimal public API surface changes, but requires many internal implementation details to be updated to support caching in some way.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
This propsal introduces minimal public API surface changes, but requires many internal implementation details to be updated to support caching in some way.
This proposal introduces minimal public API surface changes, but requires many internal implementation details to be updated to support caching in some way.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants