-
-
Notifications
You must be signed in to change notification settings - Fork 944
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Restore manganis optimizations #3195
Merged
Merged
Changes from all commits
Commits
Show all changes
46 commits
Select commit
Hold shift + click to select a range
3fb6d4f
Create a crate for constant serialization of config structs for manganis
ealmloff 349acab
use SerializeConst for the image asset builder
ealmloff 3fe59ed
switch to a declarative macro for assets
ealmloff 4db9c9a
clean up asset macro a bit
ealmloff b6533d0
add serializable options for each asset type
ealmloff d6c23b4
serialize const vec
ealmloff a346486
Add unique path formatting
ealmloff 96043f1
implement the new manganis macro
ealmloff 123b429
Merge branch 'main' into restore-manganis
ealmloff 5faf8d5
optimize assets in the CLI
ealmloff a6aebfa
Fix clippy
ealmloff 2c2c941
Fix assets with dioxus formattable
ealmloff f0e57c2
reduce fuzzing test limits
ealmloff 32460a4
use the new syntax in examples
ealmloff 7d92da5
fix formatting
ealmloff 3904749
Final doc and test pass on const-serialize
ealmloff cc40767
fix avif support
ealmloff 2d43173
Fix manganis doc tests
ealmloff 000d143
cache asset optimization
ealmloff fa304a3
Split out asset and bundled asset
ealmloff 713954d
Merge branch 'main' into restore-manganis
ealmloff 0a8726f
make hash pointer width independent
ealmloff bb3ad52
remove const serialize cargo lock
ealmloff 0502863
Fix manganis macro docs
ealmloff df38d3d
all tests passing
ealmloff 712e9e3
add constvec::is_empty method to fix clippy lint
ealmloff 59438c6
remove nasm feature
ealmloff 8b22f6b
simplify test_rsplit_once test so typos passes
ealmloff 50d0d41
fix range syntax for stable
ealmloff 06985f7
revert example change from clippy fix
ealmloff 67eaa4b
remove dioxus-static-site-generation workspace dependency
ealmloff 0eaca01
always accept unix paths
ealmloff fb46b91
fix windows path seperator
ealmloff 280b4de
fix folder asset hash
ealmloff c5b60a7
Optimize assets in a blocking task
ealmloff a7f7ed8
Fix asset options docs
ealmloff 7d0dba1
Document Asset and BundledAsset
ealmloff d5b0809
move manganis back into it's own folder
ealmloff da5ee69
simplify the linker macro a bit
ealmloff 14e5e53
add more docs to AssetParser expansion
ealmloff 812a37c
fix manganis core doc test
ealmloff 46c1312
add image format helpers
ealmloff d8b370c
Fill in new cargo.tomls
ealmloff 627fb07
fix folders with explicit options
ealmloff 9889aa0
Split by both unix and windows path separators and take the smallest…
ealmloff 1e70dbc
fix string length
ealmloff File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| use std::path::Path; | ||
|
|
||
| use anyhow::Context; | ||
| use lightningcss::{ | ||
| printer::PrinterOptions, | ||
| stylesheet::{MinifyOptions, ParserOptions, StyleSheet}, | ||
| }; | ||
| use manganis_core::CssAssetOptions; | ||
|
|
||
| pub(crate) fn process_css( | ||
| css_options: &CssAssetOptions, | ||
| source: &Path, | ||
| output_path: &Path, | ||
| ) -> anyhow::Result<()> { | ||
| let css = std::fs::read_to_string(source)?; | ||
|
|
||
| let css = if css_options.minified() { | ||
| minify_css(&css) | ||
| } else { | ||
| css | ||
| }; | ||
|
|
||
| std::fs::write(output_path, css).with_context(|| { | ||
| format!( | ||
| "Failed to write css to output location: {}", | ||
| output_path.display() | ||
| ) | ||
| })?; | ||
|
|
||
| Ok(()) | ||
| } | ||
|
|
||
| pub(crate) fn minify_css(css: &str) -> String { | ||
| let mut stylesheet = StyleSheet::parse(css, ParserOptions::default()).unwrap(); | ||
| stylesheet.minify(MinifyOptions::default()).unwrap(); | ||
| let printer = PrinterOptions { | ||
| minify: true, | ||
| ..Default::default() | ||
| }; | ||
| let res = stylesheet.to_css(printer).unwrap(); | ||
| res.code | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,75 @@ | ||
| use anyhow::Context; | ||
| use manganis_core::{AssetOptions, CssAssetOptions, ImageAssetOptions, JsAssetOptions}; | ||
| use std::path::Path; | ||
|
|
||
| use super::{ | ||
| css::process_css, folder::process_folder, image::process_image, js::process_js, | ||
| json::process_json, | ||
| }; | ||
|
|
||
| /// Process a specific file asset with the given options reading from the source and writing to the output path | ||
| pub(crate) fn process_file_to( | ||
| options: &AssetOptions, | ||
| source: &Path, | ||
| output_path: &Path, | ||
| ) -> anyhow::Result<()> { | ||
| // If the file already exists, then we must have a file with the same hash | ||
| // already. The hash has the file contents and options, so if we find a file | ||
| // with the same hash, we probably already created this file in the past | ||
| if output_path.exists() { | ||
| return Ok(()); | ||
| } | ||
| if let Some(parent) = output_path.parent() { | ||
| if !parent.exists() { | ||
| std::fs::create_dir_all(parent)?; | ||
| } | ||
| } | ||
| match options { | ||
| AssetOptions::Unknown => match source.extension().map(|e| e.to_string_lossy()).as_deref() { | ||
| Some("css") => { | ||
| process_css(&CssAssetOptions::new(), source, output_path)?; | ||
| } | ||
| Some("js") => { | ||
| process_js(&JsAssetOptions::new(), source, output_path)?; | ||
| } | ||
| Some("json") => { | ||
| process_json(source, output_path)?; | ||
| } | ||
| Some("jpg" | "jpeg" | "png" | "webp" | "avif") => { | ||
| process_image(&ImageAssetOptions::new(), source, output_path)?; | ||
| } | ||
| None if source.is_dir() => { | ||
| process_folder(source, output_path)?; | ||
| } | ||
| Some(_) | None => { | ||
| let source_file = std::fs::File::open(source)?; | ||
| let mut reader = std::io::BufReader::new(source_file); | ||
| let output_file = std::fs::File::create(output_path)?; | ||
| let mut writer = std::io::BufWriter::new(output_file); | ||
| std::io::copy(&mut reader, &mut writer).with_context(|| { | ||
| format!( | ||
| "Failed to write file to output location: {}", | ||
| output_path.display() | ||
| ) | ||
| })?; | ||
| } | ||
| }, | ||
| AssetOptions::Css(options) => { | ||
| process_css(options, source, output_path)?; | ||
| } | ||
| AssetOptions::Js(options) => { | ||
| process_js(options, source, output_path)?; | ||
| } | ||
| AssetOptions::Image(options) => { | ||
| process_image(options, source, output_path)?; | ||
| } | ||
| AssetOptions::Folder(_) => { | ||
| process_folder(source, output_path)?; | ||
| } | ||
| _ => { | ||
| tracing::warn!("Unknown asset options: {:?}", options); | ||
| } | ||
| } | ||
|
|
||
| Ok(()) | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| use std::path::Path; | ||
|
|
||
| use rayon::iter::{IntoParallelRefIterator, ParallelIterator}; | ||
|
|
||
| use super::file::process_file_to; | ||
|
|
||
| /// Process a folder, optimizing and copying all assets into the output folder | ||
| pub fn process_folder(source: &Path, output_folder: &Path) -> anyhow::Result<()> { | ||
| // Create the folder | ||
| std::fs::create_dir_all(output_folder)?; | ||
|
|
||
| // Then optimize children | ||
| let files: Vec<_> = std::fs::read_dir(source) | ||
| .into_iter() | ||
| .flatten() | ||
| .flatten() | ||
| .collect(); | ||
|
|
||
| files.par_iter().try_for_each(|file| { | ||
| let file = file.path(); | ||
| let metadata = file.metadata()?; | ||
| let output_path = output_folder.join(file.strip_prefix(source)?); | ||
| if metadata.is_dir() { | ||
| process_folder(&file, &output_path) | ||
| } else { | ||
| process_file_minimal(&file, &output_path) | ||
| } | ||
| })?; | ||
|
|
||
| Ok(()) | ||
| } | ||
|
|
||
| /// Optimize a file without changing any of its contents significantly (e.g. by changing the extension) | ||
| fn process_file_minimal(input_path: &Path, output_path: &Path) -> anyhow::Result<()> { | ||
| process_file_to( | ||
| &manganis_core::AssetOptions::Unknown, | ||
| input_path, | ||
| output_path, | ||
| )?; | ||
| Ok(()) | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| use image::{DynamicImage, EncodableLayout}; | ||
| use std::{ | ||
| io::{BufWriter, Write}, | ||
| path::Path, | ||
| }; | ||
|
|
||
| pub(crate) fn compress_jpg(image: DynamicImage, output_location: &Path) -> anyhow::Result<()> { | ||
| let mut comp = mozjpeg::Compress::new(mozjpeg::ColorSpace::JCS_EXT_RGBX); | ||
| let width = image.width() as usize; | ||
| let height = image.height() as usize; | ||
|
|
||
| comp.set_size(width, height); | ||
| let mut comp = comp.start_compress(Vec::new())?; // any io::Write will work | ||
|
|
||
| comp.write_scanlines(image.to_rgba8().as_bytes())?; | ||
|
|
||
| let jpeg_bytes = comp.finish()?; | ||
|
|
||
| let file = std::fs::File::create(output_location)?; | ||
| let w = &mut BufWriter::new(file); | ||
| w.write_all(&jpeg_bytes)?; | ||
| Ok(()) | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| use std::path::Path; | ||
|
|
||
| use anyhow::Context; | ||
| use jpg::compress_jpg; | ||
| use manganis_core::{ImageAssetOptions, ImageFormat, ImageSize}; | ||
| use png::compress_png; | ||
|
|
||
| mod jpg; | ||
| mod png; | ||
|
|
||
| pub(crate) fn process_image( | ||
| image_options: &ImageAssetOptions, | ||
| source: &Path, | ||
| output_path: &Path, | ||
| ) -> anyhow::Result<()> { | ||
| let mut image = image::ImageReader::new(std::io::Cursor::new(&*std::fs::read(source)?)) | ||
| .with_guessed_format()? | ||
| .decode(); | ||
|
|
||
| if let Ok(image) = &mut image { | ||
| if let ImageSize::Manual { width, height } = image_options.size() { | ||
| *image = image.resize_exact(width, height, image::imageops::FilterType::Lanczos3); | ||
| } | ||
| } | ||
|
|
||
| match (image, image_options.format()) { | ||
| (image, ImageFormat::Png) => { | ||
| compress_png(image?, output_path); | ||
| } | ||
| (image, ImageFormat::Jpg) => { | ||
| compress_jpg(image?, output_path)?; | ||
| } | ||
| (Ok(image), ImageFormat::Avif) => { | ||
| if let Err(error) = image.save(output_path) { | ||
| tracing::error!("Failed to save avif image: {} with path {}. You must have the avif feature enabled to use avif assets", error, output_path.display()); | ||
| } | ||
| } | ||
| (Ok(image), ImageFormat::Webp) => { | ||
| if let Err(err) = image.save(output_path) { | ||
| tracing::error!("Failed to save webp image: {}. You must have the avif feature enabled to use webp assets", err); | ||
| } | ||
| } | ||
| (Ok(image), _) => { | ||
| image.save(output_path)?; | ||
| } | ||
| // If we can't decode the image or it is of an unknown type, we just copy the file | ||
| _ => { | ||
| let source_file = std::fs::File::open(source)?; | ||
| let mut reader = std::io::BufReader::new(source_file); | ||
| let output_file = std::fs::File::create(output_path)?; | ||
| let mut writer = std::io::BufWriter::new(output_file); | ||
| std::io::copy(&mut reader, &mut writer).with_context(|| { | ||
| format!( | ||
| "Failed to write image to output location: {}", | ||
| output_path.display() | ||
| ) | ||
| })?; | ||
| } | ||
| } | ||
|
|
||
| Ok(()) | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it really would not be terrible to cut out the asset system (Manifest, optimizer, etc) into its own crate again... and maybe with some feature flags so we can gate
devmodedxto not have to include all this.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed. I would like to pull out asset optimization and resolution into a separate crate, but I don't want to block the release and it would take longer to make an more stable API. How about a follow up PR after more pressing issues like the base path are fixed? I have an issue ready once this is merged