Dependency aware asset digests

[wlipa]

This PR adds dependency aware asset fingerprinting/digests to address the issue of stale assets remaining in the cache if an indirect dependency changes. It specifically affects css files with embedded url references, both to image assets and in @import statements. Js files use the import-map mechanism which avoids the need for any changes there.

The basic idea is to extract the dependencies using the css asset url compiler that updates them in the first place. This is done after the load_path assets_by_path method creates all the asset objects. A new DependencyTree class gets invoked to handle any assets with dependencies in an ordered fashion that will create stable digests that incorporate dependency fingerprint information.

Here is a walkthrough of the individual class changes.

Asset

• new accessors for dependencies and dependency_fingerprint. These are set by DependencyTree once all of the assets in the LoadPath are known. Dependencies will be nil if the asset type does not support dependencies. Otherwise, it will be a Set of the logical paths of the detected dependencies. The dependency_fingerprint is set once the dependencies of this asset have known digests. It is the concatenation of the sorted digests of the dependencies.
• a may_depend? method to quicky skip asset types that don't have dependencies (non-css).
• a digest_to_soon? method that helps DependencyTree decide if this asset is ready to have its dependency_fingerprint set. It is also used to warn if we wind up calculating a digest before the dependency information is ready. In theory, this shouldn't happen.
• the digest method now concatenates the dependency_fingerprint with the content and version, so that the digest will change appropriately as depdendencies change. Note that we don't need to digest the compiled content if we include a fingerprint that varies when any dependent file changes.

Compiler

• adds a new find_dependencies action which should return a Set of the logical paths of the assets this asset depends on. Defaults to empty.

Compilers

• supports find_dependencies by merging the find_dependencies results from each relevant compiler.

CssAssetUrls

• uses the existing regex to slice out the dependent assets that will have their digests added.

Assembly

• adds a dependency_tree method to instantiate the DependencyTree, passing the compilers.
• modifies the creation of LoadPath to take a dependency tree.

LoadPath

• take a DependencyTree object when initialized.
• moved the clear_cache method to public so that it can be used to test that digests change appropriately.
• in assets_by_path, once all the Asset objects have been created, call dependency_tree#traverse on them to resolve dependent digests.

DependencyTree

• this is a new class that implements the bottom up tree traversal to calculate the dependency fingerprints. For each asset, if it could have dependencies, and if there is at least one asset known to propshaft that it depends on, keep it in our list of assets to process.
• set_dependency_fingerprints processes these assets. We iterate through the list, calculating the dependency fingerprint for any asset with children with known digests. In the first loop these will be the "leaf" assets that only depend on static assets like images. On the next loop, it will be the assets that only depend on "leaf" assets, and so on. Once we run out of dependent assets without digests, we are done. But if we notice that we aren't making any progress on an iteration, it means there is an asset dependency cycle. In that case we bail out and warn.

Fixes #123 and #90

[eirvandelden]

Just here to mention that we've switched our propshaft to wlipa's fork and it indeed does solve the dependency aware digesting 👍

[tonydewan]

Apologies for the random drive-by here, but this PR solves a real organizational problem for my CSS with Propshaft and I'd love to see it (or something like it) land in a release. @brenogazzola and/or @dhh, I'm curious what you think of this PR?

(Thanks @wlipa!)

[theodorton]

Thanks for the submission!

Overall this looks good to me, but I've left some suggestions for a some things that should be cleaned up and I'd prefer a second opinion from @dhh or @brenogazzola before merging.

[theodorton]

Any specific reason to have this exposed as a public method? I understand it makes the load_path method more legible, but I suggest this is made private.

[wlipa]

No specific reason; that was done to copy the style of the rest of the methods.

[theodorton]

Maybe I'm missing something obvious, but in what scenario would an asset have dependencies but no dependency fingerprint? It's just not clear to me what should be done by users in response to this warning, so either failing more loudly (and clearly) or not warning at all seems like better options.

[wlipa]

That should only happen if there was a bug in the dependency tree logic. Since digest bugs are insidious, I decided a warning was necessary, but going to the extent of failing a build for inadequate cache busting seemed a little much.

[dhh]

Replaced by #188.