Compile, but don't merge in debug (was: Introduce 'prefix' option on the bundle)

[rassie]

(a word of warning: this is something I've come up with to solve a problem I've had, there might be a more elegant solution, so don't hesitate to criticize)

A prefix makes it possible to redirect all or some of the static files to do some post-processing on the files. Such post-processing might be using a CSS compiler, linting, reformatting etc.

The idea is to aid development. Currently, webassets merges static files if a CSS/JS compiler is used, even in development, however, unmerged files are best suited for debugging. At the same time, compilation might be needed even in development, e.g. when some of the bundle's contents are LESS files and browser-based less.js is unsuited. Same is probably also true for Django templates.

So now, with debugging option on the bundle, files are returned as is, which is unsuitable for developing with LESS. Without it, files are combined, which is fine for production, but unsuitable for development. With the prefix option and debugging on, pseudo-bundle processing can be done in a custom view and thus be suitable for development with tools like LESS.

An example of such post-processing view for Django can be found at https://gist.github.com/3206764

[miracle2k]

Yeah, this might be a bit to specialized a solution.

I've been thinking a lot about this lately though (that is, this, and also a number of related things that are unfortunately all somewhat connected). Essentially, the current Bundle implementation always merges stuff together and writes a single output file, and there are a number of use cases where you'd want to write a separate output file for each input file (for example image versioning, #151).

One option I've come up with is adding a new debug mode that would maybe be called spew, which would apply select filters (like less), ignore the output file, and instead generate, for each source file, something like this:

 webassets-generated/${md5_of_physical_dirname}/rel/path/file.css

(I'm not quite sure yet if the hash is required).

So when adding the less filter to a bundle, in development, instead of forcing a merge, the bundle could switch to spew mode, and produce output files like the one above.

Since the webassets API is otherwise so manual (explicitly specifying outputs, filters etc), I'm a bit wary of auto-generated filenames. But short of help from a framework (like in your case), there's really no alternative.

[miracle2k]

Continuing that thought: Maybe django-assets could indeed provide a static view that serves the files from memory; I would prefer to integrate that with something like spew mode though, in favor of not needing the prefix.

[rassie]

I think the key point to understanding the problem and finding a solution is that there is no actual need to write any files anywhere in development -- generating them on the fly would work just as well, or indeed better, since every change would be applied immediately, which is just what I expect from a development mode. So some kind of pass-through mode is needed, which was exactly my motiviation for the setup described above. In such setup, there is usually no static directory (since files are transferred as-is), so there is no explicit place for anything like webassets-generated.

(On a connected note: are there many people who have a regular use for an explicit collect_static in development? I would think that most people want to change their files and refresh the browser -- and because of file affection of webassets, auto_build option has come to life, but again, with pass-through there would be no need for that option either.)

Clearly, you can't just include a pass-through view in webassets, since it's a generic package and as such has no dependencies or knowledge of the underlying frameworks, but maybe that's exactly the problem: webassets is essentially file-based, so it needs to write its files somewhere (forcing you to household with caching, regenerating, overwriting, naming etc.) A pass-through view would be possible inside django-assets and other -assets packages, but then there needs to be some support from webassets, like passing <filename, StringIO> tuples around or something very similar.

I have no conclusion or a definitive solution, since I'm not quite sure whether I'm thinking of all use cases out there, but maybe I could provide a bit of thought food for you.

[miracle2k]

The current design of Bundle, merging files while applying filters, is a bit narrow, and I suppose I'm struggling to move beyond that.

Consider that you can nest bundles, and each bundle in the hierarchy may contribute filters (sometimes I question how useful that is). Anyway, a static view, to support nesting, would have to analyse the bundle hierarchy, and reconstruct the filters that need to be applied; i.e. conceptually speaking, each source file is logically bound to it's place in that hierarchy, and it's hard to view it in isolation.

For a long time now I've been planning to have a more high-level approach (a lot of confusing notes on that in #12), where the filename itself holds the information about which filters to apply (the approach of the Rails Pipeline), i.e.: Bundle('foo.css', 'bar.less', 'baz.less.jinja2') would do the right thing all by itself.

While this will happen at some point, the low-level approach is supposed to stay as well. What I hope to do is evolve the internal code to a point where it would lend itself more easily to a one source -> one output mode (whether as part of a spew mode, or otherwise).

The current plan for #151 is that you'd be able to do:

  >>> url('some/source/file.png', tag='non-retina') 
  '/static/images/file-scaled-down.fa43b1.png'

That is not that different from:

  >>> get('some/source/file.less')
  <StringIO...>

Which would make writing a Django static view very straightforward. So my suggestion for now would be to revisit this a bit down the line.

[miracle2k]

I'm not sure about your question regarding collectstatic; it shouldn't be necessary to call it in development - django-assets has logic to use the staticfile finders to locate source files (when it sees that the staticfiles app is enabled).

[rassie]

I'm going to close this, since I've managed to implement live asset processing without a prefix and thus this PR is obsolete (as is my branch). If someone is interested, I could update the Gist with my solution.

[aconrad]

What is the status of this thread? Is compiling assets but not merging them when in debug mode still something we are considering? This is something I'd like to see happening with my SCSS files. I know this PR was closed but it seems to be the central point of the discussion (all other issues and PRs point to this one).

Here is a sample code that illustrates the problem:

import webassets
from webassets.filter import get_filter

env = webassets.Environment(
    'baked_assets/',
    '/smassets',
    debug=True,
)

scss_filter = get_filter('scss')

scss_bundle = webassets.Bundle(
    'testassetpackage/static/scss/test1.scss',
    'testassetpackage/static/scss/test2.scss',
    'testassetpackage/static/scss/test3.scss',
    filters=scss_filter,
    output='all.css',
)
env.register('test_scss', scss_bundle)
print scss_bundle.urls()  # ['/smassets/all.css?fee876b5']

[miracle2k]

I think the closest we came is #228, which I never had the time to review.