Allow Bundles to contain both CSS and Javascript files

[miracle2k]

Something I wanted to have for a long time, and something I am reminded of every time I work with webassets, is the ability to define a single Bundle that contains both Javascript and CSS files. This would allow one to define, for example, a jQueryUI bundle that contains all the files necessary, both scripts and stylesheets, in a single place.

Or, a form widget could provide a bundle with again, both the necessary JS and CSS files (this would potentially allow things like collecting all assets from all form widgets used to built the final bundle).

However, I'm unsure how to best implement this; obviously, backwards-compatibility is also a concern, although I wouldn't be opposed to breaking it for a good design.

The fact that webassets doesn't presume anything about the type of files it works with (i.e. you don't explicitly define "Javascript" and "CSS" files as in some other libs) was a very conscious decision and I'd like to keep it that way on a core architectural level; however, it might be time to add something on top to make the day to day job of working with Javascript and CSS files easier. If you define, for example, multiple JS bundles, and use the same filters for each, it's tedious to explicitly define the list of filters for each bundle again and again.

I'm opening a ticket for this as a kind of RFC; anybody stumbling over this and wanting to share their thoughts, I'd be happy to hear them.

[statico]

One strong point is that <link> and <script> resources often appear in different parts of the page (and they should, so says Yahoo), so whatever tag is used will have to appear twice on the page.

I'd suggest two tags, say, {% assets_css %} and {% assets_js %}, which would be just like {% assets %} but filter on the respective asset type.

[miracle2k]

My current thoughts on the matter: Add a number of additional classes: Package, JSBundle, CSSBundle. They would work on top of the base Bundle class, which can still be used. The docs would probably encourage people to use the new classes.

• JSBundle and CSSBundle would both use a default set of filters. The defaults are environment options. Filters can still be changed for each bundle instance as well.
• Package is an optional class, and a basic container for JSBundle and CSSBundle instances.

This would allow the following:

 AutoCompleteWidget = Package(
     JSBundle('components/autocomplete.js')
     CSSBundle('components/autocomplete.css', 'components/autocomplete.theme.css')
 )

 my_site_assets = Package(
     jQueryUI,
     AutoCompleteWidget,
     ....
 )

Then, in the template:

 {% assets_js my_site_assets %}
 {% assets_css my_site_assets %}

This would bundle the files for each respective type.

Additionally, JSBundle and CSSBundle could be smart enough that when given a Package as source content, they would only pull the correct type of assets from the package. As a result, the Package concept could truly be optional in that people wouldn't not need to have deal with packages directly, even while using a package that ships with a third party app:

 from autocomplete_app import Assets as AutoCompleteWidget

 all_js = JSPackage(
     AutoCompleteWidget,
     'js/common.js',
     'js/ajax.js',
 )

 all_css = CSSPackage(
    AutoCompleteWidget,
    'css/screen.css'
 )

A Package could not be added to the base Bundle class, however, so when using Packges, you would buy into the JSBundle/CSSBundle classes.

[miracle2k]

JSBundle/CSSBundle should look at file extensions, and use the proper filters for .sass or .coffee files, so such files can be mixed with regular js/css files in the same bundle. Declaring such a bundle should also be possible in templates (see #95).

[miracle2k]

I've put some more thought into this.

Where we are

Currently, webassets takes a very low-level approach. The Bundle class allows you (and expects you) to explicitly control the filter pipeline. You basically specify manually which filters to apply to which files.

This is because other tools at the time where rather inflexible, usually working with a global list of source files in settings.py.

Where I want to go

My thinking has changed somewhat. The fact is that in the real world, there are only two types of output files we need to generate, JS and CSS, and so there is no reason why an API should not be aware of this. There is no reason not to assume that a file ending in .js is a Javascript file, or that a .sass file needs to be preprocessed with the included sass filter. There is no reason not to assume that in most cases, the user will want to use the same Javascript minifier for all of his files.

What I propose

I propose to rename to existing Bundle class to Builder. It would represent the low-level operation, the new API would use the Builder internally, and it would continue to be supported - for backwards-compatibility, and whenever a low level approach is required.

While this rename would be backwards-incompatible (unless some clever solution can be found for the transition period, like patching __class__), upgrading would only involve changing the imports, which even for large projects, should be bearable.

The new API would then look like this:

jQueryUI = Bundle(
    'jquery.ui.accordion.js',
    'jquery.ui.accordion.css',
    'jquery.ui.datepicker.js',
    'jquery.ui.datepicker.de.js',
    'jquery.ui.datepicker.en.js',
    'jquery.ui.datepicker.css',
    jQueryUIButton
)

mySiteAssets = Bundle(
    'cssreset.css',
    jQuery,
    jQueryUI,
    'templates/*.jst',
    'pages/*.sass',
    'pages/*.coffee',
    output_js='gen/default.js', output_css='gen/screen.css',
)

Observe:

• Stylesheets and Javascript files can be mixed. This allows for libraries (like jQueryUI) to be defined as one entity.
• Files that need pre-processing can be mixed in too.

Again, in the background the Bundle class would flatten the hierarchy and construct appropriate Builder objects to do the job. The Updater classes would continue to operate on Builder instances.

It is possible to allow the new Bundles to include Builder instances. The type of the builder output (JS, CSS) could be determined via it's filters (or manually given). Only the filters explicitly specified for the builder would run on it's content.

Internally, there would probably be a baseclass which implements the overlap between Builder and Bundle (like the depends and contents properties).

The other big part of this is how Bundle determines the filters to use, while allowing the user to customize them. What needs to be known is:

• For each file extension, whether it should go to the CSS or the JS pipeline. A global registry could be provided on the environment level, and filters would be allowed to contribute to this registry.
• Which filters to use for the CSS and JS pipelines. This could be customized on the environment, or individual bundles. The default could be smart and use the best filter available. For the CSS pipeline, "cssrewrite" would be active by default. By knowing what is JS and CSS, Join Javascript files using ';' instead of a newline #100 could be addressed without user intervention.
• How to preprocess things like .sass or .coffee.

I'm not entirely sure yet how this should look in code. Something simple
might be:

class SassFilter():
    type = css
    preprocess = ('.sass', '.scss')

Or more explicit:

class SassFilter(Filter):
    extensions = {'.sass': 'sass', '.scss': 'sass'}
    preprocess = {'sass': 'css'}

In the unlikely case that a different sass filter should be used:

env.preprocess.update({
    'sass': MySassFilter
})
bundle.preprocess['sass'] = MySassFilter

While JS and CSS are exposed as hardcoded types, I would still implement it internally using s simple dict, i.e. output_js goes to output['js'].

Open Problems

How is preprocessing dealt with in debug mode? Sass, Coffeescript etc. still need to be compiled. Currently, using the future Builder class, one has to manually specify an output target for the nested Sass bundle. That would no longer be a an option. Instead, the options are, I think:

• Using preprocessors always forces a merge of all source files.
• The specified output file is used with a pre- or suffix.
• A separate directory (like .webassets-cache) is used to store the output of content that needs preprocessing in debug mode.

[tilgovi]

It's not important, for my needs, to address the bundles in Python and work directly with the classes. With that in mind, I almost exclusively use the YAML loader. For my cases, it would be enough to just take file extension argument to .urls(). Then, I would construct bundles with output files for each type of asset separately. (I would, say, a js bundle and the css bundle). These bundles would be combined (without an output file) with debug=True, with the intention that this would force them never to be combined and no output parameter is necessary. From there, calling .urls() on this bundle I could select the sub-set of assets that are appropriate for my need at the moment (e.g. link vs script tag). Other ideas are to take a regexp or a name (when sub-bundles have their own names) or full filenames (of the sub-bundle output or individual assets).

[miracle2k]

@tilgovi, I understand what you are suggesting, but not what is that you want to accomplish. What is the benefit of saying

 env['all'].urls('css')

as opposed to:

 env['css'].urls()

?

[tilgovi]

As I understand it, this issue is for bundling js and css together (or anything else, really...). One motivation is to allow libraries or widgets to specify bundles of all their dependencies.

I don't feel a strong need for this.

However, a single bundle, like env['widgets'], may come from a package. Using .urls() with an argument allows the resources to be accessed separately (for using link tag vs bottom script tag or similar). The only concern is that they aren't concatenated, and so debug=True could be forced.

I thought this sounded like minimal code changes to support the use case as I read it from the issue. Just my thoughts.

[kottenator]

Hi! How it's going with this isuue?
Last activity (commit into feature/12-pipeline branch) was 5 months ago...

[Turbo87]

The fact is that in the real world, there are only two types of output files we need to generate, JS and CSS, and so there is no reason why an API should not be aware of this.

I actually have to disagree on that. I haven't implemented it yet, but I would like to be able to for example generate PNGs from SVG files too. I like the simple, low-level approach, because it gives you all the power that you need in a very simple tool. I think that any simplification on top of that might actually make it harder for the unexperienced users. But then again I've only just started to use this library, so I could be wrong...

[miracle2k]

@Turbo87 And yet, there's on nice way to do this now, because Bundles always transform multiple input files into a single output file!

[Turbo87]

As far as I've understood the Bundles transform one or multiple input files into one output file. So you could for example just create one Bundle per Image, right?

[miracle2k]

Yes, but that's kind of ugly. In the process of supporting external assets (#151) a many-to-many transformation will hopefully be supported.