Multiple bundles

[fedevegili]

Hello, I've been facing a problem for some time and I have no clue what to do.

I work on a very large system with most of it's functionally implemented on the front end. We have more than 2mb on minified javascript files that needs to load in different parts of the system.

When I just curl every module, everything runs fine. But as you may know, it makes A LOT of http requests, around 100 just for the system to load.

So we decided to use CRAM to create bundles with all modules included. A single bundle just kept growing and growing, while most of it's modules weren't being used but had to be transfered.

Then we split this bundle into multiple by it's functions. The dashboards got one, the login page one, the graphics one and etc.

Everything was running fine until these bundles had to share modules between then.

Everytime a bundle was loaded with a module in it that was already loaded, curl would throw a "Duplicate define" error.

The only workaround that I could think was to remove the shared files from all bundles and let it load just in time, but then, there are a lot of http requests.

Do you have any suggestion by which way we should go? Is having a giant bundle the answer?

A problem with big bundles, is that all modules are loaded eagerly instead of JIT, so having a big bundle would result in poor initial performance. (#258)

Thank you all, and sorry for any english mistakes.

[KCPS]

Sounds like you're barking up the wrong tree my friend. Take a look at
AMD http://requirejs.org/. There are several implementations of which
RequireJS is considered among the best. Several tutorials are available
if you look around (David Shariff
https://www.youtube.com/watch?v=4nU9_SPBk7Y, Yazin
https://www.youtube.com/watch?v=eRqsZqLyYaU). The AMD home site is not
particularly good at explaining how to get started, so look at the
videos first. The site does have a How to get started
http://requirejs.org/docs/start.html section which you can look at
next, then look over the entire API
http://requirejs.org/docs/api.html AMD should solve your issues as
they are similar to what other large sites are dealing with. In
particular, the lazy load feature of AMD should be of considerable
assistance in getting your viewers attention without having to load the
entire code base in one, fell swoop up front.

Hope this helps.

On 3/5/2015 7:40 AM, fedevegili wrote:

Hello, I've been facing a problem for some time and I have no clue
what to do.

I work on a very large system with most of it's functionally
implemented on the front end. We have more than 2mb on minified
javascript files that needs to load in different parts of the system.

When I just curl every module, everything runs fine. But as you may
know, it makes A LOT of http requests, around 100 just for the system
to load.

So we decided to use CRAM to create bundles with all modules included.
A single bundle just kept growing and growing, while most of it's
modules weren't being used but had to be transfered.

Then we split this bundle into multiple by it's functions. The
dashboards got one, the login page one, the graphics one and etc.

Everything was running fine until these bundles had to share modules
between then.

Everytime a bundle was loaded with a module in it that was already
loaded, curl would throw a "Duplicate define" error.

The only workaround that I could think was to remove the shared files
from all bundles and let it load just in time, but then, there are a
lot of http requests.

Do you have any suggestion by which way we should go? Is having a
giant bundle the answer?

A problem with big bundles, is that all modules are loaded eagerly
instead of JIT, so having a big bundle would result in poor initial
performance. (#258 #258)

Thank you all, and sorry for any english mistakes.

—
Reply to this email directly or view it on GitHub
#284.

[SiteSplat]

I was going to say that as well. I think what you are experiencing is also circular dependencies. Which is indeed a problem and you should perhaps re-think the whole structure. RequireJS is the way to go and you can use tools like r.js , almond, and madge https://github.com/pahen/madge to find out circular dependencies...

[KCPS]

Good advice. But first things first. As per the instructional videos
and the AMD API, convert a few of your most basic modules then try out
the loading to get a base reference that's working. Add additional
modules as your proficiency and experience grows. Under no circumstance
attempt to convert your entire system in one go. Do the conversion in
little bites so that you can locate errors and improper dependencies
quickly. Also, I would recommend that you NOT involve the entire team
initially. A pair of you working together for a couple of weeks will
accomplish quite a bit, following which you can bring in other team
members and train them in what you've learned.

Good luck!

On 3/5/2015 10:05 AM, Dave wrote:

I was going to say that as well. I think what you are experiencing is
also circular dependencies. Which is indeed a problem and you should
perhaps re-think the whole structure. RequireJS is the way to go and
you can use tools like r.js , almond, and madge
https://github.com/pahen/madge to find out circular dependencies...

—
Reply to this email directly or view it on GitHub
#284 (comment).

[unscriptable]

Thanks for trying to help out, @KCPS and @SiteSplat, but you both seem confused. It sounds like @fedevegili already has a running AMD-based application and just wants to split the bundles up.

@fedevegili: there are a few ways to get this to work. I'll post one shortly.

[KCPS]

Doesn't sound like he's using AMD judging by the description he gave.
Why don't you ask him first?

On 3/5/2015 10:25 AM, John Hann wrote:

Thanks for trying to help out, @KCPS https://github.com/KCPS and
@SiteSplat https://github.com/SiteSplat, but you both seem confused.
It sounds like @fedevegili https://github.com/fedevegili already has
a running AMD-based application and just wants to split the bundles up.

@fedevegili https://github.com/fedevegili: there are a few ways to
get this to work. I'll post one shortly.

—
Reply to this email directly or view it on GitHub
#284 (comment).

[unscriptable]

Here's one possible way:

1. Figure out the entire list of modules you want to put into a common bundle.
2. Create a bundle of these modules with cram by specifying the modules via the --include option. Let's call this bundle "common.js".
3. Identify one of the modules in this bundle as the "main module". Let's call it "util/main".
4. Create an app-specific bundle, say "dashboard.js", by including the modules used by that page, but excluding the module in the "common.js" bundle.
5. Change your curl loading config to load "dashboard.js" (as you've already successfully done) and preload the main module ("util/main") in "common.js". You will also need to map a path from "util/main" to "common.js".

Try this out and let me know your thoughts. There are likely other ways to get this to work, including using RequireJS, which has more multi-bundle options than curl.js.

[fedevegili]

Hey guys, thank you for all your help. Forgive me if my question wasn't clear enought, english is not my mother language and the situation was a little bit complex to explain.

@unscriptable is right, we have been working with AMD for over a year now. We use curl as our loader. We just started facing this bundles problem as the application started to grow big.

@unscriptable Your suggestion was the approach I've been using. Here are some more explanation:

1 - It's a Single Page Application. There's just one entry point.
2 - I've made a "common" bundle as you suggested.

The problem is that some things are too specific to be in common.js. As the application grows, it'll just grow bigger. And I will have to load common.js in the applications load (all factory functions will be executed).

After some thinking I got to this:

I can change curl's core to don't throw a duplicate define error while loading a bundle. Just ignore the second defined module. I'll end up having bigger modules and some code repetition. But it seens like the only viable option.

What do you think?

[SiteSplat]

@fedevegili Taking an aspirin for a chronic headache will not solve the root cause. Undefined is probably due to the fact you are trying load a module inside another module or something along that line and returns an empty object etc etc...

You shouldn't have circular dependencies in your code to begin with. First of all You need to individuate them and write the code in a different manner. to make this easier to spot especially for larger projects like yours I suggested to give madge a shot. https://github.com/pahen/madge

[fedevegili]

@SiteSplat Thanks for the tip, but we don't have any problem with circular dependencies.

I'll try to explain the problem again:

Let's say we have three bundles (a bundle is a file that declares some modules): Utils.js, DashboardBundle.js, PageBundle.js.

If DashboardBundle and PageBundle requires "Utils.js", it'll be included in both bundles during the bundling proccess.

Later on, on the system, if I load DashboardBundle, Utils.js will get defined.

If after that I switch to a place in the system where PageBundle is required, it will try to define Utils.js again, throwing a "Duplicate Define Error".

If I remove Utils.js from both bundles, it will end up generating http requests to require this module.

[KCPS]

Mmmmm..... I'd cut him a little slack. It's easy to say you shouldn't
have circular dependencies in one's code to begin with. But when an app
grows to several dozen modules, and the guy says he's got 2MBs to load,
what he needs are tools to keep the dependencies clear, and that's what
we're trying to help with here. It's all well and good to look back and
say, "Well you shouldn't have done things the way you have." But with
the technology and demands for SPAs growing and changing so quickly,
it's not always possible to predict future needs. Let's help him solve
his problem.

Loading modules nested in a dependency chain is precisely why the AMD
protocol was created. It just has to be used as it was intended.

On 3/5/2015 12:32 PM, Dave wrote:

@fedevegili https://github.com/fedevegili Taking an aspirin for a
chronic headache will not solve the root cause. Undefined is probably
due to the fact you are trying load a module inside another module or
something along that line and returns an empty object etc etc...

You shouldn't have circular dependencies in your code to begin with.
First of all You need to individuate them and write the code in a
different manner. to make this easier to spot especially for larger
projects like yours I suggested to give madge a shot.
https://github.com/pahen/madge

—
Reply to this email directly or view it on GitHub
#284 (comment).

[sompylasar]

@fedevegili Try webpack with the CommonChunksPlugin.

[SiteSplat]

@KCPS I'm with you but there is always a "but". Seeing as I don't know the code in the modules I pointed out a very useful tool could help him. What do you think its gonna happen considering as his app keeps growing and growing? its not going to get any better.

This is the classic situation where in a Dev. stage you realized something should have done differently therefore you take a moment to evaluate the code and see if it makes sense re-factor some modules so that the future compatibility and issues are kept at bay. @sompylasar pointed out a good tool for the job as well. Another solution to make things more manageable and under control would be to break down the modules even more...

[KCPS]

@fedevegili Re:
Later on, on the system, if I load DashboardBundle, Utils.js will get defined.
If after that I switch to a place in the system where PageBundle is required, it will try to define Utils.js again, throwing a "Duplicate Define Error".

That behaviour doesn't happen with RequireJS. It would cache the first occurrence of Util.js (by DashboardBundle), then recognize it was already available when PageBundle is requested. "Duplicate Define Error" appears to be a tool (CRAM) deficiency. The deprecation update for curl/cram - Rave - suggests that there may be some functional churning going on. In any case, Rave technology does not appear to be following a formal development methodology, therefore should be suspect since it's based on an ES6 loader. I would strongly urge adoption of the RequireJS. Lazy loading and the aforementioned caching will be your friends. As I mentioned earlier, RequreJS has substantial buy-in, considerable maturity and is a complete AMD implementation, all of which should be of comfort to you.

[fedevegili]

Guys, note that the system runs perfectly when we are NOT using bundles, so we don't have circular problems.

@KCPS I'll run some tests using requireJS to see how it behaves and post it here afterwards.

In the best case scenario, I think it may not throw the error, but as I load the second bundle there will be modules being defined again.

I will try it in a few hours and feedback you.

[fedevegili]

@sompylasar That tool looks incredibly helpful. Depending on the way I go, I'll definitely give it a shot.
Thank you!

[unscriptable]

Hey @fedevegili,

Choosing AMD was a smart move. Since AMD is a de facto standard, you have choices.

1. we can continue to solve your issues with cram+curl (yes, it is possible to achieve despite the uninformed opinions above)
2. you can try r.js
3. you can try webpack.

Both r.js and webpack are designed to handle multiple, coordinated bundles better than cram.

You may want to investigate the next generation of module loaders, such as SystemJS, too.

-- John

[OOPMan]

@fedevegili A bit late commenting on this, but I would recommend Webpack.