Any way to bring tasks dependencies management back?

[mcartoixa]

Hi guys, having been away for while (from the JavaScript ecosystem I mean) I return and many things have changed, which the way things go. I think I can understand the rationale for many changes in v4, like the fact that tasks are now functions: it makes things closer to the wire and probably both easier and more powerful. The only drawback is that we are limited in the naming of tasks by the restrictions in the naming of JS functions, but I guess I can live with that.

OTOH I was very surprised by the disappearing of tasks dependencies management (cf. https://gulpjs.com/docs/en/api/series#avoid-duplicating-tasks). To summarize the example, if I define the clean task as a dependency of 2 independent tasks the clean task will now be executed twice instead of only once (before the first of the 2 independent tasks). To me this is a central feature of any build management framework, implemented by every single one I know of: from the good old Makefile to Grunt going through Ant, MSBuild, Rake, Phing, psake... It seems to me like it puts too much of a burden on the developer to describe complex build scenarios.

Please believe that the goal of this message is not to rant about what was or what is not anymore, but rather to engage in a discussion in order to either:

• understand the rationale behind this.
• or (hopefully 😄) try and convince you to bring this feature back in some way in the future.

What are your thoughts?

[rootwork]

I'm confused by what you're asking in relation to the documentation that you link to.

From how I read it, it suggests changing from the pattern of having two tasks dependent on clean() -- meaning it'd be run twice -- to a pattern of running clean() on its own.

The "from" example uses parallel(css, javascript) in which clean() is part of those tasks -- which would be like running parallel(series(clean, css), series(clean, javascript)).

The "to" example gives series(clean, parallel(css, javascript)) as the better solution -- which means clean() is only being run once.

Or am I misunderstanding what you're asking?

[mcartoixa]

Sorry for the confusion. The linked documentation refers to the new (version 4) behaviour which I indeed find problenatic: it is now the developer's job to explicitly define the whole dependency graph in the build file. I think dependency management is a central feature of any decent build tool (cf. examples in my previous post) and it makes me cringe to have to get back to using Grunt to get it.

If I have a more complex scenario where I have, say, pack() and 'test()' as independent tasks that both depend on a third clean() task. Then I can't describe the whole build as a build() task that would depend on pack() and test() because clean() would be called twice, potentially once when another task is running. A workaround would be to create partial tasks without the clean() dependency so that I could define build() as series(clean, parallel(testWithoutClean, packWithoutClean)).

What's more, in real life it could be useful to have dedicated clean tasks like testClean() and packClean() that I would want to declare as dependency of the main clean() task and I would declare a dependency grpah with the following (pseudo) code and every task would be executed at most once:

function cleanTest() {}
function doTest() {}
function cleanPack() {]
function doPack() {}

exports.clean = series(cleanTest, cleanPack);
exports.test = series(cleanTest, doTest);
exports.pack = series(cleanPack, doPack);
exports.build = series(clean, parallel(doTest, doPack));

I hope this post will bring more clarity to my previous one 😉

[phated]

The task dependency management you describe was just a poor version of async.auto. Since gulp recommends named functions over string registration, you can just implement this yourself, such as:

exports.test = function(cb) {
  async.auto({
    clean,
    test: ['clean', doTest]
  }, cb);
}

The entirety of gulp 4 was designed to better work in the larger node ecosystem. You just need to apply the patterns that already exist.

[phated]

@mcartoixa Gulp is not a build system. We are "a toolkit to automate and enhance your workflow". We provide the bare necessities to integrate with other nodejs tools.

[yocontra]

@mcartoixa At a macro level the task orchestration was too specific in how it worked, and limited the way people could manage their own async flows - by removing it and switching to a simpler and more "normal" (conventional node) system (function composition) it is much more flexible than it was before. You can achieve the same behavior using third party libraries (like async.auto), and gulp itself became much simpler for 99% of users.

Over time many of the things we were doing in gulp started landing in node core or were created as really great userland modules - when that happens our first reaction is to remove things that are no longer needed to simplify the project. We always lean towards having users install + import their own modules vs. including a bunch of stuff in the project and turning it into a kitchen sink with a huge API surface.

tl;dr "Gulp is basically just normal javascript, you don't need to learn anything new" has always been a goal for the project so removing this custom async orchestration system was a big win on that front.

[mcartoixa]

I have tried async.auto as suggested and I am sorry to say that I find it very impractical, at best, in real life scenarios.

• First of all, tasks execution is not logged as it is with series or parallel which makes the whole script very hard to debug.
• Then in order to manage dependencies I have to either:
  • either describe every global task dependency completely. In my simple example for instance I would have to describe almost every task twice:

    exports.clean = series(cleanTest, cleanPack);
    exports.test = function(cb) {
      async.auto({
        cleanTest,
        test: ['cleanTest', doTest]
      }, cb);
    };
    exports.pack = function(cb) {
      async.auto({
        cleanPack,
        test: ['cleanPack', doPack]
      }, cb);
    };
    exports.build = function(cb) {
      async.auto({
        cleanTest,
        cleanPack,
        clean: [cleanTest, cleanPack],
        test: ['cleanTest', doTest],
        pack: ['cleanPack', doPack]
      }, cb);
    };

  • or maintain a parallel trail of objects that describe those dependencies in order to combine them as needed. Something like:

    const cleanTree = {cleanTest, cleanPack};
    exports.clean = function(cb) { async.auto(cleanTree, cb); };
    
    const testTree = {
      cleanTest,
      test: ['cleanTest', doTest]
    };
    exports.test= function(cb) { async.auto(testTree, cb); };
    
    const packTree = {
      cleanPack,
      test: ['cleanPack', doPack]
    };
    exports.pack = function(cb) { async.auto(packTree, cb); };
    
    const buildTree = Object.assign({}, cleanTree, testTree, packTree);
    exports.build = function(cb) { async.auto(buildTree, cb); };

I find both solutions very cumbersome and error prone (and that is before you factor in the lack of logs). I feel that there is the possibility of an API which could manage the dependency tree behind the hood for async.auto to execute but I don't think I will have time to explore it. We'll see.

This may sound harsh, and I don't mean to belittle your hard work so please don't take it too badly, but I am seriously questioning the value of Gulp as a build tool at this point (I get @phated's point that Gulp is just "a toolkit to automate & enhance my workflow", but I don't see how it allows me to "compose my slow, repetitive workflows into efficient build pipelines" without a usable dependency tree management).

[phated]

That's perfectly fine. Gulp wasn't made for you, it seems. You should find another tool because we aren't bringing back what you want. That's the beauty of open source software, we are not your vendor and you have infinite choice.

[yocontra]

The old gulp task system was just this package: https://github.com/robrich/orchestrator

You can install this package and use it directly if you want to keep using the old task system.