feat: support incremental builds

[janvennemann]

This is a first approach to add incremental build capabilities to Alloy. I only took care of a part of the MVC builds step, specifically controller/view generation as those are usually the most files.

Other parts of the build pipeline could be handled in a similar manner. They just need to be split up in individual tasks and their logic needs to be adjusted to handle full/incremental builds.

Things that are not yet handled properly and need some more work:

• Handle deleted files and clear associated resources
• Force full builds if required when things other than input files change, like the Alloy compiler configuration for example.
• It may be possible to also speed up clean builds by doing processing in parallel instead of the current serial pipeline. This needs to be evaluated if the Alloy build process can handle that because it's completely serially right now.

[build]

	Warnings
⚠️	Please ensure to add a changelog entry for your changes. Edit the CHANGELOG.md file and add your change under the Unreleased items header

	Messages
📖	✅ All tests are passing Nice one! All 3490 tests are passing.

New dependencies added: appc-tasks.

appc-tasks

Author: Axway Appcelerator

Description: Base implementations for any kind of task in NodeJS

Homepage: https://github.com/appcelerator/appc-tasks#readme

Created	about 2 years ago
Last Updated	about 2 years ago
License	Apache-2.0
Maintainers	1
Releases	2
Direct Dependencies	file-state-monitor and fs-extra
Keywords	task, incremental, build and project

README

appc-tasks

Base implementations for any kind of task in NodeJS

Introduction

This module provides base implementations that can be used to create your own tasks. A task in this context represents some atomic piece of work. It is used within the Titanium SDK and Hyperloop build pipelines but is designed to be usable in any other project as well.

A full API documentation can be found at http://appcelerator.github.io/appc-tasks/?api

Getting started

Install via npm

npm i appc-tasks -S

and create your own tasks using the provided base implementation

import { BaseTask } from 'appc-tasks';

class MyTask extends BaseTask {
  runTaskAction() {
    // Implement task logic here
  }
}

let task = new MyTask({name: 'myTask'});
task.run().then(() => {
  console.log('Task completed');
}).catch(err => {
  console.log(`Task failed with error: ${err}`);
});

The base task

All tasks extend from the BaseTask class which defines the interface how tasks are being run. New tasks that extend from the BaseTask need to override runTaskAction and define their task action there. To customize the behavior of a task, you can also implement the beforeTaskAction and afterTaskAction methods which will automatically be called by the task's run method. Here you can do any pre- or post-processing that might be required for every instance of that specific task. In addition a task instance can be assigned a preTaskRun and postTaskRun function, which is intended to further customize a single instance of your task.

import { BaseTask } from 'appc-tasks';

class CustomTask extends BaseTask {
  beforeTaskAction() {
    this.logger.debug('beforeTaskAction');
  }

  runTaskAction() {
    this.logger.debug('runTaskAction');
  }

  afterTaskAction() {
    this.logger.debug('afterTaskAction');
  }
}

let task = new CustomTask({
  name: 'customTask';
});
taskInstance.preTaskRun = () => {
  task.logger.debug('preTaskRun');
}
taskInstance.postTaskRun = () => {
  task.logger.info('postTaskRun');
}
taskInstance.run();
// log output:
// customTask: preTaskRun
// customTask: beforeTaskAction
// customTask: runTaskAction
// customTask: afterTaskAction
// customTask: postTaskRun

All of the above methods are executed in a .then chain, allowing you to perform async operations by returning a Promise.

The base constructor can receive two options, a required name and and an optional logger. If you don't provide a logger, a default logger using console.log will be created. In the event that you want to provide your own logger, it has to be compatible to bunyan's log method API. A task will wrap the passed logger in an adapter, which will prefix every log message with the task name for better readability throughout all your tasks log messages.

✅ Always assign a unique name to a task, whereby it can be properly identified.

File based tasks

The BaseFileTask extends the BaseTask with the concept of input and output files. Tasks that implement this interface can use that to describe which input files they require and which output files they will produce.

import { BaseFileTask } from 'appc-tasks';

class FileTask extends BaseFileTask {

  constructor(taskInfo) {
    super(taskInfo);

    this._sourceDirectory = null;
    this._outputDirectory = null;
  }

  get sourceDirectory() {
    return this._sourceDirectory;
  }

  set sourceDirectory(sourceDirectory) {
    this._sourceDirectory = sourceDirectory;
    this.addInputDirectory(this.sourceDirectory);
  }

  get outputDirectory() {
    return this._outputDirectory;
  }

  set outputDirectory(outputPath) {
    this._outputDirectory = outputPath;
    this.registerOutputPath(this.outputDirectory);
  }

  runTaskAction() {
    // this.inputFiles contains every file under the source directory
    for (let inputFile of this.inputFiles) {
      // Do your stuff, e.g. process inputFiles and write them to outputDirectory
    }
  }
}

let task = new FileTask({
  name: 'fileTask'
});
task.sourceDirectory = '/path/to/some/sources';
task.outputDirectory = '/path/to/output';
task.run();

In the above example, adding of input files is masked behind setting a property for a cleaner API. You can also pass inputFiles directly via a the constructor option of the same name if you know your set of input files beforehand, or manually call the addInputFile and addInputDirectory methods.

Similar to the input files, you can also define output files and directories. Do so by calling registerOutputPath, which will register the path so the task knows where to search for generated output files. The BaseFileTask.afterTaskAction implementation will recursively scan your registered output paths and add all found files to the outputFiles property after the task finished its runTaskAction.

⚠️ Do not call addOutputFile or addOutputDirectory yourself, the afterTaskAction will do this for you using the registered output paths.

✅ Handle the adding of input files and registration of output paths behind a property setters for a clean API in your task. This also allows you to easily access the paths using fitting property names.

Incremental file tasks

The IncrementalFileTask further extends the BaseFileTask with the ability to run full and incremental task actions, depending on wether input or output files changed. There are a few slight changes in the implementation when creating a custom incremental task.

import { IncrementalFileTask } from 'appc-tasks';

class MySmartTask extends IncrementalFileTask {
  get incrementalOutputs() {
    return [this.outputDirectory];
  }

  get outputDirectory() {
    return this._outputDirectory;
  }

  set outputDirectory(outputPath) {
    this._outputDirectory = outputPath;
    this.registerOutputPath(this.outputDirectory);
  }

  doFullTaskRun() {
    // Implement your full task run action here
  }

  doIncrementalTaskRun(changedFiles) {
    // Implement your incremental task run action here
  }
}

let task = new MySmartTask({
  name: 'incrementalTask',
  incrementalDirectory: '/incremental/mytask'
});
task.addInputDirectory('/input/path');
task.outputDirectory = '/output/path';
task.run();

When creating a new incremental task instance, the constructor requires a incrementalDirectory to be passed via the options object. This directory will hold all the state data that is used to determine changed files and any other data your task might require to perform its incremental action.

The incrementalOutputs getter is used to define the output files and directories that will be checked to see if a anything changed and trigger a full run. This has to be an Array of paths you are free to set as you seem fit for your task. By default it is an empty array.

Instead of overriding runTaskAction like in the previous examples, incremental tasks need to override doFullTaskRun and doIncrementalTaskRun to define the its logic. runTaskAction already handles the detection of file changes and triggers either a full or incremental task run. The rules for this are:

• No incremental data: full task run
• Output files changed: full task run
• Input files changed: incremental task run
• Nothing changed: skip task run

The changedFiles in doIncrementalTaskRun will be a Map with the full path to the file as the key, and either the string created, changed or deleted as the value.

What's next?

• Ability to organize tasks into some sort of Project and define dependencies between those tasks. The project then manages the execution of all tasks, taking care of execution order as well as passing input and output data from and to the individual tasks.
• Make use of ES7 decorators to mark properties as inputs and outputs

Generated by 🚫 dangerJS against bccfdb0

[hansemannn]

Any update on this? It's a real win on making larger projects more agile and scalable

[ewanharris]

@hansemannn it wont make it into the next release (1.14.0) but maybe the one after that. There's still some work (as outlined in the PR) and a lot of testing to be done to ensure this doesn't cause any regressions.

[hansemannn]

I'd be happy to test it with our own set of (larger) apps and cherry-picking it before the release would also not be an issue. Let me know if I can help.

[hansemannn]

I wonder if a simple fill watcher on app/ couldn't help already. If a file changed, recompile that one, if not, skip.

The manual alloy compilation takes so much time for every run, it devastating.