[fud2] Additional Syntax to Rhai DSL

[jku20]

Description

This pull request adds new syntax for defining ops which punts most of the work of the DSL to Rhai. That is, the PR defines new syntax to define ops in Rhai by directly associating a sequence of shell commands with ops. Variables and setups can be done using Rhai's scripting instead of variables and multiple rules in ninja. This leaves only very simple, one rule for each op, ninja files.

As currently written, this style of op creation may be easier to write and read, however, it may also generate less readable ninja files.

Example of New Syntax

export const verilog_state = state("verilog", ["sv", "v"]);
export const calyx_state = state("calyx", ["futil"]);
export const verilog_noverify = state("verilog-noverify", ["sv"]);

fn calyx_setup() {
    // Rhai syntax for object maps.
    #{ 
        calyx_base: config("calyx.base"), 
        calyx_exe: config_or("calyx.exe", "${calyx_base}/target/debug/calyx"),
        args: config_or("calyx.args", ""),
    }
}

defop calyx_to_verilog(c: calyx_state) >> v: verilog_state {
    let s = calyx_setup();
    // `#var` is custom syntax for substituting file names specified at runtime.
    shell("${s.calyx_exe} -l ${s.calyx_base} -b verilog ${s.args} #c > #v");
}

defop calyx_noverify(c: calyx_state) >> v: verilog_noverify {
    let s = calyx_setup();
    shell("${s.calyx_exe} -l ${s.calyx_base} -b verilog ${s.args} --disable-verify #c > #v");
}

The strings passed to shell can be constructed with a function to remove some repetition if desired.

While this style of creating ops does clash with the current style, they should still be able to work together.

TODO

• Implement defop syntax.
• Implement shell function.
• Implement config and config_or function.
• Make the two types of substitution more distinct.

[sampsyo]

Very, very cool!!! Nice work getting this going; it already looks exciting. I made a few architectural suggestions here that may be worth talking about!

The following probably bears a more extensive conversation, outside the scope of this PR: this version of the DSL has a couple of restrictions w/r/t "manual" construction of Ninja stuff (namely, no Ninja variables, and every op is exactly Ninja rule). The fact that restrictions exist seems inevitable, and these ones aren't even so bad—especially if the "manual" way still exists alongside the new way. But (even if only for research's sake), I think it's worth thinking a tiny bit about what it would look like to relax these restrictions. Materially, in the "fancy DSL":

• Configuration no longer uses Ninja variables; instead, we directly substitute configuration variables into our commands. The only non-readability downside I can think of to this is that we now need to worry about shell escaping/interpolation, whereas we previously could just let Ninja worry about this for us.
• Ops are constrained use a single build rule; they can't generate a "chain" of build rules. A potential non-readability downside here is that it sacrifices parallelism & incrementality in the case where a multi-rule op would previously allow it.

Anyway, we might decide that both limitations are worthwhile, and we can roll with them. But let's at least think through what it would take to relax both. (From a development perspective, I think it is cool to merge this restricted version without having figured out the relaxations!!)

[jku20]

On the high level architectural stuff:
At its core, this is just a question of what should go in Ninja and what should go in Rhai (i.e. what is part of the frontend and what is part of the internal driver).

This manifests in having two different types of variables floating around the Rhai scripts. Ninja variables which are replaced at run time by real values, and Rhai variables which are baked into the rules.

It would be nice if these had separate syntaxes, but that is an issue for another pr (currently they are simply strings of the Ninja variables, though how these strings are generated is hidden from the user). Similarly, it would be nice if configs and input/output files agreed in both being Ninja variables or both being Rhai variables (currently configs are Rhai and files are Ninja).

@sampsyo, I think you lean towards both inputs/outputs and configs being Ninja variables. I have less of an opinion. So might try that now.

[jku20]

Things which would be nice but are not yet implemented:

• a cache for config variables so they are not declared twice.
• figuring out how to do that for rules would be cool (that way builds can be split into multiple rules, but this would require some extra declaration with shell so fud2/Ninja know which rules depend on each other).

[sampsyo]

This is looking great; really nice work on this!! I just have a few implementation-level suggestions within.

[sampsyo]

Lookin' pretty good; maybe this one is good to go?

[jku20]

The big thing which would be nice to add to this PR would be to have the emitter cache rules and configs so it can emit duplicated ones globally.

I don't see any good reason why this couldn't be done, but it probably would be a fair bit of engineering.

I think this would entail having the emitter figure out everything it is emitting first, see what is shared, then emit some "optimized" Ninja eliminating duplicate rules.

This seems like a problem for a different PR, I'll make an issue for it.

This one seems good to go!

[sampsyo]

Indeed, sounds like an appropriate next step. And this one looks good; feel free to merge!