fud2: Embed resources in executable, in release mode

[sampsyo]

As part of #1899 (see also #1878 for context), this change addresses fud2's need for external "resource" files. The idea is, in release builds of fud2, to embed all these files into the executable using the include_dir crate. To get access to these files, Ninja builds "extract" the resource files by calling back into fud2. Specifically, there is a new fud2 get-rsrc subcommand that Ninja files invoke to obtain specific resource files.

For example, here are some excerpts from a simulation build (fud2 lti.futil --to dat --through icarus -s sim.data=t.json -v -m emit):

build-tool = /Users/asampson/Documents/cu/research/calyx/target/debug/fud2
rule get-rsrc
  command = $build-tool get-rsrc $out

Every build now includes the above preamble to allow "callbacks" to fud2. Then, for instance, the Python script for data conversion works like this:

build json-dat.py: get-rsrc
rule hex-data
  command = $python json-dat.py --from-json $in $out
# ...
build $datadir: hex-data $sim_data | json-dat.py

Previously, we used json-dat.py directly from its location in the fud2 source tree. Now, we extract it into the build directory first. The build command needs an implicit dependency on the Python script to make this happen.

I added a utility e.rsrc("filename") to the fud2 emitter to make this dance easy to emit.

In debug mode, to avoid the build-time cost of embedding, fud2 instead obtains files from the rsrc directory. Build scripts are unchanged w/r/t release builds; they still go through fud2 get-rsrc. For consistency, fud2 now uses $CARGO_MANIFEST_DIR to find this directory—meaning that the rsrc config key is no longer required to use fud2. (So I removed it from the docs.)

One possible improvement here: I'm not sure include_dir is the best crate for this. There are many options; it would be great to go with something popular/well-tested. This is probably fine for now, since it's easy to swap out later?

[rachitnigam]

Slight note: it is now less obvious where this magical tb.sv file comes from (in this specification code). Is this intentional/do we want to fix it in the future?

One potential outcome of the long-term goal of treating secondary files as first class is that tb.sv could easily become a secondary file that is, by default, extracted from rsrc. In fact, something like this is super useful for the synthesis flows where we often need to override constriaints.xdc or synth.tcl.

[sampsyo]

Yep, this aspect is definitely worse about the new way. (But necessary, in a sense: if the files are going to be embedded in the binary, there is no way to avoid giving them a name. I suppose we could put them in /tmp or something instead of in the build directory, but that has its own downsides, including making it hard to do an ad hoc "override" of an extracted file.)

It would be great to have some tooling here to make these rules easier to write, i.e., to "thread through" dependencies on these extracted files. I am not sure yet how to accomplish that.

[rachitnigam]

Oh this is cool! We should probably do something similar to the Calyx compiler when pulling in primitives.

[sampsyo]

That would be cool indeed!

[rachitnigam]

@sampsyo anything blocking the merge?

[sampsyo]

anything blocking the merge?

Nope, just my own slowness!! I am going to check whether it makes sense for #1898 to go first, however, so I can fix the resulting conflicts.

[sampsyo]

Belay that; I think we should forge ahead here and I can help catch #1898 up to the new way of interacting with resources. @ayakayorihiro, that is a promise: let me fix the fud2-related merge conflicts there! 😃

[ayakayorihiro]

Thanks @sampsyo ! Sorry for the delay on #1898. Thanks for the offer 🙂