Live/incremental testing

[eggyal]

Further to discussion on Zulip a while back, this issue is to track what's required to achieve "Live/incremental testing" for Rust (à la Wallaby.js and Visual Studio); the goal is for RA to run only those tests that have been affected by edits and display the results inline.

The broad design in mind for this is as follows:

• tests are compiled with instrumentation that can determine the set of code regions^* that each exercises
• initial run of all tests constructs two maps:
  • one from each code region to the set of tests that cover it; and
  • one from each test to both its result (including point of failure) and the set of code regions that it covers
• each code region displays some visual indication of the aggregate results of its covering tests, with points of failure clearly indicated
• when a code region is modified^**, all its covering tests are queued for execution
• when a test is executed, the maps and visual indications are updated

^* The ultimate solution would probably instrument at the level of basic blocks, eg via -Zinstrument-coverage—but this (presently) requires LLVM, whereas cg-clif is probably a better fit for this feature overall (e.g. because hot-swapped JIT can help minimise compilation times); instrumenting at the function level may be more easily achievable at the outset.

^** The ultimate solution would probably enqueue affected tests as code is being edited—but this requires virtual files to be shared from RA to rustc for compilation, and currently rustc is not called in-process; queuing affected tests on save may be more achievable at the outset.

On the UI front, it's probably worth noting that VSCode may get some sort of API for precisely this over the next couple of months.

[eggyal]

Just dumping some further thoughts here, for my own recollection.

• in order to identify which regions of code have been modified, we'd either need very specific change descriptions from the editor (and a means of calculating how coverage regions are affected by those changes) or else some means of diffing the current state from its previously analysed version; the latter feels more robust, and indeed having glanced over the VSCode extension APIs, I don't think the former is possible anyway. Furthermore, the diffing approach fits better with the strategy of "run on save for now, respond to live editing later"—and it could also enable the components that identify and run the affected tests to be completely decoupled from both the editor and indeed RA itself.

  • initially I thought to diff the ASTs, but now I'm wondering if it doesn't make sense to leverage the compiler's existing incremental compilation to instead identify the set of HIR nodes that have changed;
  • at whatever level the diffing is performed, the mappings to/from tests (outlined in original post) should also be in terms of that level;
  • if using -Zinstrument-coverage, the coverage map provided by the compiler (that maps instrumentation counters to source code) could directly be in those terms, rather than source code positions;

• if using -Zinstrument-coverage, the test harness could include its own profiling runtime that receives the counter incrementation calls + coverage map and produces the mappings to/from tests (outlined in original post)

• as regards execution itself, the options appear to be:

  1. compile AOT with cg_llvm—this could provide the best parity with production builds, but comes at not inconsiderable build cost (pretty much ruled out for that reason)
  2. compile with cg_clif:
     • standard AOT compilation (whilst builds are materially faster than cg_llvm, this is perhaps still too slow for launching only a small number of tests each time—especially in larger workspaces where one edit could require the recompilation and relinking of multiple crates)
     • JIT compilation: not sure I have a good enough understanding of this yet... is it pretty much just performing the above AOT compilation and then executing in-process? What happens re linking?
     • lazy JIT compilation: saves lowering uninvoked MIR (which could be significant—however, compared against non-lazy JIT with incremental compilation, any benefit will be reduced after the first complete build); currently only supports single-threaded execution
     • JIT compilation (lazy or otherwise) with hot code swapping: test harness could continue running as a background process with only modified functions lowered and swapped-in before tests are reexecuted (one could even imagine memoizing calls to pure functions, especially if they're slow)
  3. interpret, e.g. with Miri?

  My personal gut feeling is that hot-swapped JIT can provide the best balance of incremental compilation and runtime speed for this application.

[bjorn3]

JIT compilation: not sure I have a good enough understanding of this yet... is it pretty much just performing the above AOT compilation and then executing in-process? What happens re linking?

No object files are created. All dependencies need to be available as dynamic libraries and will be loaded using dlopen. All code is emitted into memory and "linked" by cranelift-jit. This mostly saves on IO and linking. There are no other things that make this mode faster.

[flodiebold]

Maybe in some distant future, the incremental query system can include test execution 🤔