Add scheduler lookahead to elide buffer resizes

[fknorr]

Buffer resizes are slow and will cause OOM conditions when a required allocation size exceeds half the device memory capacity, as we saw with weak scaling experiments on wave_sim.

The current workaround is to have an initial "resize dummy kernel" which touches the full area later accessed by the program. This is very ugly since it requires internal knowledge about Celerity's work assignment and cannot easily be communicated to a user.

As a transparent solution to this issue, this PR refactors the scheduler to operate on separate task- and command queues, which allows commands to be buffered in order to merge their allocation requirements with future CG submisssions.

The default lookahead heuristic will keep delaying instruction generation until the last allocation request has passed behind two horizons. This is enough to eliminate resizes entirely in wave_sim and RSim. Experimental APIs on celerity::queue allow changing this behavior to either always or never flush commands, and an explicit flush operation is exposed as well.

New testing infrastructure is added to inspect scheduler output based on commands and instructions generated at any point in time.

[PeterTh]

Awesome to see a general fix for these types of patterns (which have been plaguing us for years at this point)!

One general remark: I wonder if we should offer the ability to set the maximum lookahead distance via an environment variable. I think this could allow for much faster experimentation/perf evaluation for new applications or HW platforms.

[github-actions]

Check-perf-impact results: (e05743d17fba63a8011ddd314448d04e)

❓ No new benchmark data submitted. ❓
Please re-run the microbenchmarks and include the results if your commit could potentially affect performance.

[fknorr]

The instruction-graph generator now emits a warning if it detects a high number of allocations or resizes in a single buffer. This doesn't catch all resizes though. Should we have a stricter test that warns on any (also the first) resize?

[psalz]

Very cool! I particularly like that the whole thing is mostly self-contained inside the new scheduler implementation!

[github-actions]

⚠️ Clang-Tidy found issue(s) with the introduced code (1/1)

[coveralls]

Pull Request Test Coverage Report for Build 11970970296

Details

• 233 of 234 (99.57%) changed or added relevant lines in 6 files are covered.
• 1 unchanged line in 1 file lost coverage.
• Overall coverage increased (+0.1%) to 94.911%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
src/config.cc	11	12	91.67%

Files with Coverage Reduction	New Missed Lines	%
src/scheduler.cc	1	98.23%

Totals
Change from base Build 11954307236:	0.1%
Covered Lines:	7049
Relevant Lines:	7161

---

💛 - Coveralls

[github-actions]

Check-perf-impact results: (f49f87b45e71230f6a4fd4037130429f)

⚠️ Significant slowdown (>1.25x) in some microbenchmark results: building command- and instruction graphs in a dedicated scheduler thread for N nodes - 1 > throttled submission to a scheduler thread at 10 us per task / expanding tree topology
🚀 Significant speedup (<0.80x) in some microbenchmark results: 4 individual benchmarks affected

Relative execution time per category: (mean of relative medians)

• command-graph : 1.06x
• graph-nodes : 1.03x
• grid : 1.01x
• instruction-graph : 1.02x
• scheduler : 1.02x
• system : 1.01x
• task-graph : 1.06x

[github-actions]

⚠️ Clang-Tidy found issue(s) with the introduced code (1/1)

[github-actions]

Check-perf-impact results: (32e33781a20e2a3976ece8f7dc81ae2b)

⚠️ Significant slowdown (>1.25x) in some microbenchmark results: building command- and instruction graphs in a dedicated scheduler thread for N nodes - 1 > throttled submission to a scheduler thread at 10 us per task / expanding tree topology
🚀 Significant speedup (<0.80x) in some microbenchmark results: 6 individual benchmarks affected

Relative execution time per category: (mean of relative medians)

• command-graph : 1.01x
• graph-nodes : 1.01x
• grid : 1.02x
• instruction-graph : 1.03x
• scheduler : 1.03x
• system : 0.95x
• task-graph : 1.05x