UI testing

[oli-obk]

fixes #20

I tried out compiletest-rs. Even though I set it up for clippy a few years back, I was unable to set it up here. I followed the instructions for finding dependencies, but nothing helped the ui tests to find the tracing_subscriber dependency. Additionally I got arbitrary rebuilds of all dependencies after tests ran and errored. This seemed very fragile and sub-optimal.

So, I wrote a quick script that makes cargo test check the .stderr files next to the examples on whether their output is the expected output. I can pull this into a separate crate if you prefer.

[davidbarsky]

I was unable to set it up here. I followed the instructions for finding dependencies, but nothing helped the ui tests to find the tracing_subscriber dependency.

D'oh, that's annoying. The script seems fine to me and I'm happy to merge as-is. Out of curiosity, do you have any opinions on https://github.com/dtolnay/trybuild? I gave it a spin a while back and I was pretty happy with it, but I'm not sure if it has any major deficiencies.

[oli-obk]

It has no deficiencies, but I misjudged what the pass tests do. I thought they just were meant to compile successfully, but they are actually run. I'll check how to make it produce stderr dumps for the program being run.

[davidbarsky]

Oh, yeah! I just remembered that I opened this issue: dtolnay/trybuild#19.

We can do the workaround (dtolnay/trybuild#19 (comment)) that dtonlay suggested or we can stick with this PR as-is—your choice.

[oli-obk]

That's actually fine, we want to execute the program and look at the output. I checked, and trybuild does just that. The problem is that trybuild only normalizes things like paths, but doesn't allow custom normalizations. If we want to test time output, we can't just turn it off on our side, so we need to normalize the output to get rid of actual timing numbers. I have implemented this here, and while we could add something to trybuild that seems like a major change that's not even sketched out and has a huge design space to explore.

[oli-obk]

or we can stick with this PR as-is—your choice.

Yea let's do this for now and push generalizing this into the future.

[davidbarsky]

Looks good, thanks!