Tests for the build system?

[angerman]

Alright, this is pretty broad. And I have no idea how to approach this. These days many use tests (quickcheck, hspec, ... )

• I guess we could on the one hand, verify our internal functions, e.g. in Base? What do you think @snowleopard, would that make sense? If so, I could try to cut my teeth on getting this in.
• On the other hand, can we somehow prove / verify some that he build system does what we expect? Maybe @ndmitchell can offer some advice here?

[ndmitchell]

Most of the internal functions are pretty simple - a few explicit test cases is probably plenty and keeps the dependencies down (currently quickcheck won't compile with GHC head so you don't want to depend on that).

For the build system, use lint and that's about it. They are very difficult to test.

[angerman]

@ndmitchell What got me thinking was chunksOfSize from yesterday :-) How would you go about testing this trivially? Sorry for asking so naïvely.

[ndmitchell]

Quickcheck plus a few examples at extreme values and typical values is the default answer. Since quickcheck causes dependency headaches here, if just for skip the quickcheck part.

[snowleopard]

@angerman Tests are welcome, of course!

@ndmitchell Could you clarify the QuickCheck issue? We do depend on it already, transitively, via Shake. I think we can rely on the bootstrapping compiler here (not GHC head), can't we?

[ndmitchell]

Shake only requires QuickCheck for the test suite, so a default install won't require it. I imagine building the build system with stage3 will be desirable at some point? Or maybe it doesn't matter.

[snowleopard]

Building the build system with stage3 sounds cool, but I guess this step can just be skipped in case QuickCheck is broken on GHC head (which I presume is just a temporary issue).

I don't think we should avoid QuickCheck just for this reason. Let's use it, writing all test cases by hand feels outdated :-)

[ndmitchell]

Yep, temporary.

[angerman]

Okay, I don't get the stage3 idea. Just my recap:

• Stage 0: The external pre exiting compiler
• Stage 1: The compiler built with the bootstrap compiler targeting the target architecture.
• Stage 2: The compiler build with the stage 1 compiler built on and targeting the target architecture.
• Stage 3: ?

Does quick check have to run with the actual stage1 or stage2 compiler every time? Wouldn't quick check just be a testing dependency? And wouldn't we run the quick check tests with the Stage 0 compiler (usually a stable release for which quick check, would normally work)?

If I'm not mistaken, we are building shaking-up-ghc with the stage0 compiler anyway, right?

The issue could with the current head only arrise if one built a regular stage2 compiler and used that as a stage0 compiler for another round? (Is this considered to be the stage3 compiler)?

[snowleopard]

As I understand stage3 compiler is sometimes built just as a self test. One could also try to build the build system with it, also as a self test. Maybe @ndmitchell had anything else in mind, beyond self testing?

If I'm not mistaken, we are building shaking-up-ghc with the stage0 compiler anyway, right?

Correct.

[angerman]

If I'm not mistaken, we are building shaking-up-ghc with the stage0 compiler anyway, right?

Correct.

Shouldn't we then also run the test suite with he compiler that we use to actually compile the code? Otherwise we might run the test suite against with a compiler that might not exhibit a subtle bug, that the compiler with which we actually compile and run the code might exhibit.

[ndmitchell]

I would suggest that we only run the tests with the stage0 compiler, and even then only on the buildbot. Most likely there would be a build.bat self-test phony target which ran the tests, and was explicitly invoked by the buildbots before building. I don't see any need to run the self-test in normal build operation, GHC has plenty of its own tests.

[angerman]

@ndmitchell 👍

[ndmitchell]

Hard to argue against QuickCheck tests after screwing up precisely in a way that would have been caught by them but not by manual tests...

[snowleopard]

Thanks to #151 we now have the infrastructure for testing. Shall we close this now? Let's all keep adding new tests gradually, as we do with haddock comments.

[ndmitchell]

That was what I was expecting - test things as they get added or changed, or cause regressions, or if something looks dodgy.

[snowleopard]

Agreed.

[Ericson2314]

jacereda/fsatrace#9 gives me the idea that it would be cool to ensure that all opened files are in fact declared dependencies (or, because of temp files and other edge cases, ensure at least that files that aren't declared dependencies aren't built by any rule).

Another route to test roughly the same thing would be to use some sort of layered filesystem or version control to snapshot and rollback the build artifacts of non-depended-on jobs.

Both these I suppose could be features of Shake to test any build-system made with it.

[ndmitchell]

@Ericson2314 in Shake HEAD there is full integration with fsatrace if you just pass --lint=fsatrace, along with exclusion patterns etc. I recommend waiting for the new Shake release, where a lot has changed in this area, and then start using it.