Directly call in-library functions to build packages

[sheaf]

Template Β: This PR does not modify cabal behaviour (documentation, tests, refactoring, etc.)

This PR modifies cabal-install to directly go through the Cabal library when building packages, instead of the Setup interface (except of course for build-type: Custom packages).

Overview of changes:

• Addition of LibraryMethod to SetupMethod(*), in which we directly invoke the Cabal configure, build, ... functions.
  • This required a bit of GADT trickery to accomodate the fact that configure returns a LocalBuildInfo which must then be passed to subsequent phases, while with the old Setup interface everything returns IO () and communication is done through the filesystem (e.g. the local build info file).
• New library hooks-exe, provisioning:
  • CallHooksExe: the API for communicating with an external hooks executable (cabal-install depends on this)
  • HooksExe: the necessary functions for creating a hooks executable (cabal-install adds a dependency on this when compiling a package with build-type: Hooks).
  • This library depends on the new CommunicationHandle functionality from process PR #308.

(*) NB: SetupMethod/SetupWrapper is now a bit of a misnomer, because the whole aim of this PR is to no longer go through the Setup interface (and its old UserHooks incarnations such as defaultMainWithHooks ). New naming conventions welcome!
The main change is in Distribution.Client.SetupWrapper: we add a new SetupMethod (probably a misnomer at this point because we no longer go through Setup; new names for this datatype are welcome) in which we directly invoke the Cabal configure, build etc functions rather than going through Setup (or the equivalent defaultMainWithUserHooks interface).

TODO:

• Rebase on top of Setup hooks #9551.
• Land process PR #308.
• Rebase on top of Haddock: use buildSettingKeepTempFiles #9966, Be more careful in handling unconfigured programs in the program database #9967, testsuite: Pass pkgdb of store used for intree Cabal #9968, Use SetupHooks for Configure build-type #9969, Add NoImplicitPrelude to buildTypeScript #9988, SetupHooks: make Location a separate data type #9992.
• Add an implicit dependency on the hooks-exe library when compiling the hooks executable for a package with build-type: Hooks.
• Fix recompilation checking for SetupHooks.hs.
• Compare the LocalBuildInfo obtained at the end of configuring, between using the in-library method and going through Setup, and make them as consistent as it makes sense to make them.
• Test building a package with build-tool-depends: blah in which blah needs to access its data directories. This is important, as data directories are another part of process-global state (like the working directory) that is set upon invoking an external Setup executable (see Distribution.Client.SetupWrapper.internalSetupMethod).
  • This has all been tackled, in a way that fixes both the old and the new behaviour, in Correctly provision build tools in all situations #9912.
• Fix BuildToolPaths test which currently occasionally triggers openBinaryFile: invalid argument (Bad file descriptor)
• Fix issue with not being able to find version of hsc2hs.
• Delay compiling Setup for build-type: Hooks unless we absolutely need it.
• Update bootstrap plans
• Build stackage with this patch, diffing the LocalBuildInfo between in-library and Setup.
• Rebase on top of Take working directory into account in preprocessor hs-boot hack #10991, Hi Haddock: don't try to copy artifacts that don't exist #10992, configureCompiler: separate compiler vs ProgramDb #10993, cabal-install: dynExe+profExe requires prof+dyn #10994.

[andreabedini]

I tried compiling this with

source-repository-package
  type: git
  location: https://github.com/sheaf/process
  tag: 61b1de95e72acda5d773c972c7484b1a792ca9e5
  post-checkout-command: autoreconf -i

but it fails because System.Process.CommunicationHandle in process does not export useCommunicationHandle. Am I missing something?

[sheaf]

I tried compiling this with

source-repository-package
  type: git
  location: https://github.com/sheaf/process
  tag: 61b1de95e72acda5d773c972c7484b1a792ca9e5
  post-checkout-command: autoreconf -i

but it fails because System.Process.CommunicationHandle in process does not export useCommunicationHandle. Am I missing something?

I've updated the PR now, but please note that this PR is still work in progress and there are quite a few issues that I have yet to resolve.

I didn't know about post-checkout-command, that's neat.

[sheaf]

I have successfully built all of the clc-stackage repository with this patched version of cabal (I included an additional patch to ignore logging handles, to avoid using self-exec instead of in-library build method).

[mmhat]

The current state of this work is that it is blocked on a deeper refactoring of the Cabal configure logic which would split up the configure function to allow us to directly call the parts we want to do on a per-package level without re-doing things such as configuring the compiler.

Are there any related issues to that effort? I am trying to figure out at when (broadly, of course) this PR here may land.

[sheaf]

Are there any related issues to that effort? I am trying to figure out at when (broadly, of course) this PR here may land.

I implemented the refactoring I was talking about in that comment in this PR, by refactoring the configure function from the Cabal library to expose the new configureFinal function, which does some necessary configuration (e.g. backpack instantiation logic) without re-doing configuration of things we already know such as rediscovering which compiler we are using, re-computing the program database we are using, etc.

I've been thoroughly testing this PR by diffing the behaviour between HEAD and this patch when building all of stackage. I've eliminated all the inconsistencies I could find, so I think this work is about ready for another round of review.

[geekosaur]

I'm not real confident that I understood everything (or that I covered everything given the size of the diffs).

[Mikolaj]

@sheaf: would it be possible to merge the other PRs this one depend on first, to make reviewing easier? In particular, it's hard to see if the test changes indicate there are behaviour changes or not. Thanks!