Unable to build spago

[masaeedu]

Hi there. I was trying out building https://github.com/spacchetti/spago using the following snack.nix file:

let
  nixpkgs = import <nixpkgs> {};
in
with nixpkgs.haskell.lib;
let
  # Necessary to patch packages that are broken in default nixpkgs
  # See: https://qiita.com/kimagure/items/c3fb87f7f71b9df99078
  hsOverrides = hp: with hp; hp // {
    async-pool = overrideCabal async-pool (old: {
      jailbreak = true;
      doCheck = false;
      broken = false;
    });
  };
  # ghc 8.6.5
  gwp  = nixpkgs.ghc.withPackages;
  gwp' = f: gwp (hp: f (hsOverrides hp));
in
rec {
  ghcWithPackages = gwp';
  pkgs = nixpkgs;
}

However I get the following error:

... lots of output ...
Building module Spago.TH
Local imports are:
Creating dependencies symtree for module Spago.TH
Creating module symlink for module Spago.TH
Compiling module Spago.TH
ghc: unrecognised flag: -main-is Spago

Usage: For basic information, try the `--help' option.
ghc: unrecognised flag: -main-is Curator
builder for '/nix/store/7qbpcbzi5ahkyxivip6ljc17dap48aac-Spago.Prelude.drv' failed with exit code 1
cannot build derivation '/nix/store/g6y30clfdlxspxf68jhmxkfg0rq3a67a-spago.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/gd7jmz6mkysa5qp13jm83vjlb71cp4d9-build.json.drv': 1 dependencies couldn't be built
error: build of '/nix/store/gd7jmz6mkysa5qp13jm83vjlb71cp4d9-build.json.drv' failed

Full output is here: https://gist.github.com/masaeedu/759b0d19591b555ebbe8e5cbf2364cfd

[nmattia]

Interesting, can you share your package.yaml/package.nix? Any idea where the -main-is is coming from? Looks like ghc is receiving it as -main-is Curator (one arg) as opposed to -main-is Curator (two args).

[masaeedu]

You can take a look here: https://github.com/spacchetti/spago/blob/master/package.yaml. The ghc options bit for both executables specifies that flag.

…

On Mon, May 27, 2019, 4:52 AM Nicolas Mattia ***@***.***> wrote: Interesting, can you share your package.yaml/package.nix? Any idea where the -main-is is coming from? Looks like ghc is receiving it as -main-is Curator (one arg) as opposed to -main-is Curator (two args). — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#124?email_source=notifications&email_token=AA4A7SHXMDUU6QJWAL7KPILPXOOLTA5CNFSM4HPTS3YKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWJGYHA#issuecomment-496135196>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA4A7SFKKO743ND7JFLH4JLPXOOLTANCNFSM4HPTS3YA> .

[masaeedu]

I changed:

executables:
  spago:
    # ...
    ghc-options:
    - -main-is Spago
    # ...
  spago-curator:
    # ...
    ghc-options:
    - -main-is Curator
    # ...

to:

executables:
  spago:
    # ...
    ghc-options:
    - -main-is
    - Spago
    # ...
  spago-curator:
    # ...
    ghc-options:
    - -main-is
    - Curator
    # ...

and it managed to get past that step. I think this is a bug, if the intent is to remain hpack (and therefore stack) compatible.

[masaeedu]

Now it fails with something related to template haskell:

Creating dependencies symtree for module Spago.Messages
Creating module symlink for module Spago.Messages
Compiling module Spago.Messages
unpacking sources
unpacking source archive /nix/store/jrvqah10gq6snhz67mbi5n377hp0zq4l-extra-files
source root is extra-files
building
Building module Spago.Templates
Local imports are:
 - Spago.TH
Creating dependencies symtree for module Spago.Templates
Creating module symlink for module Spago.Templates
Compiling module Spago.Templates
unpacking sources
unpacking source archive /nix/store/jrvqah10gq6snhz67mbi5n377hp0zq4l-extra-files
source root is extra-files
building
Building module Spago.Templates
Local imports are:
 - Spago.TH
Creating dependencies symtree for module Spago.Templates
Creating module symlink for module Spago.Templates
Compiling module Spago.Templates
unpacking sources
unpacking source archive /nix/store/jrvqah10gq6snhz67mbi5n377hp0zq4l-extra-files
source root is extra-files
building
Building module Spago.Watch
Local imports are:
 - Spago.Prelude
Creating dependencies symtree for module Spago.Watch
Creating module symlink for module Spago.Watch
Compiling module Spago.Watch
Spago/Templates.hs:10:17: fatal:
    cannot find object file for module `Spago.TH'
    while linking an interpreted expression
Spago/Templates.hs:10:17: fatal:
    cannot find object file for module `Spago.TH'
    while linking an interpreted expression
builder for '/nix/store/fhvkgk843h4kq94qjzg7d7jdz2hw2mcw-Spago.Templates.drv' failed with exit code 1
cannot build derivation '/nix/store/xm30cknpmjmz0i6dgz884m2gdb1sxd1j-spago-curator.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/if6hnmhdmpgmnylp27sjz4av21xymf5w-build.json.drv': 1 dependencies couldn't be built
error: build of '/nix/store/if6hnmhdmpgmnylp27sjz4av21xymf5w-build.json.drv' failed

Depending on how you'd prefer to track this stuff I can make this issue about the flags being passed in a particular way and open another issue for the template haskell related thing (I still don't know what's causing that).

[YuMingLiao]

I am facing similar issue after I successfully compiled a module with template haskell ($embedFile) in it. Now another module can't find it. I don't know what's causing that, either.

Building module Settings                                                                                                    [26/1846]
Local imports are:                 
Creating dependencies symtree for module Settings
Creating module symlink for module Settings
Compiling module Settings                                                                                 
Settings.hi  Settings.o
Done building module Settings
building '/nix/store/ksxazcqyqw1ybky9dm37qysn891vfrk9-Settings.StaticFiles.drv'...
unpacking sources
unpacking source archive /nix/store/zm54c848c608pc0i1j30z3qy6iyswxr5-extra-files
source root is extra-files
building
Building module Settings.StaticFiles
Local imports are:
 - Settings
Creating dependencies symtree for module Settings.StaticFiles
Creating module symlink for module Settings.StaticFiles
Compiling module Settings.StaticFiles
/build/extra-files
Filesystem               1K-blocks     Used Available Use% Mounted on
/dev/disk/by-label/nixos  92824212 66946124  21143188  76% /build
Settings/StaticFiles.hs
Settings/StaticFiles.hs:1:1: fatal:
    cannot find object file for module `Settings'
    while linking an interpreted expression
builder for '/nix/store/ksxazcqyqw1ybky9dm37qysn891vfrk9-Settings.StaticFiles.drv' failed with exit code 1
cannot build derivation '/nix/store/vq5mqqv81dskracxyy81i93xhhbz882p-main.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/x5d03dz4v80mkspabsik25dwg4vlz1fz-build.json.drv': 1 dependencies couldn't be built
error: build of '/nix/store/x5d03dz4v80mkspabsik25dwg4vlz1fz-build.json.drv' failed

[YuMingLiao]

Maybe it is related to this issue. Maybe adding an extension TemplateHaskell can fix this.

[YuMingLiao]

After I try to add TemplateHaskell in lib and main in package.nix, the problem still exists.

[nmattia]

Mh, it looks like the Settings object file is not being forwarded to the linking step of StaticFiles. Do you have the code?

[YuMingLiao]

I have simplified my yesod code to this.
And it's the staticFiles (...) line that makes the "Settings object file missing" error appear.
I'll look into it tomorrow.

[YuMingLiao]

I have added a successful cabal build in my code repository for comparison.

[YuMingLiao]

It seems that when buildPhase in buildModule, ghc didn't get the location of Settings object file. I haven't figured out how to pass obj information into buildModule yet.

[YuMingLiao]

After I took off -c flag in buildPhase and get dyn_o produced, ghc find the object file and snack successfully build my code!

No. Not like that. I rsync -r imported modules's folder. So Snack compiles all imported modules again when compiling a module. That's not Snack wants.

After I add -i flag for ghc, snack can find object files. However, it now asks module source code.
It says, "Could not find module Settings." That's weird, 'cause I can manually compile a TemplateHaskell object program with a plain ghc -c without meta program, as long as .dyn_o are there.

[YuMingLiao]

[root@nixos:~/TH/vanilla/Lib]# ls                                                                                                   
Lib.hs  Out  TH.dyn_hi  TH.dyn_o  TH.hi  TH.o

[root@nixos:~/TH/vanilla/Lib]# ghc Lib.hs -c                                                                                        

[root@nixos:~/TH/vanilla/Lib]# ls
Lib.hi  Lib.hs  Lib.o  Out  TH.dyn_hi  TH.dyn_o  TH.hi  TH.o

[root@nixos:~/TH/vanilla/Lib]# rm Lib.hi Lib.o

[root@nixos:~/TH/vanilla/Lib]# ghc Lib.hs -c -outputdir Out/
Lib.hs:6:7: fatal:
    cannot find object file for module ‘TH’
    while linking an interpreted expression

[root@nixos:~/TH/vanilla/Lib]# ghc Lib.hs -c -i -outputdir Out/                                                                     

Lib.hs:4:1: error:
    Could not find module ‘TH’
    It is not a module in the current program, or in any known package.                                                             
  |
4 | import TH
  | ^^^^^^^^^

[root@nixos:~/TH/vanilla/Lib]# ls ../
Exe  Exe.hs  Lib  my.cabal  setup.cabal.out  setup.out  setup.test.hs  TH.hs             

If -outputdir is used, ghc -c cannot find the object file for module in the working directory.

[YuMingLiao]

[root@nixos:~/TH/vanilla/Lib]# ls Out/

[root@nixos:~/TH/vanilla/Lib]# ls in/
TH.dyn_hi  TH.dyn_o  TH.hi  TH.o

[root@nixos:~/TH/vanilla/Lib]# ghc -c Lib.hs -outputdir in/

Lib.hs:4:1: error:
    Could not find module ‘TH’
    Use -v to see a list of the files searched for.
  |
4 | import TH
  | ^^^^^^^^^

[root@nixos:~/TH/vanilla/Lib]# ghc -c Lib.hs -iin/ -outputdir Out/
Lib.hs:6:7: fatal:
    cannot find object file for module ‘TH’
    while linking an interpreted expression

[root@nixos:~/TH/vanilla/Lib]# ghc -c Lib.hs -iin/ -outputdir in/

[root@nixos:~/TH/vanilla/Lib]# ls in/
Lib.hi  Lib.o  TH.dyn_hi  TH.dyn_o  TH.hi  TH.o

ghc needs those intermediate files from last module both in -i and -outputdir.

[YuMingLiao]

makeSymtree =
        if lib.lists.length depsDirs >= 1
        # TODO: symlink instead of copy
        then "rsync -r ${lib.strings.escapeShellArgs depsDirs} $out"
        else "";

ghc -tmpdir tmp/ ${moduleToFile modSpec.moduleName} -c -i$out -dynamic-too\
            -outputdir $out \
            ${ghcOptsArgs} \
            2>&1

After I change makeSymtree's folder from . to $out, and add ghc options -i$out -dynamic-too, my TemplateHaskell code got compiled. And my huge yesod code did so too. Great!

[nmattia]

@YuMingLiao wow, nice sleuthing! just got back from NixCon, catching up with notifications.

Can GHC still find the object files/shared objects (i.e. not the interface files) even if they're unpacked in $out? Does -i also impact those, or is it -dynamic-too?

[YuMingLiao]

Thanks for encouragement! I am so glad that I can help on this issue.

I am not sure what you mean by unpacked. I'll say what I know.

-dynamic-too not only make .o and .hi, but also create .dyn_o .dyn_hi.

And compiling a single module that use other TH module needs the other TH module's dyn_o.

So if I want to ghc -c THModule2.hs, I need to ghc -c THModule1.hs -dynamic-too.

However, if I add -outputdir like snack does, then ghc seems forgot to search . for object files.

Even if I add -outputdir to $out that has object files, ghc still can't find object files.

I have to open -i a folder that has object files and make sure -outputdir also has object files, then ghc will be satisfied and compile.

So I use $out both in -i and -outputdir.