buildAndWatch fails because of locale / invalid byte sequence

[andys8]

Issue

/nix/store/2md452mq5labdcxlsc5gfik97dhxabbr-haskell-org-0.1.0.0
Removing _site...
Removing _cache...
Removing _cache/tmp...
Initialising...
  Creating store...
  Creating provider...
  Running rules...
Checking for out-of-date items
Compiling
  updated 2015-11-10-Haskell-Org-Board-Resolution.pdf
  [ERROR] _cache/a98f5fd1dec8ada591e76ae4112817f9 for Hakyll.Core.Resource.Provider.MetadataCache/community.markdown/body: Store.set: invalid argument (invalid byte sequence)

Related

• People pointing out the error is related to UTF8 encoding and hakyll:
  hakyll can't handle unicode? jaspervdj/hakyll#614
• Similar issue in nixpgks: Update haskellPackages including a bump of Hackage and Stackage pin NixOS/nixpkgs#122117 (comment)
• Site not building locally on Linux #29
• Explicitly set locale to C.UTF-8 inside buildAndWatch script #236
• Fix horizontal scrolling / width on mobile #246 (comment)

System

I'm using OS: Manjaro 22.0.0 Sikaris, arch based.
Nix 2.8.1 installed as package manager, but not OS.

https://wiki.archlinux.org/title/locale

Locale

$ locale -a
C
C.UTF-8
de_DE.utf8
en_DE.utf8
en_US.utf8
POSIX

$ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC=de_DE.UTF-8
LC_TIME=de_DE.UTF-8
LC_COLLATE="en_US.UTF-8"
LC_MONETARY=de_DE.UTF-8
LC_MESSAGES="en_US.UTF-8"
LC_PAPER=de_DE.UTF-8
LC_NAME=de_DE.UTF-8
LC_ADDRESS=de_DE.UTF-8
LC_TELEPHONE=de_DE.UTF-8
LC_MEASUREMENT=de_DE.UTF-8
LC_IDENTIFICATION=de_DE.UTF-8
LC_ALL=

[TikhonJelvis]

Thanks for the detailed error report! I'll take a look at this. Frankly, I've had too many headaches from locale settings with Haskell + Nix—on another project I ended up rewriting my Haskell code to always use UTF-8 regardless of the locale settings, so I'll try that here too. Not sure whether the same approach will work with Hakyll though.

[andys8]

Sounds reasonable.

I think as long as there is this line (and maybe the others), passing an environment value from the outside is not possible as workaround.

www.haskell.org/buildAndWatch

Line 15 in 0a374cc

export LC_ALL=C.UTF-8 # fix locale error with Hakyll (see #29)

Btw. dropping this line would make the project work with my system. Maybe LC_ALL in the two default.nix files is enough for Nix? I don't know.

[TikhonJelvis]

The annoying thing is that I added that line earlier because not having it was causing the same error on a different person's system :P. I hate dealing with differences in OS environments like this—so much incidental complexity, and hard to reasonably test to boot.

Hopefully #255 fixes this problem for everyone uniformly. It's more correct behavior anyhow: the site builder will always be building the contents of this repo, which are always in UTF-8, so that would be the right behavior even on a system that was explicitly not using UTF-8 by default. (Does anybody even seriously use systems like that any more? I legitimately don't know.)