Nimble for Nim v2

[mratsim]

Nim and its ecosystem have been growing significantly in the past years.

One of the roles it grew into lately has been as the engine under Python and Javascript libraries thanks to packages like nimpy, nimporter or genny.

With the advent of Nim v2, in particular the new memory management default which
significantly improve interoperability with other languages (no need for GC_ref in libraries for example), Nim is becoming even more interesting as a backend even for compiled language such as C++, Go or Rust.

For example I have had discussions on how to use Constantine a high speed cryptography library from those 3 languages. In particular, static and generic (especially static enums), compile-time evaluation, the guarantee of no GC on value types, the performance and ergonomic has been instrumental to make such a library.
However, I now face challenges from a totally different area of computer science: package installation.

I've identified 3 areas of the current situation that would improve growth of libraries:

1. Support for monorepo
2. nimble install foo should preserve the package structure, i.e. what is on the developer machine should end up on users' machines.
3. Provide a nimble -pkg foo mytask, for example nimble -pkg constantine test
   so that users can run tasks of any installed package, in particular tests and benchmarks
   which can be used by developers for diagnostics.

Mono-repo support

For large codebases with strong versioning requirements of components, monorepos significantly reduce development friction. A single PR can update an application and all dpeendencies, instead of having to review multiple PRs.

In some cases, security or cryptographic packages, having a single dependency may even be a requirement. That requirement is motivated to prevent supply chain attacks which have grown a lot in frequency and sophistication in the past years. I indeed was asked this question "do you have any dependency besides the compiler?"

How to?

Unfortunately I'm unsure how other languages with a package manager like Rust, Go, Python, Javascript handle monorepo but I'd assume the root nimble file needs to be extended with support for:

• specifying multiple packages
• a path to those package
• a version (or the version can be in "child" .nimble file at the specified path)

Preserving the package structure

Currently if we want to have subpackages in a project we need to use

• myproject/src/subpackageA
• myproject/src/subpackageB

On installation, the src is dropped as are all folders not in src.

This is problematic:

• src being dropped requires ugly workarounds, so that you get the correct path on dev machines after a git clone and on users machine. For example in nimble itself

  import ospaths
  template thisModuleFile: string = instantiationInfo(fullPaths = true).filename
  
  when fileExists(thisModuleFile.parentDir / "src/nimblepkg/common.nim"):
    # In the git repository the Nimble sources are in a ``src`` directory.
    import src/nimblepkg/common
  else:
    # When the package is installed, the ``src`` directory disappears.
    import nimblepkg/common

• tests and benchmarks are important and help debugging correctness or performance issues on user machines. The less steps you require of the user for debugging the smoother the experience.
  For some packages with specialized code that depends on CPU architecture (assembly, endianness, GPU availablity) or for which errors are critical (cryptography), testing what is actually installed is important for quality software.
• documentation generation is broken with those src disappearing
  see @Vindaar trying to workaround this in :
  • mratsim/Arraymancer@6af866d
  • mratsim/Arraymancer@9e748ff

Lastly, implicitly exposing all those subpackageX is not always desirable.
Some of those subpackages might be internal dependencies that are not supposed to be used by the public.

Instead, the .nimble file should specify the path of the public packages.

Global namespaced nimble tasks

To continue on the topic on making supporting complex nim applications as streamlined as possible, I would like to be able to say to my users:

"Can you run nimble -pkg mypkg test and nimble -pkg mypkg bench"

Currently I have to ask them to clone, potentially github submodule, cd and run the nimble task.

Furthermore the cloned version might not be the installed version.

Related

• [RFC] Nim package structure (library author point-of-view) nimble#472
• Frictionless Nimble directory structures nimble#653
• Nimble package structure RFC Nim#6700
• [RFC] Nim package structure (library author point-of-view) #37

[Clonkk]

I will add to that the ability to package artifacts for library that are built on top of shared library (bindings for example).

[FedericoCeratto]

Regarding to the package structure, currently there isn't a very clear distinction between what files should be installed to use a library, or also run tests, or generate documentation or unneded files (e.g. screenshots, CI conf)
This is described in #179 and nim-lang/nimble#1042 and nim-lang/nimble#506

[metagn]

About -pkg, there is supposedly --package but it's only documented under "nimble run".