From 8ca735209f6d33f997efe74bf868291bd57b64b6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jakub=20=C5=BD=C3=A1dn=C3=ADk?= Date: Sat, 27 May 2023 22:51:30 +0300 Subject: [PATCH 01/23] Start a design document A WIP of a design document. Still has some parts to fill in --- DESIGN.md | 93 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 93 insertions(+) create mode 100644 DESIGN.md diff --git a/DESIGN.md b/DESIGN.md new file mode 100644 index 0000000..37d1242 --- /dev/null +++ b/DESIGN.md @@ -0,0 +1,93 @@ +# Design of nupm :warning: Work In Progress :warning: + +This file collects design ideas and directions. The intention is iterate on this document by PRs with discussion. + +## Project Structure + +A `nupm` project is defined by `project.nuon` (name inspired by Julia's Project.toml). This is where you define name of the project, version, dependencies, etc., and the type of the project. There are two types of Nushell projects (named `spam` for the example): +1. Simple script +``` +spam +├── project.nuon +└── test.nu +``` +* meant as a runnable script, equivalent of Rust's binary project (could use the `.nush` extension if we agree to support it) +* installed under `NUPM_CURRENT_OVERLAY/bin/` +2. Module +``` +spam +├── project.nuon +└── spam + └── mod.nu +``` +* meant as a library to be `use`d or `overlay use`d, equivalent of Rust's library project +* installed under `NUPM_CURRENT_OVERLAY/lib/` + +You can also install non-Nushell packages as well using a "custom" project type where you specify a `build.nu` installation script (e.g., you can install Nushell itself with it). Plugins should also be supported, preferably not requiring fully custom `build.nu`. + +## Separate virtual environments + +Inspiration: Python, [conda](https://docs.conda.io/en/latest), `cargo` + +There are two different concepts in how to handle virtual environments: +* Having global virtual environments, Python-style. We have a working prototype at https://github.com/kubouch/nuun using overlays. + * Installing a package will install it to the environment + * Possible to switch between them, they are completely isolated from each other +* Per-project virtual environment, cargo-style + * A project has its own universe (like Rust projects, for example) + +The global environments are installed as overlays in a location added by user to `NU_LIB_DIRS` (`~/.nupm/overlays`). For example project `spam` would create `~/.nupm/overlays/spam.nu`). The features of the file: +* automatically generated, managed by `nupm` +* `overlay use spam.nu` brings in all the definitions in the virtual environment, no other action needed +* `overlay hide` will restore the environment to the previous one + +Per-project environments use _identical_ framework with one difference: Instead of installing the overlay file to a global location, it is somewhere within the project. This also makes it opt-in. While `cargo` forces you to have all dependencies installed + +## Installation, bootstraping + +Requires these actions from the user (this should be kept as minimal as possible): +* Add `~/.nupm/bin` to PATH (install location for binary projects) +* Add `~/.nupm/lib` to NU_LIB_DIRS +* Add `~/.nupm/overlays` to NU_LIB_DIRS +* Make the `nupm` command available somehow (e.g., `use` inside `config.nu`) + +WIP: I have another idea in mind, need to think about it + +There are several approaches: +* bootstrap using shell script sourced from web (like rustup) +* embedded inside Nushell's binary + * The advantage of this is that it does not require user's config. The PATH and NU_LIB_DIRS could be pre-configured in Nushel +* (in the future maybe) as a compiled binary (using something like https://github.com/jntrnr/nu_app) + * This would allow us to reverse the installation steps: Instead of Nushell installing nupm, we could let user only install nupm which would in turn install Nushell + +## Dependency handling + +In compiled programming languages, there are two kinds of dependencies: static and dynamic. Static are included statically and compiled when compiling the project, dynamic are pre-compiled libraries linked to the project. Note that Nushell is [similar to compiled languages](https://www.nushell.sh/book/thinking_in_nu.html#think-of-nushell-as-a-compiled-language) rather than typical dynamic languages like Python, so these concepts are relevan for Nushell. + +Static dependencies: +* Advantages: reproducible, does not rely on system files (no more missing random.so.2), higher performance (allows joint optimization of dependencies and project itself) +* Disadvatage: increased compile time, binary size, can easily end up with multiple versions of the same library (hello Nushell dependencies) + +Dynamic dependencies are the opposite basically. Note that Nushell currently supports only static dependencies, but we might be able to add the "linking" feature at some point. + +We might want `nupm`support both types of dependencies. + +## Package repository + +Packages need to be stored somewhere. There should be one central "official" location (see https://github.com/NixOS/nixpkgs for inspiration). + +Additionally, user should be able to add 3rd party repositories as well as install local and other packages (e.g., from the web, just pointing at URL), as long as it has `project.nuon` telling `nupm` what to do. + +## API / CLI Interface + +Nushell's module design conflates CLI interface with API -- they are the same. + +WIP + +## Other + +* activations +* doc generation +* test running +* benchmark running +* configuration (do not add until we really need something to be configurable, keep it minimal) From 4fe8f6b1fbc54c123b0428e89de54806424cd8c1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jakub=20=C5=BD=C3=A1dn=C3=ADk?= Date: Sat, 27 May 2023 22:57:33 +0300 Subject: [PATCH 02/23] Update DESIGN.md --- DESIGN.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/DESIGN.md b/DESIGN.md index 37d1242..4fa7c04 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -51,7 +51,7 @@ Requires these actions from the user (this should be kept as minimal as possible * Add `~/.nupm/overlays` to NU_LIB_DIRS * Make the `nupm` command available somehow (e.g., `use` inside `config.nu`) -WIP: I have another idea in mind, need to think about it +WIP: I have another idea in mind, need to think about it. The disadvantage of this is that the default install location is not an overlay. We could make `nupm` itself an overlay that adds itself as a command. There are several approaches: * bootstrap using shell script sourced from web (like rustup) @@ -90,4 +90,4 @@ WIP * doc generation * test running * benchmark running -* configuration (do not add until we really need something to be configurable, keep it minimal) +* configuration (do not add until we really need something to be configurable, keep it minimal, case study of a project with minimal configuration: https://github.com/psf/black) From 2c98488fba7cb11777cc99828c04c259549ed641 Mon Sep 17 00:00:00 2001 From: amtoine Date: Sun, 28 May 2023 13:03:11 +0200 Subject: [PATCH 03/23] use placeholders for parts of the design we do not have names for --- DESIGN.md | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/DESIGN.md b/DESIGN.md index 4fa7c04..942abdb 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -2,13 +2,18 @@ This file collects design ideas and directions. The intention is iterate on this document by PRs with discussion. +> **Note** +> in the following, until we settle down on precise names, we use the following placeholders: +> - `METADATA_FILE`: the file containing the metadata of a package, e.g. `project.nuon`, `metadata.json` or `package.nuon` +> - `NUPM_HOME`: the location of all the `nupm` files, overlays, scripts, libraries, ..., e.g. `~/.nupm/`, `$env.XDG_DATA_HOME/nupm/` or `~/.local/share/nupm/` + ## Project Structure -A `nupm` project is defined by `project.nuon` (name inspired by Julia's Project.toml). This is where you define name of the project, version, dependencies, etc., and the type of the project. There are two types of Nushell projects (named `spam` for the example): +A `nupm` project is defined by `METADATA_FILE` (name inspired by Julia's Project.toml). This is where you define name of the project, version, dependencies, etc., and the type of the project. There are two types of Nushell projects (named `spam` for the example): 1. Simple script ``` spam -├── project.nuon +├── METADATA_FILE └── test.nu ``` * meant as a runnable script, equivalent of Rust's binary project (could use the `.nush` extension if we agree to support it) @@ -16,7 +21,7 @@ spam 2. Module ``` spam -├── project.nuon +├── METADATA_FILE └── spam └── mod.nu ``` @@ -36,7 +41,7 @@ There are two different concepts in how to handle virtual environments: * Per-project virtual environment, cargo-style * A project has its own universe (like Rust projects, for example) -The global environments are installed as overlays in a location added by user to `NU_LIB_DIRS` (`~/.nupm/overlays`). For example project `spam` would create `~/.nupm/overlays/spam.nu`). The features of the file: +The global environments are installed as overlays in a location added by user to `NU_LIB_DIRS` (`NUPM_HOME/overlays`). For example project `spam` would create `NUPM_HOME/overlays/spam.nu`). The features of the file: * automatically generated, managed by `nupm` * `overlay use spam.nu` brings in all the definitions in the virtual environment, no other action needed * `overlay hide` will restore the environment to the previous one @@ -46,9 +51,9 @@ Per-project environments use _identical_ framework with one difference: Instead ## Installation, bootstraping Requires these actions from the user (this should be kept as minimal as possible): -* Add `~/.nupm/bin` to PATH (install location for binary projects) -* Add `~/.nupm/lib` to NU_LIB_DIRS -* Add `~/.nupm/overlays` to NU_LIB_DIRS +* Add `NUPM_HOME/bin` to PATH (install location for binary projects) +* Add `NUPM_HOME/lib` to NU_LIB_DIRS +* Add `NUPM_HOME/overlays` to NU_LIB_DIRS * Make the `nupm` command available somehow (e.g., `use` inside `config.nu`) WIP: I have another idea in mind, need to think about it. The disadvantage of this is that the default install location is not an overlay. We could make `nupm` itself an overlay that adds itself as a command. @@ -76,7 +81,7 @@ We might want `nupm`support both types of dependencies. Packages need to be stored somewhere. There should be one central "official" location (see https://github.com/NixOS/nixpkgs for inspiration). -Additionally, user should be able to add 3rd party repositories as well as install local and other packages (e.g., from the web, just pointing at URL), as long as it has `project.nuon` telling `nupm` what to do. +Additionally, user should be able to add 3rd party repositories as well as install local and other packages (e.g., from the web, just pointing at URL), as long as it has `METADATA_FILE` telling `nupm` what to do. ## API / CLI Interface From 7e1819e0b457c0f0b3a3c1b09496b6b25591217b Mon Sep 17 00:00:00 2001 From: amtoine Date: Sun, 28 May 2023 13:07:02 +0200 Subject: [PATCH 04/23] add backticks to "Project.toml" and add Rust's example --- DESIGN.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/DESIGN.md b/DESIGN.md index 942abdb..c1debd1 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -9,7 +9,7 @@ This file collects design ideas and directions. The intention is iterate on this ## Project Structure -A `nupm` project is defined by `METADATA_FILE` (name inspired by Julia's Project.toml). This is where you define name of the project, version, dependencies, etc., and the type of the project. There are two types of Nushell projects (named `spam` for the example): +A `nupm` project is defined by `METADATA_FILE` (name inspired by Julia's `Project.toml` or Rust's `Cargo.toml`). This is where you define name of the project, version, dependencies, etc., and the type of the project. There are two types of Nushell projects (named `spam` for the example): 1. Simple script ``` spam From cb5673226b8c2cfb29720c660ca7e6df3f219416 Mon Sep 17 00:00:00 2001 From: amtoine Date: Mon, 29 May 2023 11:37:55 +0200 Subject: [PATCH 05/23] refactor a bit to make the file easier to edit --- DESIGN.md | 34 +++++++++++++++++++++++++--------- 1 file changed, 25 insertions(+), 9 deletions(-) diff --git a/DESIGN.md b/DESIGN.md index c1debd1..3b3ee9f 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -4,12 +4,17 @@ This file collects design ideas and directions. The intention is iterate on this > **Note** > in the following, until we settle down on precise names, we use the following placeholders: -> - `METADATA_FILE`: the file containing the metadata of a package, e.g. `project.nuon`, `metadata.json` or `package.nuon` -> - `NUPM_HOME`: the location of all the `nupm` files, overlays, scripts, libraries, ..., e.g. `~/.nupm/`, `$env.XDG_DATA_HOME/nupm/` or `~/.local/share/nupm/` +> - `METADATA_FILE`: the file containing the metadata of a package, +> e.g. `project.nuon`, `metadata.json` or `package.nuon` +> (name inspired by Julia's `Project.toml` or Rust's `Cargo.toml`) +> - `NUPM_HOME`: the location of all the `nupm` files, overlays, scripts, libraries, ..., +> e.g. `~/.nupm/`, `$env.XDG_DATA_HOME/nupm/` or `~/.local/share/nupm/` ## Project Structure -A `nupm` project is defined by `METADATA_FILE` (name inspired by Julia's `Project.toml` or Rust's `Cargo.toml`). This is where you define name of the project, version, dependencies, etc., and the type of the project. There are two types of Nushell projects (named `spam` for the example): +A `nupm` project is defined by `METADATA_FILE`. +This is where you define name of the project, version, dependencies, etc., and the type of the project. +There are two types of Nushell projects (named `spam` for the example): 1. Simple script ``` spam @@ -18,6 +23,7 @@ spam ``` * meant as a runnable script, equivalent of Rust's binary project (could use the `.nush` extension if we agree to support it) * installed under `NUPM_CURRENT_OVERLAY/bin/` + 2. Module ``` spam @@ -28,7 +34,9 @@ spam * meant as a library to be `use`d or `overlay use`d, equivalent of Rust's library project * installed under `NUPM_CURRENT_OVERLAY/lib/` -You can also install non-Nushell packages as well using a "custom" project type where you specify a `build.nu` installation script (e.g., you can install Nushell itself with it). Plugins should also be supported, preferably not requiring fully custom `build.nu`. +You can also install non-Nushell packages as well using a "custom" project type where you specify a `build.nu` installation script +(e.g., you can install Nushell itself with it). +Plugins should also be supported, preferably not requiring fully custom `build.nu`. ## Separate virtual environments @@ -41,12 +49,14 @@ There are two different concepts in how to handle virtual environments: * Per-project virtual environment, cargo-style * A project has its own universe (like Rust projects, for example) -The global environments are installed as overlays in a location added by user to `NU_LIB_DIRS` (`NUPM_HOME/overlays`). For example project `spam` would create `NUPM_HOME/overlays/spam.nu`). The features of the file: +The global environments are installed as overlays in a location added by user to `NU_LIB_DIRS` (`NUPM_HOME/overlays`). +For example project `spam` would create `NUPM_HOME/overlays/spam.nu`). The features of the file: * automatically generated, managed by `nupm` * `overlay use spam.nu` brings in all the definitions in the virtual environment, no other action needed * `overlay hide` will restore the environment to the previous one -Per-project environments use _identical_ framework with one difference: Instead of installing the overlay file to a global location, it is somewhere within the project. This also makes it opt-in. While `cargo` forces you to have all dependencies installed +Per-project environments use _identical_ framework with one difference: Instead of installing the overlay file to a global location, +it is somewhere within the project. This also makes it opt-in. While `cargo` forces you to have all dependencies installed ## Installation, bootstraping @@ -56,7 +66,8 @@ Requires these actions from the user (this should be kept as minimal as possible * Add `NUPM_HOME/overlays` to NU_LIB_DIRS * Make the `nupm` command available somehow (e.g., `use` inside `config.nu`) -WIP: I have another idea in mind, need to think about it. The disadvantage of this is that the default install location is not an overlay. We could make `nupm` itself an overlay that adds itself as a command. +WIP: I have another idea in mind, need to think about it. The disadvantage of this is that the default install location is not an overlay. +We could make `nupm` itself an overlay that adds itself as a command. There are several approaches: * bootstrap using shell script sourced from web (like rustup) @@ -67,7 +78,9 @@ There are several approaches: ## Dependency handling -In compiled programming languages, there are two kinds of dependencies: static and dynamic. Static are included statically and compiled when compiling the project, dynamic are pre-compiled libraries linked to the project. Note that Nushell is [similar to compiled languages](https://www.nushell.sh/book/thinking_in_nu.html#think-of-nushell-as-a-compiled-language) rather than typical dynamic languages like Python, so these concepts are relevan for Nushell. +In compiled programming languages, there are two kinds of dependencies: static and dynamic. Static are included statically and compiled when compiling the project, +dynamic are pre-compiled libraries linked to the project. +Note that Nushell is [similar to compiled languages][Nushell compiled] rather than typical dynamic languages like Python, so these concepts are relevan for Nushell. Static dependencies: * Advantages: reproducible, does not rely on system files (no more missing random.so.2), higher performance (allows joint optimization of dependencies and project itself) @@ -81,7 +94,8 @@ We might want `nupm`support both types of dependencies. Packages need to be stored somewhere. There should be one central "official" location (see https://github.com/NixOS/nixpkgs for inspiration). -Additionally, user should be able to add 3rd party repositories as well as install local and other packages (e.g., from the web, just pointing at URL), as long as it has `METADATA_FILE` telling `nupm` what to do. +Additionally, user should be able to add 3rd party repositories as well as install local and other packages (e.g., from the web, just pointing at URL), +as long as it has `METADATA_FILE` telling `nupm` what to do. ## API / CLI Interface @@ -96,3 +110,5 @@ WIP * test running * benchmark running * configuration (do not add until we really need something to be configurable, keep it minimal, case study of a project with minimal configuration: https://github.com/psf/black) + +[Nushell compiled]: https://www.nushell.sh/book/thinking_in_nu.html#think-of-nushell-as-a-compiled-language From ab4bb1edfcd5a4f44c70210b2c69c0cf43258b54 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jakub=20=C5=BD=C3=A1dn=C3=ADk?= Date: Mon, 29 May 2023 23:32:49 +0300 Subject: [PATCH 06/23] Add API ideas --- DESIGN.md | 44 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 43 insertions(+), 1 deletion(-) diff --git a/DESIGN.md b/DESIGN.md index 3b3ee9f..4e27c4d 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -101,7 +101,49 @@ as long as it has `METADATA_FILE` telling `nupm` what to do. Nushell's module design conflates CLI interface with API -- they are the same. -WIP +- `nupm new [--script] [--module]` + - create a new local package with template files ([`kubouch/nuun`]) +- `nupm list` + - list currently installed packages and if they're out of date +- `nupm install` + - install package into the currently active overlay (can override which overlay to install to) +- `nupm add` + - add a dependency to the current project + - it is different from `nupm install`: this one adds the dependency to the MANIFEST_FILE, `nupm install` does not +- `nupm uninstall` + - uninstall a package from a currently active overlay (can override which overlay to install to) +- `nupm update` + - update all packages in a currently active overlay overlay (can specify package and/or overlay name) + - can be used to self-update (let's avoid having both "update" and "upgrade") +- `nupm search` + - search package repository + +- `nupm check` + - parse the project to search for errors but do not run it +- `nupm test` + - run unit and integration tests of local package +- `nupm bench` + - run benchmarks +- `nupm doc` + - generate documentation +- `nupm publish` + - publish package to a repository + - (not sure about this one, for now, repository can be a github repo with packages submitted by PRs) + +- `nupm overlay new` + - create a new global overlay (Python's virtual environment style) + - `--local` flag could generate an overlay locally from the currently opened project +- `nupm overlay remove` + - deletes the overlay +- `nupm overlay list` + - list all overlays + - `nupm overlay list ` lists all packages installed within the overlay +- `nupm overlay export` + - dump all the installed package names, versions, etc. to a file +- `nupm overlay import` + - create overlay from exported file + +We could later think about being able to extend nupm, like cargo has plugins. ## Other From 32de1cd439fde6fa371f3536b41bfd9cb0d0ab6f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jakub=20=C5=BD=C3=A1dn=C3=ADk?= Date: Wed, 31 May 2023 00:01:15 +0300 Subject: [PATCH 07/23] Add notes about overlays; Misc comments --- DESIGN.md | 26 +++++++++++++++++--------- 1 file changed, 17 insertions(+), 9 deletions(-) diff --git a/DESIGN.md b/DESIGN.md index 4e27c4d..77e58b3 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -49,14 +49,14 @@ There are two different concepts in how to handle virtual environments: * Per-project virtual environment, cargo-style * A project has its own universe (like Rust projects, for example) -The global environments are installed as overlays in a location added by user to `NU_LIB_DIRS` (`NUPM_HOME/overlays`). -For example project `spam` would create `NUPM_HOME/overlays/spam.nu`). The features of the file: -* automatically generated, managed by `nupm` -* `overlay use spam.nu` brings in all the definitions in the virtual environment, no other action needed -* `overlay hide` will restore the environment to the previous one +Related to that is a lock file: It is intended to describe exactly the dependencies for a package so that it can be reproduced somewhere else. -Per-project environments use _identical_ framework with one difference: Instead of installing the overlay file to a global location, -it is somewhere within the project. This also makes it opt-in. While `cargo` forces you to have all dependencies installed +The overlays could be used to achieve all three goals at the same time. When installing a dependency for a package +* `nupm` adds entry to a **lock file** (this should be the only file you need to 100% replicate the environment) +* A .nu file (module) is auto-generated from the lock file and contains export statements like `export module NUPM_HOME/cache/packages/spam-v16.4.0-124ptnpbf/spam`. Calling `overlay use` on the file will activate your virtual environment, now you have a per-project environment +* This file can be installed into a global location that's in your `NU_LIB_DIRS` (e.g., `NUPM_HOME/overlays`) -- now you have a global Python-like virtual environment + +Each package would basically have its own overlay. This overlay file (it's just a module) could be used to also handle dependencies. If your project depends on `foo` and `bar` which both depend on `spam` but different versions, they could both import the different verions privately in their own overlay files and in your project's overlay file would be just `export use path/to/foo` and `export use path/to/bar`. This should prevent name clashing of `spam`. The only problem that needs to be figured out is how to tell `foo` to be aware of its overlay. ## Installation, bootstraping @@ -99,7 +99,7 @@ as long as it has `METADATA_FILE` telling `nupm` what to do. ## API / CLI Interface -Nushell's module design conflates CLI interface with API -- they are the same. +Nushell's module design conflates CLI interface with API -- they are the same. Not all of the below are of the same priority. - `nupm new [--script] [--module]` - create a new local package with template files ([`kubouch/nuun`]) @@ -143,7 +143,15 @@ Nushell's module design conflates CLI interface with API -- they are the same. - `nupm overlay import` - create overlay from exported file -We could later think about being able to extend nupm, like cargo has plugins. +### Other CLI-related points + +* We could later think about being able to extend nupm, like cargo has plugins. +* Mutable actions (like install) have by default Y/n prompt, but can be overriden with `--yes` +* By default, new projects are cross-platform: + * Windows + * MacOS + * Linux + * Android (if someone is willing to maintain it, we're not testing Nushell on Android, at least for now) ## Other From 8002bb7a873f156d9f4ac7b67053152f9905708a Mon Sep 17 00:00:00 2001 From: amtoine Date: Sat, 3 Jun 2023 13:29:49 +0200 Subject: [PATCH 08/23] add options for `nupm install` --- DESIGN.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/DESIGN.md b/DESIGN.md index 77e58b3..146b6bc 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -107,6 +107,9 @@ Nushell's module design conflates CLI interface with API -- they are the same. N - list currently installed packages and if they're out of date - `nupm install` - install package into the currently active overlay (can override which overlay to install to) + - `--reinstall (-r)`: reinstall package if installed + - `--update (-u)`: update local packages if outdated + - `-u` > `-r` - `nupm add` - add a dependency to the current project - it is different from `nupm install`: this one adds the dependency to the MANIFEST_FILE, `nupm install` does not From 5ce5dd3d15f0df6df839d4292b0203d0f89d08ac Mon Sep 17 00:00:00 2001 From: amtoine Date: Sat, 3 Jun 2023 13:34:33 +0200 Subject: [PATCH 09/23] mark `nupm publish` as not intended --- DESIGN.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/DESIGN.md b/DESIGN.md index 146b6bc..7c3cc0b 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -131,7 +131,7 @@ Nushell's module design conflates CLI interface with API -- they are the same. N - generate documentation - `nupm publish` - publish package to a repository - - (not sure about this one, for now, repository can be a github repo with packages submitted by PRs) + - NOT SUPPORTED FOR NOW: the repository will be a *GitHub* repo with packages submitted by PRs to start with - `nupm overlay new` - create a new global overlay (Python's virtual environment style) From 6fcebf82c2636bf68e9cecee613a4f8af497f221 Mon Sep 17 00:00:00 2001 From: amtoine Date: Sat, 3 Jun 2023 13:37:32 +0200 Subject: [PATCH 10/23] mention the self update methods --- DESIGN.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/DESIGN.md b/DESIGN.md index 7c3cc0b..2520329 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -116,8 +116,8 @@ Nushell's module design conflates CLI interface with API -- they are the same. N - `nupm uninstall` - uninstall a package from a currently active overlay (can override which overlay to install to) - `nupm update` - - update all packages in a currently active overlay overlay (can specify package and/or overlay name) - - can be used to self-update (let's avoid having both "update" and "upgrade") + - update all packages in a currently active overlay (can specify package and/or overlay name) + - can be used to self-update: `nupm update nupm`, `nupm update --self` or `nupm update --all` (would update everything) - `nupm search` - search package repository From 83f759d2ed5fe08e39ef8f56444aef8f6be3cba9 Mon Sep 17 00:00:00 2001 From: amtoine Date: Sat, 3 Jun 2023 13:39:37 +0200 Subject: [PATCH 11/23] move the design document to `docs/design/` --- DESIGN.md => docs/design/README.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename DESIGN.md => docs/design/README.md (100%) diff --git a/DESIGN.md b/docs/design/README.md similarity index 100% rename from DESIGN.md rename to docs/design/README.md From 0d28c710661d0f9a5962ba20baef0bf8c76f2b21 Mon Sep 17 00:00:00 2001 From: amtoine Date: Sat, 3 Jun 2023 13:40:59 +0200 Subject: [PATCH 12/23] link to the design in the README --- README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/README.md b/README.md index 57051a9..b59a455 100644 --- a/README.md +++ b/README.md @@ -1,2 +1,5 @@ # nupm A manager for Nushell packages. + +## :memo: design of `nupm` +please have a look at [the design document](docs/design/README.md) From 8fa88e65763350fb622e91db59ff6284b5cdf7dc Mon Sep 17 00:00:00 2001 From: amtoine Date: Sat, 3 Jun 2023 13:42:43 +0200 Subject: [PATCH 13/23] add a reference file about the `metadata` file --- docs/design/references/METADATA.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 docs/design/references/METADATA.md diff --git a/docs/design/references/METADATA.md b/docs/design/references/METADATA.md new file mode 100644 index 0000000..b50d8e1 --- /dev/null +++ b/docs/design/references/METADATA.md @@ -0,0 +1,22 @@ +# `nupm`'s package metadata reference + +## Required attributes +- `author`: Name of the developer/organization/etc. +- `files`: List of records of files being part of the package. Records reference: + - `checksum`: SHA256 sum of the file + - `name`: File path (relative to the package scope) + - `raw-url`: `GET`table link to the file + - `supported-os`: Exactly like in the top-level metadata +- `name`: Package name +- `nu-version`: Supported Nushell version. Just like `dependencies` might be either exact version or some matcher like "greater than 1.70" +- `short-description`: Short info about the package, displayed by default +- `supported-os`: Operating systems supported by the package, the most broad possibility: `{"arch": ["*"], "family": ["*"], "name": ["*"]}`. Matched by `$nu.os-info` +- `url`: Package website/GitHub repository. Basically a place where one can find some additional info about the package +- `version`: Version of the package. Semantic versioning is advised to enable users to have more generic requirements + +## Optional attribtues +- `dependencies`: Packages needed by the package — versions have to be specified. e.g. `[nupm/0.7.0]`. Semantic versioning is also supported: `[nupm/~0.7]` +- `installer`: Name of a script (relative to the package scope) that will install the package instead (or in addition to) of default `nupm` logic +- `keywords`: List of keywords used by `nupm search` in addition to `name` +- `pre-install-hook`: Name of a script (relative to the package scope) to be executed before installation. Might be simple pre-install message or more complex logic +- `post-install-hook`: Name of a script (relative to the package scope) to be executed after installation. Might be simple post-install message or more complex logic From 7cecdb5e0ecd39f5459181e2871d65cafe7cf294 Mon Sep 17 00:00:00 2001 From: amtoine Date: Sat, 3 Jun 2023 13:45:37 +0200 Subject: [PATCH 14/23] add options to `nupm search` --- docs/design/README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/design/README.md b/docs/design/README.md index 2520329..6ac9eb7 100644 --- a/docs/design/README.md +++ b/docs/design/README.md @@ -119,7 +119,9 @@ Nushell's module design conflates CLI interface with API -- they are the same. N - update all packages in a currently active overlay (can specify package and/or overlay name) - can be used to self-update: `nupm update nupm`, `nupm update --self` or `nupm update --all` (would update everything) - `nupm search` - - search package repository + - search package repository (only supported ones by default) + - `--unsupported (-u)`: would also list packages that are not supported in the user's system, e.g. due to OS incompatibilities + - `--all (-a)`: would list all packages - `nupm check` - parse the project to search for errors but do not run it From 345da642362f2e0d259f349c8b1cbe673e12f91f Mon Sep 17 00:00:00 2001 From: amtoine Date: Sat, 3 Jun 2023 13:46:49 +0200 Subject: [PATCH 15/23] add `--yes` to `install`, `uninstall` and `update` --- docs/design/README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/design/README.md b/docs/design/README.md index 6ac9eb7..0f0152f 100644 --- a/docs/design/README.md +++ b/docs/design/README.md @@ -110,14 +110,17 @@ Nushell's module design conflates CLI interface with API -- they are the same. N - `--reinstall (-r)`: reinstall package if installed - `--update (-u)`: update local packages if outdated - `-u` > `-r` + - `--yes (-y)`: do not ask for user confirmation to use `nupm install` in scripts - `nupm add` - add a dependency to the current project - it is different from `nupm install`: this one adds the dependency to the MANIFEST_FILE, `nupm install` does not - `nupm uninstall` - uninstall a package from a currently active overlay (can override which overlay to install to) + - `--yes (-y)`: do not ask for user confirmation to use `nupm uninstall` in scripts - `nupm update` - update all packages in a currently active overlay (can specify package and/or overlay name) - can be used to self-update: `nupm update nupm`, `nupm update --self` or `nupm update --all` (would update everything) + - `--yes (-y)`: do not ask for user confirmation to use `nupm update` in scripts - `nupm search` - search package repository (only supported ones by default) - `--unsupported (-u)`: would also list packages that are not supported in the user's system, e.g. due to OS incompatibilities From a5b118f57ae86bafa4c84c7ef4cdcde2454bccb4 Mon Sep 17 00:00:00 2001 From: amtoine Date: Sat, 3 Jun 2023 13:48:55 +0200 Subject: [PATCH 16/23] add a note about short / long descriptions --- docs/design/README.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/design/README.md b/docs/design/README.md index 0f0152f..c7fdf4d 100644 --- a/docs/design/README.md +++ b/docs/design/README.md @@ -101,6 +101,11 @@ as long as it has `METADATA_FILE` telling `nupm` what to do. Nushell's module design conflates CLI interface with API -- they are the same. Not all of the below are of the same priority. +> **Note** +> commands like `list`, `install`, `search`, `uninstall`, `update`, ..., i.e. should +> - give short descriptions by default +> - give long descriptions with `--long-description (-l)` + - `nupm new [--script] [--module]` - create a new local package with template files ([`kubouch/nuun`]) - `nupm list` From ad6b0cd8a06722ce05f90a2f67bf782f40c87010 Mon Sep 17 00:00:00 2001 From: amtoine Date: Sun, 4 Jun 2023 10:56:13 +0200 Subject: [PATCH 17/23] apply misc review suggestions --- docs/design/README.md | 12 ++++++------ docs/design/references/METADATA.md | 2 -- 2 files changed, 6 insertions(+), 8 deletions(-) diff --git a/docs/design/README.md b/docs/design/README.md index c7fdf4d..7317981 100644 --- a/docs/design/README.md +++ b/docs/design/README.md @@ -103,8 +103,8 @@ Nushell's module design conflates CLI interface with API -- they are the same. N > **Note** > commands like `list`, `install`, `search`, `uninstall`, `update`, ..., i.e. should -> - give short descriptions by default -> - give long descriptions with `--long-description (-l)` +> - print short descriptions by default +> - print long descriptions with `--long-description (-l)` - `nupm new [--script] [--module]` - create a new local package with template files ([`kubouch/nuun`]) @@ -114,18 +114,18 @@ Nushell's module design conflates CLI interface with API -- they are the same. N - install package into the currently active overlay (can override which overlay to install to) - `--reinstall (-r)`: reinstall package if installed - `--update (-u)`: update local packages if outdated - - `-u` > `-r` - - `--yes (-y)`: do not ask for user confirmation to use `nupm install` in scripts + - Both `-u` & `-r` flags might be specified, but updating has higher priority than reinstalling + - `--yes (-y)`: do not ask for user confirmation, e.g. to use `nupm install` in scripts - `nupm add` - add a dependency to the current project - it is different from `nupm install`: this one adds the dependency to the MANIFEST_FILE, `nupm install` does not - `nupm uninstall` - uninstall a package from a currently active overlay (can override which overlay to install to) - - `--yes (-y)`: do not ask for user confirmation to use `nupm uninstall` in scripts + - `--yes (-y)`: do not ask for user confirmation, e.g. to use `nupm uninstall` in scripts - `nupm update` - update all packages in a currently active overlay (can specify package and/or overlay name) - can be used to self-update: `nupm update nupm`, `nupm update --self` or `nupm update --all` (would update everything) - - `--yes (-y)`: do not ask for user confirmation to use `nupm update` in scripts + - `--yes (-y)`: do not ask for user confirmation, e.g. to use `nupm update` in scripts - `nupm search` - search package repository (only supported ones by default) - `--unsupported (-u)`: would also list packages that are not supported in the user's system, e.g. due to OS incompatibilities diff --git a/docs/design/references/METADATA.md b/docs/design/references/METADATA.md index b50d8e1..f849c11 100644 --- a/docs/design/references/METADATA.md +++ b/docs/design/references/METADATA.md @@ -18,5 +18,3 @@ - `dependencies`: Packages needed by the package — versions have to be specified. e.g. `[nupm/0.7.0]`. Semantic versioning is also supported: `[nupm/~0.7]` - `installer`: Name of a script (relative to the package scope) that will install the package instead (or in addition to) of default `nupm` logic - `keywords`: List of keywords used by `nupm search` in addition to `name` -- `pre-install-hook`: Name of a script (relative to the package scope) to be executed before installation. Might be simple pre-install message or more complex logic -- `post-install-hook`: Name of a script (relative to the package scope) to be executed after installation. Might be simple post-install message or more complex logic From c2da065eaf74248600e194954730d92433727b17 Mon Sep 17 00:00:00 2001 From: amtoine Date: Sun, 4 Jun 2023 11:19:13 +0200 Subject: [PATCH 18/23] fix minor typos and refactor into NOTEs and links --- docs/design/README.md | 58 +++++++++++++++++++++++++++---------------- 1 file changed, 36 insertions(+), 22 deletions(-) diff --git a/docs/design/README.md b/docs/design/README.md index 7317981..e29858a 100644 --- a/docs/design/README.md +++ b/docs/design/README.md @@ -1,4 +1,4 @@ -# Design of nupm :warning: Work In Progress :warning: +# Design of `nupm` :warning: Work In Progress :warning: This file collects design ideas and directions. The intention is iterate on this document by PRs with discussion. @@ -14,6 +14,10 @@ This file collects design ideas and directions. The intention is iterate on this A `nupm` project is defined by `METADATA_FILE`. This is where you define name of the project, version, dependencies, etc., and the type of the project. +> **Note** +> see [`METADATA.md`](references/METADATA.md) for a more in-depth description of +> the `METADATA_FILE` + There are two types of Nushell projects (named `spam` for the example): 1. Simple script ``` @@ -21,7 +25,8 @@ spam ├── METADATA_FILE └── test.nu ``` -* meant as a runnable script, equivalent of Rust's binary project (could use the `.nush` extension if we agree to support it) +* meant as a runnable script, equivalent of Rust's binary project +* could use the `.nush` extension if we agree to support it * installed under `NUPM_CURRENT_OVERLAY/bin/` 2. Module @@ -40,13 +45,13 @@ Plugins should also be supported, preferably not requiring fully custom `build.n ## Separate virtual environments -Inspiration: Python, [conda](https://docs.conda.io/en/latest), `cargo` +> Inspiration: Python, [conda](https://docs.conda.io/en/latest), `cargo` There are two different concepts in how to handle virtual environments: -* Having global virtual environments, Python-style. We have a working prototype at https://github.com/kubouch/nuun using overlays. +* Having global virtual environments, Python-style. We have a working prototype at [`kubouch/nuun`] using overlays. * Installing a package will install it to the environment * Possible to switch between them, they are completely isolated from each other -* Per-project virtual environment, cargo-style +* Per-project virtual environment, `cargo`-style * A project has its own universe (like Rust projects, for example) Related to that is a lock file: It is intended to describe exactly the dependencies for a package so that it can be reproduced somewhere else. @@ -66,29 +71,35 @@ Requires these actions from the user (this should be kept as minimal as possible * Add `NUPM_HOME/overlays` to NU_LIB_DIRS * Make the `nupm` command available somehow (e.g., `use` inside `config.nu`) -WIP: I have another idea in mind, need to think about it. The disadvantage of this is that the default install location is not an overlay. -We could make `nupm` itself an overlay that adds itself as a command. +> :warning: **WIP** +> I have another idea in mind, need to think about it. The disadvantage of this is that the default install location is not an overlay. +> We could make `nupm` itself an overlay that adds itself as a command. There are several approaches: -* bootstrap using shell script sourced from web (like rustup) +* bootstrap using shell script sourced from web (like `rustup`) * embedded inside Nushell's binary - * The advantage of this is that it does not require user's config. The PATH and NU_LIB_DIRS could be pre-configured in Nushel -* (in the future maybe) as a compiled binary (using something like https://github.com/jntrnr/nu_app) - * This would allow us to reverse the installation steps: Instead of Nushell installing nupm, we could let user only install nupm which would in turn install Nushell + * The advantage of this is that it does not require user's config. The `PATH` and `NU_LIB_DIRS` could be pre-configured in Nushell +* (in the future maybe) as a compiled binary (using something like [`jntrnr/nu_app`]) + * This would allow us to reverse the installation steps: Instead of Nushell installing `nupm`, we could let user only install `nupm` which would in turn install Nushell ## Dependency handling In compiled programming languages, there are two kinds of dependencies: static and dynamic. Static are included statically and compiled when compiling the project, dynamic are pre-compiled libraries linked to the project. -Note that Nushell is [similar to compiled languages][Nushell compiled] rather than typical dynamic languages like Python, so these concepts are relevan for Nushell. + +> **Note** +> Nushell is [similar to compiled languages][Nushell compiled] rather than typical dynamic languages like Python, so these concepts are relevant for Nushell. Static dependencies: -* Advantages: reproducible, does not rely on system files (no more missing random.so.2), higher performance (allows joint optimization of dependencies and project itself) -* Disadvatage: increased compile time, binary size, can easily end up with multiple versions of the same library (hello Nushell dependencies) +* :thumbsup:: reproducible, does not rely on system files (no more missing `random.so.2`), higher performance (allows joint optimization of dependencies and project itself) +* :thumbsdown:: increased compile time, binary size, can easily end up with multiple versions of the same library (hello Nushell dependencies) -Dynamic dependencies are the opposite basically. Note that Nushell currently supports only static dependencies, but we might be able to add the "linking" feature at some point. +Dynamic dependencies are the opposite basically. -We might want `nupm`support both types of dependencies. +> **Note** +> Nushell currently supports only static dependencies, but we might be able to add the "linking" feature at some point. + +We might want `nupm` support both types of dependencies. ## Package repository @@ -118,18 +129,17 @@ Nushell's module design conflates CLI interface with API -- they are the same. N - `--yes (-y)`: do not ask for user confirmation, e.g. to use `nupm install` in scripts - `nupm add` - add a dependency to the current project - - it is different from `nupm install`: this one adds the dependency to the MANIFEST_FILE, `nupm install` does not + - it is different from `nupm install`: this one adds the dependency to the `METADATA_FILE`, `nupm install` does not - `nupm uninstall` - uninstall a package from a currently active overlay (can override which overlay to install to) - `--yes (-y)`: do not ask for user confirmation, e.g. to use `nupm uninstall` in scripts - `nupm update` - update all packages in a currently active overlay (can specify package and/or overlay name) - - can be used to self-update: `nupm update nupm`, `nupm update --self` or `nupm update --all` (would update everything) + - can be used to self-update: `nupm update nupm`, `nupm update --self` or `nupm update --all` (the last one would update every package installed by `nupm`, including `nupm` itself) - `--yes (-y)`: do not ask for user confirmation, e.g. to use `nupm update` in scripts - `nupm search` - search package repository (only supported ones by default) - `--unsupported (-u)`: would also list packages that are not supported in the user's system, e.g. due to OS incompatibilities - - `--all (-a)`: would list all packages - `nupm check` - parse the project to search for errors but do not run it @@ -141,7 +151,7 @@ Nushell's module design conflates CLI interface with API -- they are the same. N - generate documentation - `nupm publish` - publish package to a repository - - NOT SUPPORTED FOR NOW: the repository will be a *GitHub* repo with packages submitted by PRs to start with + - **NOT SUPPORTED FOR NOW**: the repository will be a *GitHub* repo with packages submitted by PRs to start with - `nupm overlay new` - create a new global overlay (Python's virtual environment style) @@ -158,7 +168,7 @@ Nushell's module design conflates CLI interface with API -- they are the same. N ### Other CLI-related points -* We could later think about being able to extend nupm, like cargo has plugins. +* We could later think about being able to extend `nupm`, like `cargo` has plugins. * Mutable actions (like install) have by default Y/n prompt, but can be overriden with `--yes` * By default, new projects are cross-platform: * Windows @@ -172,6 +182,10 @@ Nushell's module design conflates CLI interface with API -- they are the same. N * doc generation * test running * benchmark running -* configuration (do not add until we really need something to be configurable, keep it minimal, case study of a project with minimal configuration: https://github.com/psf/black) +* configuration (do not add until we really need something to be configurable, keep it minimal, case study of a project with minimal configuration: [`psf/black`]) [Nushell compiled]: https://www.nushell.sh/book/thinking_in_nu.html#think-of-nushell-as-a-compiled-language + +[`kubouch/nuun`]: https://github.com/kubouch/nuun +[`jntrnr/nu_app`]: https://github.com/jntrnr/nu_app +[`psf/black`]: https://github.com/psf/black From 937ab9343d53b0c9dc15f17fa590db6818e2f7b8 Mon Sep 17 00:00:00 2001 From: amtoine Date: Sun, 4 Jun 2023 11:26:51 +0200 Subject: [PATCH 19/23] add a table of content --- docs/design/README.md | 26 ++++++++++++++++++-------- 1 file changed, 18 insertions(+), 8 deletions(-) diff --git a/docs/design/README.md b/docs/design/README.md index e29858a..20e1cae 100644 --- a/docs/design/README.md +++ b/docs/design/README.md @@ -10,7 +10,17 @@ This file collects design ideas and directions. The intention is iterate on this > - `NUPM_HOME`: the location of all the `nupm` files, overlays, scripts, libraries, ..., > e.g. `~/.nupm/`, `$env.XDG_DATA_HOME/nupm/` or `~/.local/share/nupm/` -## Project Structure +# Table of content +- [Project Structure](#project-structure-toc) +- [Separate virtual environments](#separate-virtual-environments-toc) +- [Installation, bootstraping](#installation-bootstraping-toc) +- [Dependency handling](#dependency-handling-toc) +- [Package repository](#package-repository-toc) +- [API / CLI Interface](#api--cli-interface-toc) + - [Other CLI-related points](#other-cli-related-points-toc) +- [Other](#other-toc) + +## Project Structure [[toc](#table-of-content)] A `nupm` project is defined by `METADATA_FILE`. This is where you define name of the project, version, dependencies, etc., and the type of the project. @@ -43,7 +53,7 @@ You can also install non-Nushell packages as well using a "custom" project type (e.g., you can install Nushell itself with it). Plugins should also be supported, preferably not requiring fully custom `build.nu`. -## Separate virtual environments +## Separate virtual environments [[toc](#table-of-content)] > Inspiration: Python, [conda](https://docs.conda.io/en/latest), `cargo` @@ -63,7 +73,7 @@ The overlays could be used to achieve all three goals at the same time. When ins Each package would basically have its own overlay. This overlay file (it's just a module) could be used to also handle dependencies. If your project depends on `foo` and `bar` which both depend on `spam` but different versions, they could both import the different verions privately in their own overlay files and in your project's overlay file would be just `export use path/to/foo` and `export use path/to/bar`. This should prevent name clashing of `spam`. The only problem that needs to be figured out is how to tell `foo` to be aware of its overlay. -## Installation, bootstraping +## Installation, bootstraping [[toc](#table-of-content)] Requires these actions from the user (this should be kept as minimal as possible): * Add `NUPM_HOME/bin` to PATH (install location for binary projects) @@ -82,7 +92,7 @@ There are several approaches: * (in the future maybe) as a compiled binary (using something like [`jntrnr/nu_app`]) * This would allow us to reverse the installation steps: Instead of Nushell installing `nupm`, we could let user only install `nupm` which would in turn install Nushell -## Dependency handling +## Dependency handling [[toc](#table-of-content)] In compiled programming languages, there are two kinds of dependencies: static and dynamic. Static are included statically and compiled when compiling the project, dynamic are pre-compiled libraries linked to the project. @@ -101,14 +111,14 @@ Dynamic dependencies are the opposite basically. We might want `nupm` support both types of dependencies. -## Package repository +## Package repository [[toc](#table-of-content)] Packages need to be stored somewhere. There should be one central "official" location (see https://github.com/NixOS/nixpkgs for inspiration). Additionally, user should be able to add 3rd party repositories as well as install local and other packages (e.g., from the web, just pointing at URL), as long as it has `METADATA_FILE` telling `nupm` what to do. -## API / CLI Interface +## API / CLI Interface [[toc](#table-of-content)] Nushell's module design conflates CLI interface with API -- they are the same. Not all of the below are of the same priority. @@ -166,7 +176,7 @@ Nushell's module design conflates CLI interface with API -- they are the same. N - `nupm overlay import` - create overlay from exported file -### Other CLI-related points +### Other CLI-related points [[toc](#table-of-content)] * We could later think about being able to extend `nupm`, like `cargo` has plugins. * Mutable actions (like install) have by default Y/n prompt, but can be overriden with `--yes` @@ -176,7 +186,7 @@ Nushell's module design conflates CLI interface with API -- they are the same. N * Linux * Android (if someone is willing to maintain it, we're not testing Nushell on Android, at least for now) -## Other +## Other [[toc](#table-of-content)] * activations * doc generation From 3bbad6ac0238e5b9980a6bdc749a6b8a750806c9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jakub=20=C5=BD=C3=A1dn=C3=ADk?= Date: Sun, 4 Jun 2023 22:48:15 +0300 Subject: [PATCH 20/23] Update README.md --- docs/design/README.md | 29 ++++++++++++++--------------- 1 file changed, 14 insertions(+), 15 deletions(-) diff --git a/docs/design/README.md b/docs/design/README.md index 20e1cae..d476a57 100644 --- a/docs/design/README.md +++ b/docs/design/README.md @@ -37,7 +37,7 @@ spam ``` * meant as a runnable script, equivalent of Rust's binary project * could use the `.nush` extension if we agree to support it -* installed under `NUPM_CURRENT_OVERLAY/bin/` +* installed under `NUPM_HOME/bin/` 2. Module ``` @@ -47,18 +47,19 @@ spam └── mod.nu ``` * meant as a library to be `use`d or `overlay use`d, equivalent of Rust's library project -* installed under `NUPM_CURRENT_OVERLAY/lib/` +* installed under `NUPM_HOME/modules/` You can also install non-Nushell packages as well using a "custom" project type where you specify a `build.nu` installation script (e.g., you can install Nushell itself with it). Plugins should also be supported, preferably not requiring fully custom `build.nu`. +More "helper" types of projects could be made, e.g., installing from GitHub, etc. We could allow users to add new project types via "templates". ## Separate virtual environments [[toc](#table-of-content)] > Inspiration: Python, [conda](https://docs.conda.io/en/latest), `cargo` -There are two different concepts in how to handle virtual environments: -* Having global virtual environments, Python-style. We have a working prototype at [`kubouch/nuun`] using overlays. +There are two different concepts how to handle virtual environments: +* Global virtual environments, Python-style. We have a working prototype at [`kubouch/nuun`] using overlays. * Installing a package will install it to the environment * Possible to switch between them, they are completely isolated from each other * Per-project virtual environment, `cargo`-style @@ -70,6 +71,7 @@ The overlays could be used to achieve all three goals at the same time. When ins * `nupm` adds entry to a **lock file** (this should be the only file you need to 100% replicate the environment) * A .nu file (module) is auto-generated from the lock file and contains export statements like `export module NUPM_HOME/cache/packages/spam-v16.4.0-124ptnpbf/spam`. Calling `overlay use` on the file will activate your virtual environment, now you have a per-project environment * This file can be installed into a global location that's in your `NU_LIB_DIRS` (e.g., `NUPM_HOME/overlays`) -- now you have a global Python-like virtual environment + * Each overlay under `NUPM_HOME/overlays` will mimic the main NUPM_HOME structure, e.g., for an overlay `spam` there will be `NUPM_HOME/overlays/spam/bin`, `NUPM_HOME/overlays/spam/modules` (`NUPM_HOME/overlays/spam/overlays`? It might not be the best idea to have it recursive) Each package would basically have its own overlay. This overlay file (it's just a module) could be used to also handle dependencies. If your project depends on `foo` and `bar` which both depend on `spam` but different versions, they could both import the different verions privately in their own overlay files and in your project's overlay file would be just `export use path/to/foo` and `export use path/to/bar`. This should prevent name clashing of `spam`. The only problem that needs to be figured out is how to tell `foo` to be aware of its overlay. @@ -77,13 +79,12 @@ Each package would basically have its own overlay. This overlay file (it's just Requires these actions from the user (this should be kept as minimal as possible): * Add `NUPM_HOME/bin` to PATH (install location for binary projects) -* Add `NUPM_HOME/lib` to NU_LIB_DIRS +* Add `NUPM_HOME/modules` to NU_LIB_DIRS * Add `NUPM_HOME/overlays` to NU_LIB_DIRS * Make the `nupm` command available somehow (e.g., `use` inside `config.nu`) > :warning: **WIP** -> I have another idea in mind, need to think about it. The disadvantage of this is that the default install location is not an overlay. -> We could make `nupm` itself an overlay that adds itself as a command. +> The disadvantage of this is that the default install location is not an overlay. We could make `nupm` itself an overlay that adds itself as a command, so that you can activate/deactivate it. We might need a few attempts to get to the right solution. There are several approaches: * bootstrap using shell script sourced from web (like `rustup`) @@ -101,19 +102,16 @@ dynamic are pre-compiled libraries linked to the project. > Nushell is [similar to compiled languages][Nushell compiled] rather than typical dynamic languages like Python, so these concepts are relevant for Nushell. Static dependencies: -* :thumbsup:: reproducible, does not rely on system files (no more missing `random.so.2`), higher performance (allows joint optimization of dependencies and project itself) +* :thumbsup:: reproducible, does not rely on system files at runtime (no more missing `random.so.2`), higher performance (allows joint optimization of dependencies and project itself) * :thumbsdown:: increased compile time, binary size, can easily end up with multiple versions of the same library (hello Nushell dependencies) Dynamic dependencies are the opposite basically. -> **Note** -> Nushell currently supports only static dependencies, but we might be able to add the "linking" feature at some point. - -We might want `nupm` support both types of dependencies. +Nushell currently supports only static dependencies, but we might be able to add the "linking" feature at some point which could unlock new interesting patterns regarding package management, like testing. ## Package repository [[toc](#table-of-content)] -Packages need to be stored somewhere. There should be one central "official" location (see https://github.com/NixOS/nixpkgs for inspiration). +Packages need to be stored somewhere. There should be one central "official" location (see https://github.com/NixOS/nixpkgs for inspiration). GitHub repository seems like a good starting point. Additionally, user should be able to add 3rd party repositories as well as install local and other packages (e.g., from the web, just pointing at URL), as long as it has `METADATA_FILE` telling `nupm` what to do. @@ -163,6 +161,7 @@ Nushell's module design conflates CLI interface with API -- they are the same. N - publish package to a repository - **NOT SUPPORTED FOR NOW**: the repository will be a *GitHub* repo with packages submitted by PRs to start with +The following are for Python-style global overlays, we might need to re-think this for local package overlays: - `nupm overlay new` - create a new global overlay (Python's virtual environment style) - `--local` flag could generate an overlay locally from the currently opened project @@ -176,7 +175,7 @@ Nushell's module design conflates CLI interface with API -- they are the same. N - `nupm overlay import` - create overlay from exported file -### Other CLI-related points [[toc](#table-of-content)] +### Other CLI-related notes [[toc](#table-of-content)] * We could later think about being able to extend `nupm`, like `cargo` has plugins. * Mutable actions (like install) have by default Y/n prompt, but can be overriden with `--yes` @@ -188,7 +187,7 @@ Nushell's module design conflates CLI interface with API -- they are the same. N ## Other [[toc](#table-of-content)] -* activations +* activations (not bringing all package's content but only parts of it) * doc generation * test running * benchmark running From 9e39e3c6b694256d34bc6fc00f4a2eb4a120e7d6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jakub=20=C5=BD=C3=A1dn=C3=ADk?= Date: Sun, 4 Jun 2023 23:05:25 +0300 Subject: [PATCH 21/23] Update METADATA.md Split required attributes --- docs/design/references/METADATA.md | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/docs/design/references/METADATA.md b/docs/design/references/METADATA.md index f849c11..e5d196d 100644 --- a/docs/design/references/METADATA.md +++ b/docs/design/references/METADATA.md @@ -1,20 +1,25 @@ # `nupm`'s package metadata reference ## Required attributes -- `author`: Name of the developer/organization/etc. -- `files`: List of records of files being part of the package. Records reference: - - `checksum`: SHA256 sum of the file - - `name`: File path (relative to the package scope) - - `raw-url`: `GET`table link to the file - - `supported-os`: Exactly like in the top-level metadata - `name`: Package name - `nu-version`: Supported Nushell version. Just like `dependencies` might be either exact version or some matcher like "greater than 1.70" +- `version`: Version of the package. Semantic versioning is advised to enable users to have more generic requirements +- `type`: Tells `nupm` how to install the package + +## Required attributes for publishing +- `author`: Name of the developer/organization/etc. - `short-description`: Short info about the package, displayed by default - `supported-os`: Operating systems supported by the package, the most broad possibility: `{"arch": ["*"], "family": ["*"], "name": ["*"]}`. Matched by `$nu.os-info` - `url`: Package website/GitHub repository. Basically a place where one can find some additional info about the package -- `version`: Version of the package. Semantic versioning is advised to enable users to have more generic requirements ## Optional attribtues - `dependencies`: Packages needed by the package — versions have to be specified. e.g. `[nupm/0.7.0]`. Semantic versioning is also supported: `[nupm/~0.7]` - `installer`: Name of a script (relative to the package scope) that will install the package instead (or in addition to) of default `nupm` logic - `keywords`: List of keywords used by `nupm search` in addition to `name` + +## Automatically generated, outside of the user-created metadata file +- `files`: List of records of files being part of the package. Records reference: + - `checksum`: SHA256 sum of the file + - `name`: File path (relative to the package scope) + - `raw-url`: `GET`table link to the file + - `supported-os`: Exactly like in the top-level metadata From 4a6d001ab4f10da8b9ea9eaf93c86f86bd356963 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jakub=20=C5=BD=C3=A1dn=C3=ADk?= Date: Sun, 4 Jun 2023 23:05:58 +0300 Subject: [PATCH 22/23] Remove long description note --- docs/design/README.md | 5 ----- 1 file changed, 5 deletions(-) diff --git a/docs/design/README.md b/docs/design/README.md index d476a57..2fc1835 100644 --- a/docs/design/README.md +++ b/docs/design/README.md @@ -120,11 +120,6 @@ as long as it has `METADATA_FILE` telling `nupm` what to do. Nushell's module design conflates CLI interface with API -- they are the same. Not all of the below are of the same priority. -> **Note** -> commands like `list`, `install`, `search`, `uninstall`, `update`, ..., i.e. should -> - print short descriptions by default -> - print long descriptions with `--long-description (-l)` - - `nupm new [--script] [--module]` - create a new local package with template files ([`kubouch/nuun`]) - `nupm list` From 7c889fedd7c083980bcbf2a40491436b61341a6d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jakub=20=C5=BD=C3=A1dn=C3=ADk?= Date: Mon, 5 Jun 2023 09:23:48 +0300 Subject: [PATCH 23/23] Put back long description --- docs/design/README.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/design/README.md b/docs/design/README.md index 2fc1835..d476a57 100644 --- a/docs/design/README.md +++ b/docs/design/README.md @@ -120,6 +120,11 @@ as long as it has `METADATA_FILE` telling `nupm` what to do. Nushell's module design conflates CLI interface with API -- they are the same. Not all of the below are of the same priority. +> **Note** +> commands like `list`, `install`, `search`, `uninstall`, `update`, ..., i.e. should +> - print short descriptions by default +> - print long descriptions with `--long-description (-l)` + - `nupm new [--script] [--module]` - create a new local package with template files ([`kubouch/nuun`]) - `nupm list`