Docs 0.3.0 (#36)

* update mod docs for new options for installing from tags, branches, local * update docs for new mod dependency types * add docs for POWERPIPE_GIT_TOKEN * proofread fixes * remove --pull none from mod reference docs
turbot · May 14, 2024 · 50205a7 · 50205a7
1 parent 4610967
commit 50205a7
Show file tree

Hide file tree

Showing 6 changed files with 196 additions and 16 deletions.
diff --git a/docs/build/mod-dependencies.md b/docs/build/mod-dependencies.md
@@ -66,3 +66,62 @@ powerpipe benchmark run aws_compliance.benchmark.cis_v140
 ```
 
 When running `powerpipe server` from a mod, all dashboards in your mod and its direct dependencies will be available to run.
+
+
+## Installing Mods
+
+### Git URLs & Private Repos
+
+Powerpipe uses `git` to install and update mods. When you run `powerpipe mod install` or `powerpipe mod update`, Powerpipe will first try using HTTPS and if that does not work it will try SSH.  If your SSH keys are configured properly for `git`, you should be able to pull from private repos that you have access to, as well as public ones.  Alternatively, you can authenticate with a GitHub personal access token or application token.  Set the [POWERPIPE_GIT_TOKEN](/docs/reference/env-vars/powerpipe_git_token) environment variable to your token and Powerpipe will use the token when installing and updating mods.
+
+
+### Mod Version Constraints
+
+When installing a mod, you may specify a [semver constraint](https://semver.org/).  The latest version that meets the constraint will be installed, and the constraint will be added to the `mod.pp` and honored by subsequent `steampipe mod update` operations.
+
+When installing the mod, append the mod repo with `@` and any valid semver constraint:
+
+```bash
+powerpipe mod install github.com/turbot/steampipe-mod-aws-insights@'^1'
+powerpipe mod install github.com/turbot/steampipe-mod-aws-insights@1
+powerpipe mod install github.com/turbot/[email protected]
+powerpipe mod install github.com/turbot/steampipe-mod-aws-insights@'>=0.20'
+```
+
+### Installing from Branches and Tags
+
+To install from a tagged commit, append the mod repo with `@` and the tag:
+```bash
+powerpipe mod install github.com/turbot/steampipe-mod-aws-insights@mycustomtag'
+```
+Note that the syntax is the same as for [semver constraints](#mod-version-constraints), and if the tag value is a valid semver string, Powerpipe will interpret it as a semver constraint and not a literal tag name.
+
+To install from a branch, append the mod repo with `#` and the branch name:
+```bash
+powerpipe mod install github.com/turbot/steampipe-mod-aws-insights#main'
+```
+
+### Installing from the local filesystem
+When developing mods, it can be useful to work from a local copy.  To install a mod from a local filesystem path, just pass the path to the install command:
+
+```bash
+powerpipe  mod install ../steampipe-mod-aws-insights
+```
+
+
+### Update Strategy
+
+It is also possible to have more granular control of the update behavior - e.g. when to check for new commits. The -`-pull` argument can be used to specify the update strategy when running `powerpipe update` or `powerpipe install`:
+
+| Strategy | Description
+|----------|---------------------------------------------------
+| `full`   | Check branches and tags for both latest and accuracy
+| `latest` | Update everything to latest, but only branches (not tags) are commit checked
+| `development` | Update branches and broken constraints to latest, leave satisfied constraints unchanged
+| `minimal`| Only update broken constraints. Do not check branches for new commits
+
+
+### Publishing & Distributing mods
+When publishing public mods, you should only depend on public mods (hosted in public repos) so that users of your mod don't encounter permissions issues - Avoid dependencies on local or private mods!
+
+When users install your mod using `powerpipe mod install`, your dependencies will get installed automatically.  As a result, it is recommended that you add the `.powerpipe` directory to your `.gitignore` file and do not check these files into git.
diff --git a/docs/powerpipe-hcl/mod.md b/docs/powerpipe-hcl/mod.md
@@ -67,7 +67,7 @@ mod "aws_cis" {
 | `title` | String | Optional | The display title of the mod.
 
 
-#### opengraph
+### opengraph
 The `opengraph` block is an optional block of metadata for use in social media applications that support [Opengraph](https://ogp.me/) metadata.
 
 | Name | Type| Description
@@ -77,11 +77,11 @@ The `opengraph` block is an optional block of metadata for use in social media a
 
 
 
-#### require
+### require
 A mod may contain a `require` block to specify version dependencies for the Powerpipe CLI, plugins, and mods.  While it is possible to edit this section manually, Powerpipe will also modify it (including reordering and removing comments) when you run a `powerpipe mod` command to install, update, or uninstall a mod.
 
 
-##### powerpipe
+#### powerpipe
 A mod may specify a dependency on the Powerpipe CLI.  Powerpipe will evaluate the dependency when the mod is loaded and will error if the constraint is not met, but it will not install or upgrade the CLI.  A `powerpipe` constraint specifies a *minimum version*, and does not support semver syntax:
 ```hcl
 require {
@@ -91,7 +91,7 @@ require {
 }
 ```
 
-##### plugin
+#### plugin
 A mod may specify a dependency on one or more Steampipe plugins.  Powerpipe will evaluate the dependency when the mod is loaded and will error if the constraint is not met, but it will not install or upgrade the plugin. A `plugin` constraint specifies a *minimum version*, and does not support semver syntax:
 ```hcl
 require {
@@ -101,23 +101,61 @@ require {
 }
 ```
 
-##### mod
-A mod may specify dependencies on other mods.  While you can manually edit the `mod` dependencies in the `mod.pp`, they are more commonly managed by Powerpipe when you install, update, or uninstall mods via the [powerpipe mod commands](/docs/reference/cli/mod).  The `version` can be an exact version<!-- ,a tag name, a branch name, a local file --> or a [semver](https://semver.org/) string:
+#### mod
+A mod may specify dependencies on other mods.  While you can manually edit the `mod` dependencies in the `mod.pp`, they are more commonly managed by Powerpipe when you install, update, or uninstall mods via the [powerpipe mod commands](/docs/reference/cli/mod).
+
+The mod dependency may specify a `version`.  This can be a [semver](https://semver.org/) or an exact version:
 
 ```hcl
 require {
+  mod "github.com/turbot/steampipe-mod-aws-insights" {
+    version = "0.20.0"
+  }
   mod "github.com/turbot/steampipe-mod-aws-compliance" {
     version = "^0.10"
   }
-  mod "github.com/turbot/steampipe-mod-aws-insights" {
-    version = "2.0"
-  }
   mod "github.com/turbot/steampipe-mod-gcp-compliance" {
     version = "*"
   }
 }
 ```
 
+While in development, it is often easier to use local dependencies or install a mod from a branch or tagged commit.
+
+To install from a branch, specify a `branch`:
+
+```hcl
+require {
+  mod "github.com/turbot/steampipe-mod-aws-compliance" {
+    branch = "issue-779"
+  }
+}
+```
+
+To install from a tagged commit, specify a `tag`:
+
+```hcl
+require {
+  mod "github.com/turbot/steampipe-mod-aws-compliance" {
+    tag = "issue-779"
+  }
+}
+```
+
+
+To install from a local directory, specify a `path` instead of a `version`:
+
+```hcl
+require {
+  mod "/Users/jsmyth/src/steampipe-mod-aws-insights" {
+    path = "/Users/jsmyth/src/steampipe-mod-aws-insights"
+  }
+}
+```
+
+Note that for a given dependency, you must specify *exactly one* constraint (`version`, `tag`, `branch` or `path`).
+
+
 You may pass `args` to set variables defined in the dependency mods.  You can pass hard-coded values (literals), but it is more common to define variables in your mod that encapsulate the variables  and optionality of your dependencies:
 
 

diff --git a/docs/reference/cli/mod.md b/docs/reference/cli/mod.md
@@ -56,42 +56,93 @@ powerpipe mod init --mod-location ~/my_mod
 ---
 
 ## powerpipe mod install
-Install one or more mods and their dependencies.
+Install one or more mods and their dependencies.  In addition to downloading the mod, Powerpipe will [add the mod dependency to the `mod.pp` file](/docs/powerpipe-hcl/mod#mod-1).
+
+Powerpipe uses `git` to install and update mods. When you run `powerpipe mod install` or `powerpipe mod update`, Powerpipe will first try using HTTPS and if that does not work it will try SSH.  If your SSH keys are configured properly for `git`, you should be able to pull from private repos that you have access to, as well as public ones.  Alternatively, you can authenticate with a GitHUb personal access token or application token.  Set the [POWERPIPE_GIT_TOKEN](/docs/reference/env-vars/powerpipe_git_token) environment variable to your token and Powerpipe will use the token when installing and updating mods.
+
+When you install a mod, the latest version is installed by default:
+```bash
+powerpipe mod install github.com/turbot/steampipe-mod-aws-insights
+```
+
+When installing a mod, you may specify a semver constraint.  The latest version that meets the constraint will be installed, and the constraint will be added to the `mod.pp` and honored by subsequent `steampipe mod update` operations.
+```bash
+powerpipe mod install github.com/turbot/[email protected]
+```
+
+To install from a tagged commit, append the mod repo with `@` and the tag:
+```bash
+powerpipe mod install github.com/turbot/steampipe-mod-aws-insights@mycustomtag
+```
+Note that the syntax is the same as for semver constraints, and if the tag value is a valid semver string, Powerpipe will interpret it as a semver constraint and not a literal tag name.
+
+To install from a branch, append the mod repo with `#` and the branch name:
+```bash
+powerpipe mod install github.com/turbot/steampipe-mod-aws-insights#main
+```
+
+When developing mods, it can be useful to work from a local copy.  To install a mod from a local filesystem path, just pass the path to the install command:
+
+```bash
+powerpipe  mod install ../steampipe-mod-aws-insights
+```
 
 ### Arguments
 | Flag | Description
 |-|-
 | `--dry-run` | Show which mods would be installed/updated/uninstalled without modifying them (default `false`).
 | `--force` | Install mods even if plugin/cli version requirements are not met (cannot be used with `--dry-run`).
 | `--prune` | Remove unused dependencies after installation is complete (default `true`).
+| `--pull string` | Specify an [update strategy](#update-strategy): `full`, `latest`, `development`, `minimal` (default `minimal`)
 
 
+#### Update Strategy
 
-### Git URLs & Private Repos
+It is also possible to have more granular control of the update behavior - e.g. when to check for new commits. The -`-pull` argument can be used to specify the update strategy when running `powerpipe update` or `powerpipe install`:
 
-Powerpipe uses `git` to install and update mods. When you run `powerpipe mod install` or `powerpipe mod update`, Powerpipe will first try using HTTPS and if that does not work it will try SSH.  If your SSH keys are configured properly for `git`, you should be able to pull from private repos that you have access to, as well as public ones.  
+| Strategy | Description
+|----------|---------------------------------------------------
+| `full`   | Check branches and tags for both latest and accuracy
+| `latest` | Update everything to latest, but only branches (not tags) are commit checked
+| `development` | Update branches and broken constraints to latest, leave satisfied constraints unchanged
+| `minimal`| Only update broken constraints. Do not check branches for new commits
 
 
 ### Examples
 
-Install the latest version of a mod and add the `require` statement to your `mod.pp`:
+Install the latest version of a mod:
 ```bash
 powerpipe mod install github.com/turbot/steampipe-mod-aws-insights
 ```
 
-Install an exact version of a mod and update the `require` statement to your `mod.pp`.  This may upgrade or downgrade the mod if it is already installed:
+Install an exact version of a mod:
 ```bash
 powerpipe mod install github.com/turbot/[email protected]
 ```
 
-Install a version of a mod using a semver constraint and update the `require` statement to your `mod.pp`.  This may upgrade or downgrade the mod if it is already installed:
+Install a version of a mod using a semver constraint:
 ```bash
 powerpipe mod install github.com/turbot/steampipe-mod-aws-insights@'^1'
 powerpipe mod install github.com/turbot/steampipe-mod-aws-insights@1
 powerpipe mod install github.com/turbot/[email protected]
 powerpipe mod install github.com/turbot/steampipe-mod-aws-insights@'>=0.20'
 ```
 
+Install a mod from a GitHub tag:
+```bash
+powerpipe mod install github.com/turbot/steampipe-mod-aws-insights@my-tag
+```
+
+Install a mod from a GitHub branch:
+```bash
+powerpipe mod install github.com/turbot/steampipe-mod-aws-insights#main
+```
+
+Install a mod from a local directory:
+```bash
+powerpipe  mod install ../steampipe-mod-aws-insights
+```
+
 Install all mods specified in the `mod.pp` and their dependencies:
 ```bash
 powerpipe mod install
@@ -102,8 +153,14 @@ Preview what `powerpipe mod install` will do, without actually installing anythi
 powerpipe mod install --dry-run
 ```
 
+Install all missing mods specified in the `mod.pp` and update all their dependencies:
+```bash
+powerpipe mod install --pull full
+```
+
 ---
 
+
 ## powerpipe mod list
 List mods from the current mod and its direct dependents.
 
@@ -185,6 +242,7 @@ Update one or more mods and their dependencies.
 |` --dry-run` | Show which mods would be updated without modifying them (default `false`).
 |` --force` | Update mods even if plugin/cli version requirements are not met (cannot be used with `--dry-run`).
 |` --prune` | Remove unused dependencies after update is complete (default `true`).
+| `--pull string` | Specify an [update strategy](#update-strategy): `full`, `latest`, `development`, `minimal` (default `latest`)
 
 
 ### Examples
@@ -200,5 +258,10 @@ Update all mods specified in the `mod.pp` and their dependencies to the latest v
 powerpipe mod update
 ```
 
-----
 
+Update all mods specified in the `mod.pp` and their dependencies to the latest versions that meet their constraints, and install any that are missing.  Check branches and tags for both latest and accuracy:
+```bash
+powerpipe mod update --pull full
+```
+
+---
diff --git a/docs/reference/env-vars/index.md b/docs/reference/env-vars/index.md
@@ -18,6 +18,7 @@ Powerpipe supports environment variables to allow you to change its default beha
 | [POWERPIPE_CONFIG_PATH](reference/env-vars/powerpipe_config_path)  | `.:$POWERPIPE_INSTALL_DIR/config` | Sets the search path for [configuration files](/docs/reference/config-files).  `POWERPIPE_CONFIG_PATH` accepts a colon-separated list of directories.  
 | [POWERPIPE_DASHBOARD_TIMEOUT](/docs/reference/env-vars/powerpipe_dashboard_timeout) | `0` (no timeout) | Set the dashboard execution timeout, in seconds.
 | [POWERPIPE_DATABASE](reference/env-vars/powerpipe_database)  | `postgres://steampipe@` <br /> `127.0.0.1:9193/steampipe` | A database connection string or [Turbot Pipes workspace](https://pipes.turbot.com) to use as the default database.  The default is a local [Steampipe](https://steampipe.io) instance.
+| [POWERPIPE_GIT_TOKEN](reference/env-vars/powerpipe_git_token)  | none | Set a GitHub personal access token or application token to use when installing mods from Github.
 | [POWERPIPE_INSTALL_DIR](reference/env-vars/powerpipe_install_dir)  | `~/.powerpipe` | Set the installation directory for powerpipe. Internal powerpipe files will be written to this path.
 | [POWERPIPE_LISTEN](reference/env-vars/powerpipe_listen)  | `network` | Specifies the IP addresses on which `powerpipe server` will listen for connections from clients. Currently supported values are `local` (localhost only) or `network` (all IP addresses).
 | [POWERPIPE_LOG_LEVEL](reference/env-vars/powerpipe_log_level)  | off | Set the logging output level.

diff --git a/docs/reference/env-vars/powerpipe_git_token.md b/docs/reference/env-vars/powerpipe_git_token.md
@@ -0,0 +1,18 @@
+---
+title: POWERPIPE_GIT_TOKEN
+sidebar_label: POWERPIPE_GIT_TOKEN
+---
+
+
+# POWERPIPE_GIT_TOKEN
+
+Set a GitHub personal access token or application token to use when installing mods from Github.
+
+Powerpipe uses `git` to install and update mods. When you run `powerpipe mod install` or `powerpipe mod update`, Powerpipe will first try using HTTPS and if that does not work it will try SSH.  If your SSH keys are configured properly for `git`, you should be able to pull from private repos that you have access to, as well as public ones.  Alternatively, you can authenticate with a GitHub personal access token or application token.  Set the `POWERPIPE_GIT_TOKEN` environment variable to your token and Powerpipe will use the token when installing and updating mods.
+
+
+## Usage 
+Set your API token:
+```bash
+export POWERPIPE_GIT_TOKEN=ghp_1t2ieaXluThisIsAFakeTokenG2ei14Q1PBJ
+```
diff --git a/docs/sidebar.json b/docs/sidebar.json
@@ -134,6 +134,7 @@
           "reference/env-vars/powerpipe_config_path",
           "reference/env-vars/powerpipe_dashboard_timeout",
           "reference/env-vars/powerpipe_database",
+          "reference/env-vars/powerpipe_git_token",
           "reference/env-vars/powerpipe_install_dir",
           "reference/env-vars/powerpipe_listen",
           "reference/env-vars/powerpipe_log_level",