Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Shorten target params #36

Merged
merged 2 commits into from
Sep 24, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
179 changes: 131 additions & 48 deletions projects/build-env/README.md
Original file line number Diff line number Diff line change
@@ -1,53 +1,136 @@
# @push-based/build-env

## To Research

- Store Verdaccio state under .nx?
- Project usage across existing Nx projects (popular open-source libraries?)
- Research if we can use Nx release Node.js API instead of custom target?
- How to ensure the setup will work in Nx Agents?

**What can be cached**

- `setup-env`
- npm workspace folder (.npmrc, package-lock.json, node_modules)
- `npm-publish`
- inputs
- build output, using dependent task output? careful though, [it is known to struggle with Nx cloud agents](https://github.com/nrwl/nx/issues/22745)
- outputs
- list of packages in registry under `.../storage/.verdaccio-db.json` e.g.: `{"list":["<package-name>"],"secret":"esKM34zA53wetObgi5f0Uu1e7iObmm+f"}``
- tarball of package under `.../storage/@push-based/<package-name>-<version>.tgz`
e.g.: `{workspaceRoot}/${environmentsDir}/{args.environmentProject}/storage/@push-based/${packageName}`,
- `package.json` of package under `.../storage/@push-based/<package-name>/package.json`
- `npm-install`
- outputs
- list of installed packages under `.../package.json`
- list of installed packages under `.../package-lock.json`
- source of package under `.../node_modules/@push-based/<package-name>/*`
mehh, we know the node_modules reputation
- inputs:
- output of npm-publish(?), package.json
- concerns: chicken-egg problem.
- when only caching `package-lock.json`, we still need to run `<packageManager> install`, separate install in 2 steps?
- when caching node_modules, obviously cache takes lot of room!
- to install dependencies and generate lock file, we need the Verdaccio server running

**Caching scenarios**

1. what can be cached?
2. is more small caches better
3. how to visualize scenarios

- what is a worse/best case for e2e tests

### Generators

#### Configuration

Adds a `build-env` target to your `project.json`.
See [configuration docs](./src/generators/configuration/README.md) for details
## Plugins

### Build Environment Plugin

Add dynamic targets to execute environment tasks.

See [build-environment plugin docs](./src/plugin/README.md) for details

Examples:

- `nx g @push-based/build-env:configuration --project=<project-name>`
- `nx g @push-based/build-env:configuration --project=<project-name> --targetName=cp`
- `nx g @push-based/build-env-env-setup` - generates NPM workspace and installs packages
- `nx g @push-based/build-env-env-setup --keepServerRunning` - keeps Verdaccio running for debug reasons

## Executor

### Setup Environment Executor

This executor helps to initiate an [environment folder](../../../../../README.md#-environment-folders-to-isolate-files-during-e2e-tests) and installs it`s dependent projects.

// project.json

```jsonc
{
"name": "my-project",
"targets": {
"build-env--env-bootstrap": {
"executor": "@code-pushup/build-env:env-bootstrap",
"options": {
"keepServerRunning": false
"envRoot": "/tmp/test-npm-workspace"
"verbose": true,
}
}
}
}
```

Read more under [setup executor docs](./projects/build-env/src/executors/setup/README.md).

### Bootstrap Environment Executor

This executor helps to initiate [environment](../../../../../README.md#-environment-folders-to-isolate-files-during-e2e-tests) into a given folder.

// project.json

```jsonc
{
"name": "my-project",
"targets": {
"build-env--env-bootstrap": {
"executor": "@code-pushup/build-env:env-bootstrap",
"options": {
"keepServerRunning": false
"envRoot": "/tmp/test-npm-workspace"
"verbose": true,
}
}
}
}
```

Read more under [bootstrap executor docs](./projects/build-env/src/executors/bootstrap/README.md).

### Kill Process Executor

This executor helps to kill processes by `ProcessID` or a JSON file containing a property `pid` as number.

// project.json

```jsonc
{
"name": "my-project",
"targets": {
"build-env--kill-process": {
"executor": "@push-based/build-env:kill-process"
"options": {
"pid": "42312"
"filePath": "/tmp/test-npm-workspace/process-id.json"
"verbose": true,
}
}
}
}
```

Read more under [kill-process executor docs](./projects/build-env/src/executors/kill-process/README.md).

### NPM Install Executor

This executor helps to install a [`pubishable`](../../../../../README.md#fine-grained-selection-of-publishable-projects) projects into a given [environment folder](../../../../../README.md#-environment-folders-to-isolate-files-during-e2e-tests).

// project.json

```jsonc
{
"name": "my-project",
"targets": {
"build-env--npm-install": {
"executor": "@code-pushup/build-env:release-install",
"options": {
"pkgVersion": "1.2.3"
"envRoot": "/tmp/test-npm-workspace"
"verbose": true,
}
}
}
}
```

Read more under [release install executor docs](./projects/build-env/src/executors/npm-install/README.md).

### NPM Publish Executor

This executor helps to publish a [`pubishable`](../../../../../README.md#fine-grained-selection-of-publishable-projects) projects into a given [environment folder](../../../../../README.md#-environment-folders-to-isolate-files-during-e2e-tests).

// project.json

```jsonc
{
"name": "my-project",
"targets": {
"build-env--npm-publish": {
"executor": "@code-pushup/build-env:release-publish",
"options": {
"pkgVersion": "1.2.3"
"envRoot": "/tmp/test-npm-workspace"
"verbose": true,
}
}
}
}
```

Read more under [release publish executor docs](./projects/build-env/src/executors/npm-publish/README.md).
6 changes: 1 addition & 5 deletions projects/build-env/src/executors/bootstrap/bootstrap-env.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,7 @@ import {
VERDACCIO_ENV_TOKEN,
} from './npm';

export type BootstrapEnvironmentOptions = StartVerdaccioOptions &
Environment & {
projectName: string;
environmentRoot: string;
};
export type BootstrapEnvironmentOptions = StartVerdaccioOptions & Environment;

export type BootstrapEnvironmentResult = Environment & {
registry: VercaddioServerResult;
Expand Down
1 change: 0 additions & 1 deletion projects/build-env/src/executors/bootstrap/executor.ts
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,6 @@ export async function bootstrapExecutor(
bootstrapResult = await bootstrapEnvironment({
projectName,
environmentRoot,
keepServerRunning,
});
} catch (error) {
logger.error(error);
Expand Down
6 changes: 4 additions & 2 deletions projects/build-env/src/executors/bootstrap/schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,14 @@
"properties": {
"environmentRoot": {
"type": "string",
"description": "The root directory of the environment"
"description": "The root directory of the environment",
"aliases": ["envRoot", "e"]
},
"keepServerRunning": {
"type": "boolean",
"description": "Keep the server running after the bootstrap commands have been executed",
"default": true
"default": true,
"aliases": ["k"]
},
"printConfig": {
"type": "boolean",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ export function parseRegistryData(stdout: string): VerdaccioProcessResult {
}

export type StarVerdaccioOnlyOptions = {
projectName?: string;
projectName: string;
verbose?: boolean;
};

Expand Down
3 changes: 2 additions & 1 deletion projects/build-env/src/executors/kill-process/schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,8 @@
},
"environmentRoot": {
"type": "string",
"description": "The root directory for the environment"
"description": "The root directory for the environment",
"aliases": ["envRoot", "e"]
},
"filePath": {
"type": "string",
Expand Down
7 changes: 4 additions & 3 deletions projects/build-env/src/executors/npm-install/schema.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"$schema": "http://json-schema.org/schema",
"$id": "KillProcessExecutorOptions",
"title": "A executor to kill processes by PID, command, or file",
"$id": "InstallExecutorOptions",
"title": "A executor to install a package",
"type": "object",
"properties": {
"printConfig": {
Expand All @@ -15,7 +15,8 @@
},
"environmentRoot": {
"type": "string",
"description": "The root directory for the environment"
"description": "The root directory for the environment",
"aliases": ["envRoot", "e"]
},
"filePath": {
"type": "string",
Expand Down
1 change: 0 additions & 1 deletion projects/build-env/src/executors/npm-install/schema.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
export type NpmInstallExecutorOptions = Partial<{
pkgVersion: string;
environmentProject: string;
verbose: boolean;
printConfig: boolean;
environmentRoot: string;
Expand Down
2 changes: 1 addition & 1 deletion projects/build-env/src/executors/npm-publish/README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# NPM Install Executor
# NPM Publish Executor

This executor helps to publish a [`pubishable`](../../../../../README.md#fine-grained-selection-of-publishable-projects) projects into a given [environment folder](../../../../../README.md#-environment-folders-to-isolate-files-during-e2e-tests).
This folder has to contain all needed configuration and files for the `npm publish` command to work.
Expand Down
7 changes: 4 additions & 3 deletions projects/build-env/src/executors/npm-publish/schema.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"$schema": "http://json-schema.org/schema",
"$id": "KillProcessExecutorOptions",
"title": "A executor to kill processes by PID, command, or file",
"$id": "PublishExecutorOptions",
"title": "A executor to publish NPM packages",
"type": "object",
"properties": {
"printConfig": {
Expand All @@ -11,7 +11,8 @@
},
"environmentRoot": {
"type": "string",
"description": "The root folder of the environment"
"description": "The root folder of the environment",
"aliases": ["envRoot", "e"]
},
"verbose": {
"type": "boolean",
Expand Down
6 changes: 4 additions & 2 deletions projects/build-env/src/executors/setup/schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,14 @@
"properties": {
"environmentRoot": {
"type": "string",
"description": "The root directory of the environment"
"description": "The root directory of the environment",
"aliases": ["envRoot", "e"]
},
"keepServerRunning": {
"type": "boolean",
"description": "Keep the server running after the bootstrap commands have been executed",
"default": true
"default": true,
"aliases": ["k"]
},
"dryRun": {
"type": "boolean",
Expand Down
42 changes: 42 additions & 0 deletions projects/build-env/src/plugin/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# BuildEnv Plugin

This plugin helps to add dynamic targets to execute environment tasks.
This distinguishes between projects that maintain publishable packages and e2e test projects that depend on an environment where the publishable projects get installed.

#### @push-based/build-env

## Usage

// `nx.json`:

```jsonc
{
"plugins": [
{
"plugin": "@push-based/build-env",
"options": {
"environments": {
"environmentsDir": "tmp/environments" // Optional
"targetNames": ["e2e"] // Optional
}
}
}
]
}
```

Now run your e2e test with `nx run utils-e2e:e2e`

## Options

| Name | type | description |
| -------------------------------- | ------------------------------------- | -------------------------------------------------------------------------------------------- |
| **environments.environmentsDir** | `string` (DEFAULT 'tmp/environments') | The folder name of the generated environments |
| **environments.targetNames** | `string[]` (REQUIRED) | The target names of projects depending on environments |
| **environments.filterByTag** | `string[]` (REQUIRED) | The tag names a projects needs to have to be considered for a environments (match is one of) |
| **publishable.filterByTag** | `string[]` (REQUIRED) | The tag names a projects needs to have to be considered for publishing (match is one of) |

**Example usage:**

- `nx run utils-e2e:e2e` - setup environment and then run E2E tests for `utils-e2e`
- `nx run utils-static-e2e:e2e --environmentRoot static-environments/user-lists` - setup NPM in existing environment and then run E2E tests for `utils-static-e2e`
15 changes: 8 additions & 7 deletions projects/build-env/src/plugin/build-env.plugin.ts
Original file line number Diff line number Diff line change
Expand Up @@ -180,9 +180,10 @@ function verdaccioTargets(
> &
StartVerdaccioOptions
): Record<string, TargetConfiguration> {
const { name: environmentProject } = projectConfig;
const { environmentsDir, ...verdaccioOptions } = options;
const environmentDir = join(environmentsDir, environmentProject);
const { name: envProject } = projectConfig;
const { environmentsDir, ...startVerdaccioOptions } = options;
const environmentDir = join(environmentsDir, envProject);

return {
[DEFAULT_START_VERDACCIO_TARGET]: {
// @TODO: consider using the executor function directly to reduce the number of targets
Expand All @@ -192,14 +193,14 @@ function verdaccioTargets(
port: uniquePort(),
storage: join(environmentDir, 'storage'),
clear: true,
...verdaccioOptions,
...startVerdaccioOptions,
},
},
[DEFAULT_STOP_VERDACCIO_TARGET]: {
executor: '@push-based/build-env:kill-process',
options: {
filePath: join(environmentsDir, VERDACCIO_REGISTRY_JSON),
...verdaccioOptions,
...startVerdaccioOptions,
},
},
};
Expand All @@ -209,9 +210,9 @@ function getEnvTargets(
projectConfig: ProjectConfiguration,
options: NormalizedCreateNodeOptions['environments']
): Record<string, TargetConfiguration> {
const { name: environmentProject } = projectConfig;
const { name: envProject } = projectConfig;
const { environmentsDir } = options;
const environmentRoot = join(environmentsDir, environmentProject);
const environmentRoot = join(environmentsDir, envProject);
return {
[DEFAULT_BOOTSTRAP_TARGET]: {
executor: '@push-based/build-env:bootstrap',
Expand Down