From 6f3d9ca39ab61b1d5c87fcd7a29d2dd4c3d1eb5c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Barroso?= Date: Thu, 24 Oct 2024 16:00:26 +0200 Subject: [PATCH 01/21] starting --- docs/design-documents/20241024-conduit-cli.md | 86 +++++++++++++++++++ 1 file changed, 86 insertions(+) create mode 100644 docs/design-documents/20241024-conduit-cli.md diff --git a/docs/design-documents/20241024-conduit-cli.md b/docs/design-documents/20241024-conduit-cli.md new file mode 100644 index 000000000..d95e65248 --- /dev/null +++ b/docs/design-documents/20241024-conduit-cli.md @@ -0,0 +1,86 @@ +# Conduit CLI + +## Introduction + +The goal of this design document is to clarify which part of this [Conduit discussion](https://github.com/ConduitIO/conduit/discussions/1642) will be part of the upcoming [0.13 release](https://github.com/ConduitIO/conduit/milestone/15). The intention is to provide a good starting experience and evolve it over time with more complex and advanced features. + +### Goals + +- We aim to provide our users with an interface that empowers them to use the **main** Conduit features effortlessly. While we have envisioned some complex features, they will be available in future releases. +- We will maintain the current installation methods by incorporating the CLI through the same established processes. +- The CLI needs to feel intuitive and align with our standards. + +## Background + +Here's how the existing Conduit CLI looks like: + +```shell +$ conduit --help +Usage of conduit: + -api.enabled + enable HTTP and gRPC API (default true) + -config string + global config file (default "conduit.yaml") + -connectors.path string + path to standalone connectors' directory (default "./connectors") + -db.badger.path string + path to badger DB (default "conduit.db") + -db.postgres.connection-string string + postgres connection string + -db.postgres.table string + postgres table in which to store data (will be created if it does not exist) (default "conduit_kv_store") + -db.type string + database type; accepts badger,postgres,inmemory (default "badger") + -grpc.address string + address for serving the gRPC API (default ":8084") + -http.address string + address for serving the HTTP API (default ":8080") + -log.format string + sets the format of the logging; accepts json, cli (default "cli") + -log.level string + sets logging level; accepts debug, info, warn, error, trace (default "info") + -pipelines.exit-on-error + exit Conduit if a pipeline experiences an error while running + -pipelines.path string + path to the directory that has the yaml pipeline configuration files, or a single pipeline configuration file (default "./pipelines") + -processors.path string + path to standalone processors' directory (default "./processors") + -version + prints current Conduit version +``` + +## Implementation plan + +### `conduit init` + +- This command will initialize a conduit pipeline that demonstrates some basic features of Conduit. +- It won't require flags or arguments. +- Additional flags could be provided to specify a source and a destination +- Optional argument will be for + +#### Arguments + +|Argument name | Description | Required | +|---|---|---| +| Name | Pipeline name | No | + +#### Flags + +|Flag name | Description | Required | +|---|---|---| +| source | Connector name | No | +| destination | | No | + + +#### `--help` + +```bash +$ conduit init [NAME] [--source plugin@version] [--destination plugin@version] + +EXAMPLES + $ conduit init + $ conduit init my-first-pipeline +``` + + +### Global flags From c81f5d5974be9fea434fe4f2fa8c0ecbab9eb14f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Barroso?= Date: Fri, 25 Oct 2024 12:21:15 +0200 Subject: [PATCH 02/21] update conduit --- docs/design-documents/20241024-conduit-cli.md | 23 ++++++------------- 1 file changed, 7 insertions(+), 16 deletions(-) diff --git a/docs/design-documents/20241024-conduit-cli.md b/docs/design-documents/20241024-conduit-cli.md index d95e65248..10ba999c3 100644 --- a/docs/design-documents/20241024-conduit-cli.md +++ b/docs/design-documents/20241024-conduit-cli.md @@ -53,34 +53,25 @@ Usage of conduit: ### `conduit init` -- This command will initialize a conduit pipeline that demonstrates some basic features of Conduit. +- This command will initialize a Conduit working environment. - It won't require flags or arguments. -- Additional flags could be provided to specify a source and a destination -- Optional argument will be for - -#### Arguments - -|Argument name | Description | Required | -|---|---|---| -| Name | Pipeline name | No | +- Additional flags could be provided to specify the path. #### Flags -|Flag name | Description | Required | -|---|---|---| -| source | Connector name | No | -| destination | | No | +|Flag name | Description | Required | Default | +|---|---|---|---| +| path | Connector name | No | `.` (current directory) | #### `--help` ```bash -$ conduit init [NAME] [--source plugin@version] [--destination plugin@version] +$ conduit init [--path PATH] EXAMPLES $ conduit init - $ conduit init my-first-pipeline + $ conduit init --path ~/code/conduit-dir ``` - ### Global flags From 5da3333bd4acd3759deb9df825bd4ef0728084d1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Barroso?= Date: Fri, 25 Oct 2024 13:26:54 +0200 Subject: [PATCH 03/21] updates --- docs/design-documents/20241024-conduit-cli.md | 315 +++++++++++++++++- 1 file changed, 310 insertions(+), 5 deletions(-) diff --git a/docs/design-documents/20241024-conduit-cli.md b/docs/design-documents/20241024-conduit-cli.md index 10ba999c3..c2142f9b6 100644 --- a/docs/design-documents/20241024-conduit-cli.md +++ b/docs/design-documents/20241024-conduit-cli.md @@ -49,11 +49,17 @@ Usage of conduit: prints current Conduit version ``` -## Implementation plan +## Suggested commands for Conduit 0.13 -### `conduit init` +The following list contains the suggested commands we propose to include in the first iteration. New and more complex commands will be added later. + +
+conduit init + +#### Description - This command will initialize a Conduit working environment. +- It does not require having conduit running. - It won't require flags or arguments. - Additional flags could be provided to specify the path. @@ -61,8 +67,7 @@ Usage of conduit: |Flag name | Description | Required | Default | |---|---|---|---| -| path | Connector name | No | `.` (current directory) | - +| path | Where to initialize Conduit | No | `.` (current directory) | #### `--help` @@ -74,4 +79,304 @@ EXAMPLES $ conduit init --path ~/code/conduit-dir ``` -### Global flags +
+ +
+ +
+conduit config + +#### Description + +- This command will output Conduit configuration (what db is configured, etc) +- It does not require having conduit running. + + +#### `--help` + +```bash +$ conduit config +``` + +
+ +
+ +
+conduit start [--pipelines.path] [...] + +#### Description + +- This command will start Conduit with all the available pipelines. +- It is equivalent to the current `conduit`. + +#### Flags + +| Name | Description | Required | Default Value | +|------|-------------|----------|---------------| +| api.enabled | enable HTTP and gRPC API | No | true | +| config | global config file | No | "conduit.yaml" | +| connectors.path | path to standalone connectors' directory | No | "./connectors" | +| db.badger.path | path to badger DB | No | "conduit.db" | +| db.postgres.connection-string | postgres connection string | Yes | | +| db.postgres.table | postgres table in which to store data | No | "conduit_kv_store" | +| db.type | database type; accepts badger,postgres,inmemory | No | "badger" | +| grpc.address | address for serving the gRPC API | No | ":8084" | +| http.address | address for serving the HTTP API | No | ":8080" | +| log.format | sets the format of the logging; accepts json, cli | No | "cli" | +| log.level | sets logging level; accepts debug, info, warn, error, trace | No | "info" | +| pipelines.exit-on-error | exit Conduit if a pipeline experiences an error while running | No | | +| pipelines.path | path to the directory that has the yaml pipeline configuration files, or a single pipeline configuration file | No | "./pipelines" | +| processors.path | path to standalone processors' directory | No | "./processors" | +| version | prints current Conduit version | No | | + + +#### `--help` + +```bash +$ conduit start +``` +
+ +
+ +
+conduit pipelines init [--pipelines.path] [...] + +#### Description + +- This command will initialize a pipeline. +- It does not require having conduit running. + +#### Flags + +| Name | Description | Required | Default Value | +|------|-------------|----------|---------------| +| path | Where to initialize a new pipeline | No | `.` (current directory) | + +#### `--help` + +```bash +$ conduit pipelines init +$ conduit pipelines init my-first-pipeline +``` +
+ +
+ +
+conduit pipelines ls + +#### Description + +- This command will list the running pipelines. +- It requires having conduit previously running. + +#### `--help` + +```bash +$ conduit pipelines ls +``` +
+ +
+ +
+conduit pipelines describe [NAME] + +#### Description + +- This command will describe the topology of the pipeline. +- It requires having conduit previously running. +- It requires the pipeline name as argument. + +#### Arguments + +| Name | Description | Required | Default Value | +|------|-------------|----------|---------------| +| name | pipeline name to describe | Yes | | + + +#### `--help` + +```bash +$ conduit pipelines describe [NAME] + +EXAMPLE: + +$ conduit pipelines describe my-pipeline +Source: kafka + Processor: avro.encode +Destination: kafka +``` +
+ +
+ +
+conduit connectors ls + +#### Description + +- This command will list all the available connectors. +- It requires having conduit previously running. + +#### `--help` + +```bash +$ conduit connectors ls +``` +
+ +
+ +
+conduit connectors describe [PLUGIN] + +#### Description + +- This command will describe the connector configuration available. +- It requires having conduit previously running. + +#### Arguments + +| Name | Description | Required | Default Value | +|------|-------------|----------|---------------| +| plugin | plugin name and version | Yes | | + + +#### `--help` + +```bash +$ conduit connnectors describe [PLUGIN] + +EXAMPLE: + +$ conduit connectors describe conduit-connector-http@0.1.0 +NAME DESCRIPTION REQUIRED DEFAULT VALUE EXAMPLE +url HTTP URL to send requests to. true https://... +... +``` +
+ +
+ +
+conduit processors ls + +#### Description + +- This command will list all the available processors. +- It requires having conduit previously running. + +#### `--help` + +```bash +$ conduit processors ls +``` +
+
+ +
+conduit processor describe [NAME] + +#### Description + +- This command will describe the processor configuration available. +- It requires having conduit previously running. +- It requires the processor name as argument. + +#### Arguments + +| Name | Description | Required | Default Value | +|------|-------------|----------|---------------| +| name | processor name | Yes | | + + +#### `--help` + +```bash +$ conduit processor describe [NAME] + +EXAMPLE: + +$ conduit connectors describe avro.decode +# auth.basic.password +type: string +description: This option is required if auth.basic.username contains a value. If both auth.basic.username and auth.basic.password are empty basic authentication is disabled. + +# auth.basic.username +type: string +... +``` +
+ +
+ +
+conduit doctor + +#### Description + +- This will check whether there’s a more up to date version of conduit and if some connectors / processors could also be updated. +- It requires having conduit previously running. + +#### `--help` + +```bash +$ conduit doctor +# returns version, checks if there's a newer version of conduit, plugin versions, etc. +``` +
+ +
+ +
+conduit version + +#### Description + +- This will return the existing Conduit version. +- It requires having conduit previously running. + +#### `--help` + +```bash +$ conduit version +``` +
+ +
+ +#### Global flags + +| Name | Description | Required | Default Value | +|------|-------------|----------|---------------| +| api.enabled | enable HTTP and gRPC API | No | true | +| config | global config file | No | "conduit.yaml" | +| connectors.path | path to standalone connectors' directory | No | "./connectors" | +| db.badger.path | path to badger DB | No | "conduit.db" | +| db.postgres.connection-string | postgres connection string | Yes | | +| db.postgres.table | postgres table in which to store data | No | "conduit_kv_store" | +| db.type | database type; accepts badger,postgres,inmemory | No | "badger" | +| grpc.address | address for serving the gRPC API | No | ":8084" | +| http.address | address for serving the HTTP API | No | ":8080" | +| log.format | sets the format of the logging; accepts json, cli | No | "cli" | +| log.level | sets logging level; accepts debug, info, warn, error, trace | No | "info" | +| pipelines.exit-on-error | exit Conduit if a pipeline experiences an error while running | No | | +| pipelines.path | path to the directory that has the yaml pipeline configuration files, or a single pipeline configuration file | No | "./pipelines" | +| processors.path | path to standalone processors' directory | No | "./processors" | +| version | prints current Conduit version | No | | + + +### TBD + +1. What about these? +- `connectors install` +- `connectors uninstall` +- `pipelines edit` +- `processor build` +- Other processor utilities? +1. Global flags? +1. Actions such as `pipelines start | stop`? +1. `config` vs `doctor`? From fab6002328857d63f12472e59087e94c7b5d297a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Barroso?= Date: Fri, 25 Oct 2024 14:41:48 +0200 Subject: [PATCH 04/21] fix and examples --- docs/design-documents/20241024-conduit-cli.md | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/docs/design-documents/20241024-conduit-cli.md b/docs/design-documents/20241024-conduit-cli.md index c2142f9b6..45339ed9a 100644 --- a/docs/design-documents/20241024-conduit-cli.md +++ b/docs/design-documents/20241024-conduit-cli.md @@ -88,7 +88,7 @@ EXAMPLES #### Description -- This command will output Conduit configuration (what db is configured, etc) +- This command will output the [Conduit configuration](https://github.com/ConduitIO/conduit/blob/05dbc275a724526f02779abb47b0ecc53f711485/pkg/conduit/config.go#L34) based on the default values and the user's configured settings that Conduit would use. - It does not require having conduit running. @@ -107,8 +107,8 @@ $ conduit config #### Description -- This command will start Conduit with all the available pipelines. -- It is equivalent to the current `conduit`. +- This command will start Conduit with all the configured pipelines, connectors, etc. +- It is equivalent to the current `conduit` command. #### Flags @@ -141,7 +141,7 @@ $ conduit start
-conduit pipelines init [--pipelines.path] [...] +conduit pipelines init [--path] [...] #### Description @@ -213,7 +213,7 @@ Destination: kafka
-
+
conduit connectors ls #### Description @@ -225,6 +225,9 @@ Destination: kafka ```bash $ conduit connectors ls +PLUGIN TYPE PIPELINE +postgres@v0.2.0 builtin file-to-postgres +conduit-connector-http@0.1.0. standalone my-other-pipeline ```
@@ -273,6 +276,10 @@ url HTTP URL to send requests to. true https://... ```bash $ conduit processors ls +NAME TYPE PIPELINE +avro.decode builtin my-pipeline +avro.encode builtin my-pipeline +base64.decode builtin my-other-pipeline ```

From dfb4832751a94c4f4cd2a384c951d388ef7911b7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Barroso?= Date: Fri, 25 Oct 2024 14:42:16 +0200 Subject: [PATCH 05/21] Update 20241024-conduit-cli.md --- docs/design-documents/20241024-conduit-cli.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design-documents/20241024-conduit-cli.md b/docs/design-documents/20241024-conduit-cli.md index 45339ed9a..7ea3f1a6d 100644 --- a/docs/design-documents/20241024-conduit-cli.md +++ b/docs/design-documents/20241024-conduit-cli.md @@ -213,7 +213,7 @@ Destination: kafka
-
+
conduit connectors ls #### Description From 50ef1f026c9a26671ccefeb90df3330e612c61fe Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Barroso?= Date: Fri, 25 Oct 2024 15:32:29 +0200 Subject: [PATCH 06/21] more updates --- docs/design-documents/20241024-conduit-cli.md | 101 +++++++++++------- 1 file changed, 62 insertions(+), 39 deletions(-) diff --git a/docs/design-documents/20241024-conduit-cli.md b/docs/design-documents/20241024-conduit-cli.md index 7ea3f1a6d..0d41527c4 100644 --- a/docs/design-documents/20241024-conduit-cli.md +++ b/docs/design-documents/20241024-conduit-cli.md @@ -58,7 +58,7 @@ The following list contains the suggested commands we propose to include in the #### Description -- This command will initialize a Conduit working environment. +- This command will initialize a Conduit working environment creating the `conduit.yaml` configuration file, and the three directories: processors, pipelines, and connectors. - It does not require having conduit running. - It won't require flags or arguments. - Additional flags could be provided to specify the path. @@ -103,21 +103,35 @@ $ conduit config
-conduit start [--pipelines.path] [...] +conduit run [...] #### Description -- This command will start Conduit with all the configured pipelines, connectors, etc. +- This command will run Conduit with all the configured pipelines, connectors, etc. - It is equivalent to the current `conduit` command. +- `config.path` will be the root of the working enviornment. Example: + +```bash +$ pwd +/usr/code + +$ ls +conduit.yaml +connectors/ +pipelines/ +processors/ +``` + +- Other flags such as `connectors.path`, etc. will overwrite the existing configuration on `conduit.yaml`. This will need to be evaluated before specifying to conduit to accomodate both scenarios (absolute and relative paths). #### Flags | Name | Description | Required | Default Value | |------|-------------|----------|---------------| -| api.enabled | enable HTTP and gRPC API | No | true | -| config | global config file | No | "conduit.yaml" | | connectors.path | path to standalone connectors' directory | No | "./connectors" | | db.badger.path | path to badger DB | No | "conduit.db" | +| processors.path | path to standalone processors' directory | No | "./processors" | +| pipelines.path | path to the directory that has the yaml pipeline configuration files, or a single pipeline configuration file | No | "./pipelines" | | db.postgres.connection-string | postgres connection string | Yes | | | db.postgres.table | postgres table in which to store data | No | "conduit_kv_store" | | db.type | database type; accepts badger,postgres,inmemory | No | "badger" | @@ -126,40 +140,49 @@ $ conduit config | log.format | sets the format of the logging; accepts json, cli | No | "cli" | | log.level | sets logging level; accepts debug, info, warn, error, trace | No | "info" | | pipelines.exit-on-error | exit Conduit if a pipeline experiences an error while running | No | | -| pipelines.path | path to the directory that has the yaml pipeline configuration files, or a single pipeline configuration file | No | "./pipelines" | -| processors.path | path to standalone processors' directory | No | "./processors" | -| version | prints current Conduit version | No | | - #### `--help` ```bash -$ conduit start +$ conduit run ```

-conduit pipelines init [--path] [...] +conduit pipelines init [NAME] [--pipelines.path] #### Description -- This command will initialize a pipeline. +- This command will initialize a pipeline in the previously configured pipelines path (existing on `conduit.yaml`). In other words, even if you aren't on the pipelines directory, this pipeline will be initialized there. +- In the event of not having a `conduit.yaml` configuration file already, we should prompt to initialize a working conduit environment. - It does not require having conduit running. +#### Arguments + +| Name | Description | Required | Default Value | +|------|-------------|----------|---------------| +| name | Pipeline file name and pipeline name | No | `pipeline-#` (`pipeline-#.yaml`) | + + #### Flags | Name | Description | Required | Default Value | |------|-------------|----------|---------------| -| path | Where to initialize a new pipeline | No | `.` (current directory) | +| destination | Plugin name of the destination connector | No | `log` | +| pipelines.path | Where to initialize a new pipeline | No | `.` (current directory) | +| source | Plugin name of the source connector | No | `generator` | #### `--help` ```bash $ conduit pipelines init $ conduit pipelines init my-first-pipeline +$ conduit pipelines init my-first-pipeline --pipelines.path ~/my-other-path +$ conduit pipelines init --source file@v1.0 --destination file ``` +

@@ -172,6 +195,12 @@ $ conduit pipelines init my-first-pipeline - This command will list the running pipelines. - It requires having conduit previously running. +#### Flags + +| Name | Description | Required | Default Value | +|------|-------------|----------|---------------| +| grpc.address | address for serving the gRPC API | No | ":8084" | + #### `--help` ```bash @@ -182,25 +211,26 @@ $ conduit pipelines ls
-conduit pipelines describe [NAME] +conduit pipelines describe [ID] #### Description - This command will describe the topology of the pipeline. - It requires having conduit previously running. -- It requires the pipeline name as argument. +- It requires the pipeline id as argument. #### Arguments | Name | Description | Required | Default Value | |------|-------------|----------|---------------| -| name | pipeline name to describe | Yes | | +| id | pipeline id to describe | Yes | | +| grpc.address | address for serving the gRPC API | No | ":8084" | #### `--help` ```bash -$ conduit pipelines describe [NAME] +$ conduit pipelines describe [ID] EXAMPLE: @@ -246,6 +276,7 @@ conduit-connector-http@0.1.0. standalone my-other-pipeline | Name | Description | Required | Default Value | |------|-------------|----------|---------------| | plugin | plugin name and version | Yes | | +| grpc.address | address for serving the gRPC API | No | ":8084" | #### `--help` @@ -257,7 +288,7 @@ EXAMPLE: $ conduit connectors describe conduit-connector-http@0.1.0 NAME DESCRIPTION REQUIRED DEFAULT VALUE EXAMPLE -url HTTP URL to send requests to. true https://... +url HTTP URL to send requests to. true https://... ... ```
@@ -300,6 +331,7 @@ base64.decode builtin my-other-pipeline | name | processor name | Yes | | + #### `--help` ```bash @@ -359,31 +391,22 @@ $ conduit version | Name | Description | Required | Default Value | |------|-------------|----------|---------------| -| api.enabled | enable HTTP and gRPC API | No | true | -| config | global config file | No | "conduit.yaml" | -| connectors.path | path to standalone connectors' directory | No | "./connectors" | -| db.badger.path | path to badger DB | No | "conduit.db" | -| db.postgres.connection-string | postgres connection string | Yes | | -| db.postgres.table | postgres table in which to store data | No | "conduit_kv_store" | -| db.type | database type; accepts badger,postgres,inmemory | No | "badger" | +| config.path | path to the conduit working environment | No | "." | +| version | prints current Conduit version (alias to `conduit version`) | No | | + + + + + + ### TBD -1. What about these? -- `connectors install` -- `connectors uninstall` -- `pipelines edit` -- `processor build` -- Other processor utilities? -1. Global flags? -1. Actions such as `pipelines start | stop`? 1. `config` vs `doctor`? +1. Plugins vs Names (for connectors and processors) \ No newline at end of file From 5d60f9df7bcc59e4da1353bb4f438dfdc31bfcdd Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Barroso?= Date: Fri, 25 Oct 2024 17:57:54 +0200 Subject: [PATCH 07/21] updtes --- docs/design-documents/20241024-conduit-cli.md | 195 ++++++------------ 1 file changed, 58 insertions(+), 137 deletions(-) diff --git a/docs/design-documents/20241024-conduit-cli.md b/docs/design-documents/20241024-conduit-cli.md index 0d41527c4..71e03f343 100644 --- a/docs/design-documents/20241024-conduit-cli.md +++ b/docs/design-documents/20241024-conduit-cli.md @@ -53,63 +53,44 @@ Usage of conduit: The following list contains the suggested commands we propose to include in the first iteration. New and more complex commands will be added later. -
-conduit init +### `conduit init` -#### Description - -- This command will initialize a Conduit working environment creating the `conduit.yaml` configuration file, and the three directories: processors, pipelines, and connectors. -- It does not require having conduit running. +- This command will initialize a Conduit working environment creating the `conduit.yaml` configuration file, and the three necessary directories: processors, pipelines, and connectors. +- It does not require having a conduit instance running. - It won't require flags or arguments. -- Additional flags could be provided to specify the path. - -#### Flags - -|Flag name | Description | Required | Default | -|---|---|---|---| -| path | Where to initialize Conduit | No | `.` (current directory) | +- An additional global flag named `--config.path` could specify the path where this configuration will be created. #### `--help` ```bash -$ conduit init [--path PATH] +$ conduit init [--config.path PATH] EXAMPLES $ conduit init - $ conduit init --path ~/code/conduit-dir + $ conduit init --config.path ~/code/conduit-dir ``` -
+### `conduit config` -
+- This command will output the [Conduit configuration](https://github.com/ConduitIO/conduit/blob/05dbc275a724526f02779abb47b0ecc53f711485/pkg/conduit/config.go#L34) based on the existing configuration. This will take into account the default values and the user's configured settings that Conduit will use. -
-conduit config - -#### Description - -- This command will output the [Conduit configuration](https://github.com/ConduitIO/conduit/blob/05dbc275a724526f02779abb47b0ecc53f711485/pkg/conduit/config.go#L34) based on the default values and the user's configured settings that Conduit would use. - It does not require having conduit running. - #### `--help` ```bash $ conduit config -``` -
- -
- -
-conduit run [...] +DB ... +API.HTTP: ... +... +``` -#### Description +### `conduit run [--connectors.path] [...]` -- This command will run Conduit with all the configured pipelines, connectors, etc. -- It is equivalent to the current `conduit` command. -- `config.path` will be the root of the working enviornment. Example: +- This command will run Conduit with all the configured pipelines, connectors, and processors. +- It is equivalent to the current `conduit` command as of time of this writing. +- Conduit will run based on `--config.path` or current directory. Example: ```bash $ pwd @@ -141,23 +122,12 @@ processors/ | log.level | sets logging level; accepts debug, info, warn, error, trace | No | "info" | | pipelines.exit-on-error | exit Conduit if a pipeline experiences an error while running | No | | -#### `--help` - -```bash -$ conduit run -``` -
+### `conduit pipelines init [NAME] [--pipelines.path] [--source] [--destination]` -
- -
-conduit pipelines init [NAME] [--pipelines.path] - -#### Description - -- This command will initialize a pipeline in the previously configured pipelines path (existing on `conduit.yaml`). In other words, even if you aren't on the pipelines directory, this pipeline will be initialized there. -- In the event of not having a `conduit.yaml` configuration file already, we should prompt to initialize a working conduit environment. +- This command will initialize a pipeline based on the working environment. Optionally, a user could provide a different flag if they want to specify a different path. +- In the event of not being able to to read a `conduit.yaml` configuration file based on current directory or `--config.path`, we should prompt to set up a working conduit environment via `conduit init`. - It does not require having conduit running. +- A source and a destination could be provided using the same connectors reference as described [here](https://conduit.io/docs/using/connectors/referencing). #### Arguments @@ -183,14 +153,7 @@ $ conduit pipelines init my-first-pipeline --pipelines.path ~/my-other-path $ conduit pipelines init --source file@v1.0 --destination file ``` -
- -
- -
-conduit pipelines ls - -#### Description +### conduit pipelines ls - This command will list the running pipelines. - It requires having conduit previously running. @@ -205,15 +168,12 @@ $ conduit pipelines init --source file@v1.0 --destination file ```bash $ conduit pipelines ls +NAME STATUS +my-pipeline running +my-other degraded ``` -
- -
- -
-conduit pipelines describe [ID] -#### Description +### `conduit pipelines describe ID` - This command will describe the topology of the pipeline. - It requires having conduit previously running. @@ -226,7 +186,6 @@ $ conduit pipelines ls | id | pipeline id to describe | Yes | | | grpc.address | address for serving the gRPC API | No | ":8084" | - #### `--help` ```bash @@ -239,14 +198,8 @@ Source: kafka Processor: avro.encode Destination: kafka ``` -
- -
-
-conduit connectors ls - -#### Description +### `conduit connectors ls` - This command will list all the available connectors. - It requires having conduit previously running. @@ -255,26 +208,22 @@ Destination: kafka ```bash $ conduit connectors ls -PLUGIN TYPE PIPELINE -postgres@v0.2.0 builtin file-to-postgres -conduit-connector-http@0.1.0. standalone my-other-pipeline +ID PLUGIN TYPE PIPELINE +my-source postgres@v0.2.0 builtin file-to-postgres +my-destination conduit-connector-http@0.1.0. standalone my-other-pipeline ``` -
- -
-
-conduit connectors describe [PLUGIN] - -#### Description +### `conduit connectors describe [--plugin] [--id]` - This command will describe the connector configuration available. -- It requires having conduit previously running. +- When using `--plugin` will describe the configuration for that connector plugin. +- When using `--id` will require having conduit previously running since it'll describe the existing running connector. -#### Arguments +#### Flags | Name | Description | Required | Default Value | |------|-------------|----------|---------------| +| id | connector identifier (returned by `connectors ls`) | Yes | | | plugin | plugin name and version | Yes | | | grpc.address | address for serving the gRPC API | No | ":8084" | @@ -282,27 +231,31 @@ conduit-connector-http@0.1.0. standalone my-other-pipeline #### `--help` ```bash -$ conduit connnectors describe [PLUGIN] +$ conduit connnectors describe [--plugin] [--id] [--grpc.address] EXAMPLE: -$ conduit connectors describe conduit-connector-http@0.1.0 +$ conduit connectors describe --plugin conduit-connector-http@0.1.0 NAME DESCRIPTION REQUIRED DEFAULT VALUE EXAMPLE url HTTP URL to send requests to. true https://... ... -``` -
- -
-
-conduit processors ls +$ conduit connectors describe --id my-source +NAME URL PIPELINE +my-source https://... my-pipeline +``` -#### Description +### `conduit processors ls` - This command will list all the available processors. - It requires having conduit previously running. +#### Flags + +| Name | Description | Required | Default Value | +|------|-------------|----------|---------------| +| grpc.address | address for serving the gRPC API | No | ":8084" | + #### `--help` ```bash @@ -312,13 +265,8 @@ avro.decode builtin my-pipeline avro.encode builtin my-pipeline base64.decode builtin my-other-pipeline ``` -
-
-
-conduit processor describe [NAME] - -#### Description +### `conduit processor describe [NAME]` - This command will describe the processor configuration available. - It requires having conduit previously running. @@ -330,8 +278,6 @@ base64.decode builtin my-other-pipeline |------|-------------|----------|---------------| | name | processor name | Yes | | - - #### `--help` ```bash @@ -348,32 +294,26 @@ description: This option is required if auth.basic.username contains a value. If type: string ... ``` -
- -
-
-conduit doctor - -#### Description +### `conduit doctor` - This will check whether there’s a more up to date version of conduit and if some connectors / processors could also be updated. - It requires having conduit previously running. +#### Flags + +| Name | Description | Required | Default Value | +|------|-------------|----------|---------------| +| grpc.address | address for serving the gRPC API | No | ":8084" | + #### `--help` ```bash $ conduit doctor # returns version, checks if there's a newer version of conduit, plugin versions, etc. ``` -
- -
-
-conduit version - -#### Description +### conduit version - This will return the existing Conduit version. - It requires having conduit previously running. @@ -383,30 +323,11 @@ $ conduit doctor ```bash $ conduit version ``` -
- -
#### Global flags | Name | Description | Required | Default Value | |------|-------------|----------|---------------| -| config.path | path to the conduit working environment | No | "." | +| config.path | path to the conduit working environment | No | `.` | +| json | output json | No | | | version | prints current Conduit version (alias to `conduit version`) | No | | - - - - - - - - -### TBD - -1. `config` vs `doctor`? -1. Plugins vs Names (for connectors and processors) \ No newline at end of file From 65b70de967026a8b451e878fd8538bdbd72fc1c5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Barroso?= Date: Fri, 25 Oct 2024 18:02:39 +0200 Subject: [PATCH 08/21] md fixes --- docs/design-documents/20241024-conduit-cli.md | 94 +++++++++---------- 1 file changed, 46 insertions(+), 48 deletions(-) diff --git a/docs/design-documents/20241024-conduit-cli.md b/docs/design-documents/20241024-conduit-cli.md index 71e03f343..e159e63de 100644 --- a/docs/design-documents/20241024-conduit-cli.md +++ b/docs/design-documents/20241024-conduit-cli.md @@ -18,35 +18,35 @@ Here's how the existing Conduit CLI looks like: $ conduit --help Usage of conduit: -api.enabled - enable HTTP and gRPC API (default true) + enable HTTP and gRPC API (default true) -config string - global config file (default "conduit.yaml") + global config file (default "conduit.yaml") -connectors.path string - path to standalone connectors' directory (default "./connectors") + path to standalone connectors' directory (default "./connectors") -db.badger.path string - path to badger DB (default "conduit.db") + path to badger DB (default "conduit.db") -db.postgres.connection-string string - postgres connection string + postgres connection string -db.postgres.table string - postgres table in which to store data (will be created if it does not exist) (default "conduit_kv_store") + postgres table in which to store data (will be created if it does not exist) (default "conduit_kv_store") -db.type string - database type; accepts badger,postgres,inmemory (default "badger") + database type; accepts badger,postgres,inmemory (default "badger") -grpc.address string - address for serving the gRPC API (default ":8084") + address for serving the gRPC API (default ":8084") -http.address string - address for serving the HTTP API (default ":8080") + address for serving the HTTP API (default ":8080") -log.format string - sets the format of the logging; accepts json, cli (default "cli") + sets the format of the logging; accepts json, cli (default "cli") -log.level string - sets logging level; accepts debug, info, warn, error, trace (default "info") + sets logging level; accepts debug, info, warn, error, trace (default "info") -pipelines.exit-on-error - exit Conduit if a pipeline experiences an error while running + exit Conduit if a pipeline experiences an error while running -pipelines.path string - path to the directory that has the yaml pipeline configuration files, or a single pipeline configuration file (default "./pipelines") + path to the directory that has the yaml pipeline configuration files, or a single pipeline configuration file (default "./pipelines") -processors.path string - path to standalone processors' directory (default "./processors") + path to standalone processors' directory (default "./processors") -version - prints current Conduit version + prints current Conduit version ``` ## Suggested commands for Conduit 0.13 @@ -56,7 +56,7 @@ The following list contains the suggested commands we propose to include in the ### `conduit init` - This command will initialize a Conduit working environment creating the `conduit.yaml` configuration file, and the three necessary directories: processors, pipelines, and connectors. -- It does not require having a conduit instance running. +- It does not require having a Conduit instance running. - It won't require flags or arguments. - An additional global flag named `--config.path` could specify the path where this configuration will be created. @@ -74,7 +74,7 @@ EXAMPLES - This command will output the [Conduit configuration](https://github.com/ConduitIO/conduit/blob/05dbc275a724526f02779abb47b0ecc53f711485/pkg/conduit/config.go#L34) based on the existing configuration. This will take into account the default values and the user's configured settings that Conduit will use. -- It does not require having conduit running. +- It does not require having Conduit running. #### `--help` @@ -103,18 +103,18 @@ pipelines/ processors/ ``` -- Other flags such as `connectors.path`, etc. will overwrite the existing configuration on `conduit.yaml`. This will need to be evaluated before specifying to conduit to accomodate both scenarios (absolute and relative paths). +- Other flags such as `connectors.path`, etc. will overwrite the existing configuration on `conduit.yaml`. This will need to be evaluated before specifying to Conduit to accomodate both scenarios (absolute and relative paths). #### Flags | Name | Description | Required | Default Value | |------|-------------|----------|---------------| | connectors.path | path to standalone connectors' directory | No | "./connectors" | -| db.badger.path | path to badger DB | No | "conduit.db" | +| db.badger.path | path to badger DB | No | "Conduit.db" | | processors.path | path to standalone processors' directory | No | "./processors" | | pipelines.path | path to the directory that has the yaml pipeline configuration files, or a single pipeline configuration file | No | "./pipelines" | | db.postgres.connection-string | postgres connection string | Yes | | -| db.postgres.table | postgres table in which to store data | No | "conduit_kv_store" | +| db.postgres.table | postgres table in which to store data | No | "Conduit_kv_store" | | db.type | database type; accepts badger,postgres,inmemory | No | "badger" | | grpc.address | address for serving the gRPC API | No | ":8084" | | http.address | address for serving the HTTP API | No | ":8080" | @@ -125,8 +125,8 @@ processors/ ### `conduit pipelines init [NAME] [--pipelines.path] [--source] [--destination]` - This command will initialize a pipeline based on the working environment. Optionally, a user could provide a different flag if they want to specify a different path. -- In the event of not being able to to read a `conduit.yaml` configuration file based on current directory or `--config.path`, we should prompt to set up a working conduit environment via `conduit init`. -- It does not require having conduit running. +- In the event of not being able to to read a `conduit.yaml` configuration file based on current directory or `--config.path`, we should prompt to set up a working Conduit environment via `conduit init`. +- It does not require having Conduit running. - A source and a destination could be provided using the same connectors reference as described [here](https://conduit.io/docs/using/connectors/referencing). #### Arguments @@ -135,7 +135,6 @@ processors/ |------|-------------|----------|---------------| | name | Pipeline file name and pipeline name | No | `pipeline-#` (`pipeline-#.yaml`) | - #### Flags | Name | Description | Required | Default Value | @@ -147,16 +146,16 @@ processors/ #### `--help` ```bash -$ conduit pipelines init -$ conduit pipelines init my-first-pipeline -$ conduit pipelines init my-first-pipeline --pipelines.path ~/my-other-path -$ conduit pipelines init --source file@v1.0 --destination file +conduit pipelines init +conduit pipelines init my-first-pipeline +conduit pipelines init my-first-pipeline --pipelines.path ~/my-other-path +conduit pipelines init --source file@v1.0 --destination file ``` -### conduit pipelines ls +### Conduit pipelines ls - This command will list the running pipelines. -- It requires having conduit previously running. +- It requires having Conduit previously running. #### Flags @@ -176,7 +175,7 @@ my-other degraded ### `conduit pipelines describe ID` - This command will describe the topology of the pipeline. -- It requires having conduit previously running. +- It requires having Conduit previously running. - It requires the pipeline id as argument. #### Arguments @@ -202,14 +201,14 @@ Destination: kafka ### `conduit connectors ls` - This command will list all the available connectors. -- It requires having conduit previously running. +- It requires having Conduit previously running. #### `--help` ```bash $ conduit connectors ls ID PLUGIN TYPE PIPELINE -my-source postgres@v0.2.0 builtin file-to-postgres +my-source postgres@v0.2.0 builtin file-to-postgres my-destination conduit-connector-http@0.1.0. standalone my-other-pipeline ``` @@ -217,7 +216,7 @@ my-destination conduit-connector-http@0.1.0. standalone my-other-pipeline - This command will describe the connector configuration available. - When using `--plugin` will describe the configuration for that connector plugin. -- When using `--id` will require having conduit previously running since it'll describe the existing running connector. +- When using `--id` will require having Conduit previously running since it'll describe the existing running connector. #### Flags @@ -227,7 +226,6 @@ my-destination conduit-connector-http@0.1.0. standalone my-other-pipeline | plugin | plugin name and version | Yes | | | grpc.address | address for serving the gRPC API | No | ":8084" | - #### `--help` ```bash @@ -236,8 +234,8 @@ $ conduit connnectors describe [--plugin] [--id] [--grpc.address] EXAMPLE: $ conduit connectors describe --plugin conduit-connector-http@0.1.0 -NAME DESCRIPTION REQUIRED DEFAULT VALUE EXAMPLE -url HTTP URL to send requests to. true https://... +NAME DESCRIPTION REQUIRED DEFAULT VALUE EXAMPLE +url HTTP URL to send requests to. true https://... ... $ conduit connectors describe --id my-source @@ -248,7 +246,7 @@ my-source https://... my-pipeline ### `conduit processors ls` - This command will list all the available processors. -- It requires having conduit previously running. +- It requires having Conduit previously running. #### Flags @@ -260,16 +258,16 @@ my-source https://... my-pipeline ```bash $ conduit processors ls -NAME TYPE PIPELINE -avro.decode builtin my-pipeline -avro.encode builtin my-pipeline -base64.decode builtin my-other-pipeline +NAME TYPE PIPELINE +avro.decode builtin my-pipeline +avro.encode builtin my-pipeline +base64.decode builtin my-other-pipeline ``` ### `conduit processor describe [NAME]` - This command will describe the processor configuration available. -- It requires having conduit previously running. +- It requires having Conduit previously running. - It requires the processor name as argument. #### Arguments @@ -297,8 +295,8 @@ type: string ### `conduit doctor` -- This will check whether there’s a more up to date version of conduit and if some connectors / processors could also be updated. -- It requires having conduit previously running. +- This will check whether there’s a more up to date version of Conduit and if some connectors / processors could also be updated. +- It requires having Conduit previously running. #### Flags @@ -313,21 +311,21 @@ $ conduit doctor # returns version, checks if there's a newer version of conduit, plugin versions, etc. ``` -### conduit version +### Conduit version - This will return the existing Conduit version. -- It requires having conduit previously running. +- It requires having Conduit previously running. #### `--help` ```bash -$ conduit version +conduit version ``` #### Global flags | Name | Description | Required | Default Value | |------|-------------|----------|---------------| -| config.path | path to the conduit working environment | No | `.` | +| config.path | path to the Conduit working environment | No | `.` | | json | output json | No | | | version | prints current Conduit version (alias to `conduit version`) | No | | From f4017d5363ee19ba212b18f9dff070cc07942de2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Barroso?= Date: Fri, 25 Oct 2024 18:05:19 +0200 Subject: [PATCH 09/21] split lines --- docs/design-documents/20241024-conduit-cli.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/design-documents/20241024-conduit-cli.md b/docs/design-documents/20241024-conduit-cli.md index e159e63de..49c5138cf 100644 --- a/docs/design-documents/20241024-conduit-cli.md +++ b/docs/design-documents/20241024-conduit-cli.md @@ -2,7 +2,8 @@ ## Introduction -The goal of this design document is to clarify which part of this [Conduit discussion](https://github.com/ConduitIO/conduit/discussions/1642) will be part of the upcoming [0.13 release](https://github.com/ConduitIO/conduit/milestone/15). The intention is to provide a good starting experience and evolve it over time with more complex and advanced features. +The goal of this design document is to clarify which part of this [Conduit discussion](https://github.com/ConduitIO/conduit/discussions/1642) will be part of the upcoming [0.13 release](https://github.com/ConduitIO/conduit/milestone/15). +The intention is to provide a good starting experience and evolve it over time with more complex and advanced features. ### Goals @@ -72,7 +73,8 @@ EXAMPLES ### `conduit config` -- This command will output the [Conduit configuration](https://github.com/ConduitIO/conduit/blob/05dbc275a724526f02779abb47b0ecc53f711485/pkg/conduit/config.go#L34) based on the existing configuration. This will take into account the default values and the user's configured settings that Conduit will use. +- This command will output the [Conduit configuration](https://github.com/ConduitIO/conduit/blob/05dbc275a724526f02779abb47b0ecc53f711485/pkg/conduit/config.go#L34) based on the existing configuration. +This will take into account the default values and the user's configured settings that Conduit will use. - It does not require having Conduit running. From c9f25b542afd2b9afcfb4ce8ee57f64e3eb09dcf Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Barroso?= Date: Fri, 25 Oct 2024 18:10:20 +0200 Subject: [PATCH 10/21] updates --- docs/design-documents/20241024-conduit-cli.md | 72 ++++++++++++------- 1 file changed, 48 insertions(+), 24 deletions(-) diff --git a/docs/design-documents/20241024-conduit-cli.md b/docs/design-documents/20241024-conduit-cli.md index 49c5138cf..4d219218b 100644 --- a/docs/design-documents/20241024-conduit-cli.md +++ b/docs/design-documents/20241024-conduit-cli.md @@ -7,9 +7,12 @@ The intention is to provide a good starting experience and evolve it over time w ### Goals -- We aim to provide our users with an interface that empowers them to use the **main** Conduit features effortlessly. While we have envisioned some complex features, they will be available in future releases. -- We will maintain the current installation methods by incorporating the CLI through the same established processes. -- The CLI needs to feel intuitive and align with our standards. +We aim to provide our users with an interface that empowers them to use the **main** Conduit features effortlessly. +While we have envisioned some complex features, they will be available in future releases. + +We will maintain the current installation methods by incorporating the CLI through the same established processes. + +The CLI needs to feel intuitive and align with our standards. ## Background @@ -56,10 +59,13 @@ The following list contains the suggested commands we propose to include in the ### `conduit init` -- This command will initialize a Conduit working environment creating the `conduit.yaml` configuration file, and the three necessary directories: processors, pipelines, and connectors. -- It does not require having a Conduit instance running. -- It won't require flags or arguments. -- An additional global flag named `--config.path` could specify the path where this configuration will be created. +This command will initialize a Conduit working environment creating the `conduit.yaml` configuration file, and the three necessary directories: processors, pipelines, and connectors. + +It does not require having a Conduit instance running. + +It won't require flags or arguments. + +An additional global flag named `--config.path` could specify the path where this configuration will be created. #### `--help` @@ -73,7 +79,9 @@ EXAMPLES ### `conduit config` -- This command will output the [Conduit configuration](https://github.com/ConduitIO/conduit/blob/05dbc275a724526f02779abb47b0ecc53f711485/pkg/conduit/config.go#L34) based on the existing configuration. +This command will output the [Conduit configuration](https://github.com/ConduitIO/conduit/blob/05dbc275a724526f02779abb47b0ecc53f711485/pkg/conduit/config.go#L34) +based on the existing configuration. + This will take into account the default values and the user's configured settings that Conduit will use. - It does not require having Conduit running. @@ -90,9 +98,11 @@ API.HTTP: ... ### `conduit run [--connectors.path] [...]` -- This command will run Conduit with all the configured pipelines, connectors, and processors. -- It is equivalent to the current `conduit` command as of time of this writing. -- Conduit will run based on `--config.path` or current directory. Example: +This command will run Conduit with all the configured pipelines, connectors, and processors. + +It is equivalent to the current `conduit` command as of time of this writing. + +Conduit will run based on `--config.path` or current directory. Example: ```bash $ pwd @@ -105,7 +115,8 @@ pipelines/ processors/ ``` -- Other flags such as `connectors.path`, etc. will overwrite the existing configuration on `conduit.yaml`. This will need to be evaluated before specifying to Conduit to accomodate both scenarios (absolute and relative paths). +Other flags such as `connectors.path`, etc. will overwrite the existing configuration on `conduit.yaml`. +This will need to be evaluated before specifying to Conduit to accomodate both scenarios (absolute and relative paths). #### Flags @@ -126,10 +137,16 @@ processors/ ### `conduit pipelines init [NAME] [--pipelines.path] [--source] [--destination]` -- This command will initialize a pipeline based on the working environment. Optionally, a user could provide a different flag if they want to specify a different path. -- In the event of not being able to to read a `conduit.yaml` configuration file based on current directory or `--config.path`, we should prompt to set up a working Conduit environment via `conduit init`. -- It does not require having Conduit running. -- A source and a destination could be provided using the same connectors reference as described [here](https://conduit.io/docs/using/connectors/referencing). +This command will initialize a pipeline based on the working environment. Optionally, a user could provide a different flag +if they want to specify a different path. + +In the event of not being able to to read a `conduit.yaml` configuration file based on current directory or `--config.path`, +we should prompt to set up a working Conduit environment via `conduit init`. + +It does not require having Conduit running. + +A source and a destination could be provided using the same connectors reference as described +[here](https://conduit.io/docs/using/connectors/referencing). #### Arguments @@ -216,9 +233,11 @@ my-destination conduit-connector-http@0.1.0. standalone my-other-pipeline ### `conduit connectors describe [--plugin] [--id]` -- This command will describe the connector configuration available. -- When using `--plugin` will describe the configuration for that connector plugin. -- When using `--id` will require having Conduit previously running since it'll describe the existing running connector. +1This command will describe the connector configuration available. + +When using `--plugin` will describe the configuration for that connector plugin. + +When using `--id` will require having Conduit previously running since it'll describe the existing running connector. #### Flags @@ -268,9 +287,11 @@ base64.decode builtin my-other-pipeline ### `conduit processor describe [NAME]` -- This command will describe the processor configuration available. -- It requires having Conduit previously running. -- It requires the processor name as argument. +This command will describe the processor configuration available. + +It requires having Conduit previously running. + +It requires the processor name as argument. #### Arguments @@ -297,8 +318,11 @@ type: string ### `conduit doctor` -- This will check whether there’s a more up to date version of Conduit and if some connectors / processors could also be updated. -- It requires having Conduit previously running. +This will check whether there’s a more up to date version of Conduit. + +Also will check iff some connectors or processors could be updated. + +It requires having Conduit previously running. #### Flags From f5d129295d3b568c5ecf481c029b91fb73094002 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Barroso?= Date: Fri, 25 Oct 2024 18:11:46 +0200 Subject: [PATCH 11/21] more fixes --- docs/design-documents/20241024-conduit-cli.md | 36 +++++++++++-------- 1 file changed, 21 insertions(+), 15 deletions(-) diff --git a/docs/design-documents/20241024-conduit-cli.md b/docs/design-documents/20241024-conduit-cli.md index 4d219218b..a8cfb2dd5 100644 --- a/docs/design-documents/20241024-conduit-cli.md +++ b/docs/design-documents/20241024-conduit-cli.md @@ -84,7 +84,7 @@ based on the existing configuration. This will take into account the default values and the user's configured settings that Conduit will use. -- It does not require having Conduit running. +It does not require having Conduit running. #### `--help` @@ -171,10 +171,11 @@ conduit pipelines init my-first-pipeline --pipelines.path ~/my-other-path conduit pipelines init --source file@v1.0 --destination file ``` -### Conduit pipelines ls +### `conduit pipelines ls` + +This command will list the running pipelines. -- This command will list the running pipelines. -- It requires having Conduit previously running. +It requires having Conduit previously running. #### Flags @@ -193,9 +194,11 @@ my-other degraded ### `conduit pipelines describe ID` -- This command will describe the topology of the pipeline. -- It requires having Conduit previously running. -- It requires the pipeline id as argument. +This command will describe the topology of the pipeline. + +It requires having Conduit previously running. + +It requires the pipeline id as argument. #### Arguments @@ -219,8 +222,9 @@ Destination: kafka ### `conduit connectors ls` -- This command will list all the available connectors. -- It requires having Conduit previously running. +This command will list all the available connectors. + +It requires having Conduit previously running. #### `--help` @@ -233,7 +237,7 @@ my-destination conduit-connector-http@0.1.0. standalone my-other-pipeline ### `conduit connectors describe [--plugin] [--id]` -1This command will describe the connector configuration available. +This command will describe the connector configuration available. When using `--plugin` will describe the configuration for that connector plugin. @@ -266,8 +270,9 @@ my-source https://... my-pipeline ### `conduit processors ls` -- This command will list all the available processors. -- It requires having Conduit previously running. +This command will list all the available processors. + +It requires having Conduit previously running. #### Flags @@ -337,10 +342,11 @@ $ conduit doctor # returns version, checks if there's a newer version of conduit, plugin versions, etc. ``` -### Conduit version +### `conduit version` + +This will return the existing Conduit version. -- This will return the existing Conduit version. -- It requires having Conduit previously running. +It requires having Conduit previously running. #### `--help` From 1fa465b8c90a10d3d49c1fd6b1bca895feed32f6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Barroso?= Date: Fri, 25 Oct 2024 18:15:42 +0200 Subject: [PATCH 12/21] more fixes --- docs/design-documents/20241024-conduit-cli.md | 21 ++++++++++++------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/docs/design-documents/20241024-conduit-cli.md b/docs/design-documents/20241024-conduit-cli.md index a8cfb2dd5..3d59ced77 100644 --- a/docs/design-documents/20241024-conduit-cli.md +++ b/docs/design-documents/20241024-conduit-cli.md @@ -2,12 +2,15 @@ ## Introduction -The goal of this design document is to clarify which part of this [Conduit discussion](https://github.com/ConduitIO/conduit/discussions/1642) will be part of the upcoming [0.13 release](https://github.com/ConduitIO/conduit/milestone/15). +The goal of this design document is to clarify which part of this +[Conduit discussion](https://github.com/ConduitIO/conduit/discussions/1642) will be part of the +upcoming [0.13 release](https://github.com/ConduitIO/conduit/milestone/15). + The intention is to provide a good starting experience and evolve it over time with more complex and advanced features. ### Goals -We aim to provide our users with an interface that empowers them to use the **main** Conduit features effortlessly. +We aim to provide our users with an interface that empowers them to use the **main** Conduit features effortlessly. While we have envisioned some complex features, they will be available in future releases. We will maintain the current installation methods by incorporating the CLI through the same established processes. @@ -55,11 +58,14 @@ Usage of conduit: ## Suggested commands for Conduit 0.13 -The following list contains the suggested commands we propose to include in the first iteration. New and more complex commands will be added later. +The following list contains the suggested commands we propose to include in the first iteration. + +New and more complex commands will be added later. ### `conduit init` -This command will initialize a Conduit working environment creating the `conduit.yaml` configuration file, and the three necessary directories: processors, pipelines, and connectors. +This command will initialize a Conduit working environment creating the `conduit.yaml` configuration file, +and the three necessary directories: processors, pipelines, and connectors. It does not require having a Conduit instance running. @@ -79,8 +85,7 @@ EXAMPLES ### `conduit config` -This command will output the [Conduit configuration](https://github.com/ConduitIO/conduit/blob/05dbc275a724526f02779abb47b0ecc53f711485/pkg/conduit/config.go#L34) -based on the existing configuration. +This command will output the [Conduit configuration](https://github.com/ConduitIO/conduit/blob/05dbc275a724526f02779abb47b0ecc53f711485/pkg/conduit/config.go#L34) based on the existing configuration. This will take into account the default values and the user's configured settings that Conduit will use. @@ -115,7 +120,7 @@ pipelines/ processors/ ``` -Other flags such as `connectors.path`, etc. will overwrite the existing configuration on `conduit.yaml`. +Other flags such as `connectors.path`, etc. will overwrite the existing configuration on `conduit.yaml`. This will need to be evaluated before specifying to Conduit to accomodate both scenarios (absolute and relative paths). #### Flags @@ -137,7 +142,7 @@ This will need to be evaluated before specifying to Conduit to accomodate both s ### `conduit pipelines init [NAME] [--pipelines.path] [--source] [--destination]` -This command will initialize a pipeline based on the working environment. Optionally, a user could provide a different flag +This command will initialize a pipeline based on the working environment. Optionally, a user could provide a different flag if they want to specify a different path. In the event of not being able to to read a `conduit.yaml` configuration file based on current directory or `--config.path`, From 1901b12dcf94f928dd614f6efebbe341dcd7bca4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Barroso?= Date: Fri, 25 Oct 2024 18:19:28 +0200 Subject: [PATCH 13/21] fixes --- docs/design-documents/20241024-conduit-cli.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/design-documents/20241024-conduit-cli.md b/docs/design-documents/20241024-conduit-cli.md index 3d59ced77..c0abb4392 100644 --- a/docs/design-documents/20241024-conduit-cli.md +++ b/docs/design-documents/20241024-conduit-cli.md @@ -58,13 +58,13 @@ Usage of conduit: ## Suggested commands for Conduit 0.13 -The following list contains the suggested commands we propose to include in the first iteration. +The following list contains the suggested commands we propose to include in the first iteration. New and more complex commands will be added later. ### `conduit init` -This command will initialize a Conduit working environment creating the `conduit.yaml` configuration file, +This command will initialize a Conduit working environment creating the `conduit.yaml` configuration file, and the three necessary directories: processors, pipelines, and connectors. It does not require having a Conduit instance running. @@ -85,7 +85,8 @@ EXAMPLES ### `conduit config` -This command will output the [Conduit configuration](https://github.com/ConduitIO/conduit/blob/05dbc275a724526f02779abb47b0ecc53f711485/pkg/conduit/config.go#L34) based on the existing configuration. +This command will output the [Conduit configuration](https://github.com/ConduitIO/conduit/blob/05dbc275a724526f02779abb47b0ecc53f711485/pkg/conduit/config.go#L34) +based on the existing configuration. This will take into account the default values and the user's configured settings that Conduit will use. @@ -145,12 +146,12 @@ This will need to be evaluated before specifying to Conduit to accomodate both s This command will initialize a pipeline based on the working environment. Optionally, a user could provide a different flag if they want to specify a different path. -In the event of not being able to to read a `conduit.yaml` configuration file based on current directory or `--config.path`, +In the event of not being able to to read a `conduit.yaml` configuration file based on current directory or `--config.path`, we should prompt to set up a working Conduit environment via `conduit init`. It does not require having Conduit running. -A source and a destination could be provided using the same connectors reference as described +A source and a destination could be provided using the same connectors reference as described [here](https://conduit.io/docs/using/connectors/referencing). #### Arguments From 12a8ee3a705965c17548ee6b50756d88d9cf97ef Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Barroso?= Date: Fri, 25 Oct 2024 18:22:58 +0200 Subject: [PATCH 14/21] fix `conduit config` --- docs/design-documents/20241024-conduit-cli.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/design-documents/20241024-conduit-cli.md b/docs/design-documents/20241024-conduit-cli.md index c0abb4392..0eda69a90 100644 --- a/docs/design-documents/20241024-conduit-cli.md +++ b/docs/design-documents/20241024-conduit-cli.md @@ -90,7 +90,13 @@ based on the existing configuration. This will take into account the default values and the user's configured settings that Conduit will use. -It does not require having Conduit running. +It requires having Conduit running. + +#### Flags + +| Name | Description | Required | Default Value | +|------|-------------|----------|---------------| +| grpc.address | address for serving the gRPC API | No | ":8084" | #### `--help` From 90c197456affb83468ab43048f395da314c942c1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Barroso?= Date: Fri, 25 Oct 2024 18:38:35 +0200 Subject: [PATCH 15/21] update --- docs/design-documents/20241024-conduit-cli.md | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/docs/design-documents/20241024-conduit-cli.md b/docs/design-documents/20241024-conduit-cli.md index 0eda69a90..041d79791 100644 --- a/docs/design-documents/20241024-conduit-cli.md +++ b/docs/design-documents/20241024-conduit-cli.md @@ -227,9 +227,19 @@ $ conduit pipelines describe [ID] EXAMPLE: $ conduit pipelines describe my-pipeline -Source: kafka - Processor: avro.encode -Destination: kafka +ID: generator-to-log +State: + Status: STATUS_STOPPED +Config: + Name: generator-to-log + Description: Postgres source, file destination +Connector IDs: + - generator-to-log:postgres-source + - generator-to-log:destination +Processor IDs: + - my-processor:avro.encode +Created At: 2024-10-09T13:55:17.113068Z +Updated At: 2024-10-09T13:55:17.113802Z ``` ### `conduit connectors ls` From d51f0974248eaee2b81c3c032da734a11b9ba444 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Barroso?= Date: Fri, 25 Oct 2024 18:50:49 +0200 Subject: [PATCH 16/21] add connector-plugins --- docs/design-documents/20241024-conduit-cli.md | 55 ++++++++++++++----- 1 file changed, 40 insertions(+), 15 deletions(-) diff --git a/docs/design-documents/20241024-conduit-cli.md b/docs/design-documents/20241024-conduit-cli.md index 041d79791..bf2acf318 100644 --- a/docs/design-documents/20241024-conduit-cli.md +++ b/docs/design-documents/20241024-conduit-cli.md @@ -253,43 +253,68 @@ It requires having Conduit previously running. ```bash $ conduit connectors ls ID PLUGIN TYPE PIPELINE -my-source postgres@v0.2.0 builtin file-to-postgres +my-source postgres@v0.2.0 builtin file-to-postgres my-destination conduit-connector-http@0.1.0. standalone my-other-pipeline ``` -### `conduit connectors describe [--plugin] [--id]` +### `conduit connectors describe ID` -This command will describe the connector configuration available. +This command will describe the existing running connector. -When using `--plugin` will describe the configuration for that connector plugin. - -When using `--id` will require having Conduit previously running since it'll describe the existing running connector. +It requires having Conduit previously running. #### Flags | Name | Description | Required | Default Value | |------|-------------|----------|---------------| -| id | connector identifier (returned by `connectors ls`) | Yes | | -| plugin | plugin name and version | Yes | | | grpc.address | address for serving the gRPC API | No | ":8084" | #### `--help` ```bash -$ conduit connnectors describe [--plugin] [--id] [--grpc.address] +$ conduit connnectors describe ID [--grpc.address] + +EXAMPLE: + +$ conduit connectors describe my-source +NAME URL PIPELINE PLUGIN +my-source https://... my-pipeline postgres@v0.2.0 +``` + +### `conduit connector-plugins ls` + +This command will list all the available connector plugins. + +It does not require having Conduit previously running. + +#### `--help` + +```bash +$ conduit connector-plugins ls +PLUGIN +postgres@v0.2.0 +conduit-connector-http@0.1.0 +``` + +### `conduit connector-plugins PLUGIN` + +This command will describe the configuration for that connector plugin. + +It does not require having Conduit previously running. + +#### `--help` + +```bash +$ conduit connnector-plugins describe PLUGIN EXAMPLE: -$ conduit connectors describe --plugin conduit-connector-http@0.1.0 +$ conduit connector-plugins describe conduit-connector-http@0.1.0 NAME DESCRIPTION REQUIRED DEFAULT VALUE EXAMPLE url HTTP URL to send requests to. true https://... -... - -$ conduit connectors describe --id my-source -NAME URL PIPELINE -my-source https://... my-pipeline ``` + ### `conduit processors ls` This command will list all the available processors. From f0d6815f157a730e6b639e74b757c1359278cecd Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Barroso?= Date: Fri, 25 Oct 2024 18:55:54 +0200 Subject: [PATCH 17/21] add processor plugins --- docs/design-documents/20241024-conduit-cli.md | 51 ++++++++++++++++--- 1 file changed, 44 insertions(+), 7 deletions(-) diff --git a/docs/design-documents/20241024-conduit-cli.md b/docs/design-documents/20241024-conduit-cli.md index bf2acf318..9f5513ac8 100644 --- a/docs/design-documents/20241024-conduit-cli.md +++ b/docs/design-documents/20241024-conduit-cli.md @@ -314,7 +314,6 @@ NAME DESCRIPTION REQUIRED DEFAULT VALUE EXAMPLE url HTTP URL to send requests to. true https://... ``` - ### `conduit processors ls` This command will list all the available processors. @@ -337,28 +336,66 @@ avro.encode builtin my-pipeline base64.decode builtin my-other-pipeline ``` -### `conduit processor describe [NAME]` +### `conduit processors describe ID` -This command will describe the processor configuration available. +This command will describe the existing running processor. It requires having Conduit previously running. -It requires the processor name as argument. +#### Arguments + +| Name | Description | Required | Default Value | +|------|-------------|----------|---------------| +| ID | processor ID | Yes | | + +#### `--help` + +```bash +$ conduit processors describe ID + +EXAMPLE: + +$ conduit processors describe my-processor +NAME PLUGIN PIPELINE +my-processor base64.decode my-pipeline +``` + +### `conduit processor-plugins ls` + +This command will list all the available processors. + +It does not require having Conduit previously running. + +#### `--help` + +```bash +$ conduit processor-plugins ls +NAME +avro.decode +avro.encode +base64.decode +``` + +### `conduit processor-plugins describe PLUGIN` + +This command will describe the processor plugin configuration available. + +It does not require having Conduit previously running. #### Arguments | Name | Description | Required | Default Value | |------|-------------|----------|---------------| -| name | processor name | Yes | | +| PLUGIN | processor plugin name | Yes | | #### `--help` ```bash -$ conduit processor describe [NAME] +$ conduit processor-plugins describe PLUGIN EXAMPLE: -$ conduit connectors describe avro.decode +$ conduit processor-plugins describe avro.decode # auth.basic.password type: string description: This option is required if auth.basic.username contains a value. If both auth.basic.username and auth.basic.password are empty basic authentication is disabled. From fc53df80314cd89404a8a1b2abf8506107515940 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Barroso?= Date: Fri, 25 Oct 2024 19:01:12 +0200 Subject: [PATCH 18/21] fix --- docs/design-documents/20241024-conduit-cli.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design-documents/20241024-conduit-cli.md b/docs/design-documents/20241024-conduit-cli.md index 9f5513ac8..e38d5de9b 100644 --- a/docs/design-documents/20241024-conduit-cli.md +++ b/docs/design-documents/20241024-conduit-cli.md @@ -296,7 +296,7 @@ postgres@v0.2.0 conduit-connector-http@0.1.0 ``` -### `conduit connector-plugins PLUGIN` +### `conduit connector-plugins describe PLUGIN` This command will describe the configuration for that connector plugin. From df1fb31a2b693a3714fc102aa949eb32c166ff40 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Barroso?= Date: Fri, 25 Oct 2024 19:04:50 +0200 Subject: [PATCH 19/21] =?UTF-8?q?=F0=9F=91=8B=F0=9F=8F=BB=20=F0=9F=91=A8?= =?UTF-8?q?=F0=9F=8F=BB=E2=80=8D=E2=9A=95=EF=B8=8F?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/design-documents/20241024-conduit-cli.md | 21 ------------------- 1 file changed, 21 deletions(-) diff --git a/docs/design-documents/20241024-conduit-cli.md b/docs/design-documents/20241024-conduit-cli.md index e38d5de9b..5a1166853 100644 --- a/docs/design-documents/20241024-conduit-cli.md +++ b/docs/design-documents/20241024-conduit-cli.md @@ -405,27 +405,6 @@ type: string ... ``` -### `conduit doctor` - -This will check whether there’s a more up to date version of Conduit. - -Also will check iff some connectors or processors could be updated. - -It requires having Conduit previously running. - -#### Flags - -| Name | Description | Required | Default Value | -|------|-------------|----------|---------------| -| grpc.address | address for serving the gRPC API | No | ":8084" | - -#### `--help` - -```bash -$ conduit doctor -# returns version, checks if there's a newer version of conduit, plugin versions, etc. -``` - ### `conduit version` This will return the existing Conduit version. From 010dabf57255c7d12b5b725f06bc99ab4d87548f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Barroso?= Date: Fri, 25 Oct 2024 19:09:53 +0200 Subject: [PATCH 20/21] add missing info --- docs/design-documents/20241024-conduit-cli.md | 25 ++++++++++++++++--- 1 file changed, 22 insertions(+), 3 deletions(-) diff --git a/docs/design-documents/20241024-conduit-cli.md b/docs/design-documents/20241024-conduit-cli.md index 5a1166853..25780de4b 100644 --- a/docs/design-documents/20241024-conduit-cli.md +++ b/docs/design-documents/20241024-conduit-cli.md @@ -217,12 +217,17 @@ It requires the pipeline id as argument. | Name | Description | Required | Default Value | |------|-------------|----------|---------------| | id | pipeline id to describe | Yes | | + + +#### Flags + +| Name | Description | Required | Default Value | | grpc.address | address for serving the gRPC API | No | ":8084" | #### `--help` ```bash -$ conduit pipelines describe [ID] +$ conduit pipelines describe ID EXAMPLE: @@ -263,6 +268,13 @@ This command will describe the existing running connector. It requires having Conduit previously running. +#### Arguments + +| Name | Description | Required | Default Value | +|------|-------------|----------|---------------| +| id | connector id to describe | Yes | | + + #### Flags | Name | Description | Required | Default Value | @@ -302,6 +314,13 @@ This command will describe the configuration for that connector plugin. It does not require having Conduit previously running. +#### Arguments + +| Name | Description | Required | Default Value | +|------|-------------|----------|---------------| +| PLUGIN | connector plugin to describe | Yes | | + + #### `--help` ```bash @@ -346,7 +365,7 @@ It requires having Conduit previously running. | Name | Description | Required | Default Value | |------|-------------|----------|---------------| -| ID | processor ID | Yes | | +| id | processor ID | Yes | | #### `--help` @@ -386,7 +405,7 @@ It does not require having Conduit previously running. | Name | Description | Required | Default Value | |------|-------------|----------|---------------| -| PLUGIN | processor plugin name | Yes | | +| plugin | processor plugin name | Yes | | #### `--help` From 4e731eb67e9828d12047306696a2d7c93a0d13e4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ra=C3=BAl=20Barroso?= Date: Fri, 25 Oct 2024 19:17:09 +0200 Subject: [PATCH 21/21] =?UTF-8?q?fix=20empty=20lines=20=F0=9F=AB=A1?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/design-documents/20241024-conduit-cli.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/docs/design-documents/20241024-conduit-cli.md b/docs/design-documents/20241024-conduit-cli.md index 25780de4b..fcdea65c8 100644 --- a/docs/design-documents/20241024-conduit-cli.md +++ b/docs/design-documents/20241024-conduit-cli.md @@ -218,7 +218,6 @@ It requires the pipeline id as argument. |------|-------------|----------|---------------| | id | pipeline id to describe | Yes | | - #### Flags | Name | Description | Required | Default Value | @@ -274,7 +273,6 @@ It requires having Conduit previously running. |------|-------------|----------|---------------| | id | connector id to describe | Yes | | - #### Flags | Name | Description | Required | Default Value | @@ -320,7 +318,6 @@ It does not require having Conduit previously running. |------|-------------|----------|---------------| | PLUGIN | connector plugin to describe | Yes | | - #### `--help` ```bash