DP-1602 Add data pool export / batch import commands #133

IvanGandacov · 2023-08-01T09:51:18Z

Description

Ticket: https://celonis.atlassian.net/browse/DP-1602

Added data pool batch push support
Added data pool export version 2, that's compatible with the data pool batch push.
Updated the documentation.

As already discussed in our meeting, the implementation will be adjusted to be more user-friendly. (ticket created: https://celonis.atlassian.net/browse/DP-1576)
For now, this request should be good for the Batch Import UI.

Import Data Pool

node content-cli.js import data-pools --jsonFile ./initial.json --profile dev1

info:    Data Pools batch import succeeded!
info:    Batch import report: 
{
    "dataModelIdMappings": {
        "14ded083-82de-4ca1-80a4-9d1b8d9d8fd2": "de7f22a7-41cf-45a7-ac53-6c77afe8cafd",
        "ea0ebb32-5701-46e3-8015-ae04412a2e80": "b31fde18-ba3a-48a0-98b2-3e53add083db"
    }
}

node content-cli.js import data-pools --jsonFile ./initial.json --profile dev1 --outputToJsonFile

info:    Data Pools batch import succeeded!
info:    Batch import report file: batch_import_report_1cf4d888-2ce2-4c9d-ae50-57f84c8a9a9d.json

Export Data Pool

node content-cli.js export data-pool --id 850728cc-c679-4925-954a-87fb39abb12b --profile local 

info:    Data Pools export succeeded!
info:    Exported Data Pool: 
{
    "schemaVersion": "3.0.0",
    "jobs": [
        {
            "id": "33530cfb-835f-42fb-.....
.....

node content-cli.js export data-pool --id 850728cc-c679-4925-954a-87fb39abb12b --profile local --outputToJsonFile

info:    Data Pools export succeeded!
info:    Export file: data_pool_850728cc-c679-4925-954a-87fb39abb12b.json

Checklist

I have self-reviewed this PR
I have tested the change and proved that it works in different scenarios
I have updated docs if needed

AlbinHoxha · 2023-08-01T13:47:04Z

src/content/factory/data-pool-manager.factory.ts

+    }
+
+    public createPullManager(id: string, filename: string, pullVersion: number): DataPoolManager {
+        const dataPoolManager = this.createBaseManager(id, filename);


nit: would we want to append ".json" if the user did not type it?

Thank you. Could be a way, but I don't see this as an absolute necessity for now. If we get customer requests on this maybe we could add it later.

src/content-cli-push.ts

fisgeci · 2023-08-01T14:24:20Z

Also moving forward we are trying to migrate to a different architecture for the content-cli. We are using services to implement the functionality of each command. Could you migrate only the changes for batch push and pull to this approach?
It would be similar to this case.

IvanGandacov · 2023-08-02T15:23:49Z

Also moving forward we are trying to migrate to a different architecture for the content-cli. We are using services to implement the functionality of each command. Could you migrate only the changes for batch push and pull to this approach? It would be similar to this case.

I adjusted according to the new architecture. 👍
Please see the updated description: #133 (comment)

IvanGandacov · 2023-08-02T15:27:50Z

src/services/file-service.ts

@@ -19,14 +18,22 @@ export class FileService {
    }

    public readManifestFile(importedFileName: string): Promise<ManifestNodeTransport[]> {
-        const manifest: ManifestNodeTransport[] = YAML.parse(fs.readFileSync(path.resolve(importedFileName + "/manifest.yml"), { encoding: "utf-8" }));
+        const manifest: ManifestNodeTransport[] = YAML.parse(


just prettifying

IvanGandacov · 2023-08-02T15:28:14Z

src/content-cli-import.ts

    public static packages(program: CommanderStatic): CommanderStatic {
        program
            .command("packages")
            .description("Command to import all given packages")
            .option("-p, --profile <profile>", "Profile which you want to use to list packages")
-            .option("--spaceMappings <spaceMappings...>", "List of mappings for importing packages to different target spaces. Mappings should follow format 'packageKey:targetSpaceKey'")
+            .option(


just prettifying

IvanGandacov · 2023-08-02T15:28:42Z

src/services/http-client-service.v2.ts

-            request.post(this.resolveUrl(url) + "?" + querystring.stringify(parameters), this.makeOptions(contextService.getContext().profile, body), (err, res) => {
-                this.handleResponse(res, resolve, reject);
-            });
+            request.post(


just prettifying

IvanGandacov · 2023-08-02T15:28:44Z

src/services/http-client-service.v2.ts

-            request.get(this.resolveUrl(url), this.makeOptions(contextService.getContext().profile, null), (err, res) => {
-                this.handleResponse(res, resolve, reject);
-            });
+            request.get(


just prettifying

IvanGandacov · 2023-08-02T15:29:13Z

src/services/http-client-service.v2.ts

-            request.put(this.resolveUrl(url), this.makeOptionsJson(contextService.getContext().profile, JSON.stringify(body), "application/json;charset=utf-8"), (err, res) => {
-                this.handleResponse(res, resolve, reject);
-            });
+            request.put(


just prettifying

IvanGandacov · 2023-08-02T15:29:32Z

src/util/context-initializer.ts

+import * as commander from "commander";
+
+export class ContextInitializer {
+    public static async initContext(): Promise<void> {


copied into its own class to avoid repeating code.

fisgeci · 2023-08-03T08:20:27Z

src/content-cli-export.ts

+};
+
+ContextInitializer.initContext()
+    .then(loadCommands, loadCommands)


Are we calling the loadCommands twice here? I haven't seen this syntax before/

it's exactly as it was before, just instead of writting the () => { getCommands(); } twice, I put it into a variable.

IvanGandacov added 5 commits August 1, 2023 11:50

DP-1602 Add data pool batch push support

9b46e92

DP-1602 extend documentation

1180428

DP-1602 extend documentation

91474b1

DP-1602 small change

9134cce

DP-1602 small change

ba8c667

IvanGandacov marked this pull request as ready for review August 1, 2023 12:17

IvanGandacov requested review from GencBlakqoriP9 and a team as code owners August 1, 2023 12:17

AlbinHoxha reviewed Aug 1, 2023

View reviewed changes

AlbinHoxha previously approved these changes Aug 1, 2023

View reviewed changes

fisgeci reviewed Aug 1, 2023

View reviewed changes

src/content-cli-push.ts Outdated Show resolved Hide resolved

DP-931 Small refactoring

bb74ddf

IvanGandacov dismissed AlbinHoxha’s stale review via bb74ddf August 2, 2023 09:13

IvanGandacov marked this pull request as draft August 2, 2023 11:06

IvanGandacov added 11 commits August 2, 2023 15:50

DP-931 Add import data pools command and refactor

c882837

DP-931 cleanup

b856308

DP-931 add export data-pool command + cleanup + refactor

b8a027a

DP-931 cleanup

b3c7ca7

DP-931 cleanup

0a13a69

DP-931 cleanup

1c99107

DP-931 update DOCUMENTATION.md

92fc0d9

DP-931 merge master

cde8b39

DP-931 cleanup

685d551

DP-931 cleanup

593aefc

DP-931 cleanup

d3a1da4

IvanGandacov commented Aug 2, 2023

View reviewed changes

IvanGandacov requested review from fisgeci and AlbinHoxha August 2, 2023 15:30

IvanGandacov changed the title ~~DP-1602 Add data pool batch push support~~ DP-1602 Add data pool export / batch import support Aug 2, 2023

IvanGandacov changed the title ~~DP-1602 Add data pool export / batch import support~~ DP-1602 Add data pool export / batch import commands Aug 2, 2023

IvanGandacov marked this pull request as ready for review August 2, 2023 15:33

fisgeci previously approved these changes Aug 3, 2023

View reviewed changes

Bump to v0.4.3

5ab2929

IvanGandacov dismissed fisgeci’s stale review via 5ab2929 August 3, 2023 08:59

fisgeci approved these changes Aug 3, 2023

View reviewed changes

IvanGandacov merged commit a6d5f4f into master Aug 4, 2023
2 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DP-1602 Add data pool export / batch import commands #133

DP-1602 Add data pool export / batch import commands #133

IvanGandacov commented Aug 1, 2023 •

edited

Loading

AlbinHoxha Aug 1, 2023

IvanGandacov Aug 2, 2023

fisgeci commented Aug 1, 2023

IvanGandacov commented Aug 2, 2023 •

edited

Loading

IvanGandacov Aug 2, 2023

IvanGandacov Aug 2, 2023

IvanGandacov Aug 2, 2023

IvanGandacov Aug 2, 2023

IvanGandacov Aug 2, 2023

IvanGandacov Aug 2, 2023

fisgeci Aug 3, 2023

IvanGandacov Aug 3, 2023 •

edited

Loading

DP-1602 Add data pool export / batch import commands #133

DP-1602 Add data pool export / batch import commands #133

Conversation

IvanGandacov commented Aug 1, 2023 • edited Loading

Description

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fisgeci commented Aug 1, 2023

IvanGandacov commented Aug 2, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

IvanGandacov Aug 3, 2023 • edited Loading

Choose a reason for hiding this comment

IvanGandacov commented Aug 1, 2023 •

edited

Loading

IvanGandacov commented Aug 2, 2023 •

edited

Loading

IvanGandacov Aug 3, 2023 •

edited

Loading