[Feature][Zeta engine] Added the metrics information of table statistics in multi-table mode #7088

hawk9821 · 2024-07-01T00:39:14Z

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to connector-v2
2. Update plugin-mapping.properties and add new connector information in it
3. Update the pom file of seatunnel-dist
Update the release-note.

…ics in multi-table mode apache#6959

Hisoka-X

Thanks @hawk9821 created this PR!

Hisoka-X · 2024-07-01T02:39:10Z

...nnel-engine-client/src/test/java/org/apache/seatunnel/engine/client/SeaTunnelClientTest.java

+                                                            .listJobStatus(true)
+                                                            .contains("FINISHED")));
+
+            String jobMetrics = jobClient.getJobMetrics(jobId);


how about directly assert jobMetrics string value instead of use code to check the value? This way more easier to know what's metrics look likes. eg:

Assertions.assertTrue(jobMetrics.contains(" .... the multi table metrics table1 read value = 1 table 2 read value = 2 "))

Hisoka-X · 2024-07-01T02:42:23Z

...erver/src/main/java/org/apache/seatunnel/engine/server/rest/RestHttpGetCommandProcessor.java

        try {
            JsonNode jobMetricsStr = new ObjectMapper().readTree(jobMetrics);
+            StreamSupport.stream(


please add test case to assert the metrics value use rest api in

seatunnel/seatunnel-e2e/seatunnel-engine-e2e/connector-seatunnel-e2e-base/src/test/java/org/apache/seatunnel/engine/e2e/ClusterSeaTunnelContainer.java

Line 447 in c645d92

.body("[" + i.get() + "].metrics.SourceReceivedCount", equalTo("100"))

The MultiTableMetricsIT test case has been added

Hisoka-X · 2024-07-01T02:46:56Z

...e-server/src/main/java/org/apache/seatunnel/engine/server/task/SeaTunnelSourceCollector.java

+            tableNames.forEach(
+                    tableName ->
+                            sourceReceivedCountPerTable.put(
+                                    TablePath.of(tableName).getTableName(),


why not direct use tableName? It should contains more information like database name and schema name.

Hisoka-X · 2024-07-01T02:49:57Z

...e-server/src/main/java/org/apache/seatunnel/engine/server/task/SeaTunnelSourceCollector.java

+                            sourceReceivedCountPerTable.put(
+                                    tableName,
+                                    metricsContext.counter(
+                                            SOURCE_RECEIVED_COUNT + "#" + tableName));


only put but not invoke inc? Seem like it will lose 1 row.

Hisoka-X · 2024-07-01T02:51:30Z

...engine-server/src/main/java/org/apache/seatunnel/engine/server/task/SourceSeaTunnelTask.java

+                tableNames =
+                        producedCatalogTables.stream()
+                                .map(CatalogTable::getTableId)
+                                .map(TableIdentifier::getTableName)
+                                .filter(StringUtils::isNotBlank)
+                                .collect(Collectors.toList());


ditto. Please use TablePath.

Suggested change

tableNames =

producedCatalogTables.stream()

.map(CatalogTable::getTableId)

.map(TableIdentifier::getTableName)

.filter(StringUtils::isNotBlank)

.collect(Collectors.toList());

tableNames =

producedCatalogTables.stream()

.map(CatalogTable::getTableId)

.map(TableIdentifier::toTablePath)

.collect(Collectors.toList());

Hisoka-X · 2024-07-01T02:52:08Z

...ine-server/src/main/java/org/apache/seatunnel/engine/server/task/flow/SinkFlowLifeCycle.java

+                    try {
+                        String tableId = ((SeaTunnelRow) record.getData()).getTableId();
+                        if (StringUtils.isNotBlank(tableId)) {
+                            String tableName = TablePath.of(tableId).getTableName();
+                            if (StringUtils.isNotBlank(tableName)) {
+                                Counter sinkTableCounter = sinkWriteCountPerTable.get(tableName);
+                                if (Objects.nonNull(sinkTableCounter)) {
+                                    sinkTableCounter.inc();
+                                } else {
+                                    sinkWriteCountPerTable.put(
+                                            tableName,
+                                            metricsContext.counter(
+                                                    SINK_WRITE_COUNT + "#" + tableName));
+                                }
+                            }
+                        }
+                    } catch (Exception e) {
+                        log.error("====================== {}", e.getMessage());
+                        throw new RuntimeException(e);
+                    }


Hisoka-X · 2024-07-01T02:55:55Z

seatunnel-engine/seatunnel-engine-server/pom.xml

@@ -33,6 +33,11 @@
            <artifactId>seatunnel-engine-core</artifactId>
            <version>${project.version}</version>
        </dependency>
+        <dependency>


I I suggest that we can move multitablesink package to seatunnel-common module instead of referencing connector-common. Because our connector is engine independent

move multitablesink package to seatunnel-common module, will the impact be a little big

Never mind. we have enough test case.

The classes in multitablesink package refer to the seateunel-api module , which when moved causes seatunnel-common and seatunnel-api loop dependencies.

…ics in multi-table mode apache#6959

Hisoka-X · 2024-07-11T07:44:49Z

Please resolve conflicts. Thanks.

Hisoka-X · 2024-07-11T08:07:28Z

.github/workflows/backend.yml

@@ -313,7 +313,7 @@ jobs:
      - name: run updated modules integration test (part-1)
        if: needs.changes.outputs.api == 'false' && needs.changes.outputs.it-modules != ''
        run: |
-          sub_modules=`python tools/update_modules_check/update_modules_check.py sub_update_it_module ${{needs.changes.outputs.it-modules}} 8 0`
+          sub_modules=`python tools/update_modules_check/update_modules_check.py sub_update_it_module ${{needs.changes.outputs.it-modules}} 10 0`


why add new update_modules_check module? I think 8 modules are enough for now.

The CI subtask is interrupted due to timeout.
https://github.com/hawk9821/seatunnel/actions/runs/9689364040/job/26756996209

…apache#7045)

…nt error (apache#7034)

… be load error (apache#7069)

…dule runs repeatedly apache#7090 (apache#7091)

…kafka apache#4415 (apache#4428)

… mode reader/writer support batch (apache#7087)

…apache#7076)

…to chinese (apache#7015)

…ics in multi-table mode apache#6959

Bumps [org.xerial.snappy:snappy-java](https://github.com/xerial/snappy-java) from 1.1.8.3 to 1.1.10.4. - [Release notes](https://github.com/xerial/snappy-java/releases) - [Commits](xerial/snappy-java@1.1.8.3...v1.1.10.4) --- updated-dependencies: - dependency-name: org.xerial.snappy:snappy-java dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…ache#7181) * [Improve] Improve error message when can not parse datetime value * update

…ics in multi-table mode apache#6959

hawk9821 added 4 commits June 26, 2024 17:46

[Feature][Zeta engine] Added the metrics information of table statist…

a15db44

…ics in multi-table mode apache#6959

[Feature][Zeta engine] Added the metrics information of table statist…

993bb8d

…ics in multi-table mode apache#6959

[Feature][Zeta engine] Added the metrics information of table statist…

1b7ad46

…ics in multi-table mode apache#6959

[Feature][Zeta engine] Added the metrics information of table statist…

891e2f5

…ics in multi-table mode apache#6959

Hisoka-X added feature New feature Zeta metrics labels Jul 1, 2024

Hisoka-X reviewed Jul 1, 2024

View reviewed changes

hawk9821 added 7 commits July 1, 2024 14:11

[Feature][Zeta engine] Added the metrics information of table statist…

0cdb1f4

…ics in multi-table mode apache#6959

[Feature][Zeta engine] Added the metrics information of table statist…

182bc48

…ics in multi-table mode apache#6959

[Feature][Zeta engine] Added the metrics information of table statist…

4e5624c

…ics in multi-table mode apache#6959

[Feature][Zeta engine] Added the metrics information of table statist…

3053810

…ics in multi-table mode apache#6959

[Feature][Zeta engine] Added the metrics information of table statist…

519433c

…ics in multi-table mode apache#6959

[Feature][Zeta engine] Added the metrics information of table statist…

ab976a1

…ics in multi-table mode apache#6959

[Feature][Zeta engine] Added the metrics information of table statist…

a8a806c

…ics in multi-table mode apache#6959

Hisoka-X reviewed Jul 11, 2024

View reviewed changes

Hisoka-X added the Waiting for code update label Jul 12, 2024

liunaijie and others added 12 commits July 13, 2024 09:45

[Feature][Zeta] Add tag to node used to filter worker when submit job (…

cac45fe

…apache#7045)

[Improve][Zeta] Fix JobMaster reset app classloader twice (apache#7063)

36d4fb6

[Fix][Connector-V2] Fix starrocks Content-Length header already prese…

4888b75

…nt error (apache#7034)

[Improve][Doc] Fix snowflake jdbc url with emoji (apache#7061)

f3efe20

[Hotfix][Hive Connector] Fix Hive hdfs-site.xml and hive-site.xml not…

805c36e

… be load error (apache#7069)

[Feature] [Github CI] connector-kafka-e2e , connector-rocketmq-e2e mo…

de96c28

…dule runs repeatedly apache#7090 (apache#7091)

[Doc] correct the document (apache#7099)

a9f2318

typo (apache#7100)

0ecc5d9

[Feature][connector][kafka] Support read Maxwell format message from …

79243a2

…kafka apache#4415 (apache#4428)

[Improve][Redis] Redis reader use scan cammnd instead of keys, single…

db209c2

… mode reader/writer support batch (apache#7087)

[Improve][E2E][Doris] Add no schema test case for connector-doris-e2e (…

d8644bd

…apache#7076)

[Improve] Fix plugin_config (apache#7104)

9eb7657

Hisoka-X and others added 5 commits July 13, 2024 09:47

[Improve][CI] Add milvus to label (apache#7186)

65bd3e3

[Docs] Update concept related docs info (apache#7184)

710cd5a

[Docs]translate clickhousefile,phoenix,rabbitmq,starrocks sink doc in…

03dc7ed

…to chinese (apache#7015)

[Feature][Zeta engine] Added the metrics information of table statist…

4085589

…ics in multi-table mode apache#6959

[Feature][Zeta engine] Added the metrics information of table statist…

9b57d41

…ics in multi-table mode apache#6959

github-actions bot added document dependencies Pull requests that update a dependency file CI&CD core SeaTunnel core module connectors-v2 Zeta Rest API e2e format api labels Jul 13, 2024

tcodehuber and others added 14 commits July 15, 2024 16:59

[Docs] Update transform about and faq related docs info (apache#7187)

7046e7a

[Docs] Update contribution and readme related docs info (apache#7188)

efcfbc8

Fix iceberg e2e image tag (apache#7189)

637f9a2

[Improve][Jdbc] Remove user info in catalog-table options (apache#7178)

1eaf04d

[Improve][Hive] Close resources when exception occurs (apache#7205)

c840f6b

[Improve] Improve error message when can not parse datetime value (ap…

ce79ac5

…ache#7181) * [Improve] Improve error message when can not parse datetime value * update

Update backend.yml (apache#7122)

0cfb2e6

[Feature][Zeta engine] Added the metrics information of table statist…

556d772

…ics in multi-table mode apache#6959

[Feature][Zeta engine] Added the metrics information of table statist…

a3f2266

…ics in multi-table mode apache#6959

[Feature][RestAPI] overview support tag filter (apache#7173)

02a81da

[Feature][Zeta engine] Added the metrics information of table statist…

242cc36

…ics in multi-table mode apache#6959

[Feature][Zeta engine] Added the metrics information of table statist…

f971e89

…ics in multi-table mode apache#6959

[Feature][Zeta engine] Added the metrics information of table statist…

573488f

…ics in multi-table mode apache#6959

hawk9821 mentioned this pull request Jul 16, 2024

[Feature][Zeta] Added the metrics information of table statistics in multi-table mode #7212

Merged

4 tasks

hawk9821 closed this Jul 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature][Zeta engine] Added the metrics information of table statistics in multi-table mode #7088

[Feature][Zeta engine] Added the metrics information of table statistics in multi-table mode #7088

hawk9821 commented Jul 1, 2024

Hisoka-X left a comment

Hisoka-X Jul 1, 2024

hawk9821 Jul 10, 2024

Hisoka-X Jul 1, 2024

hawk9821 Jul 11, 2024

Hisoka-X Jul 1, 2024

Hisoka-X Jul 1, 2024

hawk9821 Jul 10, 2024

Hisoka-X Jul 1, 2024

hawk9821 Jul 10, 2024

Hisoka-X Jul 1, 2024

Hisoka-X Jul 1, 2024

hawk9821 Jul 10, 2024

Hisoka-X Jul 11, 2024

hawk9821 Jul 12, 2024

Hisoka-X commented Jul 11, 2024

Hisoka-X Jul 11, 2024

hawk9821 Jul 12, 2024

[Feature][Zeta engine] Added the metrics information of table statistics in multi-table mode #7088

[Feature][Zeta engine] Added the metrics information of table statistics in multi-table mode #7088

Conversation

hawk9821 commented Jul 1, 2024

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

Hisoka-X left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Hisoka-X commented Jul 11, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment