Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature] Split transform and move jar into connectors directory #7218

Merged
merged 6 commits into from
Aug 10, 2024
Merged

[Feature] Split transform and move jar into connectors directory #7218

merged 6 commits into from
Aug 10, 2024

Conversation

liunaijie
Copy link
Member

@liunaijie liunaijie commented Jul 17, 2024 •
edited
Loading

Purpose of this pull request

close #7171

Consider transform same as source/sink plugin.

Does this PR introduce any user-facing change?

How was this patch tested?

use the existing test

Check list

@github-actions github-actions bot added transform-v2 core SeaTunnel core module and removed api labels Aug 2, 2024
@liunaijie liunaijie marked this pull request as ready for review August 6, 2024 01:15
@liunaijie
Copy link
Member Author

@Hisoka-X PTAL

@Hisoka-X Hisoka-X added this to the 2.3.7 milestone Aug 7, 2024
Hisoka-X
Hisoka-X reviewed Aug 7, 2024
View reviewed changes
Copy link
Member

@Hisoka-X Hisoka-X left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @liunaijie !

seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/sql/SQLTransform.java Outdated
Comment on lines 74 to 79
List<String> sourceTableNames = config.get(CommonOptions.SOURCE_TABLE_NAME);
if (sourceTableNames != null && !sourceTableNames.isEmpty()) {
this.inputTableName = sourceTableNames.get(0);
} else {
this.inputTableName = catalogTable.getTableId().getTableName();
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should keep this at now. The logs displayed by the two different names will be slightly different.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/sql/SQLTransform.java
@@ -97,9 +80,9 @@ public String getPluginName() {
public void open() {
sqlEngine = SQLEngineFactory.getSQLEngine(engineType);
sqlEngine.init(
inputTableName,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

...forms-v2/src/main/java/org/apache/seatunnel/transform/common/AbstractSeaTunnelTransform.java
* limitations under the License.
*/

package org.apache.seatunnel.transform.common;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes we can remove this.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

seatunnel-dist/src/main/assembly/assembly-bin-ci.xml
<!-- SeaTunnel connectors -->
<dependencySet>
<useProjectArtifact>false</useProjectArtifact>
<useTransitiveDependencies>true</useTransitiveDependencies>
<unpack>false</unpack>
<includes>
<include>org.apache.seatunnel:connector-*:jar</include>
<include>org.apache.seatunnel:seatunnel-transforms-v2:jar</include>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PTAL. @hailin0 @EricJoy2048

Hisoka-X
Hisoka-X reviewed Aug 8, 2024
View reviewed changes
...-v2/src/main/java/org/apache/seatunnel/transform/common/AbstractCatalogSupportTransform.java Outdated
Comment on lines 34 to 35
protected String inputTableName;
protected SeaTunnelRowType inputRowType;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Please only keep inputTableName.
  2. We can move it to SQLTransform because only it use this field now.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@Hisoka-X Hisoka-X changed the title Feature/split transform [Feature] Split transform and move jar into connectors directory Aug 8, 2024
Hisoka-X
Hisoka-X approved these changes Aug 8, 2024
View reviewed changes
Copy link
Member

@Hisoka-X Hisoka-X left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks @liunaijie !

@hailin0
Copy link
Member

hailin0 commented Aug 9, 2024

Please upload a screenshot of the binary package

@liunaijie
Copy link
Member Author

Please upload a screenshot of the binary package

截屏2024-08-09 13 09 27

@hailin0 hailin0 merged commit d46cf16 into apache:dev Aug 10, 2024
6 checks passed
jovezhong added a commit to jovezhong/seatunnel that referenced this pull request Aug 12, 2024
@jovezhong
Squashed commit of the following:
e8e887b 
commit 9d161d2
Author: lizhenglei <[email protected]>
Date:   Mon Aug 12 14:03:21 2024 +0800

    [Fix] Fix http e2e case (apache#7356)

    * 1

    * fix

    * fix

    * fix

    * fix

    * fix

    ---------

    Co-authored-by: lizhenglei <[email protected]>

commit 2489f64
Author: hailin0 <[email protected]>
Date:   Mon Aug 12 13:36:15 2024 +0800

    [Improve][Connector] Add multi-table sink option check (apache#7360)

    * [Improve][Connector] Add multi-table sink option check

    * fix

commit fa34ac9
Author: Jia Fan <[email protected]>
Date:   Mon Aug 12 13:35:43 2024 +0800

    [Improve][API] Check catalog table fields name legal before send to downstream (apache#7358)

    * [Improve][API] Check catalog table fields name legal before send to downstream

    * update

commit 068c5e3
Author: Tyrantlucifer <[email protected]>
Date:   Mon Aug 12 12:55:41 2024 +0800

    [Core][Flink] refactor flink proxy source/sink (apache#7355)

commit e23e3ac
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Sun Aug 11 17:40:02 2024 +0800

    Bump org.apache.activemq:activemq-client (apache#7323)

    Bumps org.apache.activemq:activemq-client from 5.14.5 to 5.15.16.

commit d46cf16
Author: Jarvis <[email protected]>
Date:   Sat Aug 10 23:47:12 2024 +0800

    [Feature] Split transform and move jar into connectors directory (apache#7218)

commit 862e205
Author: Jia Fan <[email protected]>
Date:   Sat Aug 10 15:21:36 2024 +0800

    [Improve][Document] Change deprecated connector name in setup.md (apache#7366)

commit 2769ed5
Author: hailin0 <[email protected]>
Date:   Sat Aug 10 09:58:03 2024 +0800

    [Hotfix][Connector] Fix jdbc compile error (apache#7359)

commit 3130ae0
Author: xxsc0529 <[email protected]>
Date:   Fri Aug 9 20:02:06 2024 +0800

    [Fix][Connector-V2][OceanBase] Remove OceanBase catalog's dependency on mysql driver (apache#7311)

commit 9216627
Author: chaos <[email protected]>
Date:   Fri Aug 9 13:53:28 2024 +0800

    Update ConsoleSinkFactory.java (apache#7350)

    Add existing rules to the optionRule

commit dc3c239
Author: dailai <[email protected]>
Date:   Fri Aug 9 13:11:30 2024 +0800

    [Improve][Jdbc] Skip all index when auto create table to improve performance of write (apache#7288)

commit 16950a6
Author: Jia Fan <[email protected]>
Date:   Thu Aug 8 17:27:47 2024 +0800

    [Improve][Connector-V2] Improve doris error msg (apache#7343)

commit a18fca8
Author: Alex Ting <[email protected]>
Date:   Thu Aug 8 11:22:41 2024 +0800

    [Fix][Connector-tdengine] Fix sql exception and concurrentmodifyexception when connect to taos and read data

commit 064fcad
Author: lizhenglei <[email protected]>
Date:   Wed Aug 7 21:57:57 2024 +0800

    [Improve][Transform] Improve DynamicCompile transform (apache#7319)

commit 16eeb1c
Author: He Wang <[email protected]>
Date:   Wed Aug 7 14:54:46 2024 +0800

    [Improve][Jdbc] Remove MysqlType references in JdbcDialect (apache#7333)

commit 191d9e1
Author: ChunFuWu <[email protected]>
Date:   Wed Aug 7 14:53:51 2024 +0800

    [FIX][SFTP] Fix username parameter error in sftp sink document (apache#7334)

commit 764d8b0
Author: Jia Fan <[email protected]>
Date:   Wed Aug 7 12:46:30 2024 +0800

    [Improve][API] Make sure the table name in TablePath not be null (apache#7252)

commit 8916523
Author: hailin0 <[email protected]>
Date:   Wed Aug 7 12:46:12 2024 +0800

    [Hotfix][Connector] Fix kafka consumer log next startup offset (apache#7312)

commit 819c685
Author: hailin0 <[email protected]>
Date:   Wed Aug 7 12:45:54 2024 +0800

    [Improve][Jdbc] Merge user config primary key when create table (apache#7313)

commit 7c3cd99
Author: hailin0 <[email protected]>
Date:   Wed Aug 7 12:44:41 2024 +0800

    [Improve] Remove unused code (apache#7324)

commit 460e73e
Author: hailin0 <[email protected]>
Date:   Wed Aug 7 12:44:24 2024 +0800

    [E2E] Enable fakesource e2e of spark/flink (apache#7325)

commit f6a1e51
Author: hailin0 <[email protected]>
Date:   Wed Aug 7 12:44:13 2024 +0800

    [E2E] Enable JdbcPostgresIdentifierIT (apache#7326)

commit a12786b
Author: Carl-Zhou-CN <[email protected]>
Date:   Wed Aug 7 12:27:50 2024 +0800

    [DOC][Oss] fix document configuration is rectified when the oss is selected as the checkpoint base (apache#7332)

commit a4db64d
Author: zhangdonghao <[email protected]>
Date:   Wed Aug 7 11:52:19 2024 +0800

    [Improve][E2E] Support windows for the e2e of paimon (apache#7329)

commit 855254e
Author: Jia Fan <[email protected]>
Date:   Wed Aug 7 11:03:06 2024 +0800

    [Feature][Transform] Add LLM transform (apache#7303)

    * [Feature][Transform] Add LLM transform

    * update

    * update

    * retrigger

commit f0cefbe
Author: Arin <[email protected]>
Date:   Tue Aug 6 19:23:38 2024 +0530

    [Feature] [Activemq] Added activemq sink  (apache#7251)

commit bb2c912
Author: Guangdong Liu <[email protected]>
Date:   Tue Aug 6 21:46:12 2024 +0800

    [Fix][Doris] Fix the abnormality of deleting data in CDC scenario. (apache#7315)

commit c94ea32
Author: Jia Fan <[email protected]>
Date:   Tue Aug 6 11:23:52 2024 +0800

    [Fix][Doc] Fix miss sink-options-placeholders.md in sidebars (apache#7310)

commit 73632ba
Author: hailin0 <[email protected]>
Date:   Tue Aug 6 10:08:45 2024 +0800

    [Hotfix][Zeta] Fix task cannot be stopped when system is busy (apache#7292)

commit 9d56cc3
Author: virvle <[email protected]>
Date:   Mon Aug 5 17:51:26 2024 +0800

    Update Oracle-CDC.md (apache#7285)

commit 4f120ff
Author: Jia Fan <[email protected]>
Date:   Mon Aug 5 14:34:01 2024 +0800

    [Improve] Update snapshot version to 2.3.7 (apache#7305)

commit 2fd4eec
Author: Jia Fan <[email protected]>
Date:   Mon Aug 5 11:35:02 2024 +0800

    [Fix][Doc] Fix hybrid cluster deployment document display error (apache#7306)

commit 82f5d8c
Author: hailin0 <[email protected]>
Date:   Sat Aug 3 12:02:19 2024 +0800

    [Improve][SQL-Transform] Remove escape identifier from output fields (apache#7297)

commit 34a6b8e
Author: Jast <[email protected]>
Date:   Wed Jul 31 14:43:11 2024 +0800

    [hotfix][connector-v2-hbase]fix and  optimize hbase source problem (apache#7148)

    * [hotfix][improve][doc]optimize connector hbase source

    * [doc]add dependent document

    * [doc]update dependent document

    * [improve]improve static use

    * [hotfix]add test case

    * [hotfix]add test case

    ---------

    Co-authored-by: Jia Fan <[email protected]>

commit 9df557c
Author: lizhenglei <[email protected]>
Date:   Tue Jul 30 21:37:04 2024 +0800

    [Improve][Transform] Improve DynamicCompile transform  (apache#7264)

commit b9acb57
Author: hailin0 <[email protected]>
Date:   Tue Jul 30 16:12:57 2024 +0800

    [Feature][Transforms] Support cast to bytes function of sql (apache#7284)

commit ec1c319
Author: Nothing. <[email protected]>
Date:   Mon Jul 29 21:53:01 2024 +0800

    [Docs]translate event-listener doc into chinese (apache#7274)

commit f012b2a
Author: dailai <[email protected]>
Date:   Mon Jul 29 21:51:59 2024 +0800

    [Improve][Connector-v2] Optimize the way of databases and tables are checked for existence (apache#7261)

commit 4897491
Author: 卢宗柱 <[email protected]>
Date:   Mon Jul 29 17:29:23 2024 +0800

    fix [Bug] Unable to create a source for identifier 'Iceberg'. apache#7182 (apache#7279)

    Co-authored-by: luzongzhu <[email protected]>

commit 21c4f52
Author: corgy-w <[email protected]>
Date:   Mon Jul 29 17:28:10 2024 +0800

    [Improve][Core] Improve base on plugin name of lookup strategy (apache#7278)

    * [bug][plugin-discovery] fix multi plugin discovery

    * [bug][plugin-discovery] optimize code

    ---------

    Co-authored-by: wangchao <[email protected]>

commit 3ccc6a8
Author: hailin0 <[email protected]>
Date:   Mon Jul 29 10:41:09 2024 +0800

    [Hotfix][Zeta] Fix task cannot be stopped when system is busy (apache#7280)

commit d886495
Author: Guangdong Liu <[email protected]>
Date:   Fri Jul 26 10:45:32 2024 +0800

    [Fix][Connector-V2] Fix doris TRANSFER_ENCODING header error (apache#7267)

commit 31e59cd
Author: Xuzz <[email protected]>
Date:   Fri Jul 26 10:44:50 2024 +0800

    [Feature][Jdbc] Support hive compatibleMode add inceptor dialect (apache#7262)

commit d003bd8
Author: zhangdonghao <[email protected]>
Date:   Fri Jul 26 10:44:09 2024 +0800

    [Feature][Zeta] Added the metrics information of table statistics in multi-table mode (apache#7212)

commit 876d2f0
Author: Jia Fan <[email protected]>
Date:   Thu Jul 25 10:24:55 2024 +0800

    [Fix][Dist] Remove aws jar dependency in release package (apache#7257)

    * [Fix][Dist] Remove aws jar dependency in release package

    * [Fix][Dist] Remove aws jar dependency in release package

commit 0d08b20
Author: dailai <[email protected]>
Date:   Thu Jul 25 10:00:13 2024 +0800

    [Improve][Connector-v2] Optimize the count table rows for jdbc-oracle and oracle-cdc (apache#7248)

commit cc59499
Author: Jia Fan <[email protected]>
Date:   Tue Jul 23 14:27:07 2024 +0800

    [Improve][Core] Move MultiTableSink to seatunnel-api module (apache#7243)

    * [Improve][Core] Move MultiTableSink to seatunnel-api module

    * [Improve][Core] Move MultiTableSink to seatunnel-api module

commit fe0c477
Author: Eric <[email protected]>
Date:   Tue Jul 23 08:36:58 2024 +0800

    [Hotfix][Zeta] Fix taskgroup failed log lost (apache#7241)

commit b0fe432
Author: hawk9821 <[email protected]>
Date:   Mon Jul 22 16:12:12 2024 +0800

    The isNullable attribute is true when the primary key field in the Paimon table converts the Column object. apache#7231 (apache#7242)

commit 4b3af9b
Author: Guangdong Liu <[email protected]>
Date:   Mon Jul 22 13:04:19 2024 +0800

    [Improve][Doris Connector] Unified serialization method,Use RowToJsonConverter and TextSerializationSchema (apache#7229)

    * 1

    * 1

    * 1

    * Update seatunnel-connectors-v2/connector-doris/src/main/java/org/apache/seatunnel/connectors/doris/source/serialization/RowBatch.java

    Co-authored-by: Jia Fan <[email protected]>

    ---------

    Co-authored-by: gdliu3 <[email protected]>
    Co-authored-by: Jia Fan <[email protected]>

commit 4ec25f3
Author: Jia Fan <[email protected]>
Date:   Mon Jul 22 12:01:36 2024 +0800

    [Fix][Zeta] Fix release slot resource twice (apache#7236)

commit 7dc3fa8
Author: litiliu <[email protected]>
Date:   Mon Jul 22 11:41:33 2024 +0800

    [Fix][Connector kafka]Fix Kafka consumer stop fetching after TM node restarted (apache#7233)

commit 063f83c
Author: Guangdong Liu <[email protected]>
Date:   Mon Jul 22 10:40:58 2024 +0800

    [Improve][Restapi] Add ip and port to monitoring information (apache#7203)

commit c4ca741
Author: hailin0 <[email protected]>
Date:   Sat Jul 20 21:00:06 2024 +0800

    [Feature][Core] Support using upstream table placeholders in sink options and auto replacement (apache#7131)

    、

commit 821dfc8
Author: lizhenglei <[email protected]>
Date:   Sat Jul 20 20:25:50 2024 +0800

    [Feature][Transform] DynamicCompile add transform  (apache#7170)

commit 1e7c78d
Author: hilo <[email protected]>
Date:   Fri Jul 19 19:44:18 2024 +0800

    [Improve][Doc] Add IMap OSS config jars into document (apache#7029)

commit d39ad93
Author: Jia Fan <[email protected]>
Date:   Fri Jul 19 19:42:03 2024 +0800

    [Fix][Zeta] Fix hybrid deployment can not get worker when init (apache#7235)

commit 2058f5d
Author: ChunFuWu <[email protected]>
Date:   Fri Jul 19 14:21:23 2024 +0800

    [Chore][NOTICE] Correct year in NOTICE file (apache#7232)

commit 6e33a97
Author: Jia Fan <[email protected]>
Date:   Thu Jul 18 23:03:27 2024 +0800

    [Fix] Fix Hana type converter decimal scale is 0 convert to int error (apache#7167)

commit d1b4a7f
Author: Jia Fan <[email protected]>
Date:   Thu Jul 18 15:31:30 2024 +0800

    [Improve][Zeta] Add check for submit duplicate job id (apache#7021)

commit e44e8b9
Author: hailin0 <[email protected]>
Date:   Thu Jul 18 15:30:49 2024 +0800

    [Improve][Jdbc] Support write unicode text into sqlserver (apache#7159)

commit f40f11a
Author: tcodehuber <[email protected]>
Date:   Thu Jul 18 00:03:33 2024 +0800

    [Docs] Update engine related docs info (apache#7228)
@liunaijie liunaijie deleted the feature/split_transform branch August 15, 2024 00:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hailin0 hailin0 hailin0 approved these changes

@Hisoka-X Hisoka-X Hisoka-X approved these changes

Assignees
No one assigned
Labels
approved core SeaTunnel core module e2e reviewed transform-v2 Zeta
Projects
None yet
Milestone
2.3.7
Development

Successfully merging this pull request may close these issues.

[Feature][Transform] Move transform from seatunnel-starter.jar to transform directory
3 participants
@liunaijie @hailin0 @Hisoka-X