Skip to content

基于阿里Datax改版web datax ,支持管理平台与restful风格API

Notifications You must be signed in to change notification settings

xk11961677/skycloud-datax

Folders and files

NameName
Last commit message
Last commit date

Latest commit

54d4174 · Nov 4, 2019

History

10 Commits
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Nov 4, 2019
Nov 4, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Aug 28, 2019
Nov 4, 2019
Aug 28, 2019
Aug 28, 2019

Repository files navigation

ETL WEB 版本

使用方式

1. 在父工程目录下使用maven打包

 mvn -U clean package assembly:assembly -Dmaven.test.skip=true 

2. 在打包完成的target目录下进入datax-web,可以看到skycloud-datax-web-0.0.1-SNAPSHOT

cd  datax/datax/plugin/web

3. 运行启动命令

 java  -Ddatax.home=当前目录/target/datax/datax  -jar skycloud-datax-web-0.0.1-SNAPSHOT.jar

 或者 通过启动脚本启动
 打完包后的 bin/start.sh

需要配上环境变量-Ddatax.home,此处参照上述配置mvn打包后的目录即可

4. 测试

前提条件:

系统信息:

图1

(数据库类型:mysql 数据大小: 561M 数据条数:500W 数据格式如下:)

图2

相同数据库实例(不同库数据迁移)

图3

详细日志请点我

job配置
{
  "job": {
    "setting": {
      "speed": {
        "channel": 3
      },
      "errorLimit": {
        "record": 0,
        "percentage": 0.02
      }
    },
    "content": [
      {
        "reader": {
          "name": "mysqlreader",
          "parameter": {
            "username": "root",
            "password": "123456",
            "column": [
              "id",
              "Name"
            ],
            "splitPk": "id",
            "connection": [
              {
                "table": [
                  "appnamespace"
                ],
                "jdbcUrl": [
                  "jdbc:mysql://127.0.0.1:3306/datax_test"
                ]
              }
            ]
          }
        },
        "writer": {
          "name": "mysqlwriter",
          "parameter": {
            "writeMode": "insert",
            "username": "root",
            "password": "123456",
            "column": [
              "id",
              "Name"
            ],
            "session": [
              "set session sql_mode='ANSI'"
            ],
            "preSql": [
              "delete from appnamespace"
            ],
            "connection": [
              {
                "jdbcUrl": "jdbc:mysql://127.0.0.1:3306/datax_test1",
                "table": [
                  "appnamespace"
                ]
              }
            ]
          }
        }
      }
    ]
  }
}

5. 平台界面展示

platform platform platform

6. 来源(20190705)

https://github.com/alibaba/DataX/

FAQ

  1. skycloud-base-config.jar 此jar包在skycloud-base项目中,配置中心(apollo) 如不使用配置中心,请删除此jar包与application.yml,并将 doc/application.properties拷贝到项目资源目录下 或者使用doc/settings.xml maven配置下载

About

基于阿里Datax改版web datax ,支持管理平台与restful风格API

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages