Skip to content

Exchangis 1.0.0-rc1 Release Notes
Compare
Choose a tag to compare
@FinalTarget FinalTarget released this 15 Jun 18:36
· 49 commits to dev-1.0.0-rc since this release
release-1.0.0-rc1
5aa1f45

The Exchangis-1.0.0-RC release includes all Project Exchangis-1.0.0-RC.

This release is a new RC version of the Exchangis data synchronization task platform, which supports data synchronization between different types of data sources. The platform splits the data exchange process, abstracts concepts such as data sources, data exchange tasks, and task scheduling, and strengthens the linkage with linkis and DSS. It is a reconstructed version after version 0.X.

This version mainly includes the following contents:

  1. [Submit data synchronization job] Refactoring data synchronization job execution

  2. [Sub-task running status] View their various indicators information for the running or completed sub-jobs, so as to monitor the tasks in real time.

  3. [Get real-time log] Data synchronization is in progress/finished to display real-time logs of jobs and tasks, and troubleshoot problems accurately and quickly

  4. [Data source management module] Perform a series of management operations on data synchronization data sources

  5. [Synchronization History Module] Manage the historical information of the synchronization data tasks that have been completed or are being executed.

  6. [Project Management Module] Configure and manage data synchronization projects and tasks

  7. [Data synchronization job building module] Build the information of data synchronization job configuration and form a form that can be submitted to linkis

  8. Exchangis1.0 Appconn is connected to DSS, and supports the execution of data synchronization tasks in the form of workflow

abbreviation：

EJS: Exchangis Job Server
EJB:Exchangis Job Builder
EP:Exchangis Project
EDS:Exchangis Datasource Server
EXAPP:Exchangis Appconn

Feature：

  • [EJS][Exchangis-102] Added a new exchangis-job-server module, added data synchronization task scheduling class, execution class, Submission task class, new policy task addition, deletion, modification and inspection function, log monitoring function, synchronization history management function
  • [EJB][Exchangis-110] Add a new exchangis-job-builder module, add a construction management class for data synchronization jobs, and convert data synchronization jobs into a form that can be submitted to linkis.
  • [EP][Exchangis-88] Added a new exchangis-project module, which provides data synchronization project management services, and provides functions such as additions, deletions, and changes to data synchronization projects through the restful interface.
  • [EDS][Exchangis-88] Added a new exchangis-datasource-server module, which provides data source management services for linkis-datasouce, and provides functions such as adding, deleting, modifying and checking data sources, connection testing, and version management through the restful interface.
  • [EJS][Exchangis-160] Added management service for submitted data synchronization tasks, and provides functions such as log monitoring of historical tasks, viewing of indicator information, and aborting submitted tasks through restful interface.
  • [EJS][Exchangis-131] Refactored the execution process of data synchronization jobs to provide asynchronous execution, multi-tenancy and high availability of tasks.
  • [EXAPP][Exchangis-179] Added a new exchangis-appconn module to connect with DSS in the form of a third-party component through appconn, to implement data synchronization tasks in the form of workflow, and to synchronize addition, deletion, modification, and query tasks.

Enhancement：

  • [EJS][Exchangis-140] For a running/completed sub-job, various metrics can be displayed.
  • [Exchangis][Exchangis-284] Install deployment script optimization.
  • [Exchangis][Exchangis-284] Improve exchangis ddl, dml, configuration files.
  • [Exchangis][Exchangis-281] Add support for ISSUE template, Pull request template and github CI action.
  • [Exchangis][Exchangis-224] Upgrade exchange's dependency on linkis from 1.0.3 to 1.1.0.
  • [EJS][Exchangis-222] Refactor job-server module to fit new database table.
  • [EP][Exchangeis-189] Refactored data synchronization project management module.
  • [EJB][Exchangis-188] Enhance sqoop engine data synchronization job construction function.
  • [EXAPP][Exchangis-179] Refactor exchangis appconn module.
  • [EJS][Exchangis-140] Data synchronization in progress/end can display real-time logs of jobs/ and tasks.
  • [EJS,EP][Exchangis-167] Add job update interface, add job and project paging function, optimize restful layer.
  • [Exchangis][Exchangis-141] Add front-end exception handling for data synchronization task logs.
  • [EJS][Exchangis-143] Add interface to get quest log, get all historical quests and kill quests.
  • [EJS][Exchangis-147] Adjust the acquisition logic of the log component.
  • [EJS][Exchangis-150] Added the function of obtaining hive partition information.
  • [EDS][Exchangis-165] Implementation of client RPC and modification of restful class namespace.

Bug Fix：

  • [Exchangis-227] [EJS] Solve the problem that the job info service does not fit due to changes in the library table.

  • [Exchangis-185] [EJS-Execution] Avoid deadlock during netscape call in map calculation method: adjust scheduling task selection rules.

  • [Exchangeis-158] [EJS-Execution] Fix bugs updating progress, status and metrics.

  • [Exchangeis-165] [EDS] Implementation of client-side RPC and modification of namespaces for restful classes.

  • [Exchangis-292] [EJS] Fix the issue that hive's run_date time partition system variable fails to get.

  • [Exchangis-293] [EJS-Execution] Fix the problem that sqoop workflow node still executes in the background after canceling the task.

  • [Exchangis-282] [EJS-Execution] Fix the problem that the data source database table is changed after copying the subtask, causing the field mapping of the atomic task to be overwritten.

  • [Exchangis-140] [EJS-Execution] Fix the problem that submitting tasks to wtss for scheduling proxy users fails.

  • [Exchangis-241] [EJS] Fix missing subtask log.

  • [Exchangis-221] [EJS] Fix duplicate request issue when viewing subtasks.

  • [Exchangis-204] [EJB] When selecting the data source library table in the task configuration, add a search box for quick search.

  • [Exchangis-210] [EJS] Fix the problem that after clicking the execution ID to check the progress of the task, it is closed, and it switches to another page in time, and still pulls the progress of the task non-stop.

Contributors：

The release of EXCHANGIS 1.0.0-RC is inseparable from the contributors of the EXCHANGIS community, thanks to all the community contributors! Including but not limited to the following Contributors：wushengyeyouya、Dlimeng、Davidhua1996、mingfengwang、peacewong、Casion、xiaojie19852006、yuxin-No1、ryanqin01、lucaszhu2zgf、FinalTarget、Liveipool、gjy1043、

Assets 6
Loading