0.3.1 release (#115)

* 1. add stream.log.enable in log-collector * remove unused file. * refactor JobLogRestfulApi, add lock for updating register info * 告警信息查询支持分页 * upd: 告警表格接口与交互改为分页展示 * fix not alarm bug * upd: 执行历史表格接口与交互改为分页展示 * 作业历史信息查询支持分页 * fix: 表格loading持续存在异常修复 * jobName空判断，长度最长支持2000 * Revert "upd: 执行历史表格接口与交互改为分页展示" * add support for skipping uris in job-audit-log module * fix comments in linkis.properties * refactor StreamisHeartbeatHandler * add ddl for table 'linkis_stream_register_info' * update version from '0.3.6-wedatasphere-SNAPSHOT' to '0.3.7-wedatasphere-SNAPSHOT' * fix bug in RegularUtil * upd: 执行历史表格接口与交互改为分页展示 * upd: 支持现有执行历史任务metajson查看与下载，接口待联调 * add cost time for audit log. * add metajson to StreamTask on job starting * fix val bug * fix jobconfig bug * upd: 支持job级资源下载，接口待联调 * upd: 优化启动确认检查逻辑，增加批量启动、停止相关交互 * job materials download * fix job materials download * add product name * fix path name problem * fix job material download * upd: 支持job级资源下载，接口联调 * add error codes solutions * change 'contains' to 'equals' in AuditLogAspect * Revert "change 'contains' to 'equals' in AuditLogAspect" This reverts commit 00fba0a. * change 'contains' to 'equals' in AuditLogAspect * fix some problems in material download * fix root path * fix execute statement * add costTimeMills in StreamAuditLogMapper.xml * add field 'job_startup_config' in linkis_stream_task upgrade_dml.sql * rename field 'job_startup_config' to 'job_start_config' in table linkis_stream_task * add log in DefaultStreamTaskService * upd: 审计日志信息增加接口耗时 * fix: metajson 弹窗文案修改 * fix error code and delete linkis error code * fix error code statement * fix error code solution in task table * change root path * fix execute problem * upd: 执行历史任务metajson查看与下载接口联调 * fix root path * fix problems in root path * fix: metajson 弹窗文案修改 * errorCode solution optimization * upd: 任务列表增加解决方案跳转链接 * product param optimization * upd: 批量停止弹窗footer按钮与批量启动保持统一 * upd: 批量启动失败时不显示任务完成弹窗 * fix error code match * fix String interception * fix some problems * fix error code default comments * 修改配置名称 * fix: 修改baseurl，解决环境异常 * remove fields 'metainfo' and 'worksapcename' in josn of MetaJsonInfo * fix error code default comments * fix error code default comments * fix metajson * fix metajson * add flink manager ec init extra params * fix value * fix AbstractLinkisJobStateFetcher * set productName cache * fix error msg * upd: 知识库URL改为文件配置式 * fix: 任务列表：重启操作检查失败时关闭所有浮窗而不是仅关闭检查浮窗导致卡死 * fix: 任务列表：批量修改标签后清空选中列表 * fix high available inspect * 1. remove unused resources. 2. change application.yml to application.properties in streamis-server * exclude snakeyaml in streamis-server and appconn * 1. remove unused resources. * 1. remove unused yml resources. * 1. exclude yml resources. * upd: 任务列表：增加对上传的控制 * fix: 批量停止成功后隐藏批量停止弹窗 * update version from '0.3.7-wedatasphere-SNAPSHOT' to '0.3.8-wedatasphere-SNAPSHOT' * add hook interface. * add gitignore * add token authentication for job upload * add token authentication for project files upload * add token authentication switch * fix npe in map put * upd: 添加水印 * upd: 水印出现页面调整 * add job template upload * upd: streamis子页面获取用户相关base信息存入localstorage * job template-change jobContent * job template-upload jobContent * upd: job配置与args参数是否支持编辑与上传逻辑一致由配置文件配置 * fix job template-upload * fix job template-upload * fix job template upload * fix job template upload in projectServer * add template in jobConf * add template in jobContent * fix template in jobContent * add param separates * fix job template in jobContent * add interface for getting username * add JobShutdownHook and DefaultEmptyJobShutdownHook * change type of timeoutMills from 'int' to 'long' * fix job template in job config * fix npe as project does not have a template * add jsonUtil * add download interface for job template * change task param * upd: 接入用户名接口，水印颜色加深 * upd: job 任务详情增加项目模板查看与下载 * add job shutdown hook * fix Object * add configuration in jobContent and jobConfig * upd: joblist增加是否弹出跳过hook异常弹窗相关逻辑 * upd: jobconfig、detail模板编辑权限调整，配置内容实用模板与自定义并集结果 * upd: job 任务详情模板调整 * chore: 更新依赖与配置文件 * fix jobTemplate download * register default hook 'emptyJobShutdownHook' * change registering default hook 'emptyJobShutdownHook' * fix error code match * upd: 调整joblist跳过hook异常逻辑 * fix: 解决editable获取的参数错误问题 * change wait radix on executing the hook * fix: 上传接口content-type设置调整 * fix: joblist 停止poptip改为挂载到body解决被table遮断的问题 * fix: 执行弹窗关闭时同步关闭停止信息弹窗 * fix: 字段错误修复 * fix: hasHook接口字段类型变更为数组，同步更新判断TrueFalse逻辑 * refactor executeJobShutdownHook * upd: jardetail 项目模板增加为null时的处理 * fix some problems * fix some problems * refactor executeJobShutdownHook * fix job template problems * fix jetty-server * upd: jobconfig增加模板与job标记 * upd: jobdetail 模板下载改为前端下载 * fix job shutdown hook * Revert "fix jetty-server" This reverts commit 27d23d3. * fix: jobconfig 模板字段问题修复 * fix job shutdown hook * fix job template * fix job template * fix job shutdown hook * remove cache pool in DefaultStreamTaskService * fix: jobconfig 模版字段问题修复 * upd: flinks参数添加重复相关处理逻辑，job不可重复，template可和job重复 * fix job template * fix:jobconfig 保存阶段自定义参数判重逻辑去除 * fix error code match * change the type of flink separate * fix 0.3.8 code review * 1. add cancel in StreamisJobShutdownHook 2. refactor msg in hook execution * fix 0.3.8 code review * fix 0.3.8 code review * fix 0.3.8 code review * fix 0.3.8 code review * add flink yarn log switch and log * 1. update version to '0.3.1-SNAPSHOT' 2. update linkis version to '1.5.0' * 1. update version to '0.3.1' 2. update linkis version to '1.5.0' 3. refactor pom.xml * fix dss version * update docs * update sql --------- Co-authored-by: gelxiogong <[email protected]> Co-authored-by: v_nikeyang <[email protected]> Co-authored-by: bert_cai <[email protected]>
WeBankFinTech · Jun 18, 2024 · ffa724e · ffa724e
1 parent 95f2218
commit ffa724e
Show file tree

Hide file tree

Showing 268 changed files with 36,081 additions and 934 deletions.
diff --git a/.gitignore b/.gitignore
@@ -28,3 +28,8 @@ streamis-appconn/target
 /streamis-jobmanager/streamis-job-log/job-log-collector/xspark-streamis-log-collector/target/
 /streamis-jobmanager/streamis-job-log/streamis-job-log-common/target/
 /streamis-jobmanager/streamis-job-log/streamis-job-log-server/target/
+/streamis-jobmanager/streamis-job-errorcode/target/
+/streamis-jobmanager/streamis-job-audit-log/target/
+
+**/._*
+**/.flattened-pom.xml
diff --git a/README-ZH.md b/README-ZH.md
@@ -52,7 +52,7 @@ Dimension节点、Transform节点、Sink节点 和 [Visualis](https://github.com
 
 ## 依赖的生态组件
 
-| 依赖的应用工具    | 描述                                                         | Streamis 兼容版本                               |
+| 依赖的应用工具    | 描述                                                         | Streamis 兼容版本 |
 |--------------|---------------------------------------------------------------|---------------------------------------------|
 | [DataSphereStudio](https://github.com/WeBankFinTech/DataSphereStudio) | 数据应用开发管理集成框架。以工作流式的图形化拖拽开发体验，将满足从数据交换、脱敏清洗、分析挖掘、质量检测、可视化展现、定时调度到数据输出应用等，数据应用开发全流程场景需求。 | &gt;= DSS1.1.2（已发布）                         |
 | [Linkis](https://github.com/apache/incubator-linkis) | 计算中间件 Apache Linkis，通过提供 REST/WebSocket/JDBC/SDK 等标准接口，上层应用可以方便地连接访问 MySQL/Spark/Hive/Presto/Flink 等底层引擎。 | &gt;= Linkis1.4.0（已发布） |

diff --git a/README.md b/README.md
@@ -52,7 +52,7 @@ The Dimension node, Transform node, Sink node and [Visualis](https://github.com/
 
 ## Depended ecosystems
 
-| Depended Component | Description | Streamis compatibility       |
+| Depended Component | Description | Streamis compatibility |
 | -------------- | -------------------------------------------------------------- |------------------------------|
 | [DataSphereStudio](https://github.com/WeBankFinTech/DataSphereStudio) | Data application development management framework. With a unified UI, the workflow-like graphical drag-and-drop development experience meets the entire lifecycle of data application development from data import, desensitization cleaning, data analysis, data mining, quality inspection, visualization, scheduling to data output applications, etc. | &gt;= DSS1.1.2 (Released)    |
 | [Linkis](https://github.com/apache/linkis) | Apache Linkis, builds a layer of computation middleware, by using standard interfaces such as REST/WS/JDBC provided by Linkis, the upper applications can easily access the underlying engines such as MySQL/Spark/Hive/Presto/Flink, etc. | &gt;= Linkis1.4.0 (Released) |
@@ -115,7 +115,7 @@ please refer to [Streamis Installation and Deployment Document](docs/en_US/0.3.0
 
 &nbsp; &nbsp; &nbsp; &nbsp;For any questions or suggestions, please kindly submit an [issue](https://github.com/WeBankFinTech/Streamis/issues).
 
-&nbsp; &nbsp; &nbsp; &nbsp;You can scan the QR code below to join our WeChat group to get more immediate response.
+&nbsp; &nbsp; &nbsp; &nbsp;You can scan the QR code below to join our WeChat and QQ group to get more immediate response.
 
 ![comminicate](images/zh_CN/readme/communication.png)
 

diff --git a/assembly/pom.xml b/assembly/pom.xml
@@ -20,7 +20,7 @@
     <parent>
         <artifactId>streamis</artifactId>
         <groupId>com.webank.wedatasphere.streamis</groupId>
-        <version>0.3.0</version>
+        <version>0.3.1</version>
     </parent>
     <modelVersion>4.0.0</modelVersion>
 

diff --git a/conf/config.sh b/conf/config.sh
@@ -40,6 +40,6 @@ GATEWAY_PORT=9001
 STREAMIS_SERVER_INSTALL_IP=127.0.0.1
 STREAMIS_SERVER_INSTALL_PORT=9400
 
-STREAMIS_VERSION=0.3.0
+STREAMIS_VERSION=0.3.1
 
 STREAMIS_FILE_NAME="STREAMIS-$STREAMIS_VERSION"
diff --git a/db/streamis_ddl.sql b/db/streamis_ddl.sql
@@ -222,6 +222,7 @@ CREATE TABLE `linkis_stream_task` (
   `submit_user` varchar(50) DEFAULT NULL,
   `linkis_job_id` varchar(200) DEFAULT NULL,
   `linkis_job_info` mediumtext,
+  `job_start_config` text CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL COMMENT 'streamis job startup config',
   PRIMARY KEY (`id`) USING BTREE
 ) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 ROW_FORMAT=COMPACT COMMENT='任务表';
 
@@ -252,9 +253,39 @@ CREATE TABLE `linkis_stream_project_privilege` (
 ) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COMMENT='项目权限表';
 
 
+DROP TABLE IF EXISTS `linkis_stream_error_code`;
+
+CREATE TABLE `linkis_stream_error_code` (
+  `id` bigint(20) NOT NULL AUTO_INCREMENT,
+  `error_code` varchar(50) COLLATE utf8_bin NOT NULL,
+  `error_desc` varchar(1024) COLLATE utf8_bin NOT NULL,
+  `error_regex` varchar(1024) COLLATE utf8_bin DEFAULT NULL,
+  `error_type` int(3) DEFAULT '0',
+  PRIMARY KEY (`id`)
+) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COLLATE=utf8_bin;
+
+DROP TABLE IF EXISTS `linkis_stream_audit_log`;
+
+CREATE TABLE `linkis_stream_audit_log` (
+  `id` bigint(20) NOT NULL AUTO_INCREMENT,
+  `user` varchar(100) COLLATE utf8_bin DEFAULT NULL,
+  `proxy_user` varchar(100) COLLATE utf8_bin DEFAULT NULL,
+  `api_name` varchar(100) COLLATE utf8_bin DEFAULT NULL,
+  `api_desc` varchar(100) COLLATE utf8_bin DEFAULT NULL,
+  `api_type` varchar(100) COLLATE utf8_bin DEFAULT NULL,
+  `operate_time` datetime DEFAULT NULL,
+  `input_parameters` text COLLATE utf8_bin,
+  `output_parameters` text COLLATE utf8_bin,
+  `project_name` varchar(100) COLLATE utf8_bin DEFAULT NULL,
+  `client_ip` varchar(100) COLLATE utf8_bin DEFAULT NULL,
+  `cost_time_mills` bigint(20) NULL,
+  PRIMARY KEY (`id`)
+) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8 COLLATE=utf8_bin;
+
+
 ALTER TABLE `linkis_stream_project` ADD create_time datetime DEFAULT NULL;
 ALTER TABLE `linkis_stream_project` ADD last_update_by varchar(50) DEFAULT NULL;
 ALTER TABLE `linkis_stream_project` ADD last_update_time datetime DEFAULT NULL;
 ALTER TABLE `linkis_stream_project` ADD is_deleted tinyint unsigned DEFAULT 0;
-
+ALTER TABLE linkis_stream_project_files ADD source varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NULL;
 SET FOREIGN_KEY_CHECKS = 1;