Skip to content

Commit

Permalink
[doc][taier-website] update doc
Browse files Browse the repository at this point in the history
  • Loading branch information
vainhope committed Mar 1, 2024
1 parent 5d54c0e commit cef36bb
Show file tree
Hide file tree
Showing 11 changed files with 139 additions and 19 deletions.
8 changes: 4 additions & 4 deletions k8s/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,11 +30,11 @@ k8s-n1 Ready,SchedulingDisabled master 165d v1.19.4

快速部署的组件

| 服务 | 版本 |
| ----------------- |--------|
| mysql(单机版) | 5.7 |
| 服务 | 版本 |
|---------------|--------|
| mysql(单机版) | 5.7 |
| zookeeper(集群) | 3.5.7 |
| taier 前后端 | master |
| Taier 前后端 | master |

## 2. 快速部署

Expand Down
4 changes: 2 additions & 2 deletions taier-ui/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@
<yarnVersion>${yarn.version}</yarnVersion>

<!-- If it is inconvenient to visit the official website, you can use the domestic Taobao mirror-->
<nodeDownloadRoot>https://npm.taobao.org/dist/</nodeDownloadRoot>
<nodeDownloadRoot>https://mirrors.aliyun.com/nodejs-release/</nodeDownloadRoot>
<yarnDownloadRoot>https://mirrors.huaweicloud.com/yarn/</yarnDownloadRoot>
</configuration>
</execution>
Expand All @@ -51,7 +51,7 @@
<goal>yarn</goal>
</goals>
<configuration>
<arguments>config set registry https://registry.npm.taobao.org</arguments>
<arguments>config set registry https://registry.npmmirror.com</arguments>
</configuration>
</execution>

Expand Down
2 changes: 1 addition & 1 deletion website/docs/expand/task.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ OCEANBASE_SQL(8, "OceanBaseSQL", EJobType.SQL.getType(), 4, EComponentType.OCEAN
:::

### 保存任务
在taier页面上配置好对应的属性参数和任务内容之后,点击保存即可
在Taier页面上配置好对应的属性参数和任务内容之后,点击保存即可
默认会对sqlText内容去除注释
:::tip
如果需要对任务保存做自定义的参数拼接或处理
Expand Down
4 changes: 0 additions & 4 deletions website/docs/functions/component/datax.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,6 @@ DataX 计算组件用于运行 DataX 脚本任务,DataX不依赖任务前置组
:::tip
DataX任务运行在Taier本地服务器,通过Taier来执行,并获取执行结果信息

```shell
python datax.py task.temp/task.json
```

:::

:::caution
Expand Down
16 changes: 13 additions & 3 deletions website/docs/functions/component/flink-on-standalone.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,16 @@ sidebar_label: Flink On Standalone

* flink官网下载 [release 1.12.7](https://flink.apache.org/downloads.html)

:::tip
如果不了解Flink Standalone 可以参考官网文档 [Flink Standalone](https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/deployment/resource-providers/standalone/overview/#standalone-cluster-reference)
:::
**2、chunjun的插件**

* 编译1.12版本的chunjun [master](https://github.com/DTStack/chunjun)插件,将生成的chunjun-dist 拷贝至 $FLINK_HOME/lib

:::tip
chunjun 有些依赖包依赖是provided,需要将缺省的jar放到$FLINK_HOME/lib
例如:在chunjun-core最新版本中将logback的依赖改为了provided, 需要将以下logback包放到$FLINK_HOME/lib logback-core-1.2.11.jar
logback-classic-1.2.11.jar
chunjun 有些依赖包依赖是provided,需要将缺省的jar放到$FLINK_HOME/lib
例如:在chunjun-core最新版本中将logback的依赖改为了provided,需要将以下logback包放到$FLINK_HOME/lib logback-core-1.2.11.jar logback-classic-1.2.11.jar
:::

**3、flink1.12 standalone服务参数**
Expand Down Expand Up @@ -78,6 +80,14 @@ lib/

```

部署完Flink Standalone 打开Flink Web页面
确认部署正常
1. 页面正常打开
2. chunjun的插件包 在`jobmanger`的log中显示正常加载

![Flink Standalone](../../../static/img/readme/flink-standalone.png)



## Standalone 控制台参数

Expand Down
2 changes: 1 addition & 1 deletion website/docs/functions/multi-cluster.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ sidebar_label: 集群配置

## 集群配置
1. 集群配置按照`公共组件`-->`资源调度组件`-->`存储组件`-->`计算组件`的顺序进行配置
2. 不同厂商Hadoop集群环境各不一样,遇到适配问题,可以尝试修改taier源码支持
2. 不同厂商Hadoop集群环境各不一样,遇到适配问题,可以尝试修改Taier源码支持

## 环境依赖
- [x] SFTP服务器
Expand Down
6 changes: 4 additions & 2 deletions website/docs/quickstart/build.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,9 @@ oracle 插件因为商业版本原因,官方没有提供 ojdbc 的驱动,开
```shell
$ yarn install
$ yarn run build
# or
```
或者使用
- ```shell
$ pnpm install
$ pnpm run build
```
Expand All @@ -60,6 +62,6 @@ oracle 插件因为商业版本原因,官方没有提供 ojdbc 的驱动,开
- taier-ui/dist

:::tip
编译前端代码,建议node版本为`v16.14.0`,包管理工具为`pnpm` / `yarn`
编译前端代码,建议node版本为`v16.16.0`,包管理工具为`pnpm` / `yarn`
源码中包含部分scala的代码,通过idea源码启动需要添加scala的sdk 建议scala版本为`2.11.1`
:::
66 changes: 64 additions & 2 deletions website/docs/quickstart/faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,55 @@ sidebar_label: 常见问题


## 常见问题
### 任务运行 直接报错
```java
com.dtstack.taier.common.exception.ClientAccessException: Client access exception.
at com.dtstack.taier.common.client.ClientCache.getClient(ClientCache.java:114)
at com.dtstack.taier.common.client.ClientOperator.judgeSlots(ClientOperator.java:155)
at com.dtstack.taier.scheduler.WorkerOperator.judgeSlots(WorkerOperator.java:93)
at com.dtstack.taier.scheduler.WorkerOperator$$FastClassBySpringCGLIB$$b285a6bd.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:769)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747)
at org.springframework.aop.framework.adapter.AfterReturningAdviceInterceptor.invoke(AfterReturningAdviceInterceptor.java:55)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:747)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689)
at com.dtstack.taier.scheduler.WorkerOperator$$EnhancerBySpringCGLIB$$9dd74da7.judgeSlots(<generated>)
at com.dtstack.taier.scheduler.jobdealer.JobSubmitDealer.submitEngineJob(JobSubmitDealer.java:340)
at com.dtstack.taier.scheduler.jobdealer.JobSubmitDealer.submitJob(JobSubmitDealer.java:329)
at com.dtstack.taier.scheduler.jobdealer.JobSubmitDealer.lambda$run$0(JobSubmitDealer.java:220)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```

:::tip
taier部署目录下没有worker-plugins
:::

### 新建任务下拉框为空 或者 没有数据同步任务类型
:::tip
任务下拉框为空 查看初始化SQL是否正常执行 `dict`表中type = 30 是否正常 不包含重复数据
没有数据同步任务类型 或者没有其他任务类型 确认前置组件是否正常配置 [组件任务依赖关心](./functions/multi-cluster.md)
:::


### 不依赖hadoop 可以实现异构数据源之间的数据同步吗
:::tip
可以本地部署`Flink Standalone``DataX` 通过数据同步任务或者DataX来实现同步
:::

### 数据同步版本可以用其他版本吗
:::tip
不行 Chunjun对接的版本为1.12_release Taier项目依赖的也为1.12 建议使用Flink版本1.12.7
:::

### 任务提交报错 提示找不到core-jar
```java
Expand Down Expand Up @@ -94,7 +143,7 @@ Spark组件`sparkSqlProxyPath` 配置的路径下上传对应依赖jar包
### 本地部署登录之后提示未登录
![login_error](../../static/img/example/login_error.png)
:::tip
本地通过访问http://localhost:8090/
本地通过访问http://ip:8090/
:::


Expand All @@ -113,6 +162,13 @@ com.dtstack.taier.datasource.api.exception.InitializeException: get classloader
确保taier部署环境存在同级目录datasource-plugins
:::

### 数据同步任务一直等待运行
:::tip
session 模式 确认 flink session是否拉起 可以参考github issue
standalone 默认 确认 standalone是否正常启动 组件参数配置是否正常
等待运行的信息可以在`taier.log`日志中查看
:::

### 数据同步任务运行class not found

![login_error](../../static/img/example/class_not_find.png)
Expand Down Expand Up @@ -153,7 +209,7 @@ chunjun的connector更换过之后,需要重启Taier和Flink Session
### Taier-UI编译失败

:::tip
手动通过npm等方式编译或者删除Taier-ui下node_modules目录再次编译
手动通过npm等方式编译或者删除Taier-ui下node_modules目录再次编译 npm版本建议v16.16
:::

### Taier等插件包编译失败
Expand Down Expand Up @@ -194,4 +250,10 @@ application.properties 配置hadoop.user.name=hdfs 重启Taier 更多参数参
### 任务提交显示资源不足
:::tip
确认集群的cpu、内存或flink slot数量是否足够
:::

### 如果访问页面提示空白
:::tip
![img.png](/img/readme/no_dist.png)
如果访问页面提示空白 原因为前端dist文件不存在 需要编译后在启动 参考本地启动文档
:::
50 changes: 50 additions & 0 deletions website/docs/quickstart/idea.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,11 +26,61 @@ $ build/mvn-build-plugins.sh
|-- datasource-plugins
```

- 构建前端项目文件
前端文件默认指向到data-develop模块下的/resource/static/ 下 需要构建出来才能正常访问页面
```java
public void addResourceHandlers(ResourceHandlerRegistry registry) {
registry.addResourceHandler("/static/**").addResourceLocations("classpath:/static/static/");
registry.addResourceHandler("/images/**").addResourceLocations("classpath:/static/images/");
registry.addResourceHandler("/assets/**").addResourceLocations("classpath:/static/assets/");
registry.addResourceHandler("swagger-ui.html").addResourceLocations(
"classpath:/META-INF/resources/");
registry.addResourceHandler("/webjars/**").addResourceLocations(
"classpath:/META-INF/resources/webjars/");
registry.addResourceHandler("/taier/**").addResourceLocations("classpath:/static/");
registry.addResourceHandler("/**").addResourceLocations("classpath:/static/");
super.addResourceHandlers(registry);
}
```

- 前端文件构建分为2种方式
1. 通过datadevelop.sh脚本构建
datadevelop.sh会构建前端脚本文件并复制到data-develop模块下的/resource/static/的目录下
```shell
$ build/mvn-build-datadevelop.sh
```

static目录结构
```shell
static/
├── assets
├── images
├── static
├── **.js
├── **.js
```
:::tip
如果datadevelop.sh脚本构建失败 请检查是否是node代理仓库无法访问导致
:::
2. 手动通过npm构建
参考 [源码编译](./quickstart/build.md)
编译完成之后 将taier-ui下`dist`目录拷贝至data-develop模块下的`/resource/`
`dist`文件目录重命名为`static`即可




- 在conf/application.properties配置好正确的Zookeeper、MySQL信息
- IDEA选择TaierApplication启动项目
![idea-run](/img/readme/idea-run.png)


:::tip
源码中包含部分scala的代码,通过idea源码启动需要添加scala的sdk 建议scala版本为`2.11.1`
Module Setting > Global Libraries > scala sdk
:::

:::tip
![img.png](/img/readme/no_dist.png)
如果访问页面提示空白 原因为前端文件不存在 需要编译后在启动
:::
Binary file added website/static/img/readme/flink-standalone.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added website/static/img/readme/no_dist.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit cef36bb

Please sign in to comment.