Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Streamis自动获取Checkpoint文件信息 #90

Open
1 of 2 tasks
yangwenzea opened this issue Jun 28, 2023 · 0 comments
Open
1 of 2 tasks

[Feature] Streamis自动获取Checkpoint文件信息 #90

yangwenzea opened this issue Jun 28, 2023 · 0 comments
Labels
enhancement New feature or request

Comments

@yangwenzea
Copy link
Contributor

Search before asking

  • I had searched in the issues and found no similar feature requirement.

Problem Description

Streamis维护的Flink应用作业在失败自动拉起和停止主动拉起的过程中,可能会发生作业状态(数据)丢失的现象,需要借助checkpoint文件来对作业的状态进行保存和恢复,所以要提供获取checkpoint文件信息的方法。

Description

  1. 使用Linkis的HttpClient,对应Linkis的linkis-storage-script-dev-server模块的FsRestfulApi入口类下的getDirFileTrees方法,构建相应的客户端Request和Response。Streamis侧应该通过配置文件等形式保存默认的checkpoint/savepoint文件根目录,与具体的任务名称做拼接后,得到该任务的checkpoint/savepoint根目录,并以此为请求参数入参,调用getDirFileTrees方法,获得该目录下的子目录树。

  2. 根据子目录树,拿到符合要求的子目录列表,取最新的子目录作为恢复目录,该子目录的信息即为最新的checkpoint文件信息。

Use case

No response

solutions

No response

Anything else

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!
@yangwenzea yangwenzea added the enhancement New feature or request label Jun 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant