Imprv/158220 158223 add page bulk export job cron #9461

arafubeatbox · 2024-12-01T03:56:25Z

実装内容

既存の PageBulkExportJobCron (終了/失敗した PageBulkExportJob の clean up を担当) を PageBulkExportJobCleanUpCron に変更
PageBulkExportService の PageBulkExportJob 実行部分を PageBulkExportJobCron として抽出
- 実行内容: 実行中のステータス (initializing, exporting, uploading) の PageBulkExportJob を探し出し、実行可能であれば次の実行ステップを開始する
  - PageBulkExportJob は古いものから順に n 件 (parallelExecLimit) に絞っている
- initializing, exporting, uploading のステップで実行する内容はそれぞれ個別ファイルに抽出している
  - initializing: create-page-snapshots-async.ts
  - exporting: export-pages-to-fs-async.ts
  - uploading: compress-and-upload-async.ts
- 10 秒おきに実行する。ただし、進行中の PageBulkExportJob がない場合は CheckPageBulkExportJobInProgressCronService によって停止される
- PageBulkExportService の内容の変更は Imprv/158220 158276 clean up page bulk export service #9460 で行っている
CheckPageBulkExportJobInProgressCronService を追加
- 3分おきに進行中の PageBulkExportJob があるかを確認し、ある場合は PageBulkExportJobCron を開始、ない場合は PageBulkExportJobCron を停止

task

https://redmine.weseek.co.jp/issues/158223

changeset-bot · 2024-12-01T03:56:28Z

⚠️ No Changeset found

Latest commit: 96d2dc1

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

arafubeatbox · 2024-12-01T08:46:48Z

apps/app/src/features/page-bulk-export/server/service/page-bulk-export-job-cron/index.ts

+   */
+  getTmpOutputDir(pageBulkExportJob: PageBulkExportJobDocument): string {
+    return `${this.tmpOutputRootDir}/${pageBulkExportJob._id}`;
+  }


PageBulkExportService からの抽出

arafubeatbox · 2024-12-01T08:47:42Z

apps/app/src/features/page-bulk-export/server/service/page-bulk-export-job-cron/index.ts

+   * @param action whether the export was successful
+   * @param pageBulkExportJob the page bulk export job
+   */
+  async notifyExportResultAndCleanUp(


PageBulkExportService からの抽出

arafubeatbox · 2024-12-01T08:48:03Z

apps/app/src/features/page-bulk-export/server/service/page-bulk-export-job-cron/index.ts

+   * - remove the temporal output directory
+   * - abort multipart upload
+   */
+  async cleanUpExportJobResources(pageBulkExportJob: PageBulkExportJobDocument, restarted = false) {


PageBulkExportService からの抽出

arafubeatbox · 2024-12-01T08:48:09Z

apps/app/src/features/page-bulk-export/server/service/page-bulk-export-job-cron/index.ts

+    });
+  }
+
+  private async notifyExportResult(


PageBulkExportService からの抽出

arafubeatbox · 2024-12-01T08:49:21Z

...page-bulk-export/server/service/page-bulk-export-job-cron/steps/compress-and-upload-async.ts

+import type { PageBulkExportJobDocument } from '../../../models/page-bulk-export-job';
+
+const logger = loggerFactory('growi:service:page-bulk-export-job-cron:compress-and-upload-async');
+


このファイル内の関数は全て PageBulkExportService からの抽出

arafubeatbox · 2024-12-01T08:49:31Z

...ge-bulk-export/server/service/page-bulk-export-job-cron/steps/create-page-snapshots-async.ts

+import type { PageBulkExportJobDocument } from '../../../models/page-bulk-export-job';
+import PageBulkExportJob from '../../../models/page-bulk-export-job';
+import PageBulkExportPageSnapshot from '../../../models/page-bulk-export-page-snapshot';
+


このファイル内の関数は全て PageBulkExportService からの抽出

arafubeatbox · 2024-12-01T08:49:39Z

.../page-bulk-export/server/service/page-bulk-export-job-cron/steps/export-pages-to-fs-async.ts

+import type { PageBulkExportJobDocument } from '../../../models/page-bulk-export-job';
+import type { PageBulkExportPageSnapshotDocument } from '../../../models/page-bulk-export-page-snapshot';
+import PageBulkExportPageSnapshot from '../../../models/page-bulk-export-page-snapshot';
+


このファイル内の関数は全て PageBulkExportService からの抽出

arafubeatbox · 2024-12-01T12:48:20Z

apps/app/src/features/page-bulk-export/server/service/page-bulk-export-job-cron/index.ts

+ * Manages cronjob which proceeds PageBulkExportJobs in progress.
+ * If PageBulkExportJob finishes the current step, the next step will be started on the next cron execution.
+ */
+class PageBulkExportJobCronService extends CronService implements IPageBulkExportJobCronService {


executeJob (cronjob の実行対象) で実行される proceedBulkExportJob がキモ。それ以外はほとんど PageBulkExportService からの抽出。

PageBulkExportJob が中断された際に実行中の stream も停止する必要があるため、streamInExecutionMemo を追加 (PageBulkExportService では PageBulkExportJobManager が担っていた)

yuki-takei · 2024-12-03T12:16:47Z

apps/app/src/features/page-bulk-export/server/service/page-bulk-export-job-cron/index.ts

+ */
+class PageBulkExportJobCronService extends CronService implements IPageBulkExportJobCronService {
+
+  crowi: any;


Crowi 型を参照させてください

yuki-takei · 2024-12-03T12:17:42Z

apps/app/src/features/page-bulk-export/server/service/page-bulk-export-job-cron/index.ts

+    this.crowi = crowi;
+    this.activityEvent = crowi.event('activity');
+    this.pageModel = mongoose.model<IPage, PageModel>('Page');
+    this.userModel = mongoose.model<IUser>('User');


mongoose.model() によるモデルオブジェクト取得は、それを利用するメソッドの冒頭で実行してください

arafubeatbox added 3 commits December 1, 2024 12:39

change PageBulkExportJobCron to PageBulkExportJobCleanUpCron

aab8371

add PageBulkExportJobCron

1622009

add restartFlag and statusOnPreviousCronExec to PageBulkExportJob

8f6ebbc

arafubeatbox mentioned this pull request Dec 1, 2024

Imprv/158220 158276 clean up page bulk export service #9460

Merged

arafubeatbox commented Dec 1, 2024

View reviewed changes

arafubeatbox added 3 commits December 1, 2024 20:11

add comments

27485da

add CheckPageBulkExportJobInProgressCronService

79f0706

add comments

dab56c9

arafubeatbox marked this pull request as ready for review December 1, 2024 12:39

arafubeatbox commented Dec 1, 2024

View reviewed changes

yuki-takei requested changes Dec 3, 2024

View reviewed changes

arafubeatbox added 2 commits December 3, 2024 21:37

add type for crowi and get mongoose model before queries

96d2dc1

remove swc dir

81992cf

arafubeatbox merged commit 7f8190c into feat/page-bulk-export Dec 3, 2024
10 checks passed

arafubeatbox deleted the imprv/158220-158223-add-page-bulk-export-job-cron branch December 3, 2024 12:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Imprv/158220 158223 add page bulk export job cron #9461

Imprv/158220 158223 add page bulk export job cron #9461

arafubeatbox commented Dec 1, 2024 •

edited

Loading

changeset-bot bot commented Dec 1, 2024 •

edited

Loading

arafubeatbox Dec 1, 2024

arafubeatbox Dec 1, 2024

arafubeatbox Dec 1, 2024

arafubeatbox Dec 1, 2024

arafubeatbox Dec 1, 2024

arafubeatbox Dec 1, 2024

arafubeatbox Dec 1, 2024

arafubeatbox Dec 1, 2024

yuki-takei Dec 3, 2024

yuki-takei Dec 3, 2024

		import type { PageBulkExportJobDocument } from '../../../models/page-bulk-export-job';

		const logger = loggerFactory('growi:service:page-bulk-export-job-cron:compress-and-upload-async');

Imprv/158220 158223 add page bulk export job cron #9461

Imprv/158220 158223 add page bulk export job cron #9461

Conversation

arafubeatbox commented Dec 1, 2024 • edited Loading

実装内容

task

changeset-bot bot commented Dec 1, 2024 • edited Loading

⚠️ No Changeset found

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arafubeatbox commented Dec 1, 2024 •

edited

Loading

changeset-bot bot commented Dec 1, 2024 •

edited

Loading