Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

limit parallel bulk export execution #9031

Conversation

arafubeatbox
Copy link
Contributor

@arafubeatbox arafubeatbox commented Aug 18, 2024

実装内容

  • PageBulkExportJobStreamManager を PageBulkExportJobManager に変更し、実行中の job 数を制限する queue 機能を追加。
  • PageBulkExportJobStreamManager のテストを記述

task

https://redmine.weseek.co.jp/issues/152623

Copy link

changeset-bot bot commented Aug 18, 2024

⚠️ No Changeset found

Latest commit: ef11a3c

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

Base automatically changed from feat/150418-151978-enable-restarting-bulk-export to feat/page-bulk-export August 26, 2024 11:53
@arafubeatbox arafubeatbox marked this pull request as ready for review August 26, 2024 13:05
let jobManager: PageBulkExportJobManager;

beforeAll(() => {
vi.spyOn(configManager, 'getConfig').mockReturnValue(3);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

これ雑すぎるね

beforeAll ではなく個別の it block 内ならギリギリ許容だが、beforeAll だと getConfig の別の引数で呼び出しをするテストを追加した時に意図しない動きをするだろう


describe('addJob', () => {
it('should add the job to jobsInProgress if under the parallelExecLimit', () => {
const job = { _id: 'job1' } as any;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as any 回避できないかな?
HydratedDocument や HasObjectId を使うなど

catch (error) {
expect(error).toBeInstanceOf(Error);
expect(destroySpy).toHaveBeenCalledWith(expect.any(Error));
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

toThrowError で確認する方が望ましいかと
https://vitest.dev/api/expect.html#tothrowerror

job3: { stream: undefined },
};
expect(jobManager.canExecuteNextJob()).toBe(false);
});
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

全ての it について、

// assign
...

// act
...

// assert
...

というコメントを入れた構成にしてください

const job = { _id: 'job1' } as any;
jobManager.addJob(job, { endpoint: '/test/endpoint' });
expect(jobManager.jobsInProgress[job._id.toString()]).toEqual({ stream: undefined });
expect(pageBulkExportServiceMock.executePageBulkExportJob).toHaveBeenCalledWith(job, { endpoint: '/test/endpoint' });
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

// assign// assert 両方で jobQueue.length を expect する行を入れてください
つまり、act 前後で期待する数になっているかどうかを確かめる

expect(jobManager.jobsInProgress[jobId]).toBeUndefined();
expect(jobManager.jobsInProgress[nextJob._id.toString()]).toEqual({ stream: undefined });
expect(pageBulkExportServiceMock.executePageBulkExportJob).toHaveBeenCalledWith(nextJob, undefined);
});
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

// assign// assert 両方で jobQueue.length を expect する行を入れてください
つまり、act 前後で期待する数になっているかどうかを確かめる

@arafubeatbox arafubeatbox merged commit 59b9233 into feat/page-bulk-export Aug 28, 2024
5 checks passed
@arafubeatbox arafubeatbox deleted the imprv/143599-152623-limit-parallel-bulk-export-execution branch August 28, 2024 15:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants