Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add batch size field for jira source #5348

Merged
merged 6 commits into from
Jan 23, 2025
Merged

add batch size field for jira source #5348

merged 6 commits into from
Jan 23, 2025

Conversation

Galactus22625
Copy link
Member

@Galactus22625 Galactus22625 commented Jan 22, 2025 •
edited
Loading

Description

Allow user to specify batch_size field in jira source.
Also works for all future saas_crawler sources.
Default value is 50 in jira source

cleanup unnecessary config fields

Check List

  • New functionality includes testing.
  • New functionality has a documentation issue. Please link to it in this PR.
    • New functionality has javadoc added
  • Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@Galactus22625 Galactus22625 marked this pull request as ready for review January 22, 2025 20:59
san81
san81 reviewed Jan 22, 2025
View reviewed changes
...ins/jira-source/src/main/java/org/opensearch/dataprepper/plugins/source/jira/JiraSource.java Outdated
@@ -55,7 +55,7 @@ public JiraSource(final PluginMetrics pluginMetrics,
final AcknowledgementSetManager acknowledgementSetManager,
Crawler crawler,
PluginExecutorServiceProvider executorServiceProvider) {
super(PLUGIN_NAME, pluginMetrics, jiraSourceConfig, pluginFactory, acknowledgementSetManager, crawler, executorServiceProvider);
super(PLUGIN_NAME, pluginMetrics, jiraSourceConfig, pluginFactory, acknowledgementSetManager, crawler, executorServiceProvider, jiraSourceConfig.getBatchSize());
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are already passing the entire source config object. I think, we don't need to pass the new additional argument?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh I see why you did it this way. You can add getBatchSize() method in CrawlerSourceConfig then you can avoid this additional argument.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will add to interface

@Galactus22625 Galactus22625 requested a review from san81 January 22, 2025 23:48
san81
san81 previously approved these changes Jan 23, 2025
View reviewed changes
Copy link
Collaborator

@san81 san81 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this addition

san81
san81 previously approved these changes Jan 23, 2025
View reviewed changes
sb2k16
sb2k16 reviewed Jan 23, 2025
View reviewed changes
...ra-source/src/main/java/org/opensearch/dataprepper/plugins/source/jira/JiraSourceConfig.java Outdated
@JsonProperty("filter")
private FilterConfig filterConfig;
@JsonProperty("batch_size")
private int batchSize = 50;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would suggest you add a private static final variable at the top of the class and set it there. Assign that variable here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

...src/test/java/org/opensearch/dataprepper/plugins/source/source_crawler/base/CrawlerTest.java Outdated
verify(coordinator, never()).createPartition(any(SaasSourcePartition.class));
}

@Test
void testCrawlWithNonEmptyList() throws NoSuchFieldException, IllegalAccessException {
Instant lastPollTime = Instant.ofEpochMilli(0);
List<ItemInfo> itemInfoList = new ArrayList<>();
int maxItemsPerPage = getMaxItemsPerPage();
int maxItemsPerPage = 50;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be moved out of the test and directly reference the default from the class.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

...src/test/java/org/opensearch/dataprepper/plugins/source/source_crawler/base/CrawlerTest.java Outdated
@@ -91,27 +91,50 @@ void testCrawlWithNonEmptyList() throws NoSuchFieldException, IllegalAccessExcep
void testCrawlWithMultiplePartitions() throws NoSuchFieldException, IllegalAccessException {
Instant lastPollTime = Instant.ofEpochMilli(0);
List<ItemInfo> itemInfoList = new ArrayList<>();
int maxItemsPerPage = getMaxItemsPerPage();
int maxItemsPerPage = 50;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be moved out of the test and directly reference the default from the class.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

...src/test/java/org/opensearch/dataprepper/plugins/source/source_crawler/base/CrawlerTest.java Outdated
}

@Test
void testCrawlWithNullItemsInList() throws NoSuchFieldException, IllegalAccessException {
Instant lastPollTime = Instant.ofEpochMilli(0);
List<ItemInfo> itemInfoList = new ArrayList<>();
int maxItemsPerPage = getMaxItemsPerPage();
int maxItemsPerPage = 50;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be moved out of the test and directly reference the default from the class.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

...src/test/java/org/opensearch/dataprepper/plugins/source/source_crawler/base/CrawlerTest.java Outdated
@@ -122,7 +145,8 @@ void testUpdatingPollTimeNullMetaData() {
ItemInfo testItem = createTestItemInfo("1");
itemInfoList.add(testItem);
when(client.listItems()).thenReturn(itemInfoList.iterator());
Instant updatedPollTime = crawler.crawl(lastPollTime, coordinator);
int maxItemsPerPage = 50;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be moved out of the test and directly reference the default from the class.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

...src/test/java/org/opensearch/dataprepper/plugins/source/source_crawler/base/CrawlerTest.java Outdated
@@ -133,17 +157,11 @@ void testUpdatedPollTimeNiCreatedLarger() {
ItemInfo testItem = createTestItemInfo("1");
itemInfoList.add(testItem);
when(client.listItems()).thenReturn(itemInfoList.iterator());
Instant updatedPollTime = crawler.crawl(lastPollTime, coordinator);
int maxItemsPerPage = 50;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be moved out of the test and directly reference the default from the class.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@Galactus22625
default batch size comments
65e1c46 
Signed-off-by: Maxwell Brown <[email protected]>
@san81 san81 merged commit 19d9dae into opensearch-project:main Jan 23, 2025
45 of 47 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sb2k16 sb2k16 sb2k16 approved these changes

@san81 san81 san81 approved these changes

@chenqi0805 chenqi0805 Awaiting requested review from chenqi0805 chenqi0805 is a code owner

@engechas engechas Awaiting requested review from engechas engechas is a code owner

@srikanthjg srikanthjg Awaiting requested review from srikanthjg srikanthjg is a code owner

@graytaylor0 graytaylor0 Awaiting requested review from graytaylor0 graytaylor0 is a code owner

@dinujoh dinujoh Awaiting requested review from dinujoh dinujoh is a code owner

@kkondaka kkondaka Awaiting requested review from kkondaka kkondaka is a code owner

@KarstenSchnitter KarstenSchnitter Awaiting requested review from KarstenSchnitter KarstenSchnitter is a code owner

@dlvenable dlvenable Awaiting requested review from dlvenable dlvenable is a code owner

@oeyh oeyh Awaiting requested review from oeyh oeyh is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@Galactus22625 @san81 @sb2k16