Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for source connector to control the poll interval from s3 #1293

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

asafglassbox
Copy link

Added support for controlling the poll interval on the source connector. The default poll interval is 1 second. This update allows the interval to be changed using the parameter connect.s3.poll.interval.ms (in millisecond)

@stheppi
Copy link
Contributor

stheppi commented Jul 4, 2024

@asafglassbox

Thank you for the PR.

I believe the delay should be applied only when there are no records to be returned. Otherwise, it would unnecessarily delay the delivery of available records. We recommend implementing a backoff strategy. Instead of using thread sleep, you could return an empty list when no records are available:

if there are no more records, set the task in "backoff mode". Track the currentTimestamp + backoff_interval, and for any subsequent poll method calls within this interval, just return empty records.

This approach prevents blocking the thread in the Connect execution thread pool. According to the documentation, the poll method "should block but return control to the caller regularly (by returning null)."

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants