-
Notifications
You must be signed in to change notification settings - Fork 110
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[DATA-2672] Refactor data capture and data sync #4161
Merged
nicksanford
merged 60 commits into
viamrobotics:main
from
nicksanford:refactor-data-and-sync2
Sep 26, 2024
Merged
[DATA-2672] Refactor data capture and data sync #4161
nicksanford
merged 60 commits into
viamrobotics:main
from
nicksanford:refactor-data-and-sync2
Sep 26, 2024
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nicksanford
force-pushed
the
refactor-data-and-sync2
branch
from
July 2, 2024 20:57
1090acd
to
9f35632
Compare
viambot
added
safe to test
This pull request is marked safe to test from a trusted zone
and removed
safe to test
This pull request is marked safe to test from a trusted zone
labels
Jul 2, 2024
nicksanford
force-pushed
the
refactor-data-and-sync2
branch
from
July 5, 2024 14:36
9f35632
to
af78673
Compare
viambot
added
safe to test
This pull request is marked safe to test from a trusted zone
and removed
safe to test
This pull request is marked safe to test from a trusted zone
labels
Jul 5, 2024
nicksanford
force-pushed
the
refactor-data-and-sync2
branch
from
July 10, 2024 21:49
af78673
to
a8f41e4
Compare
viambot
added
safe to test
This pull request is marked safe to test from a trusted zone
and removed
safe to test
This pull request is marked safe to test from a trusted zone
labels
Jul 10, 2024
nicksanford
force-pushed
the
refactor-data-and-sync2
branch
from
July 10, 2024 22:03
a8f41e4
to
753b65f
Compare
viambot
added
safe to test
This pull request is marked safe to test from a trusted zone
and removed
safe to test
This pull request is marked safe to test from a trusted zone
labels
Jul 10, 2024
nicksanford
force-pushed
the
refactor-data-and-sync2
branch
from
July 12, 2024 14:47
a12fed5
to
03e6342
Compare
viambot
added
safe to test
This pull request is marked safe to test from a trusted zone
and removed
safe to test
This pull request is marked safe to test from a trusted zone
labels
Jul 12, 2024
nicksanford
force-pushed
the
refactor-data-and-sync2
branch
from
September 26, 2024 21:57
5a20c35
to
03edc6f
Compare
viambot
added
safe to test
This pull request is marked safe to test from a trusted zone
and removed
safe to test
This pull request is marked safe to test from a trusted zone
labels
Sep 26, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Ticket
Also fixes the following flakey tests:
https://viam.atlassian.net/browse/DATA-2880
https://viam.atlassian.net/browse/RSDK-8249
https://viam.atlassian.net/browse/RSDK-7934
Also fixes pathological behavior around retrying uploading arbitrary files:
https://viam.atlassian.net/browse/DATA-3114
Also changes the max_sync_threads default from 1000 to
runtime.NumCPU()
:https://viam.atlassian.net/browse/DATA-3064
Also changes data collectors to only call the capture method from a single goroutine rather than spawning a new goroutine per capture interval:
https://viam.atlassian.net/browse/DATA-2771
For why see: https://viam.atlassian.net/browse/DATA-2771?focusedCommentId=34912
Also changes
sync_interval_mins
to a default of0.1
if unset:https://viam.atlassian.net/browse/DATA-2506
This PR refactors the builtin datamanager into datamanager.builtin.sync and datamanager.builtin.capture packages.
builtin is responsible for taking the deps & config from resource graph and validating that they are in a state where initialization / reconfiguration are possible (they are only impossible if new errors are introduced at compile time).
It also refactors tests to:
It also moves data collector functionality previously under the
builtin
data manager package into thedata
package, where the only users of that functionality were data collectors.It also changes the signature of the builtin datamanager constructor to allow us to test how the builtin datamanager behaves when it's connection with app:
Not managing the lifecycle of the connection with the app was the source of several bugs in the past few months, so investing the time to manage that connection appropriately seemed like an important goal of this refactor.
I'm splitting this PR into sub-PRs. Once each of these are merged, I'll rebase this PR on main & it will continue to shrink until there is nothing more we can split out of this PR.
Changes in this PR:
I believe all of these are fixing bugs / risks in the existing code:
Builtin:
sync_interval_mins
,file_last_modified_millis
,maximum_capture_file_size_bytes
,delete_every_nth_when_disk_full
,Sync:
Capture:
Linked PRs: