Implement State #29
Labels
enhancement
New feature or request
Comments
|
Yes that would be much better, even better if we had incremental replication setup. Hard coding the last 7 days isn't the best :/ Not sure what the limits are to history and how that works with Google Ads specifically. |
Closed
visch
changed the title
Merged
|
We don't have State / Incremental replication but we do have History available now which solves the core issue! I'll leave this open but it's now just for implementing the State functionality of Singer |
visch
changed the title
guilherme-maioli
pushed a commit
to dadosfera/tap-googleads
that referenced
this issue
Dec 11, 2024
* Revert removal of metric compatibility removal * Whitespace cleanup
guilherme-maioli
pushed a commit
to dadosfera/tap-googleads
that referenced
this issue
Dec 11, 2024
* add conversion window test * add conversion window test * wip updated tests to worka with currently syncing dev branch [skip ci] * Revert removal of metric compatibility removal (AutoIDM#29) * Revert removal of metric compatibility removal * Whitespace cleanup * Add currently syncing (AutoIDM#24) * Sync all customers for a given stream * Add logging to see when we retry requests * Update currently_syncing with customerId too. Write state as soon as we update it * Add the customerId to the bookmark keys * Add shuffle for customerId and tap_stream_id; add shuffle unit tests * Bug fix for when currently_syncing is null * Fix exception handling typeError * Fix none cases for currently_syncing * Fix currently_syncing to write a tuple we can read in later * Add get_customer_ids so we can use it in the tests * Fix manipulated_state to account for customer_ids * Update assertion for currently_syncing * Fix currently syncing assertion * Move bookmark access into Full Table assertions section Full Table doesn't need the "stream_name and customer id" key logic * Remove duplicate assertion * Revert 6db016e7ec29c2b00973b671c1efdf9451aca9c2 * Update bookmark to read stream->customer->replication_key * Update tap to write bookmarks as stream->customer->replication_key * Update manipulated state to nest stream->customer->replication_key * Run bookmark assertions for every customer * Fix dict comprehension typo * Fix conflict with main * Remove `get_state_key` again, use env var instead of hardcoded value * Add missing dependency * Move currently-syncing-null-out to the end of sync to prevent gaps * Sort selected_streams and customers to guarantee consistency across runs * Don't let the tap write (None, None) * Sort selected_streams and customers effectively * Update currently_syncing test assertions * Add sort functions for streams and customers * Update `shuffle` to handle a missing value * Update unit tests to use sort_function, add a test for shuffling streams * Add end date (AutoIDM#28) * Add optional end date, add unit tests Co-authored-by: Andy Lu <[email protected]> * Test functions can't be named run_test apparently * Rename do_thing * Extract `get_queries_from_sync` as a function * Remove unused variable * Refactor tests to be more explicit * Mock singer.utils.now to return a specific date Co-authored-by: Andy Lu <[email protected]> * add conversion_window test * fixed conversion window unittests, bug removed Co-authored-by: dylan-stitch <[email protected]> Co-authored-by: Andy Lu <[email protected]> Co-authored-by: kspeer <[email protected]> * Bump to v0.2.0, update changelog (AutoIDM#31) * Bump to v0.2.0, update changelog * Add link for this PR, fix link syntax * Update changelog format * expanded conversion window testing for error case, BUG linked * parallelism 8 -> 12 * added unittest for start date within conversion window Co-authored-by: kspeer <[email protected]> Co-authored-by: Dylan <[email protected]> Co-authored-by: dylan-stitch <[email protected]> Co-authored-by: Andy Lu <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This is a new feature request. It would be good to be able to control the number of days pulled from Google Ads in certain taps. Most of time this is controlled in the Google Ads query with something like
WHERE segments.date DURING LAST_7_DAYS. Changing to asegments.date BETWEEN '2019-01-01' AND '2019-01-31'strategy controlled by the configuration parametersstart_dateandend_datewould be ideal.The text was updated successfully, but these errors were encountered: