Skip to content

Commit

Permalink
Use latest version (#6)
Browse files Browse the repository at this point in the history
* Bump to v1.0.0, update changelog (singer-io#45)

* Qa/future testing (singer-io#48)

* documented reasons for untested streams

* list high level scenarios for manual qa checks

* remove stitch specific pr template

Co-authored-by: kspeer <[email protected]>

* Add call_details stream (singer-io#49)

* Add call_view core stream, filter non-attribute fields from core streams

* Change `call_view` stream name to `call_details`

* Make pylint happy

* Fix integration tests

* Remove campaign_id from foreign keys, split call_details foreign keys

* Update tests to exclude call_details as needed

* Add context around excluding call_details from tests

* Remove outdated TODO related to addition of call_details stream.

Co-authored-by: dsprayberry <[email protected]>

* Add core LABELS streams and campaign.labels fields to relevant reports (singer-io#53)

* Add core LABELS streams and campaign.labels fields to relevant reports

* Update test exclusions to exclude new core streams.

* Update discover to include campaign_label in reports list

* Update foreign_key expected metadata to include attributed resource foreign_keys

* Update tests to re-include campaign_labels and labels as we now have test data.

* Handles case where state does not have currently_syncing (singer-io#54)

* Handles case where state does not have currently_syncing

* tap-tester test added

* takeout unused imports in test

Co-authored-by: kspeer <[email protected]>

* Version bump and changelog entry (singer-io#52)

* Version bump and changelog entry

* Update to include PR 53.

* Add PR 54 to changelog

* Implement Automatic Keys (singer-io#55)

* Set geographic_view.location_type as automatic to account for reporting discrepancies

* Update Automatic Fields test happy / error paths

* Update base to include automatic_keys metadata for use in tests

* WIP Add Automatic Report Fields

* Pass automatic_keys to BaseStream; use automatic_keys for inclusion

* Fix bad field name in streams; Start updating tests

* Rename function to match base.py

* Add closing brace -_-

* Fix tests; rename ad_group_criterion_criterion_id in automatic_keys; reorder test metadata; add campaign.id to campaign_audience_performance_report

* Rename report_field_parts to split_report_field

* Update transform_keys to raise ad_group_ad.ad fields

* Update happy path streams_to_test to exclude streams with no data

* Remove field name change for ad_performance_report stream

* Accept Andy's Suggestion

Co-authored-by: Andy Lu <[email protected]>

* Reverting Andy's change because it affects core streams

* Explicitly install grpcio-status to avoid from_call attribute errors

* Update setup.py with docs explaining required but unused dep

Co-authored-by: Andy Lu <[email protected]>
Co-authored-by: Arthur Gorka <[email protected]>

* Cleanup (singer-io#56)

* Remove useless function

* Rename "REPORTS" to "STREAMS" for accuracy / readability

* Version bump for PRs 56 and 55 (singer-io#57)

* Version bump for PRs 56 and 55

* Update to exclude ad_group_ad change for ad_performance_report

* [Feature] Add more core streams (singer-io#58)

* WIP add new core streams

* Set geographic_view.location_type as automatic to account for reporting discrepancies

* Update Automatic Fields test happy / error paths

* Update base to include automatic_keys metadata for use in tests

* WIP Add Automatic Report Fields

* Pass automatic_keys to BaseStream; use automatic_keys for inclusion

* Fix bad field name in streams; Start updating tests

* Rename function to match base.py

* Add closing brace -_-

* Fix tests; rename ad_group_criterion_criterion_id in automatic_keys; reorder test metadata; add campaign.id to campaign_audience_performance_report

* Rename report_field_parts to split_report_field

* Update transform_keys to raise ad_group_ad.ad fields

* Update happy path streams_to_test to exclude streams with no data

* WIP remove IPDB and try except

* Fix report_definition typo and add handling for user_interest_id field

* Remove field name change for ad_performance_report stream

* Accept Andy's Suggestion

Co-authored-by: Andy Lu <[email protected]>

* Reverting Andy's change because it affects core streams

* Explicitly install grpcio-status to avoid from_call attribute errors

* Remove ipdb -_-

* Update setup.py with docs explaining required but unused dep

* WIP w/ failing tests and attributed_resource foreign_keys

* WIP with failing tests; remove extraneous attributed_resoruce automatic fields

* Create UserInterestStream class to handle its edge case

* Add transform_keys to UserInterestStream class; start test updates

* Rename obj to json_message; rename variables accordingly; fix UserInterestStream transform_keys

* update auto fields test to account for compound pks

* Fix campaigns typo

* Remove exclusion of feed and feed items from sync canary test

Co-authored-by: Andy Lu <[email protected]>
Co-authored-by: Arthur Gorka <[email protected]>
Co-authored-by: kspeer <[email protected]>
Co-authored-by: atribed <[email protected]>

* V1.3.0 (singer-io#59)

* Version bump for PRs 56 and 55

* Update to exclude ad_group_ad change for ad_performance_report

* Version bump for 1.3.0

Co-authored-by: Arthur Gorka <[email protected]>

* Qa/fix build notification (singer-io#65)

* fix slack notif for build

* remove click_performance_report from tests

* remove click_performance_report from sync canary test

* run only streams that are untested in canary sync

* just skip canaray test

* put back the assert in the skipped test

Co-authored-by: kspeer <[email protected]>

* Crest master (singer-io#66)

* Tdl 19235 handle uncaught exceptions (singer-io#61)

* Added backoff for 5xx, 429 and ReadTimeout errors.

* Resolved pylint error.

* Updated comments in the unittest cases.

* Updated error handling.

* TDL-18749 Implement interruptible full table streams. (singer-io#60)

* Implemented interruptible full table streams.

* Resolved pylint error

* Resolved error in full table sync test case.

* Updated config.yml to pass cci

* Updated query building logic.

* Updated integration test case.

* Resolved review comments.

* Resolved comments.

* Implemeted logic to skip the duplicate records.

* Resolved unittest case error.

* Resolved pylint error

* Resolved integration test case error

* Added empty filter param for call_details and campaign_label stream.

* Added unit test cases for should_sync method.

* Revert "Implemeted logic to skip the duplicate records."

This reverts commit cd06e11.

* Added logger message for debugging purpose

* Updated integration test case.

* Replaced .format with f string.

* Updated comment in integration test.

Co-authored-by: KrishnanG <[email protected]>

* Bump version (singer-io#67)

Co-authored-by: KrishnanG <[email protected]>

* TDL-18524 updated readme and added sample config (singer-io#51)

* updated readme and added sample config

* updated endpoints

* add new streams

* resolved PR comments

* fixed a typo

* added the

* Add timeout parameter to gas.search (singer-io#64)

* Add timeout parameter to search gas.search

* Increase timeout to 15 minutes for safety.

* Add get_request_timeout function, add config to make_request as needed

* Update on_giveup to raise specific exception text for timeoutexception class; add unit test

* Make Pylint happy take 1

* Update make_request signature in unittests.

* Another Unittest update

* More unittest fixes for signature

* Add default config param value for ease of implementation in future tests

* Fix pylint dangerous-defaul-value

* Fix stupid error

* Version bump and changelog (singer-io#70)

* TDL-19486 Add limit clause to core stream queries (singer-io#68)

* Initial commit for add page limit.

* Added limit parameter in sync method of ReportStream class.

* Fixed issue for call_details stream.

* Resolved unit test case error.

* Added test cases.

* Updated code comments.

* Fixed keyerror issue.

* Updated default query limit.

* Updated pagination test case.

* Modify config name to be more explicit.

* Update comment for accuracy.

* Update property name in base and propogate name change to tests

* Exclude feed from streams_to_test because of lack of data

* Committing stuff from main that should have already been in

Co-authored-by: dsprayberry <[email protected]>

* Version Bump and Changelog Update (singer-io#72)

* Reintroduce unintentionally removed files. (singer-io#73)

* add schemaless

* fix schemaless config

* update login_customer_ids

* fix missing key from dict

* add google ads api version in config
add log to show the version

* use version from config

* use version global

Co-authored-by: bryantgray <[email protected]>
Co-authored-by: Kyle Speer <[email protected]>
Co-authored-by: kspeer <[email protected]>
Co-authored-by: dsprayberry <[email protected]>
Co-authored-by: Andy Lu <[email protected]>
Co-authored-by: Arthur Gorka <[email protected]>
Co-authored-by: atribed <[email protected]>
Co-authored-by: Prijen Khokhani <[email protected]>
Co-authored-by: KrishnanG <[email protected]>
Co-authored-by: KrisPersonal <[email protected]>
Co-authored-by: namrata270998 <[email protected]>
Co-authored-by: bbaltagi-dtsl <[email protected]>
  • Loading branch information
13 people committed Aug 2, 2022
1 parent 6b279c4 commit 1112a96
Show file tree
Hide file tree
Showing 5 changed files with 22 additions and 21 deletions.
5 changes: 2 additions & 3 deletions tap_google_ads/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,7 @@
"refresh_token",
"customer_ids",
"developer_token",
"schemaless",
"version"
"schemaless"
]


Expand All @@ -27,7 +26,7 @@ def main_impl():
resource_schema = create_resource_schema(args.config)
state = {}

LOGGER.info(f"Sync Starting with google-ads api {args.config['version']} version.")
LOGGER.info(f"Sync Starting with google-ads api.")
if args.state:
state.update(args.state)
if args.discover:
Expand Down
5 changes: 3 additions & 2 deletions tap_google_ads/streams.py
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,9 @@ def get_selected_fields(stream_mdata):
if mdata["breadcrumb"]:
inclusion = mdata["metadata"].get("inclusion")
selected = mdata["metadata"].get("selected")
if utils.should_sync_field(inclusion, selected) and mdata["breadcrumb"][1] != "_sdc_record_hash" and "tap-google-ads.api-field-names" in mdata["metadata"]:
if utils.should_sync_field(inclusion, selected) \
and mdata["breadcrumb"][1] != "_sdc_record_hash" \
and "tap-google-ads.api-field-names" in mdata["metadata"]:
selected_fields.update(mdata["metadata"]["tap-google-ads.api-field-names"])

return selected_fields
Expand Down Expand Up @@ -756,7 +758,6 @@ def sync(self, sdk_client, customer, stream, config, state, query_limit, schemal
transformed_obj = self.transform_keys(json_message)
record = transformer.transform(transformed_obj, stream["schema"])
record["_sdc_record_hash"] = generate_hash(record, stream_mdata)

singer.write_record(stream_name, record)
else:
singer.write_record(stream_name, json_message)
Expand Down
1 change: 1 addition & 0 deletions tap_google_ads/sync.py
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,7 @@ def get_query_limit(config):

def do_sync(config, catalog, resource_schema, state):
# QA ADDED WORKAROUND [START]

try:
customers = json.loads(config["customer_ids"])
except TypeError: # falling back to raw value
Expand Down
20 changes: 10 additions & 10 deletions tap_google_ads/tests/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -46,14 +46,14 @@ def get_properties(self):
'end_date': '2018-04-15T00:00:00Z',
'conversion_window_days' : '-1',
'user_id': 'not used?',
'customer_ids': os.getenv('TAP_ADWORDS_CUSTOMER_IDS')}
'customer_ids': os.getenv('TAP_ADS_CUSTOMER_IDS')}

@staticmethod
def get_credentials(self):
return {'developer_token': os.getenv('TAP_ADWORDS_DEVELOPER_TOKEN'),
'oauth_client_id': os.getenv('TAP_ADWORDS_OAUTH_CLIENT_ID'),
'oauth_client_secret': os.getenv('TAP_ADWORDS_OAUTH_CLIENT_SECRET'),
'refresh_token': os.getenv('TAP_ADWORDS_REFRESH_TOKEN')}
return {'developer_token': os.getenv('TAP_ADS_DEVELOPER_TOKEN'),
'oauth_client_id': os.getenv('TAP_ADS_OAUTH_CLIENT_ID'),
'oauth_client_secret': os.getenv('TAP_ADS_OAUTH_CLIENT_SECRET'),
'refresh_token': os.getenv('TAP_ADS_REFRESH_TOKEN')}

def expected_metadata(self):
"""The expected streams and metadata about the streams"""
Expand Down Expand Up @@ -235,11 +235,11 @@ def expected_replication_method(self):


def setUp(self):
missing_envs = [x for x in [os.getenv('TAP_ADWORDS_DEVELOPER_TOKEN'),
os.getenv('TAP_ADWORDS_OAUTH_CLIENT_ID'),
os.getenv('TAP_ADWORDS_OAUTH_CLIENT_SECRET'),
os.getenv('TAP_ADWORDS_REFRESH_TOKEN'),
os.getenv('TAP_ADWORDS_CUSTOMER_IDS')] if x == None]
missing_envs = [x for x in [os.getenv('TAP_ADS_DEVELOPER_TOKEN'),
os.getenv('TAP_ADS_OAUTH_CLIENT_ID'),
os.getenv('TAP_ADS_OAUTH_CLIENT_SECRET'),
os.getenv('TAP_ADS_REFRESH_TOKEN'),
os.getenv('TAP_ADS_CUSTOMER_IDS')] if x == None]
if len(missing_envs) != 0:
raise Exception("Missing environment variables: {}".format(missing_envs))

Expand Down
12 changes: 6 additions & 6 deletions tests/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ def get_properties(self, original: bool = True):
'query_limit': 2,
# 'conversion_window_days': '30',
'login_customer_ids': [{"customerId": os.getenv('TAP_GOOGLE_ADS_CUSTOMER_ID'),
"loginCustomerId": os.getenv('TAP_GOOGLE_ADS_LOGIN_CUSTOMER_ID'),}],
"loginCustomerId": os.getenv('TAP_GOOGLE_ADS_LOGIN_CUSTOMER_ID')}],
}

if original:
Expand Down Expand Up @@ -458,11 +458,11 @@ def expected_replication_method(self):


def setUp(self):
missing_envs = [x for x in [os.getenv('TAP_ADWORDS_DEVELOPER_TOKEN'),
os.getenv('TAP_ADWORDS_OAUTH_CLIENT_ID'),
os.getenv('TAP_ADWORDS_OAUTH_CLIENT_SECRET'),
os.getenv('TAP_ADWORDS_REFRESH_TOKEN'),
os.getenv('TAP_ADWORDS_CUSTOMER_IDS')] if x == None]
missing_envs = [x for x in [os.getenv('TAP_ADS_DEVELOPER_TOKEN'),
os.getenv('TAP_ADS_OAUTH_CLIENT_ID'),
os.getenv('TAP_ADS_OAUTH_CLIENT_SECRET'),
os.getenv('TAP_ADS_REFRESH_TOKEN'),
os.getenv('TAP_ADS_CUSTOMER_IDS')] if x == None]
if len(missing_envs) != 0:
raise Exception("Missing environment variables: {}".format(missing_envs))

Expand Down

0 comments on commit 1112a96

Please sign in to comment.