Add date type processing #2
Conversation
| @@ -100,6 +110,15 @@ def process_worksheet(gsheets_loader, sheet_name, worksheet, start_from_row, con | |||
| singer.write_record(stream_name, record_transformed) | |||
|
|
|||
|
|
|||
| def non_standard_date_execution(records): | |||
| for record in records: | |||
| for field in record: | |||
There was a problem hiding this comment.
So, basically this is looping through every record and every field trying to parse the date, right?
How many records we have in there?
I am just a bit concerned, because this could be quite impactful in the performance.
There was a problem hiding this comment.
As an alternative, it should be possible to look for a date column using only the first record. Would that work?
There was a problem hiding this comment.
I'll think about it. i believe that's this way incorrect. All date fields will be transformed to date_format. need to change whole logic
tap_gsheets/__init__.py
Outdated
| @@ -64,9 +70,13 @@ def process_worksheet(gsheets_loader, sheet_name, worksheet, start_from_row, con | |||
| else: | |||
| stream_name = tableize(parameterize(name_with_worksheet)) | |||
|
|
|||
| records = gsheets_loader.get_records_as_json(sheet_name, worksheet, start_from_row) | |||
|
|
|||
| if gsheet_loader.date_format: | |||
There was a problem hiding this comment.
If I followed the logic correctly, this field will always exist, right? If that is the case, then this if is unnecessary. 🙂
There was a problem hiding this comment.
if we need to transform date column we will set date template to this field otherwise it will empty all the time
tap_gsheets/__init__.py
Outdated
| if "date_format" in sheet: | ||
| gsheet_loader.date_format = sheet["date_format"] | ||
| else: | ||
| gsheet_loader.date_format = "" |
There was a problem hiding this comment.
What about this suggestion:
| if "date_format" in sheet: | |
| gsheet_loader.date_format = sheet["date_format"] | |
| else: | |
| gsheet_loader.date_format = "" | |
| gsheet_loader.date_format = "" | |
| if "date_format" in sheet: | |
| gsheet_loader.date_format = sheet["date_format"] |
There was a problem hiding this comment.
we can delete the second condition 'else'
because default value of this var it's none
https://github.com/miroapp/tap-gsheets/blob/add_date_type_processing/tap_gsheets/gsheet_loader.py#L10
No description provided.