You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi, something we observed in the schema generation of schema2000, which causes some trouble. It still might be correct what schema2000 does due to the heterogeneous nature of the underlaying data.
For some timestamp field, we got a schema like this:
The problematic part is the "string" part, which in turn complicates the analyzed data structure when used in BI tools (in this case Bigquery). See the following screenshot, where we wrote parquet data with a schema generated by schema2000.
Once we load the data in Bigquery and let Bigquery automatically derive the schema, this results look like the following screenshot:
For myDateTime2 Bigquery inserted new subfields (e.g. member0, member1). These subfield in turn are resulting in more complicated queries. If we could avoid this, that would be really great.
Maybe one could set a command line parameter and schema2000 could somehow avoid adding the additional "string" type??
A user reported something is wrong with dates/datetime, maybe invalid schema, maybe detection is off.
The text was updated successfully, but these errors were encountered: