Timezone Aware Timestamp Parsing

[tustvold]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

string_to_timestamp_nanos contains logic to parse timestamp-like string to nanoseconds since UTC epoch.

The semantics for this are well defined for timestamps including a timezone, e.g. 1997-01-31 09:26:56.123Z or 1997-01-31T09:26:56.123-05:00. However, the semantics get confused for timestamps of the form 1997-01-31T09:26:56.123

As pointed out by @MachaelLee on #3787 prior to #2814 timestamp string without a timezone would be interpreted as being in the system's local timezone, and this continues to be what the function docs state happens. This was changed in #2814 by @waitingkuo to instead be parsed in the UTC timezone.

There are at least three "correct" behaviours when parsing strings without an embedded timezone 1997-01-31T09:26:56.123 depending on the context

• When parsing a user-provided timestamp, use the system Local timezone and convert back to UTC what it used to do
• When parsing a string to a datatype without a timezone assume UTC what it currently does
• When parsing a string to a Timestamp column with a timezone, should assume the timestamp is in the given Tz and convert back to UTC I think this is correct??

Describe the solution you'd like

Provide a function with the signature

pub fn string_to_datetime<T: Timezone>(t: &T, s: &str) -> Result<DateTime<T>>

This could then be used with Local, Utc, or Tz as appropriate.

We could then update string_to_timestamp_nanos to be something like

pub fn string_to_timestamp_nanos(s: &str) -> Result<i64, ArrowError> {
    to_timestamp_nanos(string_to_datetime(Utc, s)?.naive_utc())
}

And possibly deprecate it as it has rather confusing semantics

Describe alternatives you've considered

Additional context

[alamb]

pub fn string_to_datetime<T: Timezone>(t: &T, s: &str) -> Result<DateTime>

Makes sense to me

@waitingkuo what do you think?

[stuartcarnie]

Am I correct this is an improvement to the API, and the semantics of interpreting a timestamp without an offset is still to interpret as UTC?

[tustvold]

the semantics of interpreting a timestamp without an offset is still to interpret as UTC

This does not alter the current behaviour, however, #1936 tracks using this functionality to interpret such timestamps with respect to the datatype's timezone, if any

[alamb]

Proposed implementation in #3795

[tustvold]

label_issue.py automatically added labels {'arrow'} from #3795