Changed 'wind_2min_timestamp' variable type from character to date-ti… #52
Conversation
…me when downloading hourly and daily data. Changed date-time assignment to 'date_datetime' variable in hourly data. Previously was UTC, now MST
R/az_hourly.R
Outdated
| dplyr::mutate(date_datetime = lubridate::force_tz(lubridate::ymd_hms(.data$date_datetime), tzone = "America/Phoenix")) %>% | ||
| dplyr::mutate(wind_2min_timestamp = lubridate::with_tz(lubridate::parse_date_time(.data$wind_2min_timestamp, orders = "ymdHMSz"), tzone = "America/Phoenix")) %>% |
There was a problem hiding this comment.
What's the difference between force_tz() and with_tz()?
There was a problem hiding this comment.
My understanding is that both handle date_datetime and wind_2min_timestamp without any warning message while assigning the correct time zone to each variable.
With date_datetime, there is no time zone assigned to values in the database. Using force_tz directly assigns tzone = "America/Phoenix", while keeping dates and hours correct. I tested with_tz here, too, but it assigned UTC as the time zone and the corresponding dates and hours, which is wrong.
With wind_2min_timestamp, there is a time zone already assigned to values in the database. Using with_tz and tzone = "America/Phoenix" correctly assigns the desired time zone while keeping corresponding dates and hours correct.
There was a problem hiding this comment.
NEWS.md file updated
There was a problem hiding this comment.
Can you commit and push the NEWS.md changes? I don't see them in this pull request
There was a problem hiding this comment.
Ah, they're there now. But, I managed to make a conflict
There was a problem hiding this comment.
I can show you how to deal with the merge conflict next time we meet (next Friday), or you can pick a time tomorrow or earlier next week: https://calendar.app.google/eCWobQteVLViq5NJ8
It'll be an quick fix, just something easier to show over Zoom than via email.
| @@ -86,6 +86,7 @@ az_daily <- function(station_id = NULL, start_date = NULL, end_date = NULL) { | |||
| )) %>% | |||
| dplyr::filter(.data$meta_station_id != "az99") %>% | |||
| dplyr::mutate(datetime = lubridate::ymd(.data$datetime)) %>% | |||
| dplyr::mutate(wind_2min_timestamp = lubridate::with_tz(lubridate::parse_date_time(.data$wind_2min_timestamp, orders = "ymdHMSz"), tzone = "America/Phoenix")) %>% | |||
| #convert NAs | |||
| dplyr::mutate( | |||
| dplyr::across( | |||
There was a problem hiding this comment.
Just below this is code to change values like -9999 to NA, but it currently skips wind_2min_timestamp. I think all you need to do is:
...
dplyrr::across(
c(wind_2min_timestamp, tidyselect(where(is.numeric))),
function(x)
...There was a problem hiding this comment.
Without adding the above code to include wind_2min_timestamp in the conversion of NAs, existing code with wind_2min_timestamp as character automatically converts corresponding nodata values of -99999 to NA while throwing a warning 27 failed to parse, as with
az_daily(start_date = "2023-03-22", end_date = "2023-03-22")
Only one of the 28 stations at that point had wind_2min_timestamp daily values on that date.
I think the nodata value conversion needs to happen somehow in conjunction with character-to-datetime conversion of this variable.
There was a problem hiding this comment.
Also, the above code has trouble with wind_2min_timestamp, as it is an object of class "c('POSIXct', 'POSIXt')". This is after the character-to-datetime conversion
There was a problem hiding this comment.
@Aariq please review solutions in az_daily() and az_hourly() I just pushed on this branch. I think I figured it out
| ) %>% | ||
| dplyr::mutate( | ||
| wind_2min_timestamp = lubridate::with_tz( | ||
| lubridate::parse_date_time(.data$wind_2min_timestamp, orders = "ymdHMSz"), |
There was a problem hiding this comment.
ahhh, I understand this now. So pars_date_time does actually understand the timezone, it just converts it automatically to UTC and with_tz() converts it back to MST. Cool!
| dplyr::mutate(date_datetime = lubridate::ymd_hms(.data$date_datetime)) %>% | ||
| dplyr::mutate( | ||
| date_datetime = | ||
| lubridate::force_tz( |
There was a problem hiding this comment.
but here, ymd_hms() doesn't see a timezone and you need force_tz() to keep the time and only change the timezone. Nice!
Co-authored-by: Eric R. Scott <[email protected]>
Co-authored-by: Eric R. Scott <[email protected]>
…me when downloading hourly and daily data.
Changed date-time assignment to 'date_datetime' variable in hourly data. Previously was UTC, now MST