Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Write the episode file as a partitioned dataset #750

Merged
merged 2 commits into from
Jul 18, 2023
Merged

Write the episode file as a partitioned dataset #750

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f6854d6
Revert "Write out as a partitioned arrow dataset (#726)"
Moohan Jul 18, 2023
54c1636
Write the episode file out as a partitioned dataset
Moohan Jul 18, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 0 additions & 10 deletions R/run_episode_file.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,16 +128,6 @@
)

write_file(episode_file, slf_episode_path)

arrow::write_dataset(
dataset = episode_file,
path = fs::path_ext_remove(slf_episode_path),
format = "parquet",
# Should correspond to the available slfhelper filters
partitioning = c("recid", "hscp2018"),
compression = "zstd",
version = "latest"
)
}

return(episode_file)
Expand Down Expand Up @@ -218,12 +208,12 @@
fill_missing_cij_markers <- function(data) {
fixable_data <- data %>%
dplyr::filter(
.data[["recid"]] %in% c("01B", "04B", "GLS", "02B", "DD") & !is.na(.data[["chi"]])

Check warning on line 211 in R/run_episode_file.R

View workflow job for this annotation

GitHub Actions / lint-changed-files 

file=R/run_episode_file.R,line=211,col=81,[line_length_linter] Lines should not be more than 80 characters.
)

non_fixable_data <- data %>%
dplyr::filter(
!(.data[["recid"]] %in% c("01B", "04B", "GLS", "02B", "DD")) | is.na(.data[["chi"]])

Check warning on line 216 in R/run_episode_file.R

View workflow job for this annotation

GitHub Actions / lint-changed-files 

file=R/run_episode_file.R,line=216,col=81,[line_length_linter] Lines should not be more than 80 characters.
)

fixed_data <- fixable_data %>%
Expand Down Expand Up @@ -356,8 +346,8 @@
})

# This 'blocks' the code until they have both finished executing
value_demographic <- future::value(future_demographic)

Check warning on line 349 in R/run_episode_file.R

View workflow job for this annotation

GitHub Actions / lint-changed-files 

file=R/run_episode_file.R,line=349,col=3,[object_usage_linter] local variable ‘value_demographic’ assigned but may not be used
value_service_use <- future::value(future_service_use)

Check warning on line 350 in R/run_episode_file.R

View workflow job for this annotation

GitHub Actions / lint-changed-files 

file=R/run_episode_file.R,line=350,col=3,[object_usage_linter] local variable ‘value_service_use’ assigned but may not be used

return(data)
}
Expand Down
12 changes: 12 additions & 0 deletions _targets.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -547,6 +547,18 @@ list(
data = episode_file,
year = year
)
),
tar_target(
episode_file_dataset,
arrow::write_dataset(
dataset = episode_file,
path = fs::path_ext_remove(slf_episode_path),
format = "parquet",
# Should correspond to the available slfhelper filters
partitioning = c("recid", "hscp2018"),
compression = "zstd",
version = "latest"
)
)
)
)
Loading