Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pick up latest geography file, and save out with compression #983

Merged
merged 3 commits into from
Jul 24, 2024
Merged

pick up latest geography file, and save out with compression #983

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
885b88a
pick up latest geography file, and save out with compression
Jul 15, 2024
6ee1227
Style code
SwiftySalmon Jul 15, 2024
84c13e4
use `get_spd_path`
Jennit07 Jul 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 15 additions & 13 deletions extract_new_nsu_cohort/filter_nsu_duplicates.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
# extract from the CHILI team.
#
# Steps for requesting a new NSU extract for SLFs:
# 1. Send an email to [[email protected]] to request a new NSU

Check failure on line 14 in extract_new_nsu_cohort/filter_nsu_duplicates.R

View workflow job for this annotation

GitHub Actions / Check Spelling 

`recordlinkage` is not a recognized word. (unrecognized-spelling)
# extract after the JUNE update.
# 2. Prepare a service use extract. Run script `get_service_use_cohort.R` to
# extract a list of CHI's from the most recent 'full' file.
Expand All @@ -29,12 +29,12 @@
library(dplyr)
library(purrr)
library(stringr)
library(PostcodesioR)

Check failure on line 32 in extract_new_nsu_cohort/filter_nsu_duplicates.R

View workflow job for this annotation

GitHub Actions / Check Spelling 

`Postcodesio` is not a recognized word. (unrecognized-spelling)
library(janitor)
library(fs)
library(haven)
library(glue)


## Setup------------------------------------------------------------------------

## Update line 41##
Expand All @@ -43,12 +43,14 @@

# Update lines 45-46 ##
# Analysts username and schema to collect the data.
analyst <- "ALEXRE01"
schema <- "FINAL_1"
analyst <- "ROBERM18"
schema <- "FINAL_2"

# setup directory
nsu_dir <- path("/conf/hscdiip/SLF_Extracts/NSU")

# latest geography file
spd_path <- get_spd_path()

# Set up connection to SMRA-----------------------------------------------------
db_connection <- odbc::dbConnect(
Expand Down Expand Up @@ -78,10 +80,7 @@
filter(postcode_count > 1)

# Get the latest SPD
spd <-
readr::read_rds(
"/conf/linkage/output/lookups/Unicode/Geography/Scottish Postcode Directory/Scottish_Postcode_Directory_2021_2.rds"
) %>%
spd <- read_file(spd_path) %>%
select(pc7, date_of_introduction, date_of_deletion)

# Load some regex to check if a postcode is valid
Expand Down Expand Up @@ -120,8 +119,8 @@
pull(postcode) %>%
unique() %>%
list(postcodes = .) %>%
# This function will fail if more than 100 pcs

Check failure on line 122 in extract_new_nsu_cohort/filter_nsu_duplicates.R

View workflow job for this annotation

GitHub Actions / Check Spelling 

`pcs` is not a recognized word. (unrecognized-spelling)
PostcodesioR::bulk_postcode_lookup() %>%

Check failure on line 123 in extract_new_nsu_cohort/filter_nsu_duplicates.R

View workflow job for this annotation

GitHub Actions / Check Spelling 

`Postcodesio` is not a recognized word. (unrecognized-spelling)
# Parse the result, we only want the country
map_dfr(~ tibble(
postcode = .x$query,
Expand Down Expand Up @@ -166,10 +165,7 @@
anti_join(nsu_pc_duplicates_checked %>%
filter(keep_priority > 1)) %>%
# Filter any remaining duplicates (none on this test)
distinct(upi_number, .keep_all = TRUE)

# Save data out to be used
final_data %>%
distinct(upi_number, .keep_all = TRUE) %>%
select(
chi = upi_number,
dob = date_of_birth,
Expand All @@ -184,7 +180,13 @@
) %>%
arrange(chi) %>%
# Save as anon chi on disk
slhelper::get_anon_chi() %>%
arrow::write_parquet(path(nsu_dir, glue("anon-All_CHIs_20{year}.parquet")), compress = TRUE)
slfhelper::get_anon_chi()

# Save data out to be used
final_data %>%
arrow::write_parquet(path(nsu_dir, glue::glue("anon-All_CHIs_20{year}.parquet")),
compression = "zstd"
)


## End of Script ##
Loading