Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fill ch names #946

Merged
merged 22 commits into from
May 21, 2024
Merged

Fill ch names #946

merged 22 commits into from
May 21, 2024

Conversation

lizihao-anu
Copy link
Contributor

@lizihao-anu lizihao-anu commented Apr 29, 2024 •
edited
Loading

There are issues in care home extracts that increase the difficulty of matching care home names and postcodes. For the sake of information governance and data protect, the detailed explanation with descriptive summary statistics is provided in the PHS Sharepoint.

matching_quality_indicator is created to help to solve matching issues and monitor the matching quality.

@lizihao-anu lizihao-anu changed the base branch from master to June-24-update April 29, 2024 15:08
@github-actions GitHub Actions

This comment has been minimized.

Sign in to view
@github-actions GitHub Actions

This comment has been minimized.

Sign in to view
@lizihao-anu

This comment has been minimized.

Sign in to view
@github-actions GitHub Actions

This comment has been minimized.

Sign in to view
@github-actions GitHub Actions

This comment has been minimized.

Sign in to view
@github-actions GitHub Actions

This comment has been minimized.

Sign in to view
@github-actions GitHub Actions

This comment has been minimized.

Sign in to view
@lizihao-anu

This comment has been minimized.

Sign in to view
@github-actions GitHub Actions

This comment has been minimized.

Sign in to view
@github-actions GitHub Actions

This comment has been minimized.

Sign in to view
@github-actions GitHub Actions

This comment has been minimized.

Sign in to view
SwiftySalmon
SwiftySalmon reviewed May 2, 2024
View reviewed changes
R/fill_ch_names.R Outdated Show resolved Hide resolved
SwiftySalmon
SwiftySalmon reviewed May 2, 2024
View reviewed changes
R/fill_ch_names.R Outdated Show resolved Hide resolved
SwiftySalmon
SwiftySalmon requested changes May 2, 2024
View reviewed changes
Copy link
Collaborator

@SwiftySalmon SwiftySalmon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Logic seems good to me!

Things to update:
English postcodes,
Add more notes
Maybe rename the dataframes?

@github-actions GitHub Actions

This comment has been minimized.

Sign in to view
@lizihao-anu

This comment has been minimized.

Sign in to view
@github-actions GitHub Actions

This comment has been minimized.

Sign in to view
@github-actions GitHub Actions

This comment has been minimized.

Sign in to view
@lizihao-anu lizihao-anu merged commit 1f9d84e into June-24-update May 21, 2024
12 checks passed
@lizihao-anu lizihao-anu deleted the fill_ch_names branch May 21, 2024 14:58
github-merge-queue bot pushed a commit that referenced this pull request Jun 11, 2024
@lizihao-anu @Jennit07
@SwiftySalmon @rchlv
June 24 update (#962)
f047aab 
* Remove redundant code

* Update documentation

* Style code

* Reorder when we match on client variables
This was causing NSUs to show a social care id. This now resolves this.

* Update documentation

* Style code

* Revert "Update logic to use end of Quarter"

This reverts commit 004e831.

* Style code

* Update documentation

* add check comment (TO DO for this PR)

* Remove `check_quarter_format` function

* Remove `check_quarter_format`

* Add chi parameter to `create_demog_test_flags`

* Style code

* Use CHI parameter for ep/indiv tests

* Use CHI parameter for extract tests (chi)

* Change test sheet names to lowercase

* Change date to lowercase

* Update documentation

* Update documentation

* Update documentation

* Style code

* Fix pick variables
This was not taking the correct variables, leading to NSUs being assigned psychiatry

* SC Demographics and SDS (#900)

* Style code

* # read in sc demographics

different variables - removed extract date as not accurate, using chi over upi after discussion with social care data management. Added in date of death just for fun.

* social care demographics first draft

removed a lot of the submitted variables and instead using chi variables from chi seeding. Other changes:
- Fill in missing values,
- create flag for latest social care id (one from database is not accurate), this makes sure that each chi only has ONE sc id as the latest to stop it creating duplicates
- change postcode to choose chi over submitted

* Style code

* had a github error? Not sure what happened but commiting first draft of sc demographics

* Style code

* first draft sds.
No major changes - only how demographics is matched on and how latest social care id is selected

* Update documentation

* demographics - add sending location to group by

* Style code

* Update documentation

* Added ungroup()

* Remove comments

* Remove comments

* Style code

---------

Co-authored-by: SwiftySalmon <[email protected]>
Co-authored-by: marjom02 <[email protected]>
Co-authored-by: Jennit07 <[email protected]>
Co-authored-by: Jennit07 <[email protected]>
Co-authored-by: Zihao Li <[email protected]>

* Sc all at speedup (#904)

* speed up process_sc_all_alarms_telecare function with data.table package

* Update documentation

---------

Co-authored-by: lizihao-anu <[email protected]>
Co-authored-by: Megan McNicol <[email protected]>
Co-authored-by: Jennit07 <[email protected]>

* Add case_when statement for `high_cc` cohort

* Bug - `high_cc` in demographic cohort showing `NAs` instead of `TRUE/FALSE` (#911)

Add case_when statement for `high_cc` cohort

* added a casewhen to update property type description for homelessness

* Update documentation

* Style code

* Bug - deal with missing variables (#914)

* Add missing sc variables for no sc data

* Fix code for including `_inc_dna` variables

* Remove commented line

* Bug - Fix get pop path failing and preventing the indiv file from running.  (#913)

Fix bug - pop file paths breaking indiv file

* correct file hscp file path

* Update process_sc_all_home_care.R

A small issue was identified when running targets. Linked with changes to the function `fix_sc_end_dates()`

* Update process_sc_all_alarms_telecare.R

* remove duplicate columns

* Fix targets (#892)

* fix sc_client_lookup sc_send_lca

* fix an issue of get_pop_path

* Style code

* fix the rest of get_pop_path from get_datazone_pop_path

* Update documentation

* fix sc_send_lca

* add missing year column

* explicitly specify the argument year to avoid corruption of targets

* Update documentation

* new data pipeline with targets
remove create_individual_files from targets and append it to run_targets script

* minor changes

* Style code

* undo sc_send_lca bit

* Update targets scripts

* Remove top level targets scripts

---------

Co-authored-by: lizihao-anu <[email protected]>
Co-authored-by: Megan McNicol <[email protected]>
Co-authored-by: Jennit07 <[email protected]>
Co-authored-by: Jennifer Thom <[email protected]>

* remove cases that start date is later than end date

* Update Refs for March24 SLF update

* update documentation

* Update sc connection name

* Update documentation

* 936 - Update parameters with file paths  (#939)

Specify file paths in sc function parameters

* Add test for `n_records` in ep file tests

* remove and merge overlapping records in GP OoHs

* Style code

* update spelling to lowercases

* update spelling

* Add function for reading Dev SLF file
Uses SLFhelper for easy access to Source_Linkage_Files

* Add cross year tests using SLFhelper WIP
WIP - still need to add write to disk and possibly develop visuals

* Create tests for social care sandpit extracts (#943)

* Update `write_tests_xlsx`

* Update documentation

* Add in sandpit tests where the extract is saved

* Setup tests for sandpit
Further checks needed for writing to disk

* Update documentation

* Amend case_when statement

* rename function to include 'sc'

* Update documentation

* Use `is.null` instead of `missing`

* Update documentation

* Add `year` as a parameter

* Update documentation

* Setup for writing sandpit tests to disk

* Update parameters for sandpit tests

* Update documentation

* Use `process_tests_sc_sandpit`

* Apply styling

* Style code

* update documentation

Co-authored-by: Zihao Li <[email protected]>

* Rename variable sc_id

Co-authored-by: Zihao Li <[email protected]>

* Rename variable

Co-authored-by: Zihao Li <[email protected]>

* Rename variable

Co-authored-by: Zihao Li <[email protected]>

* Update documentation

* [check-spelling] Update metadata

Update for https://github.com/Public-Health-Scotland/source-linkage-files/actions/runs/8689503990/attempts/1
Accepted in #943 (comment)

Signed-off-by: check-spelling-bot <[email protected]>
on-behalf-of: @check-spelling <[email protected]>

* update spelling

* update spelling expect variant

---------

Signed-off-by: check-spelling-bot <[email protected]>
Co-authored-by: Jennit07 <[email protected]>
Co-authored-by: Zihao Li <[email protected]>
Co-authored-by: Zihao Li <[email protected]>

* Remove filtering between 90-105% completeness

* Keep percentage comparison

* Add new variable pre/post hl1 application

* re-write the logic of fill_ch_names

* Update documentation

* Style code

* minor typo fix

* [check-spelling] Update metadata

Update for https://github.com/Public-Health-Scotland/source-linkage-files/actions/runs/8881311681/attempts/1
Accepted in #946 (comment)

Signed-off-by: check-spelling-bot <[email protected]>
on-behalf-of: @check-spelling <[email protected]>

* update spelling expect

* update spelling expect

* fix R CMD warning of no visible binding

* Style code

* [check-spelling] Update metadata

Update for https://github.com/Public-Health-Scotland/source-linkage-files/actions/runs/8893412405/attempts/1
Accepted in #946 (comment)

Signed-off-by: check-spelling-bot <[email protected]>
on-behalf-of: @check-spelling <[email protected]>

* spelling seems not recognize variants

* only select columns we want in ltc raw data

* [check-spelling] Update metadata

Update for https://github.com/Public-Health-Scotland/source-linkage-files/actions/runs/8897746003/attempts/1
Accepted in #947 (comment)

Signed-off-by: check-spelling-bot <[email protected]>
on-behalf-of: @check-spelling <[email protected]>

* fix care home cancelled dates might be 1900-01-01

* for some reason the latest scid code was overwritten after the march update?? anyway, now it is fixed.

* Style code

* add checking ch_postcode in England, quality 15

* Update documentation

* Style code

* [check-spelling] Update metadata

Update for https://github.com/Public-Health-Scotland/source-linkage-files/actions/runs/8971882687/attempts/1
Accepted in #946 (comment)

Signed-off-by: check-spelling-bot <[email protected]>
on-behalf-of: @check-spelling <[email protected]>

* spelling metadata

* Merge May24 NI update into June update branch (#949)

Collect data before manipulations

* update metadata for fill_ch_names

* Update documentation

* add rounding to one decimal place on percentage

* Add write to disk

* update `write_tests_xlsx`

* Style code

* Update documentation

* Add to targets pipeline

* Update NEWS.md

* Update NEWS.md

* Added function for get_all_slf_deaths_lookup_path

* Update documentation

* Style code

* Add vars for activity after death flag

* Add activity after death flag

* Join data back to episode file

* Style code

* Update documentation

* fix a bug for quality 21

* Update `00_sort_bi_extracts` to write anon_chi (#952)

* Update `00_sort_BI_extracts`
Save a new file with `anon-` prefix and use slfhelper to get the anon_chi

* remove file copy

* Update `00_sort_bi_extracts` note

* Style code

* Update chi when this is different e.g UPI number or PAT_UPI

* remove storing as a dataframe

* Add condition if CHI exists in data file

* update 00_Sort_BI_Extracts
replace for loop by function to enable parallel computing with lapply

* Style code

* merge similar code

* simplify sort_bi_extracts

---------

Co-authored-by: Jennit07 <[email protected]>
Co-authored-by: Zihao Li <[email protected]>
Co-authored-by: lizihao-anu <[email protected]>

* Update refs

* changes to activity after death flag

* Update documentation

* Update R/add_activity_after_death_flag.R

Co-authored-by: Jennit07 <[email protected]>

* Update R/add_activity_after_death_flag.R

Co-authored-by: Jennit07 <[email protected]>

* added .data$ to variables

* Update documentation

* Style code

* comment out cross_year_tests for now

* Update anon_chi for dn and cmh

* Update boxi filepath ("anon-")

* remove file copy

* Update `00_sort_bi_extracts` note

* Style code

* Update `get_source_extract_path` (anon- prefix)

* Update chi when this is different e.g UPI number or PAT_UPI

* Change `read` functions to read anon_chi

* change `process` functions to read `anon_chi`

* remove storing as a dataframe

* Add condition if CHI exists in data file

* Update dd path

* switch between chi - ooh and dd

* Update chi when this is different e.g UPI number or PAT_UPI

* remove storing as a dataframe

* Add condition if CHI exists in data file

* update 00_Sort_BI_Extracts
replace for loop by function to enable parallel computing with lapply

* Style code

* merge similar code

* simplify sort_bi_extracts

* update sparra/hhg paths (anon_chi)

* use anon_chi for sc demogs

* Update documentation

* Update `create_episode_file`

* update NSU path

* Use `get_chi` before phs methods check - ooh

* Update LTCs

* Style code

* Update sc paths to `anon-` prefix

* update cohorts paths

* Update deaths paths with `anon-` prefix

* sc client anon_chi

* match files with chi

* Update `create_episode_file` joins

* Update documentation

* update get sandpit extracts

* update tests to use `chi`

* Style code

* Update IT extracts to maintain chi

* Update sort_bi_extracts

* Update bracket

* update parameter

* Update documentation

* bugs fix

* fix reading data from plateform and homelessness chi

* update sc demog path

* update homelessness lookup

* Update documentation

* supply get_chi() where needed in targets

* Style code

* Update documentation

* Update targets with get_chi()

* Update targets with get_chi()

* Update client script

* Update documentation

* fix fill_ch_names

* add anon- and update targets

* fix add_activity_after_death in create_episode_file

* Style code

* process_tests_sc_client_lookup fix

* fix anon-chi issues in create_episode_file

* Update documentation

* fix typo

* Update documentation

* fix write_tests_xlsx path

* minor fix

* fix R package build warnings

* Style code

* aligning

* [check-spelling] Update metadata

Update for https://github.com/Public-Health-Scotland/source-linkage-files/actions/runs/9419296266/attempts/1
Accepted in #962 (comment)

Signed-off-by: check-spelling-bot <[email protected]>
on-behalf-of: @check-spelling <[email protected]>

* remove version 3.6
arrow package requries 4.0 or newer

* spelling checking fix trial

* Revert "spelling checking fix trial"

This reverts commit 1df8bc4.

* new github spell check workflows

* Revert "new github spell check workflows"

This reverts commit a35dc65.

* trial spell checking

* update expected word list

* update word list

* Update metadata

check-spelling run (push) for 966-github-action-spell-checking-issues-cannot-properly-recognize-variants

Signed-off-by: check-spelling-bot <[email protected]>
on-behalf-of: @check-spelling <[email protected]>

* spell checking update

* Update metadata

check-spelling run (pull_request_target) for June-24-update

Signed-off-by: check-spelling-bot <[email protected]>
on-behalf-of: @check-spelling <[email protected]>

---------

Signed-off-by: check-spelling-bot <[email protected]>
Co-authored-by: Jennifer Thom <[email protected]>
Co-authored-by: Jennit07 <[email protected]>
Co-authored-by: Jennit07 <[email protected]>
Co-authored-by: Megan McNicol <[email protected]>
Co-authored-by: SwiftySalmon <[email protected]>
Co-authored-by: marjom02 <[email protected]>
Co-authored-by: lizihao-anu <[email protected]>
Co-authored-by: rchlv <[email protected]>
Co-authored-by: rachev04 <[email protected]>
Co-authored-by: rchlv <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@SwiftySalmon SwiftySalmon Awaiting requested review from SwiftySalmon

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@lizihao-anu @SwiftySalmon