Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pick up latest geography file, and save out with compression #983

Merged
merged 3 commits into from
Jul 24, 2024

Conversation

SwiftySalmon
Copy link
Collaborator

No description provided.

This comment has been minimized.

Co-authored-by: James McMahon <[email protected]>
Copy link

@check-spelling-bot Report

🔴 Please review

See the 📂 files view, the 📜action log, or 📝 job summary for details.

Unrecognized words (3)

pcs
Postcodesio
recordlinkage

To accept these unrecognized words as correct, you could run the following commands

... in a clone of the [email protected]:Public-Health-Scotland/source-linkage-files.git repository
on the postcode_filepath branch (ℹ️ how do I use this?):

curl -s -S -L 'https://raw.githubusercontent.com/check-spelling/check-spelling/main/apply.pl' |
perl - 'https://github.com/Public-Health-Scotland/source-linkage-files/actions/runs/9991969066/attempts/1'

OR

To have the bot accept them for you, reply quoting the following line:
@check-spelling-bot apply updates.

Available 📚 dictionaries could cover words (expected and unrecognized) not in the 📘 dictionary

This includes both expected items (307) from .github/actions/spelling/expect.txt and unrecognized words (3)

Dictionary Entries Covers Uniquely
cspell:fullstack/dict/fullstack.txt 419 3 3
cspell:k8s/dict/k8s.txt 153 4 1
cspell:php/dict/php.txt 1689 4
cspell:node/dict/node.txt 891 3 1
cspell:npm/dict/npm.txt 302 3

Consider adding them (in .github/workflows/spelling.yml) for uses: check-spelling/check-spelling@main in its with:

      with:
        extra_dictionaries:
          cspell:fullstack/dict/fullstack.txt
          cspell:k8s/dict/k8s.txt
          cspell:php/dict/php.txt
          cspell:node/dict/node.txt
          cspell:npm/dict/npm.txt

To stop checking additional dictionaries, add (in .github/workflows/spelling.yml) for uses: check-spelling/check-spelling@main in its with:

check_extra_dictionaries: ''
Errors (1)

See the 📂 files view, the 📜action log, or 📝 job summary for details.

❌ Errors Count
❌ ignored-expect-variant 3

See ❌ Event descriptions for more information.

If the flagged items are 🤯 false positives

If items relate to a ...

  • binary file (or some other file you wouldn't want to check at all).

    Please add a file path to the excludes.txt file matching the containing file.

    File paths are Perl 5 Regular Expressions - you can test yours before committing to verify it will match your files.

    ^ refers to the file's path from the root of the repository, so ^README\.md$ would exclude README.md (on whichever branch you're using).

  • well-formed pattern.

    If you can write a pattern that would match it,
    try adding it to the patterns.txt file.

    Patterns are Perl 5 Regular Expressions - you can test yours before committing to verify it will match your lines.

    Note that patterns can't match multiline strings.

@Jennit07 Jennit07 merged commit 6ceaa72 into add_new_nsu Jul 24, 2024
9 of 10 checks passed
@Jennit07 Jennit07 deleted the postcode_filepath branch July 24, 2024 13:10
Jennit07 added a commit that referenced this pull request Aug 16, 2024
* Add NSU code to github
Includes extracting the service user cohort to send to the chili team and then NSU extraction.

* Style code

* Add compression and package library

* Style code

* pick up latest geography file, and save out with compression (#983)

* pick up latest geography file, and save out with compression

* Style code

* use `get_spd_path`

Co-authored-by: James McMahon <[email protected]>

---------

Co-authored-by: marjom02 <[email protected]>
Co-authored-by: SwiftySalmon <[email protected]>
Co-authored-by: Jennit07 <[email protected]>
Co-authored-by: James McMahon <[email protected]>

---------

Co-authored-by: Jennit07 <[email protected]>
Co-authored-by: Megan McNicol <[email protected]>
Co-authored-by: marjom02 <[email protected]>
Co-authored-by: SwiftySalmon <[email protected]>
Co-authored-by: James McMahon <[email protected]>
github-merge-queue bot pushed a commit that referenced this pull request Sep 17, 2024
* update documentation

* Update sc connection name

* Update documentation

* 936 - Update parameters with file paths  (#939)

Specify file paths in sc function parameters

* remove and merge overlapping records in GP OoHs

* Style code

* update spelling to lowercases

* update spelling

* Create tests for social care sandpit extracts (#943)

* Update `write_tests_xlsx`

* Update documentation

* Add in sandpit tests where the extract is saved

* Setup tests for sandpit
Further checks needed for writing to disk

* Update documentation

* Amend case_when statement

* rename function to include 'sc'

* Update documentation

* Use `is.null` instead of `missing`

* Update documentation

* Add `year` as a parameter

* Update documentation

* Setup for writing sandpit tests to disk

* Update parameters for sandpit tests

* Update documentation

* Use `process_tests_sc_sandpit`

* Apply styling

* Style code

* update documentation

Co-authored-by: Zihao Li <[email protected]>

* Rename variable sc_id

Co-authored-by: Zihao Li <[email protected]>

* Rename variable

Co-authored-by: Zihao Li <[email protected]>

* Rename variable

Co-authored-by: Zihao Li <[email protected]>

* Update documentation

* [check-spelling] Update metadata

Update for https://github.com/Public-Health-Scotland/source-linkage-files/actions/runs/8689503990/attempts/1
Accepted in #943 (comment)

Signed-off-by: check-spelling-bot <[email protected]>
on-behalf-of: @check-spelling <[email protected]>

* update spelling

* update spelling expect variant

---------

Signed-off-by: check-spelling-bot <[email protected]>
Co-authored-by: Jennit07 <[email protected]>
Co-authored-by: Zihao Li <[email protected]>
Co-authored-by: Zihao Li <[email protected]>

* only select columns we want in ltc raw data

* [check-spelling] Update metadata

Update for https://github.com/Public-Health-Scotland/source-linkage-files/actions/runs/8897746003/attempts/1
Accepted in #947 (comment)

Signed-off-by: check-spelling-bot <[email protected]>
on-behalf-of: @check-spelling <[email protected]>

* for some reason the latest scid code was overwritten after the march update?? anyway, now it is fixed.

* Style code

* Merge May24 NI update into June update branch (#949)

Collect data before manipulations

* Update NEWS.md

* link GP-OoH with CUP markers

* Style code

* update gp ooh cup

* link cup to acute

* Update documentation

* adding the death dates to activity after death cases (#972)

adding the death dates to the cases where there is activity after death

Co-authored-by: marjom02 <[email protected]>

* Add sys time to functions (#971)

* adding in syst_time alerts for all functions in create episode and create individual. So that when it runs manually I can see where it is and where it's getting stuck

* Style code

---------

Co-authored-by: marjom02 <[email protected]>
Co-authored-by: SwiftySalmon <[email protected]>
Co-authored-by: Jennit07 <[email protected]>

* Update slf deaths lookup function name

* automate combined deaths lookup

* Update documentation

* Update targets script

* Update years

* Update running process manually

* re-arrange brackets

* Style code

* Update run targets scripts

* Set up all-targets script

* Style code

* Update documentation

* Update targets script

* Update years

* Update running process manually

* re-arrange brackets

* Style code

* Update run targets scripts

* Set up all-targets script

* Style code

* Style code

* Automate the combined slf deaths lookup (#973)

Closes #957 and #968

* Update documentation

* Update Run_SLF_Files_targets/run_all_targets.R

Co-authored-by: Zihao Li <[email protected]>

* Style code

* remove combined_deaths_lookup from targets

* Style code

* fix acute_cup and gp_ooh_cup paths

* Update documentation

* fix typo

* adapt acute_cup for anon_chi

* Style code

* minor changes

* Style code

* Update documentation

* Person id sds (#981)

* added back in missing person_id for SDS.
also added latest_flag back in to client lookup in targets

* Update documentation

* added back in missing person_id for SDS.
also added latest_flag back in to client lookup in targets

* Update documentation

* change as suggested by Jen

* Update documentation

---------

Co-authored-by: marjom02 <[email protected]>
Co-authored-by: SwiftySalmon <[email protected]>
Co-authored-by: Jennifer Thom <[email protected]>
Co-authored-by: Jennit07 <[email protected]>

* Update NEWS.md

* unify file names for cup files

* Add client flags (#979)

* New methodology for social care client data.

- removed code that wasn't needed.
- updated housing codes
- latest social care ID
- changed "mental health problems" to "mental health disorders" in line with PHS style guide

* Update documentation

* Style code

* add person ID to client so it carries through to match on to all cases

* New methodology for social care client data.

- removed code that wasn't needed.
- updated housing codes
- latest social care ID
- changed "mental health problems" to "mental health disorders" in line with PHS style guide

* Update documentation

* Style code

* add person ID to client so it carries through to match on to all cases

---------

Co-authored-by: marjom02 <[email protected]>
Co-authored-by: SwiftySalmon <[email protected]>
Co-authored-by: Jennifer Thom <[email protected]>
Co-authored-by: Jennit07 <[email protected]>

* Update lookup to use anon-chi

* Remove redundant code
This uses the NRS Weekly dates and if this is blank use the chi death date. This methodology is wrong. We want to use the monthly nrs boxi date by default and chi date if there is an issue

* Update documentation

* remove weekly nrs date variable

* Use boxi nrs date or chi death date

* Use `get_combined_slf_deaths_path`

* add catch for NAs

* add notes

* Fix typo

* remove redundant code

* Style code

* add a function of combine nrs and it_chi death

* Update documentation

* Style code

* minor changes

* remove process_slf_deaths_lookup

* Update documentation

* Major update of Care Home script (#945)

* # major changes to care home script

see document on sharepoint for description

also:
- added in type of admission description
- updated care home contact in fill_ch_name script

* minor note updates

* Style code

* Update documentation

* couple of note updates

* Update R/process_sc_all_care_home.R

Co-authored-by: Jennit07 <[email protected]>

* Update R/process_sc_all_care_home.R

Co-authored-by: Jennit07 <[email protected]>

* Style code

* Update R/process_sc_all_care_home.R

* change to ch name lookup

* Update documentation

* remove fill ch provider fill line

* update fill ch names so it works with new ch methodology

* Style code

* Update documentation

* Style code

* Update documentation

* Remove redundant variable `latest_sc_id`

* use slfhelper::get_chi

* new section for sc_ch_id_markers

* Style code

* Update documentation

* Remove extra text and white space

* add rename to use death_date_chi

* use `read_excel` function

* Update documentation

* Return the paths only for SPD and ch name

* Update documentation

* Remove rename - no longer needed

* fix typo

* remove variables that dont exist

---------

Co-authored-by: marjom02 <[email protected]>
Co-authored-by: SwiftySalmon <[email protected]>
Co-authored-by: Jennit07 <[email protected]>
Co-authored-by: Jennit07 <[email protected]>
Co-authored-by: Jennifer Thom <[email protected]>

* update `ch_chi_cis` methodology

* update `ch_sc_id_cis` methodology

* Update notes

* Use `right_join`

* Update process_sc_all_care_home.R

added in missing variable at the end

* Add new nsu (#991)

* Add NSU code to github
Includes extracting the service user cohort to send to the chili team and then NSU extraction.

* Style code

* Add compression and package library

* Style code

* pick up latest geography file, and save out with compression (#983)

* pick up latest geography file, and save out with compression

* Style code

* use `get_spd_path`

Co-authored-by: James McMahon <[email protected]>

---------

Co-authored-by: marjom02 <[email protected]>
Co-authored-by: SwiftySalmon <[email protected]>
Co-authored-by: Jennit07 <[email protected]>
Co-authored-by: James McMahon <[email protected]>

---------

Co-authored-by: Jennit07 <[email protected]>
Co-authored-by: Megan McNicol <[email protected]>
Co-authored-by: marjom02 <[email protected]>
Co-authored-by: SwiftySalmon <[email protected]>
Co-authored-by: James McMahon <[email protected]>

* update reference

* Reduce dependencies (#984)

* removing packages that I don't think get used anywhere. and removing references to fst and spss files

* Update documentation

* Update authors in description

* [check-spelling] Update metadata

Update for https://github.com/Public-Health-Scotland/source-linkage-files/actions/runs/10469297653/attempts/1
Accepted in #984 (comment)

Signed-off-by: check-spelling-bot <[email protected]>
on-behalf-of: @check-spelling <[email protected]>

* [check-spelling] Update metadata

Update for https://github.com/Public-Health-Scotland/source-linkage-files/actions/runs/10469690723/attempts/1https://github.com/Public-Health-Scotland/source-linkage-files/actions/runs/10469690723/attempts/1
Accepted in #984 (comment)

Signed-off-by: check-spelling-bot <[email protected]>
on-behalf-of: @check-spelling <[email protected]>

---------

Signed-off-by: check-spelling-bot <[email protected]>
Co-authored-by: marjom02 <[email protected]>
Co-authored-by: SwiftySalmon <[email protected]>
Co-authored-by: Jennifer Thom <[email protected]>
Co-authored-by: Jennit07 <[email protected]>
Co-authored-by: Jennit07 <[email protected]>

* minor changes to social care code

* Style code

* Update process_sc_all_care_home.R

* Update NEWS.md

* merge Sep2024 fix into sep24 branch (#1003)

* update write_tests_xlsx

* update process_refined_death

* fix tests by removing get_chi

* add 2425

* Style code

* fix NA matches in refined_death

* move latest_cost_year() to cost_uplift()

* improve automation

* Update documentation

* fix `cij_ppa` in DD data

* fix bugs of dd and populate cij_delay back to episodes

* Style code

* keep all variable for delayed discharge episodes

* remove dummy variable names from dd_date

* Style code

* remove `deceased_boxi` variable - bug

* remove `create_person_id`. Its matched in client

* remove `create_person_id`

* Update `run_slf_manually` scripts

* further remove person_id

* fix duplicate row introduced by adding death

* remove duplicated chi when joining death data

* TODO: check distinct death data by chi while keeping chi==NA records

* add parameter for year

* fix duplicate in add_activity_after_death_flag

* Update `check_year_valid`

* Declare DN variables

* Style code

* remove redundant variables

---------

Co-authored-by: Zihao Li <[email protected]>
Co-authored-by: lizihao-anu <[email protected]>
Co-authored-by: Jennit07 <[email protected]>

* update copy_to_hscdiip.R

* Update older years to bring the data in line with our newest processes.  (#988)

* fix sc_client_lookup sc_send_lca

* fix an issue of get_pop_path

* Style code

* fix the rest of get_pop_path from get_datazone_pop_path

* Update documentation

* fix sc_send_lca

* add missing year column

* Remove redundant code

* Update documentation

* Style code

* explicitly specify the argument year to avoid corruption of targets

* Update documentation

* Reorder when we match on client variables
This was causing NSUs to show a social care id. This now resolves this.

* Update documentation

* Style code

* Add chi parameter to `create_demog_test_flags`

* Style code

* Use CHI parameter for ep/indiv tests

* Use CHI parameter for extract tests (chi)

* Change test sheet names to lowercase

* Change date to lowercase

* Update documentation

* new data pipeline with targets
remove create_individual_files from targets and append it to run_targets script

* minor changes

* Style code

* Update documentation

* Update documentation

* Style code

* undo sc_send_lca bit

* Add code for running years available

* Update `_targets.R` script for running old years

* Style code

* Update `check_year_valid` for running old years

* Use `check_year_valid` where no data for old yrs

* Style code

* Fix pick variables
This was not taking the correct variables, leading to NSUs being assigned psychiatry

* SC Demographics and SDS (#900)

* Style code

* # read in sc demographics

different variables - removed extract date as not accurate, using chi over upi after discussion with social care data management. Added in date of death just for fun.

* social care demographics first draft

removed a lot of the submitted variables and instead using chi variables from chi seeding. Other changes:
- Fill in missing values,
- create flag for latest social care id (one from database is not accurate), this makes sure that each chi only has ONE sc id as the latest to stop it creating duplicates
- change postcode to choose chi over submitted

* Style code

* had a github error? Not sure what happened but commiting first draft of sc demographics

* Style code

* first draft sds.
No major changes - only how demographics is matched on and how latest social care id is selected

* Update documentation

* demographics - add sending location to group by

* Style code

* Update documentation

* Added ungroup()

* Remove comments

* Remove comments

* Style code

---------

Co-authored-by: SwiftySalmon <[email protected]>
Co-authored-by: marjom02 <[email protected]>
Co-authored-by: Jennit07 <[email protected]>
Co-authored-by: Jennit07 <[email protected]>
Co-authored-by: Zihao Li <[email protected]>

* Sc all at speedup (#904)

* speed up process_sc_all_alarms_telecare function with data.table package

* Update documentation

---------

Co-authored-by: lizihao-anu <[email protected]>
Co-authored-by: Megan McNicol <[email protected]>
Co-authored-by: Jennit07 <[email protected]>

* Add case_when statement for `high_cc` cohort

* Bug - `high_cc` in demographic cohort showing `NAs` instead of `TRUE/FALSE` (#911)

Add case_when statement for `high_cc` cohort

* added a casewhen to update property type description for homelessness

* Update documentation

* Style code

* Bug - deal with missing variables (#914)

* Add missing sc variables for no sc data

* Fix code for including `_inc_dna` variables

* Remove commented line

* Bug - Fix get pop path failing and preventing the indiv file from running.  (#913)

Fix bug - pop file paths breaking indiv file

* correct file hscp file path

* Declare missing variables for older years

* setup targets scripts for old years

* Style code

* Include `check_year_valid` for sc client path

* Add check year valid to join sc client

* Add if else statement

* WIP - TO DO - fix dummy path for `get_chi()`

* Style code

* update dummy data file to read empty tibble

* Update `check_year_valid`

* Update declared `NA` variables

* Update documentation

* declare `count_not_known` as NA

* supply year as default in `aggregate_by_chi`

* Decalre unused variables

* Style code

* Update sc client with sept update new code

* Specify code for running older years

* Style code

* Add Running SLF files manually scripts

* Style code

* update write_tests_xlsx

* update process_refined_death

* fix tests by removing get_chi

* add 2425

* Style code

* fix NA matches in refined_death

* move latest_cost_year() to cost_uplift()

* improve automation

* Update documentation

* fix `cij_ppa` in DD data

* fix bugs of dd and populate cij_delay back to episodes

* Style code

* keep all variable for delayed discharge episodes

* remove dummy variable names from dd_date

* Style code

* remove `deceased_boxi` variable - bug

* remove `create_person_id`. Its matched in client

* remove `create_person_id`

* Update `run_slf_manually` scripts

* further remove person_id

* fix duplicate row introduced by adding death

* remove duplicated chi when joining death data

* TODO: check distinct death data by chi while keeping chi==NA records

* add parameter for year

* fix duplicate in add_activity_after_death_flag

* Update `check_year_valid`

* Declare DN variables

* Style code

* Declare client variables

* remove extra dd variables

* remove redundant variables

* remove fy variable

* Remove redundant variable `count_not_known`

* Remove duplicate code

* revert commit - remove fy

* update manual run

* declare missing sc variables indiv file

* Style code

---------

Co-authored-by: Zihao Li <[email protected]>
Co-authored-by: lizihao-anu <[email protected]>
Co-authored-by: Jennit07 <[email protected]>
Co-authored-by: Megan McNicol <[email protected]>
Co-authored-by: SwiftySalmon <[email protected]>
Co-authored-by: Zihao Li <[email protected]>
Co-authored-by: marjom02 <[email protected]>

* Update `replace_sc_id_with_latest` function

* Style code

---------

Signed-off-by: check-spelling-bot <[email protected]>
Co-authored-by: Jennit07 <[email protected]>
Co-authored-by: Zihao Li <[email protected]>
Co-authored-by: Zihao Li <[email protected]>
Co-authored-by: lizihao-anu <[email protected]>
Co-authored-by: marjom02 <[email protected]>
Co-authored-by: Megan McNicol <[email protected]>
Co-authored-by: SwiftySalmon <[email protected]>
Co-authored-by: rchlv <[email protected]>
Co-authored-by: James McMahon <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants