December 2024 Update

[lizihao-anu]

Issues:

• Improve and automate check_year_valid() Re-write check_year_valid function #995
• Add temporary files for processing and debugging #1011
• Bug: warning type does not work properly as expected in data pipeline magrittr %>% #1010
• - Time consumption documentation. How long does each function take to process? How long is the process for ep file start to finish? Same for individual file.
• Following the use of temporary files, create test_mode = TRUE for only testing one year of data. This will only create temporary files for one year to save space in sourcedev. The idea is to test one year and if everything is working then we can proceed to running all years as the assumption is that the code will work because it should be identical.
• Moving delayed discharge and homelessness to the beginning within create_episode_file() (low priority and implications) #1018
• Remove person_id (including comments) from extracts code Clean up code  #1006
• handle the issue of file-in-use for extracts tests writing to xlsx handle the issue of file-in-use for extracts tests writing to xlsx  #1001
• clear out test folder in hscdiip and decide how many old files we would like to keep
• create subfolders for ep and ind tests
• contact social care team to check if we are extracting data correctly targets does not read all raw data from data warehouse properly #1000 - meeting set for 25/09/24
• create a marker linking CH to AE Create a marker for AE attendences from a care home #1008
• add tests and warning messages for reading social care raw data add tests and warning messages for reading social care raw data #1002
• check distinct death data by chi while keeping chi==NA records check distinct death data by chi while keeping chi==NA records #999
• this join_by is not 100% accurate. Consider use ep_file_row_id to join this join_by is not 100% accurate. Consider use ep_file_row_id to join #998
• Investigate 1819 processing Investigate 1819 processing #997
• Bug - Geography differences Bug - Geography differences #996
• Delayed discharges investigations Delayed discharges investigations  #994
• Change request completed for substance misuse flag Investigate ICD10 codes for drug related admissions #836
• Update documentation: which data are quarterly/annual updated, which data we need to request by ourselves. This includes Add checks for latest version of files #955
• Update documentation. We can create flow charts to help visualise the process. This was particularly helpful for understanding the deaths processing due to the complexities.
• Check the substance misuse flag in the December update and add this to the update email to make people aware this CR has completed
• add to draft email - extracts on DD data is based up to XXX month, Also for SC data- Data based up to XXX period