Create HRI Variables

[shintoLamp]

I have written some code to add the HRI variables to the individual file, but I need someone to check the functionality.

The main issue is that the individual file I was using for testing (2019/20) has 5,983,038 rows, and when the HRIs are matched on there's 5,986,774. So we've gained 3736 rows. I imagine that the left_join at the end of add_hri_variables is doing something unexpected.

Could someone run through the code please and try and figure out what's going on?

[Moohan]

Looks great, I haven't tested in R and I couldn't obviously see where you might be getting duplicate rows.

One suggestion would be to use the new error checking on the joins (https://www.tidyverse.org/blog/2023/01/dplyr-1-1-0-joins/#unmatched-rows)

For a lookup you can be really explicit and do something like:

left_join(
data,
lookup,
by = "lookup_var",
na_matches = "never",
unmatched = "drop",
relationship = "one-to-one")

I think the relationship arg is the only none default one there. https://dplyr.tidyverse.org/reference/mutate-joins.html#arguments

[lizihao-anu]

The duplicates in individual files may contribute to the issue. I am investigating it. #786

[Moohan]

The main issue is that the individual file I was using for testing (2019/20) has 5,983,038 rows, and when the HRIs are matched on there's 5,986,774. So we've gained 3736 rows. I imagine that the left_join at the end of add_hri_variables is doing something unexpected.

@lizihao-anu identified that the individual file has a number of duplicated rows, this looks to be (at least in part) because the SC client extract wasn't unique. Zihao is looking at this issue separately.

I've checked the HRI code on a 'deduplicated' individual file (I just did distinct(chi)) and the HRI code doesn't produce any new duplicates, so I'm generally happy with it pending my other comments and a final review 😄

[Jennit07]

@lizihao-anu Check this PR please, the test would be to check the numbers against the archived 1718 fst file in hscdiip

[lizihao-anu]

@lizihao-anu Check this PR please, the test would be to check the numbers against the archived 1718 fst file in hscdiip

As I check, if the duplicated rows in the 1718 individual file were eliminated, then there would not be a difference in row numbers between HRI and the individual file. It means the code is fine from this perspective.

The new individual file produced by R for 1718 is around 0.11% rows than the old file produced by SPSS. So it should be cool!

[Jennit07]

@lizihao-anu Check this PR please, the test would be to check the numbers against the archived 1718 fst file in hscdiip

As I check, if the duplicated rows in the 1718 individual file were eliminated, then there would not be a difference in row numbers between HRI and the individual file. It means the code is fine from this perspective.

The new individual file produced by R for 1718 is around 0.11% rows than the old file produced by SPSS. So it should be cool!

Thanks Zihao! Great work, we can discuss this tomorrow!

[Jennit07]

Looks good, i think this is ready to merge!

[github-actions]

@check-spelling-bot Report

🔴 Please review

See the 📂 files view or the 📜action log for details.

Unrecognized words (10)

gls
hbp
hhg
hri
lcap
nrs
prac
scotp
sparra
telecare

To accept ✔️ these unrecognized words as correct and remove the previously acknowledged and now absent words, run the following commands

... in a clone of the [email protected]:Public-Health-Scotland/source-linkage-files.git repository
on the d03_create_hri_variables branch (ℹ️ how do I use this?):

curl -s -S -L 'https://raw.githubusercontent.com/check-spelling/check-spelling/main/apply.pl' |
perl - 'https://github.com/Public-Health-Scotland/source-linkage-files/actions/runs/5952010040/attempts/1'

To have the bot do this for you, reply quoting the following line:
@check-spelling-bot apply updates.

Available 📚 dictionaries could cover words not in the 📘 dictionary

This includes both expected items (220) from .github/actions/spelling/expect.txt and unrecognized words (10)

Dictionary	Entries	Covers
cspell:sql/src/tsql.txt	455	1
cspell:python/src/python/python.txt	453	1
cspell:python/src/python/python-lib.txt	3873	1
cspell:latex/latex.txt	424	1
cspell:html/html.txt	1863	1
cspell:elixir/elixir.txt	103	1
cspell:django/django.txt	859	1

Consider adding them using (in .github/workflows/spelling.yml):

      with:
        extra_dictionaries:
          cspell:sql/src/tsql.txt
          cspell:python/src/python/python.txt
          cspell:python/src/python/python-lib.txt
          cspell:latex/latex.txt
          cspell:html/html.txt
          cspell:elixir/elixir.txt
          cspell:django/django.txt

To stop checking additional dictionaries, add:

      with:
        check_extra_dictionaries: ''

If the flagged items are 🤯 false positives

If items relate to a ...

• binary file (or some other file you wouldn't want to check at all).

  Please add a file path to the excludes.txt file matching the containing file.

  File paths are Perl 5 Regular Expressions - you can test yours before committing to verify it will match your files.

  ^ refers to the file's path from the root of the repository, so ^README\.md$ would exclude README.md (on whichever branch you're using).

• well-formed pattern.

  If you can write a pattern that would match it,
  try adding it to the patterns.txt file.

  Patterns are Perl 5 Regular Expressions - you can test yours before committing to verify it will match your lines.

  Note that patterns can't match multiline strings.