Update 00_sort_bi_extracts to write anon_chi

[Jennit07]

As part of ensuring anon-chi is used on disk, an update to sort bi extract script. Opening as a draft for now as this works for extracts with chi variable available but we do take some extracts without chi e.g. gp ooh data. Some thought is needed to expand the code to work around this. Possibly a conditional statement.

[lizihao-anu]

Update the 00_Sort_BI_Extracts.R to

• enable parallel computing
• tell whether a file have a chi column or not

It works well as I tested. But I do have a question about the new file name. Since we have added a prefix "anon-" for those files, will it break the existing data pipeline? If so, we should

• modify the functions which pick up those files. OR,
• we just do not add the prefix to file names.

[Jennit07]

Update the 00_Sort_BI_Extracts.R to

• enable parallel computing
• tell whether a file have a chi column or not

It works well as I tested. But I do have a question about the new file name. Since we have added a prefix "anon-" for those files, will it break the existing data pipeline? If so, we should

• modify the functions which pick up those files. OR,
• we just do not add the prefix to file names.

Hi @lizihao-anu Thanks for all your help with this. We want to keep the anon- prefix for consistency as the updated get_boxi_extract path has renamed this for all extracts when we are changing everything to anon_chi. I have made the changes we have discussed and also tested a file. I am happy the function is working as intended.

I will mark this PR ready for review. Could you please do a final check to confirm you are happy with this and then approve/merge into the June24 update branch. Thanks! :)

[github-actions]

@check-spelling-bot Report

🔴 Please review

See the 📂 files view, the 📜action log, or 📝 job summary for details.

Unrecognized words (1)

sourcedev

To accept these unrecognized words as correct, you could run the following commands

... in a clone of the [email protected]:Public-Health-Scotland/source-linkage-files.git repository
on the amend_sort_bi_extracts branch (ℹ️ how do I use this?):

curl -s -S -L 'https://raw.githubusercontent.com/check-spelling/check-spelling/main/apply.pl' |
perl - 'https://github.com/Public-Health-Scotland/source-linkage-files/actions/runs/9172398523/attempts/1'

OR

To have the bot accept them for you, reply quoting the following line:
@check-spelling-bot apply updates.

Errors (1)

See the 📂 files view, the 📜action log, or 📝 job summary for details.

❌ Errors	Count
❌ forbidden-pattern	1

See ❌ Event descriptions for more information.

If the flagged items are 🤯 false positives

If items relate to a ...

• binary file (or some other file you wouldn't want to check at all).

  Please add a file path to the excludes.txt file matching the containing file.

  File paths are Perl 5 Regular Expressions - you can test yours before committing to verify it will match your files.

  ^ refers to the file's path from the root of the repository, so ^README\.md$ would exclude README.md (on whichever branch you're using).

• well-formed pattern.

  If you can write a pattern that would match it,
  try adding it to the patterns.txt file.

  Patterns are Perl 5 Regular Expressions - you can test yours before committing to verify it will match your lines.

  Note that patterns can't match multiline strings.