758 investigate extracts to identify areas of code which can be cut down for processing times

[lizihao-anu]

As discussed in #758, two functions, process_sc_all_sds and process_sc_all_alarms_telecare, take a very long time to execute. I have re-written the functions using data.table to improve the efficiency, including the speed and memory usage. The improvement is ridiculously significant.

Here's a summary comparing the improvements in two functions, process_sc_all_sds and process_sc_all_alarms_telecare:

Function Name	Old Method	New Method	Improvement
process_sc_all_sds	24.94 mins	1.69 mins	94% reduction
process_sc_all_alarms_telecare	31.81 mins	0.81 mins	98% reduction

In both cases, the new methods produce identical results to the old methods but demonstrate significant reductions in running time. The process_sc_all_sds function saw a 94% reduction in running time, while the process_sc_all_alarms_telecare function experienced an even more impressive 98% reduction. These improvements are a testament to the effectiveness of the rewritten functions in optimizing their performance.

[SwiftySalmon]

Ran the sds code and it's much much faster.

Only issue is that I've made a lot of changes to the sds code! Might need to merge my pull request and then make some changes to this one?

[lizihao-anu]

Ran the sds code and it's much much faster.

Only issue is that I've made a lot of changes to the sds code! Might need to merge my pull request and then make some changes to this one?

Yes. I agree. We can merge your changes to sds first, and the I can rewrite the code for sds.

[Jennit07]

Hi @lizihao-anu PR #900 has now merged. When you are back from A/L would you be able to have a look at the SDS code again and make any changes to have this in data table code please? Thanks!

[github-actions]

@check-spelling-bot Report

🔴 Please review

See the 📂 files view, the 📜action log, or 📝 job summary for details.

Unrecognized words (3)

fcase
fifelse
setorder

To accept these unrecognized words as correct, you could run the following commands

... in a clone of the [email protected]:Public-Health-Scotland/source-linkage-files.git repository
on the 758-investigate-extracts-to-identify-areas-of-code-which-can-be-cut-down-for-processing-times branch (ℹ️ how do I use this?):

curl -s -S -L 'https://raw.githubusercontent.com/check-spelling/check-spelling/main/apply.pl' |
perl - 'https://github.com/Public-Health-Scotland/source-linkage-files/actions/runs/8079543041/attempts/1'

OR

To have the bot accept them for you, reply quoting the following line:
@check-spelling-bot apply updates.

Errors (2)

See the 📂 files view, the 📜action log, or 📝 job summary for details.

❌ Errors	Count
❌ ignored-expect-variant	13
ℹ️ no-newline-at-eof	1

See ❌ Event descriptions for more information.

If the flagged items are 🤯 false positives

If items relate to a ...

• binary file (or some other file you wouldn't want to check at all).

  Please add a file path to the excludes.txt file matching the containing file.

  File paths are Perl 5 Regular Expressions - you can test yours before committing to verify it will match your files.

  ^ refers to the file's path from the root of the repository, so ^README\.md$ would exclude README.md (on whichever branch you're using).

• well-formed pattern.

  If you can write a pattern that would match it,
  try adding it to the patterns.txt file.

  Patterns are Perl 5 Regular Expressions - you can test yours before committing to verify it will match your lines.

  Note that patterns can't match multiline strings.

[lizihao-anu]

Ready to merge

changes made