Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve memory usage of evaluator #688

Merged
merged 2 commits into from
Mar 8, 2025

Conversation

andrews05
Copy link
Collaborator

@andrews05 andrews05 commented Mar 7, 2025

This adds a new final_round parameter to the evaluator, which allows us to determine when to keep the filtered data and when to keep the compressed idat_data. By only keeping one of these we can significantly improve memory usage.

Ref #683

@andrews05
Copy link
Collaborator Author

andrews05 commented Mar 7, 2025

Some stats on a set of 10 large files. 8a44cdbc is the commit prior to refactoring optimize_raw which caused higher memory usage in 9.1.4.
At o6 the memory usage drops back to what it was before, while at o2 (fast mode) it's even lower than it was before.

memory -sao6
       75.70 real       680.67 user         2.76 sys
          2081685504  maximum resident set size
          1791613376  peak memory footprint

memory -sao2
       11.64 real        65.60 user         1.25 sys
          2752004096  maximum resident set size
          2214273280  peak memory footprint

memory -sao2 -t1
       59.53 real        58.39 user         0.76 sys
           551501824  maximum resident set size
           303793792  peak memory footprint

master -sao6
       74.11 real       680.50 user         2.79 sys
          3215491072  maximum resident set size
          2571888128  peak memory footprint

master -sao2
       11.62 real        65.42 user         1.24 sys
          3314466816  maximum resident set size
          2761025024  peak memory footprint

master -sao2 -t1
       59.11 real        58.32 user         0.76 sys
           644005888  maximum resident set size
           323684288  peak memory footprint

8a44cdbc -sao6
       74.44 real       682.79 user         2.83 sys
          2097070080  maximum resident set size
          1723193408  peak memory footprint

8a44cdbc -sao2
       11.63 real        65.88 user         1.21 sys
          3353853952  maximum resident set size
          2744837632  peak memory footprint

8a44cdbc -sao2 -t1
       59.32 real        58.52 user         0.78 sys
           697843712  maximum resident set size
           359794560  peak memory footprint

Copy link
Collaborator

@AlexTMjugador AlexTMjugador left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch!

@AlexTMjugador AlexTMjugador merged commit 7650953 into shssoichiro:master Mar 8, 2025
12 checks passed
@andrews05 andrews05 deleted the memory branch March 8, 2025 20:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants