Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: large refactoring #303

Merged
merged 6 commits into from
Mar 25, 2024
Merged

ENH: large refactoring #303

merged 6 commits into from
Mar 25, 2024

Conversation

bzah
Copy link
Member

@bzah bzah commented Jan 30, 2024

Pull Request to resolve #184

  • Unit tests cover the changes.
  • These changes were tested on real data.
  • The relevant documentation has been added or updated.
  • A short description of the changes has been added to doc/source/references/release_notes.rst.

Describe the changes you made

Goal 1: Add DCSC indices

This pull request add the indices required for DRIAS in a new StandardIndexRegistry DcscIndexRegistry.
Additionally, a new parameter offset has been added to Threshold class and build_threshold function.
This allows to easily compute indices such as count days where tasmax is 5 deg_C above normal :

thresh = build_threshold(operator=">", value="normal_tasmax.nc", offset="5 delta_deg_C")
result = icclim.count_occurrences("tasmax.nc", threshold=thresh)

Todo:

  • Finish architecture refactoring
  • Add automated tests
  • Update CI to generate the new indices.
  • Add documentation
  • Add example notebook
  • Remove icclim.dcsc.dcsc_indices_functions.py and autogenerate functions from the indices definition (similarly to ecad indices)
  • Some indices details need to be checked with MF to ensure their correctness.
  • cleanup code (todos, commented code)
  • Add missing indices:
    vents:
        FF3   Écart du cumul du nombre de jours sans vent
              (par rapport à la référence 1976-2005)1
    humidité
        HUSAV   Humidité spécifique moyenne journalière

icclim will perform a unit conversion if needed.

The following indices were added:

  • TAV
  • TXAV
  • TRAV
  • TX10
  • TX90
  • TN10
  • TN90
  • TNFD
  • TXFD
  • SD
  • TX35
  • TR
  • TXND
  • TNHT
  • TNND
  • TNCWD
  • TXHWD
  • HDD
  • CDD
  • PAV
  • PINT
  • RR
  • RR1MM
  • PN20MM
  • PXCDD
  • PXCWD
  • R99
  • PFL90
  • FFAV
  • FF98
  • PQ90
  • PQ99
  • FFQ98

Goal 2: Refactor to improve maintainability

context: by adding the DCSC indices many issues within the code base were exposed thus this PR includes to improve this situation.

Issues
  • Lack of docstrings for the public methods and for modules
    --> Use of github copilot to help generating those.
  • On our doc website, no access to the existing docstring
    --> Use autoapi to generate it once
  • Lack of type hinting in some functions and classes
    --> Enable additional ruff rules and add type hints
  • Mutable state of the Indicators when computing the metadata, corrupting the subsequent calls
    --> Add a clone method to the relevant classes
  • extract_icclim_funs script is far from easy to augment when creating new StandardIndex registries (like the DCSC one)
    --> refactor the script to ease maintenance of it.
  • The architecture does not separate a public API, which should not be updated to often to avoid breaking changes and a private API
    --> Inspired by numpy 2.0 _core module, segregate the private API in icclim._core and expose the public API in icclim module

Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

@bzah bzah marked this pull request as draft January 30, 2024 18:26
@bzah bzah force-pushed the add_dcsc_indices branch 2 times, most recently from 6b492cb to e44e455 Compare February 27, 2024 16:22
Copy link

github-actions bot commented Feb 29, 2024

Coverage

Report
FileStmtsMissCoverMissing
/opt/hostedtoolcache/Python/3.9.18/x64/lib/python3.9/site-packages/icclim
   logger.py862966%24, 44, 66, 71–96, 106, 112–143, 151–156
   main.py2703288%60–70, 131, 140, 491–492, 658–659, 727, 753–754, 756–757, 759, 761–762, 764–765, 789–796, 866–871
   rechunk.py90792%29–32, 126, 228–229, 233
/opt/hostedtoolcache/Python/3.9.18/x64/lib/python3.9/site-packages/icclim/_core
   climate_variable.py992080%31–42, 156–162, 264–265, 367–368, 377, 411, 421–422
   frequency.py1801393%39–41, 258–261, 449–453, 470, 553–557, 579, 584
   input_parsing.py2636376%28–37, 92–93, 97–98, 103–108, 140–141, 269, 274–278, 281, 301–305, 307–308, 349–350, 402–406, 409, 463, 534–539, 543, 545, 635, 686, 692–700, 731, 735–739, 754–758, 785, 799, 804–809, 816–821, 832–833, 837–844, 860
   utils.py14286%30–35
/opt/hostedtoolcache/Python/3.9.18/x64/lib/python3.9/site-packages/icclim/_core/generic
   functions.py3198374%54–63, 154, 238–239, 980–981, 987, 1021, 1027, 1081–1085, 1087–1088, 1106–1112, 1116, 1169, 1171, 1234–1235, 1244–1270, 1298–1318, 1334, 1342–1374, 1380–1383
   indicator.py1501789%36–41, 158–161, 165, 408, 416–420, 422–426, 461, 507–508, 530
/opt/hostedtoolcache/Python/3.9.18/x64/lib/python3.9/site-packages/icclim/_core/generic/threshold
   basic.py104892%31–34, 72, 172–173, 198, 324
   bounded.py551376%23–29, 54–56, 84–88, 92–96, 227
   percentile.py1341390%40–47, 215–216, 339–344, 458–460
/opt/hostedtoolcache/Python/3.9.18/x64/lib/python3.9/site-packages/icclim/_core/legacy/user_index
   model.py20200%3–43
   parse.py851582%21–26, 60–61, 79, 86, 90–94, 204, 229, 235–236
/opt/hostedtoolcache/Python/3.9.18/x64/lib/python3.9/site-packages/icclim/_core/model
   cf_calendar.py35197%63
   global_metadata.py660%3–23
   icclim_types.py14140%3–25
   in_file_dictionary.py990%3–46
   index_config.py31681%13–18
   indicator.py36878%10–11, 23, 28, 67, 72, 77, 82
   logical_link.py21195%14
   operator.py32391%13, 18–22
   standard_index.py36975%9–16, 68, 72, 77
   standard_variable.py48198%48
   threshold.py541081%15–25, 95, 100, 149
/opt/hostedtoolcache/Python/3.9.18/x64/lib/python3.9/site-packages/icclim/_generated
   _dcsc.py1015546%20–32, 136, 217, 298, 407, 527, 647, 767, 859, 943, 1027, 1111, 1195, 1413–1419, 1524–1530, 1635–1641, 1746–1752, 1834, 1918, 2002, 2083, 2167, 2248, 2332, 2416, 2500, 2612, 2733, 2854, 2975, 3083, 3193
   _ecad.py144894%19–31
   _generic.py591083%18–33
/opt/hostedtoolcache/Python/3.9.18/x64/lib/python3.9/site-packages/icclim/ecad
   binding.py58886%15–17, 45, 49, 86, 90, 127, 131
/opt/hostedtoolcache/Python/3.9.18/x64/lib/python3.9/site-packages/icclim/generic
   registry.py41490%39, 46–47, 69
/opt/hostedtoolcache/Python/3.9.18/x64/lib/python3.9/site-packages/icclim/threshold
   factory.py1691293%38, 177–178, 255, 279–280, 285, 355–356, 359–363, 403
TOTAL310150084% 

Test results

Tests Skipped Failures Errors Time
196 0 💤 0 ❌ 0 🔥 2m 3s ⏱️

@bzah
Copy link
Member Author

bzah commented Feb 29, 2024

pre-commit.ci autofix

@bzah bzah force-pushed the add_dcsc_indices branch 3 times, most recently from 7170579 to 100fae0 Compare March 20, 2024 12:27
@bzah bzah changed the title [WIP] ENH: Add dcsc indices [WIP] ENH: large refactoring Mar 21, 2024
Abel Aoun added 5 commits March 22, 2024 13:44
The goal here is to properly expose the public api and hide
icclom internals in a ``_core`` package.
Added ruff rules.
Added sphinx extensions.
@bzah bzah marked this pull request as ready for review March 25, 2024 08:14
@bzah bzah changed the title [WIP] ENH: large refactoring ENH: large refactoring Mar 25, 2024
Make use of autodoc to generate dynamically the doc for
the generated functions.
@bzah bzah merged commit 47a7152 into master Mar 25, 2024
5 checks passed
@bzah bzah deleted the add_dcsc_indices branch March 25, 2024 13:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

ENH: Add Cooling Degree Days (CDDCold{xx})
1 participant