EPIC: Roadmap v1.0

[pabloitu]

As mentioned in #239, API has stablized and we need to complete minor work for v1.0.0. Here we agree on the tasks, and issues, needed to be completed before the major release.

Features:

• Correction to M-test with resampling, and addition of new MLL test #268
• Implementation of the Brier Score and its consistency test. #232

Refactoring:

• Plot Refactoring #263, which closes Plot Refactoring  #260, Basemaps downloading requires a timeout option #258
• Clean up unused functionality (e.g., documents.py, file.py, log.py and many other functions within)
• Refactor core-modules structure (might break API): In which module should gridded non-poisson tests (brier, negbinom, binary) be placed? Should each have its own module? I believe we should abstract any coupling between scores, tests and distributions: Some users might be interested in the tests, and some only the scores of the test.
• Refactor core modules to harmonize function signatures and include type hinting (not breaking API)
• Clean-up example/tutorial folders, and decide what goes to tutorials and what to notebooks.

Minor Fixes:

• CartesianGrid2D not having magnitude attribute instantiation #257
• Link to CSEP Website #256
• Refactor binomial to binary in tests/ and csep.core/binomial_evaluations.py #216
• Add binomial_evaluations to top-level __init__.py #215
• global region does not support different resolutiosn #187
• add new-zealand and japan testing regions #176
• read basic catalog using csv.reader #90

Moderate Fixes:

• How to deal with forecast rates ≤ 0 in any LL-based test? #226
• T-test: deal with corner cases that raise RuntimeWarnings #225
• Documentation requires slight revision. Many references are broken and it raises multiple warnings/errors when building.

Major fix:

• Spatial binning doesn't properly account for floating point precision #255

Fixes needing closing (?):

• Plot comparison test failing #214
• catalog-based m-test #196
• version v0.7.0 roadmap #195
• Dealing with dependencies in pyCSEP #192 (discussion must be had)

Remaining open-issues could be dealt within the v1 versions.

[pabloitu]

@wsavran @fabiolsilva @pjmaechling @Serra314 @mherrmann3 @bayonato89 Need feedback on the roadmap, and include any tasks suggestion on the comments of this issue. We can discuss these issues in the next csep-dev meeting.

These are my work suggestion to be included in v1.0.0:

Features:

• Correction to M-test with resampling, and addition of new MLL test #268
• Implementation of the Brier Score and its consistency test. #232

Refactoring:

• Plot Refactoring #263, which closes Plot Refactoring  #260, Basemaps downloading requires a timeout option #258
• Clean up unused functionality (e.g., documents.py, file.py, log.py and many other functions within)
• Refactor core-modules structure (might break API): In which module should gridded non-poisson tests (brier, negbinom, binary) be placed? Should each have its own module? I believe we should abstract any coupling between scores, tests and distributions: Some users might be interested in the tests, and some only the scores of the test.
• Refactor core modules to harmonize function signatures and include type hinting (not breaking API)
• Clean-up example/tutorial folders, and decide what goes to tutorials and what to notebooks.

Minor Fixes:

• CartesianGrid2D not having magnitude attribute instantiation #257
• Link to CSEP Website #256
• Refactor binomial to binary in tests/ and csep.core/binomial_evaluations.py #216
• Add binomial_evaluations to top-level __init__.py #215
• global region does not support different resolutiosn #187
• add new-zealand and japan testing regions #176
• read basic catalog using csv.reader #90

Moderate Fixes:

• Spatial binning doesn't properly account for floating point precision #255
• How to deal with forecast rates ≤ 0 in any LL-based test? #226
• T-test: deal with corner cases that raise RuntimeWarnings #225
• Documentation requires slight revision. Many references are broken and it raises multiple warnings/errors when building.

Fixes needing closing (?):

• Plot comparison test failing #214
• catalog-based m-test #196
• version v0.7.0 roadmap #195
• Dealing with dependencies in pyCSEP #192 (discussion must be had)

Remaining open-issues could be dealt within the v1 versions.

[wsavran]

Hi @pabloitu,

Some thoughts here... Overall the roadmap to 1.0 looks good. We probably want to add some incremental releases in here, and we can iterate on during our next call. Also, I would elevate #255 to being a major issue. This has implications for most of the inner workings of the code and evaluations.