Data shift algorithm #142

kperrynrel · 2022-05-25T18:24:16Z

Description

Thank you for your contribution! Please provide a brief description of the problem and the proposed solution or new feature (if not already fully described in a linked issue)

Checklist

The following items must be addressed before the code can be merged.
Please don't hesitate to ask for help if you are unsure of how to accomplish any of the items.
You are free to remove any checklist items that do not apply or add additional items that are
not on this list

Added new API functions to docs/api.rst
Clearly documented all new API functions with PEP257 and numpydoc compliant docstrings
Adds description and name entries in the appropriate "what's new" file
in docs/whatsnew
for all changes. Includes link to the GitHub Issue with :issue:`num`
or this Pull Request with :pull:`num`. Includes contributor name
and/or GitHub username (link with :ghuser:`user`).
Non-API functions clearly documented with docstrings or comments as necessary
Added tests to cover all new or modified code
Pull request is nearly complete and ready for detailed review

@cwhanse

* v0.1.1 Release (pvlib#132) * change pypi classifier from pre-alpha to beta * remove unnecessary docs/requirements.txt * whatsnew v0.1.1 * include 0.1.1 in whatsnew index * link zenodo in readme * Added clipping time series example for Sphinx documentation. * added sphinx documentation + examples for running the clipping mask. * fixed pep8 formatting errors. * added a new whatsnew rst file for version 0.1.2 * removed close plot to visualize in sphinx. * removed trailing whitespace. * added tight layout for plot sizing. * Updated the docs based on @kanderso-nrel's recs. * fixed pep8 warning. * removed trailing whitespace-pep8 issue. * Added placeholder scripts for each function for Sphinx documentation. * added sphinx module for completeness score. * Update docs/examples/clipping.py Co-authored-by: Cliff Hansen <[email protected]> * Update docs/examples/clipping.py Co-authored-by: Cliff Hansen <[email protected]> * added each of the python scripts for detecting stale data, interpolated data, and check for daily data completeness. * updated the interpolated-periods documentation * cleaning up doc strings. * updated the naming conventions of the sphinx docs. * fixed pep8 error on stale data periods docs. * made updates to the Sphinx docs based on kanderso-nrel's feedback. * fixed pep8 errors. * Edited some of the language in the sphinx doc comments. * Added pv-terms json. * added the documentation files for hampel, tukey, and zscore outlier detection. * update the documentation to include separate data files for each of the different issues, to avoid further confusion. * updated the interpolated data docs to pull the correct csv. * More docstring cleanup. * Updated outlier code to use the new outlier csv file. * updated the outliers routine to handle varying indices. * Docstring cleaning. * made updates to the hovertext and the _round edits per @kanderso-nrel's comments * updated diff to round on docstring per @kanderso-nrel's comment. * updated the whatsnew doc with the outliers documentation. * updated the hovertext info. * updated the routine with the bug fix for whatsnew, and removed the initial graphing. * added prints to visualize the imported data in example docs. * Update docs/whatsnew/0.1.2.rst Co-authored-by: Kevin Anderson <[email protected]> * added new commenting based on @cwhanse's recommendations. * fixed improper spelling in comments * Day night masking sphinx documentation (pvlib#139) * update the day-night masking example. * update the day-night masking routine. * added the SERF east data for running the day-night mask examples. * added the day-night masking routine. * Added section for comparing day-night mask to PVlib sunrise-sunset times. * added separate printouts for sunrise and sunset time comparisons. * added vertical lines for sunrise + sunset in plots * update the routine to remove hardcoded file name. * added update to the whatsnew file. * removed a newline to see if we could get git actions to work. * Made updates to documentation based on @kanderso-nrel's recommendations. * Update docs/examples/day-night-masking.py Co-authored-by: Cliff Hansen <[email protected]> * Removed default kwargs for pvlib SPA sunrise-sunset function. * Updating the commenting. * fixed pep8 line length Co-authored-by: Perry <[email protected]> Co-authored-by: Cliff Hansen <[email protected]> * Irradiance sphinx documentation (pvlib#140) * added initial files for all of the irradiance documentation (need to edit). * added RMIS example data for irradiance Sphinx documentation. * added the new qcrad function. * update the examples for both qcrad functions. * added qcrad-limits documentation. * ensured outputs for all irradiance functions in examples. * added plotting functionality for some of the examples. * added graphics for all of the irradiance documentation. * added new line at end of file to stop pep8 failure. * Clean up of doc strings for irradiance documentation. * Fixed the docstring PEP8 error. * Update docs/examples/clearsky-limits-irradiance.py Co-authored-by: Kevin Anderson <[email protected]> * Update docs/examples/clearsky-limits-irradiance.py Co-authored-by: Kevin Anderson <[email protected]> * Update docs/examples/clearsky-limits-irradiance.py Co-authored-by: Kevin Anderson <[email protected]> * Update docs/examples/clearsky-limits-irradiance.py Co-authored-by: Kevin Anderson <[email protected]> * Update docs/examples/qcrad-limits-irradiance.py Co-authored-by: Kevin Anderson <[email protected]> * Update docs/examples/qcrad-limits-irradiance.py Co-authored-by: Kevin Anderson <[email protected]> * Update docs/examples/qcrad-consistency-irradiance.py Co-authored-by: Kevin Anderson <[email protected]> * Update docs/examples/qcrad-consistency-irradiance.py Co-authored-by: Kevin Anderson <[email protected]> * Removed 'sampled' reference from docstring when describing data * changed py:func to py:meth in docstring * Updated the routine to calculate extraterrestrial radition as dni_extra for check_irradiance_limits_qcrad() function. * Renamed the routine Clearsky Limits for Daily Insolation * removed pep8 issues * added the documentation to the whatsnew file. * Update docs/examples/clearsky-limits-irradiance.py Co-authored-by: Cliff Hansen <[email protected]> * Update docs/examples/clearsky-limits-irradiance.py Co-authored-by: Cliff Hansen <[email protected]> * Update docs/examples/qcrad-consistency-irradiance.py Co-authored-by: Cliff Hansen <[email protected]> * Update docs/examples/qcrad-consistency-irradiance.py Co-authored-by: Cliff Hansen <[email protected]> * Update docs/examples/qcrad-consistency-irradiance.py Co-authored-by: Cliff Hansen <[email protected]> * Update docs/examples/qcrad-limits-irradiance.py Co-authored-by: Cliff Hansen <[email protected]> * Update docs/examples/daily-insolation-limits-irradiance.py Co-authored-by: Cliff Hansen <[email protected]> * Update docs/examples/clearsky-limits-irradiance.py Co-authored-by: Cliff Hansen <[email protected]> * added day-night mask to clearsky-limits-irradiance documentation * removed hardcoded path! * Update docs/examples/daily-insolation-limits-irradiance.py Co-authored-by: Cliff Hansen <[email protected]> * Update docs/examples/daily-insolation-limits-irradiance.py Co-authored-by: Cliff Hansen <[email protected]> * Update docs/examples/daily-insolation-limits-irradiance.py Co-authored-by: Cliff Hansen <[email protected]> * switched the ordering of parameters in ) per @cwhanse's request. * rearranged the order of inputs for irradiance_consistency_qcrad function in unit test. * Update docs/examples/qcrad-consistency-irradiance.py Co-authored-by: Cliff Hansen <[email protected]> * updated clearsky-limits-irradiance example to comment on Ineichen model performance * Update docs/examples/qcrad-limits-irradiance.py Co-authored-by: Cliff Hansen <[email protected]> * Update docs/examples/qcrad-consistency-irradiance.py Co-authored-by: Cliff Hansen <[email protected]> * Update docs/examples/daily-insolation-limits-irradiance.py Co-authored-by: Cliff Hansen <[email protected]> Co-authored-by: Perry <[email protected]> Co-authored-by: Kevin Anderson <[email protected]> Co-authored-by: Cliff Hansen <[email protected]> Co-authored-by: Kevin Anderson <[email protected]> Co-authored-by: Perry <[email protected]> Co-authored-by: Cliff Hansen <[email protected]>

kperrynrel · 2022-05-25T18:26:56Z

This PR was accidentally merged and reverted in pvanalytics/master. Please see here for original PR:#124

kperrynrel · 2022-05-26T16:13:47Z

@kanderso-nrel and @cwhanse apologies for the issues with this PR--I accidentally merged and reverted this yesterday so all of the commenting is in PR #124. However, this PR looks pretty good at this point, and is ready for final review. Hoping to get this merged here in the next week before I do a poster presentation on it at PVSC. Thanks!

Updated PR number

kandersolar

I don't expect to want to make any more comments, so hitting "approve" now :)

kandersolar · 2022-05-26T17:21:29Z

pvanalytics/quality/data_shifts.py

+                     (time_series >= time_series.quantile(.99)))
+    # Filter out the associated data by masking
+    time_series = time_series[(~stale_mask) & (~negative_mask) &
+                              (~quantile_mask)]


I mention this not because I think it's a worthwhile change, but just because my brain can't help but notice it and perhaps it is interesting: the logic of (~A) & (~B) & (~C) can be reduced from five operations to three with ~(A | B | C) via De Morgan's laws. But again I would prioritize readability over the very marginal performance difference here.

pvanalytics/quality/data_shifts.py

pvanalytics/tests/quality/test_data_shifts.py

…o series

…tring

cwhanse

Thanks @kperrynrel. Apologies for the delayed review.

pvanalytics/quality/data_shifts.py

docs/whatsnew/0.1.2.rst

Co-authored-by: Cliff Hansen <[email protected]>

kperrynrel · 2022-06-24T18:00:45Z

@cwhanse no worries! I got your changes incorporated and I am squashing and merging the PR now. Thank you @kanderso-nrel and @cwhanse for all the help with getting this through!

kperrynrel and others added 3 commits May 25, 2022 11:48

Merge branch 'pvlib:master' into data_shift_algorithm

eeaa419

Merge branch 'pvlib:master' into data_shift_algorithm

841e15b

Kirsten Perry added 10 commits May 25, 2022 13:58

added back in all of the data shift files.

24b3711

Debug pep8 fixes.

4b847e7

Added data_dir path to conftest.

7894478

pep8 fix.

b6cb13b

renamed pytest fixture.

329755f

added unit test for when ruptures is not available--up test coverage.

49f22e0

retry to debug the ruptures package load

503e626

fixed my ruptures delete and re-impoort test

f2cb363

added import error test for ruptures

3838787

more ruptures package unload-reload unit tests

cec5062

kperrynrel requested review from kandersolar and cwhanse May 26, 2022 16:10

Update 0.1.2.rst

5200422

Updated PR number

kandersolar approved these changes May 26, 2022

View reviewed changes

kandersolar added this to the v0.1.2 milestone May 26, 2022

kandersolar added the enhancement New feature or request label May 26, 2022

Kirsten Perry added 3 commits May 26, 2022 14:02

updates based on @kanderso-nrel's review

1f6c786

renamed variables in the test file for data shifts from time_series t…

b022381

…o series

updated the longest segment output to tuple + updated warning in docs…

060a2c1

…tring

cwhanse approved these changes Jun 21, 2022

View reviewed changes

kperrynrel and others added 5 commits June 24, 2022 10:23

Update pvanalytics/quality/data_shifts.py

b71263f

Co-authored-by: Cliff Hansen <[email protected]>

Update pvanalytics/quality/data_shifts.py

7fc94b1

Co-authored-by: Cliff Hansen <[email protected]>

Update pvanalytics/quality/data_shifts.py

b2e5642

Co-authored-by: Cliff Hansen <[email protected]>

Update pvanalytics/quality/data_shifts.py

d747fc8

Co-authored-by: Cliff Hansen <[email protected]>

Update docs/whatsnew/0.1.2.rst

140a5d6

Co-authored-by: Cliff Hansen <[email protected]>

Perry and others added 6 commits June 24, 2022 10:33

Updated PVSC citation.:

8e0aa88

Cleaned up any existing pep8 errors

a60ce2d

Update pvanalytics/quality/data_shifts.py

8edede4

Co-authored-by: Cliff Hansen <[email protected]>

Update pvanalytics/quality/data_shifts.py

a39cf42

Co-authored-by: Cliff Hansen <[email protected]>

Update the docstring

4a54cc5

updated the docstring

00a1f7a

kperrynrel merged commit e8ce801 into pvlib:master Jun 24, 2022

kandersolar mentioned this pull request Jul 1, 2022

Comparing sensor measurements with external data to detect issues #143

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Data shift algorithm #142

Data shift algorithm #142

kperrynrel commented May 25, 2022 •

edited

Loading

kperrynrel commented May 25, 2022

kperrynrel commented May 26, 2022

kandersolar left a comment

kandersolar May 26, 2022

cwhanse left a comment

kperrynrel commented Jun 24, 2022

Data shift algorithm #142

Data shift algorithm #142

Conversation

kperrynrel commented May 25, 2022 • edited Loading

Description

Checklist

kperrynrel commented May 25, 2022

kperrynrel commented May 26, 2022

kandersolar left a comment

Choose a reason for hiding this comment

kandersolar May 26, 2022

Choose a reason for hiding this comment

cwhanse left a comment

Choose a reason for hiding this comment

kperrynrel commented Jun 24, 2022

kperrynrel commented May 25, 2022 •

edited

Loading