Account for lower GHI limit in check_irradiance_consistency_qcrad

[AdamRJensen]

I'm surprised that the check_irradiance_consistency_qcrad function does not take into account the 50 W/m^2 lower limit as specified in the reference. The reference states that for GHI values less than the 50 W/m^2 limit the test cannot be applied (see reference below).

At the very least I suggest that a comment on this is added to the warning section as otherwise we cannot really claim to use the criteria in the reference.

Perhaps an example can be shown at the bottom of the docs, demonstrating how to only apply this mask for GHI>50W/m^2.

Reference:

ICYMI @adriesse

[AdamRJensen]

I think that flagging low irradiance at nighttime is OK in the context of pvanalytics. The package provides other functions which label day/night periods, and each quality function returns a time series of Boolean for the critieria it checks. The temptation is to regard 'True' here as equivalent to a valid measurement - and that is not always the case, as the discussion brings out.
Originally posted by @cwhanse in #167 (comment)

I think this is a very valid point, thus, perhaps we should go beyond labeling flags as binary. In particular, it could be considered to use nans for data points that are outside the test criteria. For example, typically values that are less than 50 W/m^2 can't be tested, and labeling them 0 (test failed) or 1 (test passed) is not correct. In such cases, setting the flag as nan would be a reasonable option IMHO. In fact, this was the method followed in this study.

[adriesse]

perhaps we should go beyond labeling flags as binary.

I fully agree.

[cwhanse]

I would agree if, in the implementation, nan doesn't introduce spurious True values when AND-ing series of flags.

a = np.array([np.nan, True, False])
b = np.array([np.nan, True, True])
np.logical_and(a, b)

array([ True,  True, False])

[adriesse]

I suggest all integer values. No nans. Negative values could be algorithm exception codes and non-negative values diagnostic output, for example.

[cwhanse]

I'm a hard no on returning a Series of integer codes in place of a Series of booleans, since these filtering functions are meant to define filters for data. We could return a second Series of codes indicating state, in addition to returning the Series of boolean, and let the user decide which states to filter on.

[cwhanse]

@AdamRJensen I think that the lower bound on GHI (50 W/m2) is applied in the consistency check in this line and is sourced from this dict.

[AdamRJensen]

@AdamRJensen I think that the lower bound on GHI (50 W/m2) is applied in the consistency check in this line and is sourced from this dict.

Huh! Well I guess I'm just in disagreement with how the check is implemented then :) The way I interpret the 50 W/m^2 limit is that the checks aren't reliable in assessing if the measurement is incorrect in that region, hence the values below 50 W/m^2 are considered passing (and should be set to True and not False).

To cite Perez-Astudillo

One must note that some BSRN tests are not done in some conditions; the corresponding measurements are then kept here if they pass the other tests; in other words, data are only marked as ‘bad quality’ if they actually fail an applied test.